Image processing method and device for 3D display and storage medium

Abstract

The invention relates to an image processing method and device for 3D display and a storage medium. The image processing method comprises the following steps: acquiring a plurality of to-be-interleaved images; simultaneously carrying out interleaving calculation on at least two to-be-interleaved images in the plurality of to-be-interleaved images in parallel so as to obtain 3D images corresponding to the plurality of to-be-interleaved images; and performing 3D display based on the plurality of 3D images corresponding to the plurality of to-be-interleaved images.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to an image processing method, device and storage medium for 3D display. Background Art

[0002] Multi-view naked-eye 3D display devices have become the main development direction of 3D display devices in the future. They can solve the problem of multiple people and multiple positions viewing 3D images without wearing glasses, which is in line with people's viewing habits. Multi-view naked-eye 3D display devices are mainly composed of a display panel and a grating unit. The display panel can be an LCD display panel or an OLED display panel, etc. The grating unit can be a slit grating or a cylindrical lens grating, etc.

[0003] The naked-eye 3D display technology in the related art generates multiple viewpoint images from a two-dimensional image, for example, 24 images from different viewpoints, and then interleaves the 24 viewpoint images according to the hardware parameters of the 3D display device to generate a 3D image displayed by the corresponding hardware. For video, the generated 3D image is encoded and then put into the corresponding hardware device for video playback.

[0004] Interlacing calculation is the process of sampling different viewpoint images according to different 3D display hardware to convert them into an image of a specified size. The interlacing calculation process includes:

[0005] 1. Load the viewpoint images that need to be interlaced. For example, if you need to interlace a 2-minute video, 25 video frames per second, if it is 24 viewpoints, then you need to load about 24*25*60*2=72,000 viewpoint images. If it is 49 viewpoints, then you need to load 147,000 viewpoint images.

[0006] 2. Sampling is performed using the given hardware parameters and the resolution of the 3D image to be generated: first, the length and width of the 3D image are traversed to obtain the initial coordinates; then, the initial coordinates are interleaved using the hardware parameters to obtain the interleaved coordinates; then, pixel data is obtained in the corresponding viewpoint image based on the interleaved coordinates; finally, a 3D image is generated based on the obtained pixel data.

[0007] 3. Encode the 3D image corresponding to each generated video frame and put it into the corresponding hardware display device for video playback.

[0008] It can be seen that interleaving calculation is an important key in the 3D display process, so its efficient operation is crucial. Summary of the invention

[0009] The present application provides an image processing method, device and storage medium for 3D display, so as to improve the efficiency of interlaced calculation.

[0010] In a first aspect, an image processing method for 3D display is provided, comprising: acquiring a plurality of images to be interleaved; performing interleaving calculations on at least two of the plurality of images to be interleaved in parallel to obtain 3D images corresponding to the plurality of images to be interleaved; and performing 3D display based on the plurality of 3D images corresponding to the plurality of images to be interleaved.

[0011] In a second aspect, an image processing device is provided, comprising: an acquisition module, configured to acquire multiple images to be interlaced; an interlacing calculation module, configured to perform interlacing calculations on at least two of the multiple images to be interlaced in parallel to obtain 3D images corresponding to each of the multiple images to be interlaced; and a display module, configured to perform 3D display based on the multiple 3D images corresponding to the multiple images to be interlaced.

[0012] In a third aspect, an electronic device is provided, comprising: a memory; and at least one processor, wherein the memory stores a computer program, and the computer program can be executed by the at least one processor to implement the above-mentioned image processing method.

[0013] In a fourth aspect, a display system is provided, comprising: the electronic device mentioned above.

[0014] According to a fifth aspect, a non-transitory computer-readable storage medium storing a computer program is provided, wherein the computer program implements the above-mentioned image processing method when executed by a processor.

[0015] In a sixth aspect, a computer program product is provided, comprising a computer program, wherein the computer program implements the above-mentioned image processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of an image processing method in some embodiments of the present application;

[0018] Figure 2 A schematic diagram of the optical principle of a multi-view naked-eye 3D display device in some embodiments of the present application;

[0019] Figure 3 A flowchart of an image processing method in some embodiments of the present application;

[0020] Figure 4A flow chart of a method for obtaining an image to be interlaced in some embodiments of the present application;

[0021] Figure 5 A flow chart of a method for performing interlaced calculation on an interlaced image in some embodiments of the present application;

[0022] Figure 6 A schematic diagram of the image processing process in some embodiments of the present application;

[0023] Figure 7 A schematic diagram of a process of performing interlaced calculation on an interlaced image in some embodiments of the present application;

[0024] Figure 8 A block diagram of an image processing device in some embodiments of the present application;

[0025] Fig. 9 It is a block diagram of an electronic device in some embodiments of the present application. DETAILED DESCRIPTION

[0026] The technical solution of the present application will be described clearly and in detail below in conjunction with the drawings of the embodiments of the present application. Obviously, the embodiments described below are part of the embodiments of the present application rather than all of the embodiments, and are only used to more clearly illustrate the technical solution of the present application. Therefore, they are only used as examples, and the protection scope of the present application cannot be limited by this. Based on the described embodiments of the present application, all other embodiments that can be obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein shall have the meanings commonly understood by technicians in the technical field of this application; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the embodiments of this application and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the embodiments of this application are only used to distinguish different objects and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the embodiments of this application, unless otherwise stated, "multiple" means more than two.

[0028] In related technologies, in order to achieve naked-eye 3D display, a two-dimensional image can be generated into multiple viewpoint images, for example, 24 images with different viewpoints, and then the 24 viewpoint images are interleaved and calculated according to the hardware parameters of the 3D display device to generate a 3D image displayed by the corresponding hardware. For video, the generated 3D image is encoded and then put into the corresponding hardware device for video playback.

[0029] Interlacing calculation is the process of sampling different viewpoint images according to different 3D display hardware to convert them into an image of a specified size. The interlacing calculation process includes:

[0030] 1. Load the viewpoint images that need to be interlaced. For example, if you need to interlace a 2-minute video, 25 video frames per second, if it is 24 viewpoints, then you need to load about 24*25*60*2=72,000 viewpoint images. If it is 49 viewpoints, then you need to load 147,000 viewpoint images.

[0031] 2. Perform interlaced calculations using the given hardware parameters and the resolution of the 3D image to be generated: first, traverse the length and width of the 3D image to obtain the initial coordinates; then, perform interlaced calculations on the initial coordinates using the hardware parameters to obtain the interlaced coordinates; then, obtain pixel data in the corresponding viewpoint image based on the interlaced coordinates; finally, generate a 3D image based on the obtained pixel data.

[0032] 3. Encode the 3D image corresponding to each generated video frame and put it into the corresponding hardware display device for video playback.

[0033] It can be seen that the data to be processed by interleaving calculation is huge, so how to improve the efficiency of interleaving calculation becomes a problem that needs to be solved urgently.

[0034] Based on this, an image processing method for 3D display is proposed in an embodiment of the present application, which improves the calculation efficiency by performing interleaving calculation on at least two to-be-interleaved images in parallel.

[0035] The image processing method in the embodiment of the present application can be applied to a 3D display device, and the 3D display device can be but is not limited to a naked-eye 3D TV, a virtual reality device, or any other 3D image playback device.

[0036] The technical solution of the present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, according to some embodiments of the present application, an image processing method for 3D display is provided, including:

[0038] Step S101, obtaining a plurality of images to be interleaved;

[0039] Step S102, performing interleaving calculations on at least two of the multiple images to be interleaved simultaneously to obtain 3D images corresponding to the multiple images to be interleaved;

[0040] Step S103: performing 3D display based on a plurality of 3D images corresponding to the plurality of images to be interlaced.

[0041] Therefore, by performing interleaving calculations on at least two to-be-interleaved images in parallel, the calculation efficiency can be improved compared with performing interleaving calculations on a plurality of to-be-interleaved images one by one.

[0042] The image to be interlaced may include multiple viewpoint images, which may be generated based on a two-dimensional initial image. The initial image may be an image captured by a camera, or may be an image read from a storage device, or may be an image transmitted from other electronic devices.

[0043] Taking two viewpoints as an example, the steps of generating two viewpoint images based on a two-dimensional initial image include: first inputting the 2D image into a depth estimation model to perform depth estimation of the 2D image, i.e. determining the depth information of each pixel in the 2D image; then based on the depth image and the 2D image, using the viewpoint generation model to estimate and generate the image of the next viewpoint, thereby obtaining the images of the two viewpoints. It should be noted that the method of generating multiple viewpoint images based on a two-dimensional initial image is not limited to this. For example, if the depth information of the 2D image is known, there is no need to use the depth estimation model to perform depth estimation of the 2D image.

[0044] The process of performing interleaving calculation on each image to be interleaved may be: obtaining a preset resolution of the 3D display; based on the preset resolution of the 3D display, traversing the length and width of the 3D image to obtain an initial coordinate index, the initial coordinate index including pixel coordinates of all pixels in the 3D image, such as (0, 0), ... (1, 720), ... (1280, 1), etc.; performing interleaving calculation on each pixel coordinate in the initial coordinate index through hardware parameters (i.e., interleaving parameters) to obtain an interleaved coordinate index, the interleaved coordinate index including the interleaved coordinates of each pixel coordinate obtained by the interleaving calculation; obtaining pixel point data in the image to be interleaved according to the interleaved coordinate index; and generating a 3D image based on the obtained pixel point data.

[0045] In some embodiments, at least two processes may be used to perform interleaving calculations on at least two images to be interleaved simultaneously.

[0046] In some other embodiments, the interleaving calculation may be performed on at least two to-be-interleaved images in parallel by at least two threads.

[0047] The parallel interleaved calculations may be implemented by multi-processing and / or multi-threading, depending on the program programming language used.

[0048] The multi-view naked-eye 3D display device realizes multi-view display through the grating structure. Under the occlusion or light splitting effect of the corresponding grating structure, only one viewpoint image can be viewed in one direction, and the human eyes can see two viewpoint images, thus forming a stereoscopic vision. The grating structure can adopt a slit grating or a cylindrical lens grating. The following takes the cylindrical lens grating as an example to describe the principle of multi-view naked-eye 3D display.

[0049] Different multi-view naked-eye 3D display devices may have different resolutions and viewpoint numbers. The number of viewpoint images required to be included in the interlaced image may be determined according to the viewpoint number of the display device.

[0050] The interlacing parameters are related to the resolution of the display device and the parameters of the lenticular lens array. The interlacing parameters may include the tilt angle of the pixel coordinates and the offset of the pixel coordinates. The parameters of the lenticular lens array include the fitting angle between the lenticular lens array and the display screen. The tilt angle of the pixel coordinates in the interlacing parameters is the fitting angle. Figure 2 In the illustrated example, the fitting angle θ between the cylindrical lens array 200 and the display screen 100 is 90 degrees, that is, the optical axis direction of the cylindrical lens array 200 is perpendicular to the row direction of the sub-pixels of the display screen 100 .

[0051] exist Figure 2 In the illustrated example, the fitting angle θ between the cylindrical lens array 200 and the display screen 100 is 90 degrees, and the sub-pixel sequence is extracted in a direction perpendicular to the optical axis of the cylindrical lens array 200, that is, the sub-pixel sequence is extracted in the row direction of the sub-pixels. The display screen 100 may include R, G, and B sub-pixels. Taking the display device as having four viewpoints as an example, it is necessary to extract the sub-pixel sequence corresponding to the four viewpoints, and the sub-pixels of each row are cyclically numbered according to the sequence of ①②③④, then all sub-pixel sequences numbered ① in the display screen 100 correspond to the first viewpoint, all sub-pixel sequences numbered ② in the display screen 100 correspond to the second viewpoint, all sub-pixel sequences numbered ③ in the display screen 100 correspond to the third viewpoint, and all sub-pixel sequences numbered ④ in the display screen 100 correspond to the fourth viewpoint.

[0052] The upper left corner of the plane where the display screen 100 is located is taken as the origin of the coordinate system, the row direction is the X-axis, and the column direction is the X-axis. Figure 2In the illustrated example, the sub-pixel column information of each row is the same, the coordinates of the sub-pixel sequence corresponding to the first viewpoint on the X-axis are: Pw[1,5,9, ...1+(i-1)*4], the coordinates of the sub-pixel sequence corresponding to the second viewpoint on the X-axis are: Pw[2,6,10, ...2+(i-1)*4], the coordinates of the sub-pixel sequence corresponding to the third viewpoint on the X-axis are: Pw[3,7,11, ...3+(i-1)*4], and the coordinates of the sub-pixel sequence corresponding to the fourth viewpoint on the X-axis are: Pw[4,8,12, ...4+(i-1)*4]. Among them, i is the column coordinate of the sub-pixel (that is, the coordinate on the Y-axis), and Pw is the width of the sub-pixel.

[0053] It can be understood that the fitting angle θ between the cylindrical lens array and the display screen may not be 90 degrees, and the sub-pixel sequence only needs to be extracted along the direction perpendicular to the optical axis of the cylindrical lens array, and the principle is similar.

[0054] It can be seen that in the interlacing calculation, it is necessary to perform the interlacing calculation according to the initial coordinates corresponding to the sub-pixel sequence of each viewpoint in the 3D image to obtain the interlacing coordinates corresponding to each viewpoint image in the image to be interlaced, and sample the image to be interlaced based on the interlacing coordinates to obtain the pixel value of the sub-pixel sequence of each viewpoint in the 3D image. Therefore, according to the resolution of the 3D image, the pixel value of the pixel point corresponding to the sub-pixel sequence of each viewpoint is obtained from the image to be interlaced to generate a 3D image that can be displayed by the 3D display device.

[0055] The interlacing parameters in the embodiment of the present application are used for coordinate interlacing calculation, and are related to the resolution of the display device and the fitting angle between the cylindrical lens array and the display screen. The interlacing parameters include the offset and tilt angle of the pixel coordinates.

[0056] In some embodiments, interleaving parameters used for coordinate interleaving calculation may be provided by a 3D display device.

[0057] For 3D video display, a large number of viewpoint images need to be loaded through I / O during interleaving calculation, which will generate frequent IO operations and cause ineffective consumption of resources. Based on this, in the embodiment of the present application, the viewpoint images are preprocessed to reduce the IO operations when loading images and improve the calculation efficiency.

[0058] According to some embodiments of the present application, the image processing method may further include:

[0059] Step S310, obtaining an initial image;

[0060] Step S320, generating multiple viewpoint images based on the initial image;

[0061] Step S330: merge multiple viewpoint images to generate an image to be interlaced corresponding to the initial image.

[0062] Therefore, by merging multiple viewpoint images in advance to generate an image to be interlaced, multiple viewpoint images for interlaced calculation can be loaded through one I / O, which can reduce the I / O operation when loading images and improve calculation efficiency.

[0063] The initial image may be, but is not limited to, a video frame of the target video. In this case, the multiple initial images used to generate the multiple images to be interlaced may be multiple continuous video frames in the target video, and the 3D image generated based on the multiple images to be interlaced may be encoded to achieve 3D video display.

[0064] Take the example of loading a video that needs to be interleaved for 2 minutes and 25 video frames (initial images) per second. If there are 24 viewpoints, then about 24*25*60*2=72,000 viewpoint images need to be loaded. If one viewpoint image is loaded at a time, 72,000 I / O operations are required to load all viewpoint images. However, by adopting the preprocessing method in the embodiment of the present application, the 24 viewpoint images of one video frame are merged to generate an image to be interleaved, and only 3,000 I / O operations are required to load all viewpoint images, which greatly reduces the I / O operations when loading images and improves computing efficiency.

[0065] In some embodiments, merging multiple viewpoint images to generate an image to be interlaced corresponding to the initial image may include: splicing the multiple viewpoint images along the row direction of the viewpoint images to generate the image to be interlaced corresponding to the initial image. It should be noted that the multiple viewpoint images may also be merged in other ways to generate the image to be interlaced corresponding to the initial image, for example, splicing the multiple viewpoint images along the row direction and the column direction respectively, that is, the multiple viewpoint images are spliced ​​in a matrix distribution.

[0066] According to some embodiments of the present application, the image processing method may further include:

[0067] storing the plurality of to-be-interleaved images in a disk cache; and

[0068] In response to the memory cache having storage space, the to-be-interleaved image stored in the disk cache is loaded into the memory cache for storage.

[0069] like Figure 4 As shown, step S110, obtaining the image to be interleaved, includes:

[0070] Step S111, reading the current to-be-interleaved image for interleaving calculation from the memory cache;

[0071] Step S112, in response to the current image to be interleaved not existing in the memory cache, reading the current image to be interleaved from the disk cache;

[0072] Step S113, loading the current to-be-interleaved image read from the disk cache into the memory cache for storage; and

[0073] Step S114: In response to the existence of the current image to be interleaved in the memory cache, read the current image to be interleaved from the memory cache.

[0074] The access time and data transfer rate of memory cache are much better than those of disk. Disk cache usually reads and stores a portion of data in advance, so it can provide data faster when data is requested. Disk cache can reduce the random seek time of the disk because the data has been pre-stored and can be read directly from the cache without relocating the disk. Disk cache usually optimizes and compresses data to provide faster data access. This can significantly reduce read time.

[0075] The solution in the embodiment of the present application is to store multiple images to be interleaved in the disk cache, and when the memory cache has storage, the images to be interleaved cached in the disk cache are loaded into the memory cache. Therefore, after each parallel completion of the interleaving calculation of at least two images to be interleaved, the current image to be interleaved used for the interleaving calculation can be obtained from the memory cache in priority to reduce access time, data transmission rate, etc. In the case that the current image to be interleaved does not exist in the memory cache, the current image to be interleaved can be read from the disk cache, and data can be read faster than disk storage. And in the case that the current image to be interleaved does not exist in the memory cache, it is triggered to load a new image to be interleaved from the disk cache to the memory cache, so that the next interleaving calculation can read the image to be interleaved used for this interleaving calculation from the disk cache.

[0076] Therefore, the solution of the embodiment of the present application combines the advantages of disk cache and memory cache to optimize and accelerate viewpoint image loading, which can significantly reduce loading time and improve computing efficiency.

[0077] In some embodiments, the multiple images to be interleaved may be numbered in the order in which the interleaving calculation is performed on the multiple images to be interleaved, and when the multiple images to be interleaved are stored, the corresponding number of each image to be interleaved is stored at the same time. Thus, the stored multiple images to be interleaved can be read in sequence according to the numbers to obtain at least two images to be interleaved processed by each parallel interleaving calculation, and the interleaving calculation of the multiple images to be interleaved can be completed through multiple parallel processing.

[0078] Taking 3D video display as an example, multiple images to be interlaced are generated based on multiple continuous video frames of the target video, and the images to be interlaced may include multiple viewpoint images generated from one video frame, and the multiple viewpoint images may be merged to generate one image to be interlaced. In this case, the corresponding multiple images to be interlaced may be numbered according to the frame numbers of the multiple video frames.

[0079] According to some embodiments of the present application, step S110, obtaining the image to be interlaced, further includes:

[0080] Step S115: In response to the memory buffer overflow, the earliest to-be-interleaved image stored in the memory buffer is deleted.

[0081] After deleting the earliest image to be interlaced stored in the memory cache, the current image to be interlaced read from the disk cache is loaded into the memory cache for storage. In response to the memory cache not overflowing, the current image to be interlaced read from the disk cache is directly loaded into the memory cache for storage. Considering the limitation of the device hardware memory, the idea of ​​memory rolling is adopted.

[0082] Therefore, when the memory cache overflows, the earliest generated image to be interleaved can be deleted to add a new image to be interleaved, so that the next interleaving calculation can read the image to be interleaved for this interleaving calculation from the disk cache.

[0083] It is understandable that the multiple images to be interleaved may be stored and loaded in other ways. For example, the multiple images to be interleaved may be stored only in a disk cache, which may also improve the image reading efficiency to a certain extent.

[0084] For the at least two to-be-interleaved images read each time, interleaving calculations may be performed on the at least two to-be-interleaved images in parallel to obtain 3D images corresponding to the multiple to-be-interleaved images.

[0085] Each pixel point in the image to be interlaced may include pixel values ​​of multiple colors, for example, red R pixel value, green G pixel value and blue B pixel value, and color display is achieved through three primary colors. Each pixel point may also include a white W pixel value, for example.

[0086] According to some embodiments of the present application, performing interleaving calculation on each to-be-interleaved image may include:

[0087] Step S510: Generate an initial coordinate index according to a preset resolution of 3D display, where the initial coordinate index includes coordinates of all pixels in the image of the preset resolution;

[0088] Step S520: performing multiple interleaving calculations on the initial coordinate index in parallel based on the interleaving parameters corresponding to the plurality of color pixels, so as to obtain the interleaving coordinate indexes corresponding to the plurality of color pixels;

[0089] Step S530, based on the interleaved coordinate indexes corresponding to the plurality of color pixels, sampling the interleaved image multiple times in parallel to obtain sampled images corresponding to the plurality of color pixels; and

[0090] Step S540: Generate a 3D image based on the multiple sampled images corresponding to the multiple color pixels.

[0091] Since the interleaving calculations of multiple color channels of the image to be interleaved can be independent of each other, parallel processing can be used to simultaneously perform interleaving calculations on multiple color channels of each interleaved image. Compared with performing interleaving calculations on multiple color channels of the image to be interleaved sequentially, the parallel calculation scheme in the embodiment of the present application can further speed up the interleaving calculation process.

[0092] In some embodiments, according to the preset resolution of the 3D image, the initial coordinate index of the 3D image can be generated by traversing the rows and columns twice. For example, the initial coordinate index can be: (0, 0), ... (1, 720), ... (1280, 1), ... and so on.

[0093] Taking the example that a pixel point may include R pixel value, G pixel value and B pixel value, three interleaving calculations can be performed in parallel on the initial coordinate index based on the interleaving parameters corresponding to the R pixel value, G pixel value and B pixel value to obtain the interleaving coordinate index corresponding to the R pixel, G pixel and B pixel, and then the R pixel value, G pixel value and B pixel value of the interleaved image can be sampled in parallel based on the interleaving coordinate index corresponding to the R pixel, G pixel and B pixel to obtain the sampled images corresponding to the R pixel, G pixel and B pixel, and generate a 3D image based on the sampled images corresponding to the R pixel, G pixel and B pixel.

[0094] Multiple threads or multiple processes may be used to calculate multiple color channels simultaneously, including: using multiple threads or multiple processes to simultaneously calculate and obtain the interleaved coordinate indexes corresponding to the R pixels, G pixels, and B pixels, and after obtaining the interleaved coordinate indexes, using multiple threads or multiple processes to simultaneously sample and obtain the sampled images corresponding to the R pixels, G pixels, and B pixels. Each pixel point of the sampled image also includes an R pixel value, a G pixel value, and a B pixel value.

[0095] According to the above content, in the process of coordinate interleaving calculation, it is necessary to traverse each pixel point of the 3D image to be generated to generate an initial coordinate index, and traverse each pixel point to perform interleaving calculation to generate the corresponding interleaving coordinates, which requires huge amount of calculation.

[0096] Based on this, the idea of ​​vectorization is adopted in the embodiments of the present application to reduce the resource waste caused by the traversal loop, while also improving the calculation efficiency of the coordinate index.

[0097] According to some embodiments of the present application, the initial coordinate index includes an initial row coordinate vector and an initial column coordinate vector, the initial row coordinate vector includes multiple first elements, each first element includes the initial row coordinates corresponding to multiple pixels in each row of pixels, the initial column coordinate vector includes multiple second elements, each second element includes the initial column coordinates corresponding to multiple pixels in each row of pixels, and the multiple first elements and the multiple second elements correspond one to one.

[0098] The interleaving calculation of the initial coordinate index includes:

[0099] Performing a first interleaving calculation on a plurality of initial row coordinates in the initial row coordinate vector to obtain an interleaved row coordinate vector indexed by the interleaved coordinates, wherein the interleaved row coordinate vector includes a plurality of third elements, each of which includes interleaved row coordinates corresponding to a plurality of pixels in each row of pixels; and

[0100] A second interleaving calculation is performed on multiple initial column coordinates in the initial column coordinate vector to obtain an interleaved column coordinate vector indexed by the interleaved coordinates, wherein the interleaved column coordinate vector includes multiple fourth elements, each fourth element includes the interleaved column coordinates corresponding to multiple pixels in each row of pixels, and the multiple third elements and the multiple fourth elements correspond one to one.

[0101] The interlaced image is sampled based on the interlaced coordinate index, including:

[0102] Based on multiple interleaved row coordinates of each third element in the interleaved row coordinate vector and multiple interleaved column coordinates of fourth elements in the interleaved column coordinate vector corresponding one-to-one to the third element, multiple sampling coordinates are obtained, and the sampling coordinates are represented by an interleaved row coordinate in the third element and an interleaved column coordinate in the fourth element corresponding to the position of the interleaved row coordinate.

[0103] Therefore, by performing an interleaving calculation on the row coordinates of all pixel points in the initial coordinate index at the same time, and performing an interleaving calculation on the column coordinates of all pixel points in the initial coordinate index at the same time, the interleaved coordinate index can be obtained through two interleaving calculations without traversing each pixel point for interleaving calculation, which greatly improves the calculation efficiency and reduces resource consumption.

[0104] In some embodiments, the coordinates of the pixel points can be listed in rows and tables according to the resolution of the 3D image. For example, the coordinates of the pixel points in the first row and first column are (1, 1), the coordinates of the pixel points in the first row and second column are (1, 2), the coordinates of the pixel points in the first row and third column are (1, 3), the coordinates of the pixel points in the second row and first column are (2, 1), ... the coordinates of the pixel points in the third row and third column are (3, 3), etc.

[0105] Exemplarily, taking a 3D image with a resolution of 3*3 as an example, the initial row coordinate vector of the initial coordinate index may be: [1,1,1], [2,2,2], [3,3,3], where the first element [1,1,1] represents the initial row coordinates of the multiple pixels in the first row of pixels, the first element [2,2,2] is the initial row coordinate of all pixels in the second row, and the first element [3,3,3] is the initial row coordinate of all pixels in the third row. The initial column coordinate vector of the initial coordinate index may be [1,2,3], [1,2,3], [1,2,3], where the first second element [1,2,3] is the initial column coordinate of all pixels in the first column, the second second element [1,2,3] is the initial column coordinate of all pixels in the second row, and the third second element [1,2,3] is the initial column coordinate of all pixels in the third row. The formula for calculating the interleaved coordinate index Index can be: Index = a*[[1,1,1],[2,2,2],[3,3,3]]+b*[[1,2,3],[1,2,3],[1,2,3]], where a is an interleaved parameter for interleaving row coordinates, including the inclination and offset of the pixel row coordinates, and b is an interleaved parameter for interleaving column coordinates, including the inclination and offset of the pixel column coordinates. a and b can be obtained according to the resolution of the naked eye 3D display device and the fitting angle between the cylindrical lens array and the display screen.

[0106] After calculating the interleaved coordinate index Index, the interleaved row coordinates of three rows of pixels can be obtained from the interleaved row coordinate vector a*[[1,1,1],[2,2,2],[3,3,3]] of the interleaved coordinate index, and the interleaved column coordinates of three columns of pixels can be obtained from the interleaved column coordinate vector b*[[1,2,3],[1,2,3],[1,2,3]] of the interleaved coordinate index. Then, according to the interleaved row coordinates corresponding to each row of pixels and the interleaved column coordinates corresponding to the row, multiple pixel points of the image to be interleaved are sampled.

[0107] In some embodiments, when interleaving calculations are performed on multiple color channels of an interleaved image in parallel, for the interleaving calculation of each color channel, the interleaving calculation method using the vectorization concept in the above-mentioned embodiment can be used to obtain the corresponding interleaving coordinate index of each color channel, and the interleaved image can be sampled multiple times in parallel based on the corresponding interleaving coordinate indexes of the multiple color channels to obtain a sampled image for generating a 3D image.

[0108] It is understandable that the interleaving coordinate index corresponding to each color channel may also be obtained by using the interleaving calculation method based on the vectorization concept in the above embodiment, and then the interleaving coordinate indexes corresponding to the plurality of color channels are calculated one by one.

[0109] After obtaining the 3D images corresponding to the multiple images to be interlaced, the 3D images may be sent to a display device for display.

[0110] According to some embodiments of the present application, the plurality of 3D images may have serial number identifiers, and the image processing method may further include:

[0111] The multiple 3D images obtained by the interleaving calculation are sequentially stored in a queue according to the serial number identification;

[0112] The 3D display is performed based on multiple 3D images corresponding to the multiple images to be interlaced, including:

[0113] The 3D images are sequentially acquired from the queue according to the serial number identifiers for video encoding to realize 3D video display.

[0114] Therefore, after generating the corresponding 3D images to be interleaved, the 3D images can be synchronously obtained from the queue in order for video encoding to achieve 3D video display. That is to say, when the interleaving calculation of multiple images to be interleaved is completed, the encoding is almost completed synchronously, thus ensuring further speeding up of video interleaving and ensuring the efficiency of 3D video display.

[0115] In some embodiments, before storing a 3D image in a queue, the size of the last 3D image sequence number in the current queue may be compared first. If the sequence numbers differ by 1, the 3D image is pushed into the process sharing queue. If not, wait.

[0116] Exemplarily, in the case where the image to be interlaced is obtained based on a video frame of a target video, the sequence number of the 3D image may be a frame number of a corresponding video frame.

[0117] In some exemplary embodiments, taking 3D video display as an example, in combination with Figure 6 and Figure 7 As shown, the image processing process is:

[0118] A plurality of continuous video frames of a target video: a first video frame, a second video frame, ..., an Nth video frame;

[0119] Generate n viewpoint images of each video frame, and merge multiple viewpoint images of each video frame to generate to-be-interlaced image 1, to-be-interlaced image 2, ..., to-be-interlaced image N;

[0120] The N to-be-interleaved images are stored in a disk cache, and the to-be-interleaved images in the disk cache are loaded into a memory cache for storage;

[0121] The m processes processed in parallel obtain the m to-be-interleaved images for interleaving calculation from the memory cache, and generate 3D image 1, 3D image 2, ..., 3D image m corresponding to the m interleaved images, where m≤N;

[0122] Store m 3D images into a queue in sequence according to their serial numbers;

[0123] The m processes processed in parallel read the new m images to be interleaved from the memory cache again, generate 3D image 1, 3D image 2, ..., 3D image m corresponding to the new m interleaved images, store them in the disk cache, and then store them in the queue in sequence;

[0124] While m processes are processed in parallel, 3D images are synchronously obtained from the queue according to the sequence number for encoding.

[0125] The processing of each process may include:

[0126] Determining the resolution of the 3D image to be generated;

[0127] Generate an initial coordinate index based on the resolution of the 3D image. The initial coordinate index can be used in all process processing and does not need to be recalculated each time.

[0128] Based on the interleaving parameters corresponding to the R pixels, the G pixels, and the B pixels, multiple interleaving calculations are performed in parallel on the initial coordinate indexes to obtain the interleaving coordinate indexes corresponding to the R pixels, the G pixels, and the B pixels;

[0129] Based on the interleaved coordinate indexes corresponding to the R pixels, the G pixels and the B pixels, the interleaved image is sampled multiple times in parallel to obtain the sampled images corresponding to the R pixels, the G pixels and the B pixels;

[0130] A plurality of sampling images corresponding to the R pixels, the G pixels, and the B pixels are merged to generate a 3D image.

[0131] Therefore, by optimizing the image preprocessing and loading scheme, and through parallel interleaved computing processing, parallel sampling of multiple color channels and synchronous encoding scheme, the efficiency of interleaved computing can be greatly improved.

[0132] like Figure 8 As shown, according to some embodiments of the present application, an image processing device 800 is further provided, including: an acquisition module 810, configured to acquire a plurality of images to be interlaced; an interlacing calculation module 820, configured to perform interlacing calculations on at least two of the plurality of images to be interlaced in parallel to obtain 3D images corresponding to the plurality of images to be interlaced; and a display module 830, configured to perform 3D display based on the plurality of 3D images corresponding to the plurality of images to be interlaced.

[0133] The operation of each module of the image processing device is similar to the operation of each step of the image processing method described above, and will not be described in detail here.

[0134] According to some embodiments of the present application, there is also provided an electronic device, including: a memory; and at least one processor, wherein the memory stores a computer program, and the computer program can be executed by the at least one processor to implement the above method.

[0135] According to some embodiments of the present application, a display system is also provided, including: the above-mentioned electronic device.

[0136] According to some embodiments of the present application, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above method when executed by a processor.

[0137] According to some embodiments of the present application, a computer program product is also provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0138] See also Fig. 9 , a block diagram of an electronic device 900 that can be used as the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be a computer device of different types, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0139] Fig. 9 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Fig. 9As shown, the electronic device 900 may include at least one processor 901 , a working memory 902 , an I / O device 904 , a display device 905 , a storage device 906 , and a communication interface 907 that can communicate with each other via a system bus 903 .

[0140] The processor 901 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 901 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. The processor 901 may be configured to obtain and execute computer-readable instructions stored in a working memory 902, a storage device 906, or other computer-readable media, such as program codes of an operating system 902a, program codes of an application program 902b, and the like.

[0141] The working memory 902 and the storage device 906 are examples of computer-readable storage media for storing instructions, which are executed by the processor 901 to implement the various functions described above. The working memory 902 may include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the storage device 906 may include a hard disk drive, a solid-state drive, a removable medium, including external and removable drives, memory cards, flash memory, a floppy disk, an optical disk (e.g., CD, DVD), a storage array, a network attached storage, a storage area network, etc. The working memory 902 and the storage device 906 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by the processor 901 as a specific machine configured to implement the operations and functions described in the examples herein.

[0142] The I / O device 904 may include an input device and / or an output device. The input device may be any type of device capable of inputting information to the electronic device 900, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output device may be any type of device capable of presenting information, and may include but is not limited to a video / audio output terminal, a vibrator, and / or a printer.

[0143] The communication interface 907 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a Wi-Fi device, a WiMax device, a cellular communication device and / or the like.

[0144] The application 902b in the working register 902 can be loaded to execute the various methods and processes described above, for example Figure 1 In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the storage device 906 and / or the communication interface 907. When the computer program is loaded and executed by the processor 901, one or more steps of the image processing method described above may be performed.

[0145] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0147] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0150] A computing system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0152] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An image processing method for 3D display, characterized in that: include: Acquire a plurality of images to be interlaced; Performing interlacing calculations on at least two of the multiple images to be interlaced in parallel to obtain 3D images corresponding to the multiple images to be interlaced; A plurality of 3D images corresponding to the plurality of to-be-interleaved images are used for 3D display.

2. The image processing method according to claim 1, characterized in that: The step of performing interleaving calculation on at least two of the plurality of images to be interleaved simultaneously includes: The interleaving calculation is performed on at least two images to be interleaved simultaneously through at least two processes in parallel.

3. The image processing method according to claim 1, characterized in that: Each pixel point in the to-be-interleaved image includes pixel values ​​of multiple colors, and performing an interleaving calculation on the to-be-interleaved image includes: Generate an initial coordinate index according to a preset resolution of the 3D display, wherein the initial coordinate index includes coordinates of all pixel points in the image of the preset resolution; Based on the interleaving parameters corresponding to the plurality of color pixels, the initial coordinate index is interleaved multiple times in parallel to obtain the interleaving coordinate index corresponding to the plurality of color pixels; Based on the interleaving coordinate indexes corresponding to the plurality of color pixels, the to-be-interleaved image is sampled multiple times in parallel to obtain sampled images corresponding to the plurality of color pixels; The 3D image is generated based on the multiple sampled images corresponding to the multiple color pixels.

4. The image processing method according to claim 3, characterized in that: The initial coordinate index includes an initial row coordinate vector and an initial column coordinate vector, the initial row coordinate vector includes a plurality of first elements, each of the first elements includes initial row coordinates corresponding to a plurality of pixels in each row of pixels, the initial column coordinate vector includes a plurality of second elements, each of the second elements includes initial column coordinates corresponding to a plurality of pixels in each row of pixels, and the plurality of first elements and the plurality of second elements correspond to each other one by one; The interleaving calculation of the initial coordinate index includes: Performing a first interleaving calculation on a plurality of initial row coordinates in the initial row coordinate vector to obtain an interleaved row coordinate vector indexed by the interleaved coordinates, wherein the interleaved row coordinate vector includes a plurality of third elements, each of which includes interleaved row coordinates corresponding to a plurality of pixels in each row of pixels; and Performing a second interleaving calculation on a plurality of initial column coordinates in the initial column coordinate vector to obtain an interleaved column coordinate vector indexed by the interleaved coordinate, wherein the interleaved column coordinate vector includes a plurality of fourth elements, each of the fourth elements includes interleaved column coordinates corresponding to a plurality of pixels in each row of pixels, and the plurality of third elements correspond to the plurality of fourth elements in a one-to-one manner; The step of sampling the to-be-interleaved image based on the interleaving coordinate index comprises: Based on multiple interleaved row coordinates of each third element in the interleaved row coordinate vector and multiple interleaved column coordinates of fourth elements in the interleaved column coordinate vector corresponding one-to-one to the third elements, multiple sampling coordinates are obtained, and the sampling coordinates are represented by an interleaved row coordinate in the third element and an interleaved column coordinate in the fourth element corresponding to the position of the interleaved row coordinate.

5. The image processing method according to any one of claims 1 to 4, characterized in that: Also includes: Get the initial image; generating a plurality of viewpoint images based on the initial image; The multiple viewpoint images are combined to generate an image to be interlaced corresponding to the initial image.

6. The image processing method according to claim 5, characterized in that: The multiple viewpoint images are merged to generate an image to be interlaced corresponding to the initial image, comprising: The multiple viewpoint images are spliced ​​along the row direction of the viewpoint images to generate an image to be interlaced corresponding to the initial image.

7. The image processing method according to claim 5, characterized in that: Also includes: storing the plurality of to-be-interleaved images in a disk cache; as well as In response to the memory cache having storage space, loading the to-be-interleaved image stored in the disk cache into the memory cache for storage; Wherein, obtaining the image to be interlaced includes: Reading a current to-be-interleaved image for interleaving calculation from the memory buffer; In response to the current to-be-interleaved image not existing in the memory cache, reading the current to-be-interleaved image from the disk cache; and The current to-be-interleaved image read from the disk cache is loaded into the memory cache for storage.

8. The image processing method according to claim 7, characterized in that: Acquiring the image to be interlaced further includes: In response to the memory buffer overflowing, the earliest to-be-interleaved image stored in the memory buffer is deleted.

9. The image processing method according to claim 5, characterized in that: The multiple initial images used to generate the multiple images to be interlaced are multiple continuous video frames in the target video.

10. The image processing method according to any one of claims 1 to 4, characterized in that: The plurality of 3D images have serial number identifiers, and the image processing method further comprises: The multiple 3D images obtained by the interleaving calculation are sequentially stored in a queue according to the serial number identification; The step of performing 3D display based on a plurality of 3D images corresponding to the plurality of images to be interlaced includes: The 3D images are sequentially acquired from the queue according to the serial number identifiers for video encoding to achieve 3D video display.

11. An image processing device, characterized in that: include: An acquisition module, configured to acquire a plurality of images to be interlaced; an interlacing calculation module, configured to perform interlacing calculations on at least two of the multiple images to be interlaced in parallel, so as to obtain 3D images corresponding to the multiple images to be interlaced; The display module is configured to perform 3D display based on a plurality of 3D images corresponding to the plurality of images to be interlaced.

12. An electronic device, characterized in that: include: Memory; At least one processor, the memory stores a computer program, and the computer program can be executed by the at least one processor to implement the image processing method according to any one of claims 1 to 10.

13. A display system, characterized in that: include: The electronic device as claimed in claim 12.

14. A non-transitory computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the computer program implements the image processing method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the image processing method according to any one of claims 1 to 10.

Images