Video image transmission format method with good hardware compatibility for realizing high-resolution ultra-wide visual angle three-dimensional display

Abstract

The invention discloses a video image transmission format method with good hardware compatibility for realizing high-resolution ultra-wide visual angle three-dimensional display, which comprises the following steps of: 1) acquiring a three-dimensional model or light field information of a scene, and further acquiring a plurality of viewpoint orthogonal RGBD images of the scene; 2) performing fusion processing on the obtained depth images, and sorting RGB images and images containing depth information to form an image set; and 3) rearranging the same row of pixel information of all images in the set, and storing the same row of pixel information in several rows of image data for transmission to ensure that hardware equipment can efficiently read the same row of pixel data of all viewpoints in a limited storage bandwidth. The method is adaptive to existing mainstream video compression coding, the transmission efficiency of video data is greatly improved, and the requirements of high-resolution and ultra-wide-view-angle three-dimensional display are effectively met.

Description

Technical Field

[0001] The invention relates to a video image transmission format method with good hardware compatibility to realize high-resolution and ultra-wide viewing angle three-dimensional display, belonging to the technical field of three-dimensional reconstruction. Background Art

[0002] Visual information is the main way for humans to obtain information from the outside world. About 70% of the perception of the outside world comes from visual images. However, traditional two-dimensional images can only display planar information, while the real world scene is three-dimensional, which leads to the loss of spatial information in the two-dimensional display process. In recent years, with the development of naked-eye three-dimensional display technology, three-dimensional display content has gradually developed in the direction of high resolution and ultra-wide viewing angle. At the same time, the progress of three-dimensional reconstruction algorithms (such as 3DGS, NeRF, etc.) has made it easier to obtain three-dimensional information, greatly reducing the difficulty of producing three-dimensional display content. However, three-dimensional display requires processing multi-viewpoint information and huge three-dimensional video data. In the scene of high resolution and wide field of view, the bottleneck of transmission and storage is particularly prominent. Especially in the application of real-time video streaming, the high-speed RAM storage resources of hardware devices (such as FPGA) are limited, and usually only part of the line information of the input image can be stored. Therefore, under the limited storage bandwidth and hardware access restrictions, it has become a technical problem to realize efficient processing of high-resolution and wide-field-of-view three-dimensional video data. In order to effectively solve the problem of insufficient memory and too large video data to be processed in real time, a new video image transmission format is proposed. This format inputs multi-viewpoint color and depth images, and arranges the same row of pixel information of each viewpoint image in several rows of data, ensuring that the hardware device can efficiently read the same row of pixel data of all viewpoints within the limited storage bandwidth. The format is adapted to the existing mainstream video compression encoding, and can achieve efficient video data transmission and smooth playback. Summary of the invention

[0003] The purpose of the present invention is to provide a video image transmission format method with good hardware compatibility for realizing high-resolution and ultra-wide viewing angle three-dimensional display. The method obtains the three-dimensional model or light field information of the scene, and further obtains orthogonal RGBD images of multiple viewpoints of the scene, rearranges the same row of pixel information of each viewpoint, and stores it in several rows of new image data for transmission, thereby ensuring that the hardware device can efficiently read the same row of pixel data of all viewpoints within a limited storage bandwidth. The format is adapted to the existing mainstream video compression coding, which not only greatly improves the transmission efficiency of video data, but also effectively meets the needs of high-resolution and ultra-wide viewing angle three-dimensional display.

[0004] In order to achieve the above-mentioned purpose, the technical solution of the present invention is as follows: a new video format method with good hardware compatibility to realize high-resolution ultra-wide viewing angle 3D display, comprising the following steps:

[0005] Step 1) obtaining orthogonal RGB images and corresponding depth information images of multiple viewpoints of the scene according to the three-dimensional model or three-dimensional light field information of the scene;

[0006] Step 2) The depth images of all the viewpoints are fused to form a set of multiple viewpoint RGB images and fused depth information images;

[0007] Step 3) Arrange all pixel information of the same row of all images in the set in several rows of data of the output image to ensure that the hardware can efficiently read the same row of pixel data of all viewpoints within the limited storage bandwidth and compress and encode them for transmission.

[0008] As an improvement of the present invention, the method of obtaining step 1) is as follows: directly extracting the real scene model through a three-dimensional reconstruction algorithm such as NeRF, 3DGS, or obtaining scene information through a light field camera or directly using an existing model to obtain orthogonal RGBD images of three or more viewpoints of the scene.

[0009] As an improvement of the present invention, in step 1), according to the obtained scene orthogonal RGBD image, the RGB image of each viewpoint is output in sequence from left to right, and then the corresponding Depth map is output until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1, Depth1, RGB2, Depth2, ..., RGB n ,Depth n}, according to the number of viewpoints n, the number of images in set A is 2n.

[0010] As an improvement of the present invention, the depth map fusion method in step 2) is: in order from left to right, the depth maps of every three viewpoints are fused and stored in the RGB channels of an image respectively. If the RGB channels of an image are not fully filled, the unused channels are set to 0. The fused depth maps are placed behind the corresponding viewpoint RGB images in sequence to form a new set B, which is in the form of:

[0011] B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3,RGB4,RGB5,

[0012] RGB6,Depth4 / Depth5 / Depth 6, ...}

[0013] Sort the images in set B starting from 0 and we get:

[0014] B={img0,img1,img2,img3,...}

[0015] Among them, img0=RGB1, img1=RGB2, img2=RGB3, img3=Depth1 / Depth2 / Depth3, and so on.

[0016] The number of images in set B is m and is calculated as:

[0017]

[0018] Where n is the number of viewpoints, Indicates rounding up.

[0019] As an improvement of the present invention, in step 2), the resolution h×w of the RGB image of the viewpoint and the corresponding depth image is determined according to the number of viewpoints n and the image resolution H×W of the display screen, which is collectively referred to as the resolution h×w of the sub-image. The relationship between h and w and H, W is shown in the following formula:

[0020]

[0021] The conditions that k should satisfy are:

[0022] (1) All are integers;

[0023] (2 The total resolution of all images in set B after splicing is greater than the image resolution H×W of the display screen

[0024] H×W≤m×h×w (4) Code implementation steps:

[0025] 1) Find the k value that satisfies condition (1) within the required range;

[0026] 2) Find the value that satisfies condition (1) The k value;

[0027] 3) Find the largest k value from the values ​​that satisfy condition (2) and calculate the sub-image pixels:

[0028] As an improvement of the present invention, if the amount of data to be transmitted is small, the step 2) can be simply obtained by modifying the code to implement the steps as follows:

[0029] 1) Find the k value that satisfies condition (1) within the required range;

[0030] 2) Find the value that satisfies condition (1) The k value;

[0031] 3) Select any k value that meets the conditions and calculate the sub-image pixels:

[0032] As an improvement of the present invention, the step 3) can be arranged periodically in a row-crossing manner, and the calculation formula of the pixel position is as follows:

[0033] 1) When k<3, each row stores the row information of 2 subgraphs. The specific formula is as follows

[0034] First half of the rows (1-w columns):

[0035]

[0036] T represents the crossover period, which means that every T rows of the image, another sub-image information is arranged; Represents the sub-image stored in the first half of the image in row i, and the sub-image number is represented by to calculate; Calculate the row number where the subgraph is stored; Represents round down.

[0037] The second half of the row (1+w-2w columns):

[0038]

[0039] T represents the crossover period, which means that every T rows of the image, another sub-image information is arranged; Indicates the subgraph stored in the second half of the i-th row, and the subgraph number is to calculate; Calculate the row number where the subgraph is stored; Represents round down.

[0040] 2) When k<4, each row stores row information of 3 subgraphs. The specific formula is as follows

[0041] The first 1 / 3 (columns 1-w):

[0042]

[0043] Part 2 / 3 (columns 1+w-2w):

[0044]

[0045] The last 1 / 3 (1+2w-3w columns):

[0046]

[0047] 3) When k<5, each row stores row information of 4 subgraphs. The specific formula is as follows

[0048] The first 1 / 4 (columns 1-w):

[0049]

[0050] Part 2 / 4 (columns 1+w-2w):

[0051]

[0052] Part 3 / 4 (1+2w-3w columns):

[0053]

[0054] The last 1 / 4 (1+3w-4w columns):

[0055]

[0056] As an improvement of the present invention, the step 3) selects a suitable number of cross-rows T according to the selected video transmission compression format, and T can be increased if the hardware computing power and storage resources are sufficient.

[0057] As an improvement of the present invention, the pixel positions of all sub-images (m in total) in one period of step 3) can be arranged arbitrarily, and then periodically filled according to the arrangement format.

[0058] Beneficial Effects

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The video image transmission format proposed by the present invention stores the information of the same row of pixels of each viewpoint in several rows of image data, so that the hardware device can efficiently read the same row of pixel data of all viewpoints within the limited storage bandwidth. Compared with the traditional method, it reduces the hardware cache requirements, improves the data reading speed, and simplifies the data processing process.

[0061] 2. The video image transmission format composed of multi-viewpoint sub-images is particularly suitable for video stream transmission in high-resolution and ultra-wide viewing angle scenes. The format is adapted to the existing mainstream video compression coding, and by optimizing the data arrangement method, it significantly improves the transmission efficiency of videos with a large field of view, ensuring real-time, smooth 3D display and free viewpoint video stream processing under high-resolution conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the processing flow of the present invention;

[0063] Figure 2 Taking the three viewpoints described in Embodiment 1 of the present invention as an example, a schematic diagram of an image array format where T is 4;

[0064] Figure 3 This is a schematic diagram of an image array format taking the five viewpoints described in step 3) of the present invention as an example;

[0065] Figure 4 Taking the seven viewpoints described in Embodiment 2 of the present invention as an example, a schematic diagram of an image array format where T is 4;

[0066] Figure 5 This is a schematic diagram of an image array format taking the twelve viewpoints described in step 3) of the present invention as an example. DETAILED DESCRIPTION

[0067] The invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the invention and are not used to limit the scope of the invention. After reading the invention, various equivalent modifications of the invention by those skilled in the art fall within the scope defined by the claims attached to this application.

[0068] Embodiment 1:

[0069] See also Figure 1 and Figure 3 , a new video format method with good hardware compatibility to achieve high-resolution ultra-wide viewing angle three-dimensional display, the method comprising the following steps:

[0070] Step 1) obtaining a three-dimensional model or three-dimensional light field information of a scene, obtaining orthogonal RGB images of multiple viewpoints of the scene and corresponding depth information images.

[0071] Here we choose a more convenient way to directly use the existing model. Technicians in this field can also choose to use but are not limited to 1. Directly extract the real scene model through three-dimensional reconstruction algorithms such as NeRF and 3DGS 2. Obtain scene information through light field cameras.

[0072] Here, taking three viewpoints as an example, the RGB image of each viewpoint is output in sequence from left to right, and then the corresponding Depth map is output until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1, Depth1, RGB2, Depth2, RGB3, Depth3}. According to the number of viewpoints n = 3, the number of images in set A is 6.

[0073] Step 2) Fusing the acquired depth map to form a collection of multiple viewpoint RGB images and their depth information;

[0074] The depth maps in the obtained set A are fused: from left to right, the depth maps of the three viewpoints are fused and stored in the RGB channels of an image to obtain a new image storing depth information. The fused depth maps are placed behind the corresponding viewpoint RGB images in sequence to form a new set B, which is in the form of:

[0075] B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3}

[0076] Sort the images in set B starting from 0 and we get:

[0077] B = {img0,img1,img2,img3}

[0078] Among them, img0=RGB1, img1=RGB2, img2=RGB3, img3=Depth1 / Depth2 / Depth3

[0079] The number of images m in set B is:

[0080]

[0081] Where n is the number of viewpoints, Indicates rounding up.

[0082] According to the video image resolution H×W of the display screen, 7680×4320 is taken as an example here, the number of viewpoints n=3, and the resolution h×w of the viewpoint RGB image and the corresponding depth image is determined. From the conditions, it can be obtained that the maximum k value that meets the conditions of three viewpoints is 2, then:

[0083]

[0084] Step 3) The pixels are arranged periodically in a row-crossing manner, and the calculation formula for the pixel position is as follows:

[0085] When k=2, T=4, each row stores the row information of 2 subgraphs. The specific formula is as follows

[0086] The first half of the row (columns 1-3840):

[0087]

[0088] Represents the sub-image stored in the first half of the image in row i, and the sub-image number is represented by to calculate; Calculate the row number where the subgraph is stored; Represents rounding down. The second half of the row (columns 3841-7680):

[0089]

[0090] Indicates the subgraph stored in the second half of the i-th row, and the subgraph number is to calculate; Calculate the row number where the subgraph is stored; Represents round down.

[0091] The final arrangement can be obtained by Figure 2 shown.

[0092] Embodiment 2:

[0093] See also Figure 1 and Figure 4 , a new video format method with good hardware compatibility to achieve high-resolution ultra-wide viewing angle three-dimensional display, the method comprising the following steps:

[0094] Step 1) obtaining a three-dimensional model or three-dimensional light field information of a scene, obtaining orthogonal RGB images of multiple viewpoints of the scene and corresponding depth information images.

[0095] Here we choose a more convenient way to directly use the existing model. Technicians in this field can also choose to use but are not limited to 1. Directly extract the real scene model through three-dimensional reconstruction algorithms such as NeRF and 3DGS 2. Obtain scene information through light field cameras.

[0096] Here, taking seven viewpoints as an example, the RGB image of each viewpoint is output in sequence from left to right, and then the corresponding Depth map is output until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1, Depth1, RGB2, Depth2, ..., RGB7, Depth7}. According to the number of viewpoints n = 7, the number of images in set A is 14.

[0097] Step 2) Fusing the acquired depth map to form a collection of multiple viewpoint RGB images and their depth information;

[0098] The depth maps in the obtained set A are fused: from left to right, the depth maps of each three viewpoints are fused and stored in the RGB channels of an image respectively, and three new images storing depth information are obtained. The fused depth maps are placed behind the corresponding viewpoint RGB images in sequence to form a new set B, which is in the form of:

[0099] B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3,RGB4,RGB5,RGB6,Depth4 / Depth5 / Depth6,RGB7,Depth7 / 0 / 0}

[0100] Sort the images in set B starting from 0 and we get:

[0101] B={img0,img1,img2,img3,...}

[0102] Among them, img0=RGB1, img1=RGB2, img2=RGB3, img3=Depth1 / Depth2 / Depth3, and so on.

[0103] The number of images m in set B is:

[0104]

[0105] Where n is the number of viewpoints, Indicates rounding up.

[0106] According to the video image resolution H×W of the display screen, 7680×4320 is taken as an example here, and the number of viewpoints n=10, the resolution h×w of the viewpoint RGB image and the corresponding depth image is determined. From the conditions, it can be obtained that the maximum k value that meets the conditions for three viewpoints is 3.2, then:

[0107]

[0108] Step 3) The pixels are arranged periodically in a row-crossing manner, and the calculation formula for the pixel position is as follows:

[0109] When k=3.2 and T=4, each row stores row information of 3 subgraphs. The specific formula is as follows

[0110] The first 1 / 3 (columns 1-1350):

[0111]

[0112] Part 2 / 3 (columns 1351-2700):

[0113]

[0114] The last 1 / 3 (columns 2701-4050):

[0115]

[0116] The final arrangement can be obtained by Figure 4 shown.

[0117] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A video image transmission format method with good hardware compatibility for high-resolution ultra-wide viewing angle three-dimensional display, characterized in that: The following steps are involved: Step 1) obtaining orthogonal RGB images and corresponding depth information images of multiple viewpoints of the scene according to the three-dimensional model or three-dimensional light field information of the scene; Step 2) The depth images of all the viewpoints are fused to form a set of multiple viewpoint RGB images and fused depth information images; Step 3) Arrange all pixel information of the same row of all images in the set in several rows of data of the output image to ensure that the hardware can efficiently read the same row of pixel data of all viewpoints within the limited storage bandwidth and compress and encode them for transmission.

2. The video image transmission format method with good hardware compatibility and high resolution ultra-wide viewing angle 3D display according to claim 1, characterized in that: The acquisition method of step 1) is as follows: directly extract the real scene model through NeRF, 3DGS and other three-dimensional reconstruction algorithms, or obtain scene information through a light field camera or directly use an existing model to obtain orthogonal RGBD images of three or more viewpoints of the scene.

3. The video image transmission format method with good hardware compatibility and high resolution and ultra-wide viewing angle 3D display according to claim 1, characterized in that: In step 2), according to the obtained scene orthogonal RGBD image, the RGB image of each viewpoint is output in order from left to right, and then the corresponding Depth map is output until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1, Depth1, RGB2, Depth2, ..., RGB n ,Depth n }, according to the number of viewpoints n, the number of images in set A is 2n.

4. The video image transmission format method with good hardware compatibility and high resolution ultra-wide viewing angle 3D display according to claim 1, characterized in that: The depth map fusion method in step 2) is: in order from left to right, the depth maps of every three viewpoints are fused and stored in the RGB channels of an image respectively. If the RGB channels of an image are not fully filled, the unused channels are set to 0, and the fused depth maps are placed behind the corresponding viewpoint RGB images in sequence to form a new set B, which is in the form of: B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3,RGB4,RGB5,RGB6,Depth4 / Depth5 / Depth 6, ...} Sort the images in set B starting from 0 and get: B={img0,img1,img2,img3,...} Among them, img0=RGB1, img1=RGB2, img2=RGB3, img3=Depth1 / Depth2 / Depth3, and so on. The number of images in set B is m and is calculated as: Where n is the number of viewpoints, Indicates rounding up.

5. The video image transmission format method with good hardware compatibility and high resolution ultra-wide viewing angle 3D display according to claim 1, characterized in that: In step 2), the resolution h×w of the RGB image of the viewpoint and the corresponding depth image is determined according to the number of viewpoints n and the image resolution H×W of the display screen, which is collectively referred to as the resolution h×w of the sub-image. The relationship between h and w and H and W is shown in the following formula: The conditions that k should satisfy are: (1) are all integers, (2) The total resolution of all images in set B after splicing is greater than or equal to the image resolution of the display screen H×WH×W≤m×h×w (4) Code implementation steps: 1) Find the k value that satisfies condition (1) within the required range; 2) Find the value that satisfies condition (1) The k value; 3) Find the largest k value from the values ​​that satisfy condition (2) and calculate the sub-image pixels:

6. The video image transmission format method with good hardware compatibility and high resolution ultra-wide viewing angle 3D display according to claim 1, characterized in that: In step 3), the pixels are arranged periodically in a row-crossing manner, and the calculation formula for the pixel position is as follows: 1) When k<3, each row stores the row information of 2 subgraphs. The specific formula is as follows First half of the rows (1-w columns): Where T represents the number of cross-rows, indicating that every T rows of the image, another sub-image information is arranged; Represents the sub-image stored in the first half of the image in row i, and the sub-image number is represented by to calculate; Calculate the row number where the subgraph is stored; Represents rounding down, the second half of the row (1+w-2w columns): Where T represents the number of cross-rows, indicating that every T rows of the image, another sub-image information is arranged; Indicates the subgraph stored in the second half of the i-th row, and the subgraph number is to calculate; Calculate the row number where the subgraph is stored; represents rounding down, 2) When k<4, each row stores row information of 3 subgraphs. The specific formula is as follows The first 1 / 3 (columns 1-w): Part 2 / 3 (columns 1+w-2w): The last 1 / 3 (1+2w-3w columns): 3) When k<5, each row stores row information of 4 subgraphs. The specific formula is as follows The first 1 / 4 (columns 1-w): Part 2 / 4 (columns 1+w-2w): Part 3 / 4 (1+2w-3w columns): The last 1 / 4 (1+3w-4w columns):

7. The video image transmission format method with good hardware compatibility and high resolution ultra-wide viewing angle 3D display according to claim 5, characterized in that: If the amount of data to be transmitted is small, the sub-graph steps can be easily obtained and the code implementation steps can be modified as follows: 1) Find the k value that satisfies condition (1) within the required range; 2) Find the value that satisfies condition (1) The k value; 3) Select any k value that meets the conditions and calculate the sub-image pixels:

8. The video image transmission format method with good hardware compatibility and high resolution ultra-wide viewing angle 3D display according to claim 6, characterized in that: According to the selected video transmission compression format, select the appropriate number of cross-rows T. If the hardware computing power and storage resources are sufficient, T can be increased.

9. The video image transmission format method with good hardware compatibility and high resolution ultra-wide viewing angle 3D display according to claim 6, characterized in that: The pixel positions of all sub-images (a total of m) of a period can be arranged arbitrarily, and then filled periodically according to the arrangement format.

Images