A video image transmission format method for realizing high-resolution and ultra-wide viewing angle three-dimensional display with good hardware compatibility

Abstract

This invention discloses a video image transmission format method with good hardware compatibility for high-resolution ultra-wide-viewing-angle 3D display, comprising the following steps: 1) acquiring the 3D model or light field information of the scene, and further acquiring orthogonal RGBD images of multiple viewpoints of the scene; 2) fusing the acquired depth images, and sorting the RGB images and the images containing depth information to form an image set; 3) rearranging the pixel information of the same row of all images in the set and storing it in several rows of the image data for transmission, ensuring that the hardware device can efficiently read the pixel data of the same row of all viewpoints within a limited storage bandwidth. This invention is compatible with existing mainstream video compression encoding, significantly improving the transmission efficiency of video data and effectively meeting the needs of high-resolution, ultra-wide-viewing-angle 3D display.

Description

Technical Field

[0001] This invention relates to a video image transmission format method with good hardware compatibility for high-resolution ultra-wide-view 3D display, belonging to the field of 3D reconstruction technology. Background Technology

[0002] Visual information is the primary means by which humans acquire information about the outside world, with over 70% of our perception of the external world originating from visual images. However, traditional two-dimensional images can only display planar information, while real-world scenes are three-dimensional, leading to a loss of spatial information during two-dimensional display. In recent years, with the development of glasses-free 3D display technology, 3D display content has gradually moved towards higher resolution and ultra-wider viewing angles. Simultaneously, advancements in 3D reconstruction algorithms (such as 3DGS and NeRF) have made acquiring 3D information more convenient, significantly reducing the difficulty of creating 3D display content. However, 3D displays require processing multi-viewpoint information and massive amounts of 3D video data. In high-resolution and wide-field-of-view scenarios, transmission and storage bottlenecks are particularly prominent. Especially in real-time video streaming applications, the high-speed RAM storage resources of hardware devices (such as FPGAs) are limited, typically only able to store a portion of the input image's line information. Therefore, achieving efficient processing of high-resolution, wide-field-of-view 3D video data under limited storage bandwidth and hardware access constraints has become a technical challenge. To effectively address the problems of insufficient memory and the inability to process excessively large video data in real time, a new video image transmission format is proposed. This format takes multi-viewpoint color and depth images as input and arranges the same row of pixel information for each viewpoint image in several rows of data. This ensures that the hardware device can efficiently read the same row of pixel data for all viewpoints within a limited storage bandwidth. The format is compatible with existing mainstream video compression codecs, enabling efficient video data transmission and smooth playback. Summary of the Invention

[0003] The purpose of this invention is to provide a video image transmission format method with good hardware compatibility for high-resolution ultra-wide-view 3D display. This method acquires the 3D model or light field information of the scene, and further obtains orthogonal RGBD images of multiple viewpoints of the scene. The same row of pixel information of each viewpoint is rearranged and stored in several rows of new image data for transmission. This ensures that the hardware device can efficiently read the same row of pixel data of all viewpoints within a limited storage bandwidth. The format is compatible with existing mainstream video compression encoding, which not only greatly improves the transmission efficiency of video data, but also effectively meets the needs of high-resolution and ultra-wide-view 3D display.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows: A novel video format method with good hardware compatibility for realizing high-resolution ultra-wide-view 3D display, comprising the following steps:

[0005] Step 1) Based on the 3D model or 3D light field information of the scene, obtain orthogonal RGB images and corresponding depth information images from multiple viewpoints of the scene;

[0006] Step 2) Fuse the depth maps of all acquired viewpoints to form a set of multiple viewpoint RGB images and the fused depth information image;

[0007] Step 3) Arrange all pixel information of the same row of all images in the set in several rows of data of the output image to ensure that the hardware can efficiently read the same row of pixel data of all viewpoints within the limited storage bandwidth, and transmit it in compressed encoding.

[0008] As an improvement of the present invention, the acquisition method of step 1) is as follows: the real scene model is directly extracted by three-dimensional reconstruction algorithms such as NeRF and 3DGS, or scene information is obtained by a light field camera or by directly using an existing model to obtain orthogonal RGBD images of the scene from three or more viewpoints.

[0009] As an improvement of the present invention, in step 1), based on the obtained scene orthogonal RGBD image, the RGB image of each viewpoint is output sequentially from left to right, followed by the output of the corresponding depth image, until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1, Depth1, RGB2, Depth2, ..., RGB}. n Depth n Based on the number of viewpoints n, the number of images in set A is 2n.

[0010] As an improvement to the present invention, the depth map fusion method in step 2) is as follows: Depth maps from every three viewpoints are fused in a left-to-right order and stored in the RGB channels of an image respectively. If an image's RGB channels are not completely filled, unused channels are set to 0. The fused depth maps are then sequentially placed after the corresponding viewpoint RGB images to form a new set B, which has the following form:

[0011] B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3,RGB4,RGB5,

[0012] RGB6,Depth4 / Depth5 / Depth 6, ...}

[0013] Sort the images in set B starting from 0, and we get:

[0014] B = {img0, img1, img2, img3, ...}

[0015] Where img0 = RGB1, img1 = RGB2, img2 = RGB3, img3 = Depth1 / Depth2 / Depth3, and so on.

[0016] The number of images m in set B is calculated as follows:

[0017]

[0018] Where n is the number of viewpoints. This indicates rounding up to the nearest integer.

[0019] As an improvement to this invention, in step 2), the resolution h×w of the RGB image and corresponding depth image of each viewpoint is determined based on the number of viewpoints n and the image resolution H×W of the display screen. These are collectively referred to as the resolution h×w of the sub-image. The relationship between h and w and H and W is shown in the following formula:

[0020]

[0021] The conditions that k should satisfy are:

[0022] (1) All are integers;

[0023] (2) The total resolution of all images in set B after stitching together is greater than or equal to the image resolution H×W of the display screen.

[0024] H×W≤m×h×w (4) Code implementation steps:

[0025] 1) Find a k value that satisfies condition (1) within the scope of the requirements;

[0026] 2) Find the values ​​that satisfy condition (1) The value of k;

[0027] 3) Find the largest k value among the values ​​that satisfy condition (2), and calculate the sub-image pixels:

[0028] As an improvement to the present invention, if step 2) is performed when the amount of transmitted data is small, the sub-graph step can be easily obtained. The modified code implementation steps are as follows:

[0029] 1) Find a k value that satisfies condition (1) within the scope of the requirements;

[0030] 2) Find the values ​​that satisfy condition (1) The value of k;

[0031] 3) Select any k value that meets the conditions, and calculate the sub-image pixels:

[0032] As an improvement of the present invention, step 3) can be arranged periodically in a row-interlaced manner, and the formula for calculating the pixel position is as follows:

[0033] 1) When k < 3, each row stores the row information of 2 subgraphs, and the specific formula is as follows:

[0034] First row (columns 1-w):

[0035]

[0036] Where T represents the crossover period, indicating that every T rows of the image, another sub-image information is arranged; This represents the sub-image stored in the first half of the image in the i-th row, and the sub-image number is obtained by... To calculate; Calculate the row number storing the subgraph; This represents rounding down to the nearest integer.

[0037] The second half of the row (columns 1+w-2w):

[0038]

[0039] Where T represents the crossover period, indicating that every T rows of the image, another sub-image information is arranged; This represents the subgraph stored in the second half of the i-th row, with the subgraph numbering passed through... To calculate; Calculate the row number storing the subgraph; This represents rounding down to the nearest integer.

[0040] 2) When k < 4, each row stores the row information of 3 subgraphs, and the specific formula is as follows:

[0041] First 1 / 3 (columns 1-w):

[0042]

[0043] Part 2 / 3 (columns 1+w-2w):

[0044]

[0045] The last third (columns 1 + 2w - 3w):

[0046]

[0047] 3) When k < 5, each row stores the row information of 4 subgraphs, and the specific formula is as follows:

[0048] First 1 / 4 section (columns 1-w):

[0049]

[0050] Part 2 / 4 (columns 1+w-2w):

[0051]

[0052] Part 3 / 4 (columns 1+2w-3w):

[0053]

[0054] The last quarter (columns 1 + 3w - 4w):

[0055]

[0056] As an improvement of the present invention, in step 3), a suitable number of interlaced lines T is selected according to the selected video transmission compression format. If the hardware computing power and storage resources are sufficient, T can be increased.

[0057] As an improvement of the present invention, in step 3), the pixel positions of all sub-images (m in total) in one cycle can be arranged arbitrarily, and then periodically filled according to the arrangement format.

[0058] Beneficial effects

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] 1. The video image transmission format proposed in this invention stores the information of pixels in the same row from each viewpoint in several rows of image data, enabling hardware devices to efficiently read pixel data in the same row from all viewpoints within limited storage bandwidth. Compared to traditional methods, this reduces hardware cache requirements, improves data reading speed, and simplifies the data processing flow.

[0061] 2. This video image transmission format, constructed using multi-viewpoint subgraphs, is particularly suitable for video streaming in high-resolution, ultra-wide-angle scenarios. The format is compatible with existing mainstream video compression encoding methods. By optimizing data arrangement, it significantly improves the transmission efficiency of wide-angle video, ensuring real-time, smooth 3D display and free-viewpoint video stream processing under high-resolution conditions. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the processing flow of the present invention;

[0063] Figure 2 Taking the three viewpoints described in Embodiment 1 of the present invention as an example, this is a schematic diagram of an image array format with T = 4;

[0064] Figure 3 This is a schematic diagram of an image array format using the five viewpoints described in step 3) of the present invention as an example;

[0065] Figure 4 Taking the seven viewpoints described in Embodiment 2 of the present invention as an example, this is a schematic diagram of an image array format with T = 4;

[0066] Figure 5 This is a schematic diagram of an image array format using the twelve viewpoints described in step 3) of the present invention. Detailed Implementation

[0067] The invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, those skilled in the art will find that various equivalent modifications to the invention fall within the scope defined by the appended claims.

[0068] Example 1:

[0069] See Figure 1 and Figure 3 A novel video format method with good hardware compatibility for achieving high-resolution ultra-wide-view 3D display, the method comprising the following steps:

[0070] Step 1) Obtain the 3D model or 3D light field information of the scene, and obtain orthogonal RGB images and corresponding depth information images from multiple viewpoints of the scene.

[0071] Here we choose a relatively convenient method: directly using existing models. Those skilled in the art can also choose to use, but are not limited to: 1. Directly extracting real-world scene models through 3D reconstruction algorithms such as NeRF and 3DGS; 2. Obtaining scene information through a light field camera.

[0072] Taking three viewpoints as an example, the RGB image of each viewpoint is output sequentially from left to right, followed by the corresponding depth image, until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1,Depth1,RGB2,Depth2,RGB3,Depth3}. Based on the number of viewpoints n = 3, the number of images in set A is 6.

[0073] Step 2) The acquired depth maps are fused to form a set of multiple viewpoint RGB images and their depth information;

[0074] The depth maps in set A are then fused: Following a left-to-right order, the depth maps from the three viewpoints are fused and stored separately in the RGB channels of an image, resulting in a new image storing the depth information. The fused depth maps are then sequentially placed after the corresponding viewpoint RGB images, forming a new set B, with the following form:

[0075] B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3}

[0076] Sort the images in set B starting from 0, and we get:

[0077] B = {img0, img1, img2, img3}

[0078] Where img0 = RGB1, img1 = RGB2, img2 = RGB3, and img3 = Depth1 / Depth2 / Depth3

[0079] The number of images m in set B is:

[0080]

[0081] Where n is the number of viewpoints. This indicates rounding up to the nearest integer.

[0082] Based on the resolution H×W of the video image on the display screen (taking 7680×4320 as an example), and the number of viewpoints n=3, determine the resolution h×w of the RGB image and corresponding depth image for each viewpoint. From the given conditions, the maximum k value satisfying the conditions for the three viewpoints is 2. Therefore:

[0083]

[0084] Step 3) Arrange the pixels periodically using a row-interlacing method. The formula for calculating the pixel position is as follows:

[0085] When k=2 and T=4, each row stores the row information of 2 subgraphs, and the specific formula is as follows:

[0086] First half of the row (columns 1-3840):

[0087]

[0088] This represents the sub-image stored in the first half of the image in the i-th row, and the sub-image number is obtained by... To calculate; Calculate the row number storing the subgraph; This represents rounding down. The second half of the row (columns 3841-7680):

[0089]

[0090] This represents the subgraph stored in the second half of the i-th row, with the subgraph numbering passed through... To calculate; Calculate the row number storing the subgraph; This represents rounding down to the nearest integer.

[0091] The final arrangement can be obtained from Figure 2 As shown.

[0092] Example 2:

[0093] See Figure 1 and Figure 4 A novel video format method with good hardware compatibility for achieving high-resolution ultra-wide-view 3D display, the method comprising the following steps:

[0094] Step 1) Obtain the 3D model or 3D light field information of the scene, and obtain orthogonal RGB images and corresponding depth information images from multiple viewpoints of the scene.

[0095] Here we choose a relatively convenient method: directly using existing models. Those skilled in the art can also choose to use, but are not limited to: 1. Directly extracting real-world scene models through 3D reconstruction algorithms such as NeRF and 3DGS; 2. Obtaining scene information through a light field camera.

[0096] Taking seven viewpoints as an example, the RGB image of each viewpoint is output sequentially from left to right, followed by the corresponding depth image, until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1,Depth1,RGB2,Depth2,...,RGB7,Depth7}. Based on the number of viewpoints n = 7, the number of images in set A is 14.

[0097] Step 2) The acquired depth maps are fused to form a set of multiple viewpoint RGB images and their depth information;

[0098] The depth maps in set A are then fused: Following a left-to-right order, the depth maps of every three viewpoints are fused and stored in the RGB channels of a separate image, resulting in three new images storing depth information. These fused depth maps are then sequentially placed after the corresponding viewpoint RGB images, forming a new set B, in the following form:

[0099] B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3,RGB4,RGB5,RGB6,Depth4 / Depth5 / Depth6,RGB7,Depth7 / 0 / 0}

[0100] Sort the images in set B starting from 0, and we get:

[0101] B = {img0, img1, img2, img3, ...}

[0102] Where img0 = RGB1, img1 = RGB2, img2 = RGB3, img3 = Depth1 / Depth2 / Depth3, and so on.

[0103] The number of images m in set B is:

[0104]

[0105] Where n is the number of viewpoints. This indicates rounding up to the nearest integer.

[0106] Based on the resolution H×W of the video image on the display screen (taking 7680×4320 as an example), and the number of viewpoints n=10, determine the resolution h×w of the RGB image and corresponding depth image of each viewpoint. From the given conditions, the maximum k value satisfying the conditions for three viewpoints is 3.2. Therefore:

[0107]

[0108] Step 3) Arrange the pixels periodically using a row-interlacing method. The formula for calculating the pixel position is as follows:

[0109] When k = 3.2 and T = 4, each row stores the row information of 3 subgraphs, and the specific formula is as follows:

[0110] First 1 / 3 (columns 1-1350):

[0111]

[0112] Part 2 / 3 (columns 1351-2700):

[0113]

[0114] The last third (columns 2701-4050):

[0115]

[0116] The final arrangement can be obtained from Figure 4 As shown.

[0117] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A video image transmission format method with good hardware compatibility for high-resolution ultra-wide-viewing-angle 3D display, characterized in that, Includes the following steps: Step 1) Based on the 3D model or 3D light field information of the scene, obtain orthogonal RGB images and corresponding depth information images from multiple viewpoints of the scene; Step 2) Fuse the depth maps of all acquired viewpoints to form a set of multiple viewpoint RGB images and the fused depth information image; Step 3) Arrange all pixel information of the same row of all images in the set in several rows of data of the output image to ensure that the hardware can efficiently read the same row of pixel data of all viewpoints within the limited storage bandwidth, and transmit it in compressed encoding. In step 2), based on the obtained orthogonal RGBD image of the scene, the RGB image of each viewpoint is output sequentially from left to right, followed by the corresponding Depth image, until the RGB and depth images of all viewpoints are output, forming a set A = {RGB1, Depth1, RGB2, Depth2, ..., RGB}. n Depth n Based on the number of viewpoints n, the number of images in set A is 2n; The depth map fusion method in step 2) is as follows: Depth maps from every three viewpoints are fused from left to right and stored in the RGB channels of an image. If an image's RGB channels are not completely filled, unused channels are set to 0. The fused depth maps are then placed sequentially after the corresponding viewpoint's RGB images, forming a new set B, which takes the following form: B={RGB1,RGB2,RGB3,Depth1 / Depth2 / Depth3, RGB4, RGB5, RGB6,Depth4 / Depth5 / Depth 6, ...} Sort the images in set B starting from 0, and we get: B = {img0, img1, img2, img3, ...} Where img0=RGB1, img1=RGB2, img2=RGB3, img3=Depth1 / Depth2 / Depth3, and so on. The number of images m in set B is calculated as follows: Where n is the number of viewpoints, and ⌈⋅⌉ represents rounding up; In step 2), based on the number of viewpoints n and the image resolution H×W of the display screen, the resolution h×w of the RGB image and the corresponding depth image of each viewpoint are determined, collectively referred to as the sub-image resolution h×w. The relationship between h and w and H and W is shown in the following formula: The conditions that k should satisfy are: (1) , All are integers. (2) The total resolution of all images in set B after stitching together is greater than or equal to the image resolution H×W of the display screen, that is... Code implementation steps: 1) Find a k value that satisfies condition (1) within the scope of the requirements; 2) Find the values ​​that satisfy condition (1) The value of k; 3) Find the largest k value among the values ​​that satisfy condition (2), and calculate the sub-image pixels: , ; In step 3), the pixels are periodically arranged in a row-interlaced manner, and the formula for calculating the pixel position is as follows: 1) When k < 3, each row stores the row information of 2 subgraphs, and the specific formula is as follows: First row (columns 1-w): Where T represents the number of intersecting rows, indicating that every T rows of the image, another sub-image is arranged; This represents the sub-image stored in the first half of the image in the i-th row, and the sub-image number is obtained by... To calculate; Calculate the row number storing the subgraph; This represents rounding down. The second half of the row (columns 1+w-2w): Where T represents the number of intersecting rows, indicating that every T rows of the image, another sub-image is arranged; This represents the subgraph stored in the second half of the i-th row, with the subgraph numbering passed through... To calculate; Calculate the row number storing the subgraph; This represents rounding down. 2) When k < 4, each row stores the row information of 3 subgraphs, and the specific formula is as follows: First 1 / 3 (columns 1-w): Part 2 / 3 (columns 1+w-2w): The last third (columns 1 + 2w - 3w): 3) When k < 5, each row stores the row information of 4 subgraphs, and the specific formula is as follows: First 1 / 4 section (columns 1-w): Part 2 / 4 (columns 1+w-2w): Part 3 / 4 (columns 1+2w-3w): The last quarter (columns 1 + 3w - 4w): 。 2. The video image transmission format method for achieving high-resolution ultra-wide-angle 3D display with good hardware compatibility according to claim 1, characterized in that: The acquisition method in step 1) is as follows: extract the real scene model directly through NeRF, 3DGS three-dimensional reconstruction algorithm, or obtain scene information through light field camera or directly use existing model to obtain orthogonal RGBD images of the scene from three or more viewpoints.

3. The video image transmission format method for achieving high-resolution ultra-wide-angle 3D display with good hardware compatibility according to claim 2, characterized in that: If the amount of data to be transmitted is small, the steps to obtain the subgraph can be simplified by modifying the code as follows: 1) Find a k value that satisfies condition (1) within the scope of the requirements; 2) Find the values ​​that satisfy condition (1) The value of k; 3) Select any k value that meets the conditions, and calculate the sub-image pixels: , .

4. The video image transmission format method for achieving high-resolution ultra-wide-view 3D display with good hardware compatibility according to claim 3, characterized in that: Based on the selected video transmission compression format, choose an appropriate number of interlaced lines T. If the hardware computing power and storage resources are sufficient, T can be increased.

5. The video image transmission format method for achieving high-resolution ultra-wide-viewing-angle 3D display with good hardware compatibility according to claim 4, characterized in that: The pixel positions of all sub-images in a period can be arranged arbitrarily, and then periodically filled according to this arrangement format, wherein the total number of all sub-images is m.

Images