A 360-degree video hardware rendering method based on region approximation mapping

Abstract

The present invention provides a 360-degree video hardware rendering method based on regional approximate mapping. According to the similarity of the mapping relationship in the space of a single-frame image, the pixel points of the input frame are read and the mapping relationship is calculated according to the sliding window area. The mapping calculation is simplified by simplifying the complex nonlinear mapping relationship into a linear mapping relationship. Only the mapping relationship of the center point needs to be calculated. The mapping relationship of the remaining pixel points in the sliding window area can complete the coordinate mapping from the input frame to the output frame in the sliding window area through simple shift and addition operations. At the same time, the inverse operation of the linear mapping relationship is used to complete the many-to-many mapping relationship between the input frame and the output frame, and reconstruct the output frame. Based on the mapping scheme of the input frame and the output frame, the present invention reduces the memory access overhead by changing the order of operations and rearranging the output frames to reduce the number of output memory accesses.

Description

Technical Field

[0001] The present invention relates to a 360-degree video hardware rendering method based on region approximate mapping. Background Art

[0002] Currently, VR (Virtual Reality) technology has important applications in fields such as medicine, education, and aerospace. Unlike traditional video rendering, 360° video requires rendering input frames based on real-time head motion parameters to display the scene corresponding to the user's current head position. This unique operation consumes a large amount of hardware resources and computing cycles, resulting in a huge computational overhead that can exceed 50% of the entire process and is difficult to implement with hardware.

[0003] To provide users with a more immersive experience, the rendering and display of 360-degree videos place high demands on video resolution and frame rate. The required presentation resolution exceeds 4K, and the input frame resolution far exceeds the output frame resolution, resulting in increased computational overhead, memory access overhead, and energy consumption. Rendering a 720p 360° video at 30 frames per second (FPS) consumes over 4W of power, exceeding the thermal design point (TDP) of a typical mobile device. Furthermore, due to the unique characteristics of 360-degree video, it's difficult to simultaneously read and write input and flat video frames simultaneously. This results in additional memory access overhead and even greater power consumption during hardware implementation.

[0004] There are two main existing hardware rendering solutions. One method maps the output frame coordinates, finds the corresponding pixels in the input frame, transmits the input frame pixels, and reconstructs the entire output frame. In addition to the complex mapping relationship, this approach also introduces a problem: the mapping relationship between the input and output frames reduces the image hit rate, affecting the transmission efficiency of the rendering and reducing the frame rate. The other method directly reads the input frame pixels sequentially, maps them to the output frame, and transmits the pixels simultaneously to complete the reconstruction of the output frame. This method also introduces a complex mapping relationship and incurs greater storage and write access overhead.

[0005] Existing technical solutions mainly have the following two optimization methods. The first is to reduce the number of pixels that need to be mapped and the amount of calculation required for the rendering process. Therefore, in the current rendering solution, the edge pixels of the output frame are mapped to the output frame to determine the range of the 360-degree video input frame that needs to be rendered. This range accounts for approximately 30%-60% of the input frame image, reducing the number of pixels that need to be mapped. In addition, through the focus rendering algorithm, the area outside the focus of the user's perspective is blurred to reduce the number of pixels that need to be rendered and calculated.

[0006] In addition, the correlation between images between frames can be used to reduce the rendering calculations of certain frames. Since the head rotation angle deviation of adjacent frames is relatively small, the adjacent input frame image mappings have a high linear relationship. Therefore, multiple frames are used to calculate and save the mapping relationship of only one frame. At the same time, the linear relationship of the frame image mapping relationship is calculated according to the change in the head rotation angle to reconstruct the output frame images of other frames, reducing the computational complexity of multi-frame rendering calculations.

[0007] For single-frame rendering calculations, the current solution is to pipeline several steps of the mapping process and perform pixel-level parallel rendering based on the SIMD (Single Instruction Multiple Data) instructions of the GPU (graphics processing unit). Summary of the Invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a 360-degree video hardware rendering method based on region approximate mapping, comprising the following steps:

[0009] Step 1: Perform a sliding window operation on the 360-degree video input frame image. The window size is determined according to the current resolution. The step size of the sliding window is the window size. The two-dimensional coordinates of the input frame image within the sliding window area are mapped to the output frame coordinates. The corresponding output frame range is determined by comparison.

[0010] Step 2: Based on the output frame range determined in step 1, reverse the first part of the process to obtain the corresponding input frame coordinate offset value, and obtain the mapping relationship between the input frame and the output frame through threshold comparison;

[0011] Step 3: Complete the reading and writing of image data and reconstruct the complete output frame image.

[0012] Step 1 includes:

[0013] Step 1-1: Map the two-dimensional offset coordinates dx and dy of the input frame coordinate relative to the center point (the center point offset coordinates of the input frame sliding window area are defined as (0, 0). The two-dimensional offset coordinates of the sliding window area relative to the center point are within the range of (-d / 2, d / 2), where d is the size of the sliding window area) to polar coordinates dφ and dθ. The scaling matrix I2P is used for calculation. The I2P matrix is ​​a diagonal matrix. The conversion formula is:

[0014] dφ=dx*base_width / (2*pi),

[0015] dθ=dy*base_height / (pi),

[0016] The conversion process is implemented using a matrix, and we get: I2P = diag([base_width / (2*pi), base_height / pi, 1]);

[0017] Where base_width and base_height are the width and height of the input frame respectively, pi is a constant, pi = 3.1415926; diag represents the diagonal matrix function;

[0018] Step 1-2: Map polar coordinates to three-dimensional coordinates using a sliding window;

[0019] Steps 1-3: Perform rotation operations based on the current head parameters and use the rotation matrix ROT for calculation;

[0020] Steps 1-4 map the rotated 3D matrix to the display parameter 3D coordinate domain and use the mapping matrix TRANS for calculation. The formula is as follows:

[0021]

[0022] Among them, width and height are the width and height of the output frame respectively, fov_h and fov_v are the horizontal field of view angle and vertical field of view angle respectively. These parameters are predetermined, so the matrix TRANS is also a fixed matrix;

[0023] Steps 1-5: Map the three-dimensional spherical coordinates to plane coordinates, and use division to complete the operation to obtain the mapped coordinates of the center point of the input frame;

[0024] Step 1-6, using steps 1-1 to 1-5, complete the mapping of the four corners within the sliding window range of the input frame, compare and obtain the output frame range.

[0025] In steps 1-2, the following formula is used to map polar coordinates to three-dimensional coordinates:

[0026] x=sinφ·cosθ≈sinφ0·cosθ0+cosφ0·cosθ0·dφ-sinφ0·sinθ0·dθ (2)

[0027] y=sinθ≈sinθ0+cosθ0·dθ (3)

[0028] z=cosφ·cosθ≈cosφ0·cosθ0-sinφ0·cosθ0·dφ-cosφ0·sinθ0·dθ (4)

[0029] x, y, z are three-dimensional coordinates, that is, the input frame is saved as a plane, but it reflects a three-dimensional image in space. According to the corresponding polar coordinates, the corresponding three-dimensional coordinates can be calculated;

[0030] Where φ and θ are the polar coordinates corresponding to the input frame, φ0 and θ0 are the coordinates of the center point of the corresponding input frame area, and dφ and dθ are the offsets of the rectangular area relative to the polar coordinates of the center point. Converting formulas (2) to (4) into matrix form, we get the matrix TL:

[0031]

[0032] In steps 1-3, the rotation matrix ROT is as follows:

[0033]

[0034] Among them, yaw and pitch are the yaw angle and pitch angle of the head direction respectively.

[0035] In steps 1-5, the following formula is used to map the three-dimensional coordinates to the plane coordinates:

[0036]

[0037] Where sx, sy, and sz are three-dimensional spherical coordinates.

[0038] Step 1-6 includes: after calculating formulas (2) to (4), convolving the four 3*3 matrices I2P, TL, ROT and TRANS in steps 1-1 to 1-4 to obtain matrix T to implement the coordinate transformation of steps 1-1 to 1-4, and integrating steps 1-5 into matrix T (in the specific mapping process, the coordinates need to be convolved with the T matrix first, and the results need to be operated in steps 1-5. In the implementation process, the matrix T is first operated, and the operations of steps 1-5 are added to the matrix T to obtain the matrix TR. The mapping process is equivalent to directly performing the convolution operation of the matrix TR by first convolving with the matrix T and then executing steps 1-5. The output results of the two are equivalent) to obtain the final mapping from the input frame to the output frame. The transformation matrix TR is expressed as follows:

[0039]

[0040] Among them, T 00 Represents the element in the first row and first column of matrix T.

[0041] In steps 1-6, the two-dimensional offset coordinates dx, dy of the input frame coordinates relative to the center point are convolved with the TR matrix to obtain the output frame coordinates of the four corners of the input frame. The four obtained output frame mapping coordinates are compared to determine the rectangular mapping range of the output frame. The comparison method is to find the maximum and minimum values ​​of the horizontal and vertical coordinates of the four corners of the output frame respectively.

[0042] Step 2 includes: the coordinates of the input frame are located on the plane z=1, and the offset values ​​for the three-dimensional spherical coordinates are:

[0043] dsx·F 02 +dsy·F 12 +dsz·F 22 =diz (9)

[0044] dsx=dox·sz+(ox+dox)·dsz (10)

[0045] dsy=doy·sz+(oy+doy)·dsz (11)

[0046] Where dsx, dsy, and dsz represent the offset value of the x-axis coordinate of the input spherical three-dimensional coordinate, the offset value of the y-axis coordinate of the spherical three-dimensional coordinate, and the offset value of the z-axis coordinate of the spherical three-dimensional coordinate, respectively. ox and oy are the horizontal and vertical coordinates of the output frame, respectively. dox and doy are the offset values ​​of the horizontal and vertical coordinates of the output frame, respectively. The F matrix is ​​the inverse matrix of the matrix T. 02 Represents the element in the first row and third column of the matrix F. Using equations (9) to (11), we can update diz. Since the actual value of diz should be 0, we have:

[0047]

[0048] dsz1=dsz0+ddsz (13).

[0049] Where ddsz represents the error value of the z-axis offset of the spherical 3D coordinates; dsz0 is the given initial value, and dsz1 is the result of one iteration. dsz0 is generally set to 0.

[0050] In step 2, in order to make the calculation result more accurate, the value of dsz is iterated twice using equations (9) to (13) to obtain the second iteration result dsz2, and the value of dsz2 is substituted into (10) and (11) to calculate the results of dsx and dsy. The results of dsx, dsy and dsz2 are convolved with the matrix F to obtain the offset of the input frame coordinates (dix, diy);

[0051] For the coordinates within the output frame range (the coordinate acquisition method is to obtain the coordinates using a counter based on the rectangular mapping range of the output frame determined in steps 1-6), the sliding window operation is also used. First, the offset coordinates of the center point of the sliding window area are calculated (for example, the output frame range is determined to be x∈(-8, 9), y∈(-8, 9), and the area is traversed with a 5*5 sliding window. The offset coordinates of the first sliding window center point are (-5, -5), and the offset coordinates of the second sliding window center point are (0, -5)). The convolution result of the offset coordinates and the F matrix is ​​calculated for the pixel coordinates within the sliding window area. Since the offset value from the center point of the sliding window area is p∈(-d0, d0), where d0 is half the size of the output frame sliding window area and p is an integer, the convolution calculation of the pixel coordinates is completed using shift and addition.

[0052] The step of obtaining the mapping relationship between the input frame and the output frame by comparing the threshold value specifically includes:

[0053] The offsets (dix, diy) of the obtained input frame coordinates are compared with the threshold values.

[0054] The threshold comparison method of diy is: rounding diy, and the rounding result is the offset vertical coordinate within the sliding window area of ​​the input frame corresponding to the output frame;

[0055] The threshold comparison method of dix is ​​as follows: obtain the cosθ value in formula (5), obtain the X (value ranges from 1 to 4) times of cosθ by shifting, perform four threshold comparisons, and obtain the corresponding threshold comparison result: for example, dx∈(3*cosθ,4*cosθ), the output comparison result is 3, and the threshold comparison result is the offset horizontal coordinate in the sliding window area of ​​the input frame corresponding to the output frame.

[0056] Step 3 includes: using the input frame coordinate offset obtained in step 2 to realize image transmission, rounding the obtained input frame coordinate offset, obtaining the pixel value of the corresponding position of the input frame using a nearest neighbor algorithm, and transmitting and constructing a complete output frame image;

[0057] Input frame reading and distribution: Source image data is retrieved from the double-rate synchronous dynamic random access memory (DDR) and sequentially stored (pixel data is read and stored by source image row) into the memory banks in the source data area of ​​the storage array. Eight pixels are retrieved from each memory bank at a time to perform the parallel calculations of steps 1 and 2. The source image data is read using a ping-pong operation, with the previous source image data calculation being performed simultaneously with the current source image data read.

[0058] Writing back the output frame: After collecting the image data within the sliding window area obtained in step 2 (based on the mapping relationship between the input frame and the output frame, the image data of the input frame is reconstructed into the output frame sliding window area through the MUX (data selector) to obtain the image data within the sliding window area), the image data is rearranged. The process is as follows: for a certain column of image data, according to the mapping coordinates of the input frame center point obtained in steps 1-5, each row of image data within the output frame range is arranged into an 8-pixel data packet according to 8 rows of data, and the insufficient positions are filled with 0; according to the mapping coordinates of the input frame center point obtained in steps 1-5, the DDR storage address and image data corresponding to each row are obtained, and the data packets at the corresponding positions of each column of data are written into the FIFO in row order, and the corresponding storage coordinates are generated at the same time; according to the data and coordinates stored in the FIFO, they are written sequentially into the double-rate dynamic random access memory.

[0059] Beneficial effects:

[0060] This solution reads pixels and calculates mapping relationships for the input frame according to the similarity of mapping relationships in the space of a single-frame image. This solution simplifies the mapping calculation by simplifying complex nonlinear mapping relationships into linear mapping relationships. Only the mapping relationship of the center point needs to be calculated. The mapping relationships of the remaining pixels within the sliding window region can be simply calculated through simple shift and addition operations to complete the coordinate mapping from the input frame to the output frame within the sliding window region. Simultaneously, the inverse operation of the linear mapping relationship is used to complete the many-to-many mapping relationship between the input frame and the output frame, and reconstruct the output frame. Furthermore, based on the mapping scheme of the input and output frames, the present invention reduces the number of output memory accesses and thus reduces memory access overhead by changing the order of operations and rearranging the output frames.

[0061] Existing single-frame rendering solutions use GPUs for calculations, dividing the mapping process into multiple parts for transmission and calculation to obtain the final result. When calculating high-resolution images, this simple pixel-level parallel solution consumes a large amount of hardware resources and is difficult to meet high frame rate or high resolution requirements. The rendering solution of the present invention uses linear mapping to pre-calculate the overall rendering mapping relationship within the sliding window area to complete the transmission of regional pixels. At the same throughput rate, it consumes fewer hardware resources. At the same time, the rendering solution of the present invention does not conflict with other existing solutions and can be combined with existing rendering solutions to further improve rendering speed and reduce energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0063] Figure 1 It is a flow chart of the method of the present invention.

[0064] Figure 2 This is a schematic diagram of input frame reading and distribution operations.

[0065] Figure 3 This is a schematic diagram of the distribution and storage of output frames. DETAILED DESCRIPTION

[0066] like Figure 1 As shown, the present invention provides a 360-degree video hardware rendering method based on regional approximate mapping. This method implements rendering based on a new coordinate mapping method and divides the rendering scheme into three parts. The first part performs a sliding window operation on the input frame image, determines the sliding window size according to the current resolution, and maps the two-dimensional coordinates of the input frame image within the sliding window area to the output frame coordinates. The corresponding output frame range is determined by comparison. The second part reverses the first part based on the output frame range determined by the first part to obtain the corresponding input frame coordinate offset value. The mapping relationship between the input frame and the output frame is obtained by threshold comparison. The third part completes the reading and writing of image data to reconstruct the complete output frame image.

[0067] In the first part of the mapping, the following steps are included:

[0068] Step 1-1: Map the input frame pixel offset coordinates dx, dy to polar coordinates dφ, dθ using the scaling matrix I2P.

[0069] Step 1-2: Map polar coordinates to three-dimensional coordinates, map the coordinates with a sliding window, and use Taylor expansion to simplify the calculation;

[0070] Steps 1-3: Perform rotation operations based on the current head parameters and use the rotation matrix ROT for calculation;

[0071] Steps 1-4 map the rotated 3D matrix to the display parameter 3D coordinate domain and use the mapping matrix TRANS for calculation. The formula is as follows:

[0072]

[0073] Steps 1-5: Map the three-dimensional spherical coordinates to plane coordinates, and use division to complete the operation to obtain the mapped coordinates of the center point of the input frame;

[0074] Step 1-6, using steps 1-1 to 1-5, complete the mapping of the four corners within the sliding window range of the input frame, compare and obtain the output frame range.

[0075] Steps 1-1 and 1-4 are fixed matrices, stored in memory, waiting to be read and used.

[0076] In steps 1-2, the following formula is used to map polar coordinates to three-dimensional coordinates:

[0077] x=sinφ·cosθ≈sinφ0·cosθ0+cosφ0·cosθ0·dφ-sinφ0·sinθ0·dθ (2)

[0078] y=sinθ≈sinθ0+cosθ0·dθ (3)

[0079] z=cosφ·cosθ≈cosφ0·cosθ0-sinφ0·cosθ0·dφ-cosφ0·sinθ0·dθ (4)

[0080] Among them, φ and θ are the polar coordinates corresponding to the input frame, φ0 and θ0 are the coordinates of the center point of the corresponding input frame area, and dφ and dθ are the offsets within the rectangular area relative to the polar coordinates of the center point. Convert the above formula into matrix form:

[0081]

[0082] In steps 1-3, the rotation matrix ROT is as follows:

[0083]

[0084] Among them, yaw and pitch are the yaw angle and pitch angle of the head direction respectively.

[0085] In steps 1-5, the following formula is used to map the three-dimensional coordinates to the plane coordinates:

[0086]

[0087] Where sx, sy, and sz are three-dimensional spherical coordinates, and dx and dy are the two-dimensional coordinate offsets of the input frame coordinates relative to the center point.

[0088] After expanding 1-2, the matrices of the first four steps can be convolved to obtain matrix T to implement the coordinate transformation of the first four steps. At the same time, the fifth step is also integrated into matrix T to obtain the final mapping from input frame to output frame, which is expressed as follows:

[0089]

[0090] Where T is the corresponding transformation matrix, and the subscripts are the matrix elements at the corresponding positions.

[0091] In the calculation process of the above matrices, due to the similarity between the TL matrix and the ROT matrix, and the fact that the ROT matrix only needs to be calculated once in one frame of image, the present invention uses the same circuit to calculate the TL and ROT matrices. Only different element values ​​are stored during the final matrix storage. The same operation can also complete the inverse matrix calculation of the two matrices.

[0092] In steps 1-6, the x,y coordinate offset values ​​dx,dy of the input frame are convolved with the TR matrix to obtain the output frame coordinates of the four corners. Since the values ​​of dx and dy are ±d / 2, where d is the step size of the sliding window, and since the values ​​of dx and dy are also fixed after d is determined, the convolution calculation process can be completed by shifting and adding, and the size of the obtained four corner mapping coordinates is compared to determine the rectangular mapping range of the output frame.

[0093] In the second part, the calculations in the first part are reversed to obtain the offset value of the input frame. When solving the inverse matrix of the T matrix, since the ROT matrix and the TL matrix are trigonometric matrices, their inverse matrices are also matrices composed of trigonometric functions. Therefore, the results of the ROT matrix and the TL matrix can be reused and the elements can be rearranged to form the inverse matrix.

[0094] The I2P and TRANS matrices are pre-calculated fixed matrices, and their inverse matrices can also be calculated in advance without requiring additional operations.

[0095] By performing a convolution operation on the above inverse matrix, the inverse matrix F of the T matrix can be obtained.

[0096] Since step 5 of the first part involves division and the denominator is not a fixed value, the inverse operation is performed as follows.

[0097] Since the coordinates of the input frame are located on the plane of z = 1, the offset values ​​for the three-dimensional spherical coordinates are:

[0098] dsx·F 02 +dsy·F 12 +dsz·F 22 =diz (9)

[0099] dsx=dox·sz+(ox+dox)·dsz (10)

[0100] dsy=doy·sz+(oy+doy)·dsz (11)

[0101] For the input dsz value, use equations (9) to (11) to complete the update of diz and obtain diz_part. Since the real value of diz should be 0, we have:

[0102]

[0103] dsz1=dsz0+ddsz (13)

[0104] In order to make the calculation result more accurate, the value of dsz is iterated twice using equations (9) to (13). The initial value of dsz is 0, and the final result of dsz2 is obtained. The value of dsz2 is substituted into (10) and (11) to calculate the results of dsx and dsy. The results of dsx, dsy and dsz2 are convolved with the F matrix to obtain the offset of the input frame coordinates.

[0105] For the coordinates within the output frame range, the sliding window operation is also used. First, the convolution result of the offset coordinates of the center point of the sliding window area and the F matrix is ​​calculated. For other points in the sliding window, since the offset value from the center point of the sliding window area is p∈(-d0,d0), where d0 is half the size of the sliding window area of ​​the output frame and p is an integer, the convolution calculation of the remaining pixel points is completed using shift and addition.

[0106] In the third part, the offset obtained in the second part is used to realize the transmission of the image, the obtained offset is rounded, and the pixel value of the corresponding position of the input frame is obtained using the nearest neighbor algorithm, and the complete output frame image is transmitted and constructed.

[0107] Due to the nonlinear mapping relationship between input and output frames, operations such as sequentially reading input frame data or sequentially writing back output frames will result in a low hit rate or bandwidth waste in the other memory access method. Combined with the calculation method mentioned above, some processing needs to be done on the distribution and writing back of the image to reduce the bandwidth waste of writing back the output frames.

[0108] Reading and distribution of input frames: The source image data is taken out from the double rate synchronous dynamic random access memory (DDR), and the row image data of the input frame is read and stored in the memory bank (SRAM) of the source data area of ​​the storage array. LINE0 represents the image data of the first row, LINE1 represents the image data of the second row, and so on. Each time, eight pixels (DATA0~DATAn) are taken out from each memory bank for parallel calculation. The source data is read using a ping-pong operation. The previous source data calculation is performed while the current source data is being read, which speeds up the data reading speed. The specific reading process is as follows: Figure 2 shown.

[0109] Writing back the output frame: After collecting the image data in the sliding window area, the data is rearranged. The process is as follows: for a column of image data, according to the coordinate results of the output frame, each column of data is arranged into an 8-pixel data packet (DATA0~DATAn) with 8 rows of data, and the missing positions are filled with 0; for the data packets at the corresponding positions, they are written into the FIFO (First Input First Output) in row order, and the corresponding storage coordinates are generated according to the output frame center point mapping coordinates obtained in steps 1-5; according to the data and coordinates stored in the FIFO, they are sequentially written into the double-rate dynamic random access memory, such as Figure 3 shown.

[0110] Example:

[0111] This embodiment implements the rendering of 2k resolution input frames on ZCU104, and the corresponding output frame resolution is 1k*1k. In the implementation scheme of this embodiment, the step size of the sliding window is determined to be 8, and the calculation amount for the corresponding sliding window area is as follows. For 2k resolution, the sliding window area needs to be calculated based on the current angle. When the pitch angle is 0 degrees, the number of sliding window areas determined is 4900 (the number of valid sliding windows needs to be further filtered). As for the range of the inverse operation, it also needs to be determined based on the mapping range from the input frame to the output frame. For a 5*5 area, the maximum range of the output frame is detected to be a range of 14*14. Each sliding window area requires a maximum of 9 inverse operations, as shown in Table 1.

[0112] Table 1

[0113]

[0114]

[0115] The computational complexity of the original solution (pixel-level parallelism) (8*8 area) is shown in Table 2:

[0116] Table 2

[0117] trigonometric functions 3*3 matrix convolution division multiplication addition Shift Single frame fixed calculation 2 1 0 6 0 0 Input frames are mapped to output frames 16 0 64 832 512 0 total 18 1 64 838 512 0 The mapping relationship of the output frame is determined / / / / / /

[0118] Mapping input frames to output frames represents the specific computational effort of the present invention's pixel-level parallelism implementation on an FPGA. The method for ultimately determining the mapping relationship for the output frames is not specifically provided, so the specific computational effort cannot be given. However, it can be seen that the computational effort for trigonometric functions and division is significantly reduced in this embodiment of the present invention.

[0119] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, can execute the invention content of the 360-degree video hardware rendering method based on regional approximate mapping provided by the present invention and some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0121] The present invention provides a 360-degree video hardware rendering method based on regional approximation mapping. There are numerous methods and approaches for implementing this technical solution. The above is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A 360-degree video hardware rendering method based on region approximate mapping, characterized in that: The following steps are involved: Step 1: Perform a sliding window operation on the 360-degree video input frame image. The window size is determined according to the current resolution. The step size of the sliding window is the window size. The two-dimensional coordinates of the input frame image within the sliding window area are mapped to the output frame coordinates. The corresponding output frame range is determined by comparison. Step 2: Based on the output frame range determined in step 1, reverse the first part of the process to obtain the corresponding input frame coordinate offset value, and obtain the mapping relationship between the input frame and the output frame through threshold comparison; Step 3: Complete the reading and writing of image data and reconstruct the complete output frame image; Step 1 includes: Step 1-1: Map the two-dimensional offset coordinates dx, dy of the input frame coordinates relative to the center point to polar coordinates dφ, dθ. The scaling matrix I2P is used for calculation. The I2P matrix is ​​a diagonal matrix. The conversion formula is: dφ=dx*base_width / (2*pi), dθ=dy*base_height / (pi), The conversion process is implemented using a matrix, and we get: I2P = diag([base_width / (2*pi), base_height / pi, 1]); Where base_width and base_height are the width and height of the input frame respectively, pi is a constant; diag represents the diagonal matrix function; Step 1-2: Map polar coordinates to three-dimensional coordinates using a sliding window; Steps 1-3: Perform rotation operations based on the current head parameters and use the rotation matrix ROT for calculation; Steps 1-4 map the rotated 3D matrix to the display parameter 3D coordinate domain and use the mapping matrix TRANS for calculation. The formula is as follows: Among them, width and height are the width and height of the output frame respectively, fov_h and fov_v are the horizontal field of view angle and vertical field of view angle respectively; Steps 1-5: Map the three-dimensional spherical coordinates to plane coordinates, using division to complete the operation and obtain the mapped coordinates of the center point of the input frame; Step 1-6, using steps 1-1 to 1-5, complete the mapping of the four corners within the sliding window range of the input frame, compare and obtain the output frame range.

2. The method according to claim 1, characterized in that In steps 1-2, the following formula is used to map polar coordinates to three-dimensional coordinates: x=sinφ·cosθ≈sinφ0·cosθ0+cosφ0·cosθ0·dφ-sinφ0·sinθ0·dθ (2) y=sinθ≈sinθ0+cosθ0·dθ (3) z=cosφ·cosθ≈cosφ0·cosθ0-sinφ0·cosθ0·dφ-cosφ0·sinθ0·dθ (4) x, y, z are three-dimensional coordinates, that is, the input frame is saved as a plane, reflecting the three-dimensional image in space. According to the corresponding polar coordinates, the corresponding three-dimensional coordinates are calculated; Where φ and θ are the polar coordinates corresponding to the input frame, φ0 and θ0 are the coordinates of the center point of the corresponding input frame area, and dφ and dθ are the offsets of the rectangular area relative to the polar coordinates of the center point. Converting formulas (2) to (4) into matrix form, we get the matrix TL:

3. The method according to claim 2, characterized in that In steps 1-3, the rotation matrix ROT is as follows: Among them, yaw and pitch are the yaw angle and pitch angle of the head direction respectively.

4. The method according to claim 3, characterized in that In steps 1-5, the following formula is used to map the three-dimensional coordinates to the plane coordinates: Where sx, sy, and sz are three-dimensional spherical coordinates.

5. The method according to claim 4, characterized in that Step 1-6 includes: after performing the calculations of formulas (2) to (4), convolving the four 3*3 matrices I2P, TL, ROT, and TRANS in steps 1-1 to 1-4 to obtain the matrix T to implement the coordinate transformation of steps 1-1 to 1-4. At the same time, step 1-5 is also integrated into the matrix T to obtain the final mapping from the input frame to the output frame. The transformation matrix TR is expressed as follows: Among them, T 00 Represents the element in the first row and first column of matrix T.

6. The method according to claim 5, characterized in that In steps 1-6, the two-dimensional offset coordinates dx, dy of the input frame coordinates relative to the center point are convolved with the TR matrix to obtain the output frame coordinates of the four corners of the input frame. The four obtained output frame mapping coordinates are compared to determine the rectangular mapping range of the output frame. The comparison method is to find the maximum and minimum values ​​of the horizontal and vertical coordinates of the four corners of the output frame respectively.

7. The method according to claim 6, characterized in that Step 2 includes: the coordinates of the input frame are located on the plane z=1, and the offset values ​​for the three-dimensional spherical coordinates are: dsx·F 02 +dsy·F 12 +dsz·F 22 =diz (9) dsx=dox·sz+(ox+dox)·dsz (10) dsy=doy·sz+(oy+doy)·dsz (11) Where dsx, dsy, and dsz represent the offset value of the x-axis coordinate of the input spherical three-dimensional coordinate, the offset value of the y-axis coordinate of the spherical three-dimensional coordinate, and the offset value of the z-axis coordinate of the spherical three-dimensional coordinate, respectively. ox and oy are the horizontal and vertical coordinates of the output frame, respectively. dox and doy are the offset values ​​of the horizontal and vertical coordinates of the output frame, respectively. The F matrix is ​​the inverse matrix of the matrix T. 02 Represents the element in the first row and third column of the matrix F. Using equations (9) to (11), we can update diz. Since the actual value of diz should be 0, we have: dsz1=dsz0+ddsz (13) Where ddsz represents the error value of the offset value of the z-axis coordinate of the spherical three-dimensional coordinate; dsz0 is the given initial value, and dsz1 is the result of one iteration.

8. The method according to claim 7, characterized in that In step 2, in order to make the calculation result more accurate, the value of dsz is iterated twice using equations (9) to (13) to obtain the second iteration result dsz2, and the value of dsz2 is substituted into (10) and (11) to calculate the results of dsx and dsy. The results of dsx, dsy and dsz2 are convolved with the matrix F to obtain the offset of the input frame coordinates (dix, diy); For the coordinates within the output frame range, the sliding window operation is also used. First, the convolution result of the offset coordinates of the center point of the sliding window area and the F matrix is ​​calculated. For the pixel coordinates within the sliding window area, since the offset value from the center point of the sliding window area is p∈(-d0,d0), where d0 is half the size of the output frame sliding window area and p is an integer, the convolution calculation of the pixel coordinates is completed using shift and addition. The step of obtaining the mapping relationship between the input frame and the output frame by comparing the threshold value specifically includes: The offsets (dix, diy) of the obtained input frame coordinates are compared with the threshold values. The threshold comparison method of diy is: rounding diy, and the rounding result is the offset vertical coordinate within the sliding window area of ​​the input frame corresponding to the output frame; The threshold comparison method of dix is ​​as follows: obtain the cosθ value in formula (5), obtain the X-times value of cosθ by shifting, perform threshold comparison, and obtain the corresponding threshold comparison result. The threshold comparison result is the offset horizontal coordinate in the sliding window area of ​​the input frame corresponding to the output frame.

9. The method according to claim 8, characterized in that Step 3 includes: using the input frame coordinate offset obtained in step 2 to realize image transmission, rounding the obtained input frame coordinate offset, obtaining the pixel value of the corresponding position of the input frame using a nearest neighbor algorithm, and transmitting and constructing a complete output frame image; Input frame reading and distribution: Source image data is retrieved from the double-rate synchronous dynamic random access memory and sequentially stored in the memory banks of the source data area of ​​the storage array. Eight pixels are retrieved from each memory bank at a time to perform the parallel calculations of steps 1 and 2. The source image data is read using a ping-pong operation, and the previous source image data calculation is performed simultaneously with the current source image data reading. Writing back the output frame: After collecting the image data within the sliding window area obtained in step 2, it is rearranged. The process is as follows: for a column of image data, according to the mapping coordinates of the input frame center point obtained in steps 1-5, each row of image data within the output frame range is arranged into an 8-pixel data packet as 8 rows of data, and the insufficient positions are filled with 0; according to the mapping coordinates of the input frame center point obtained in steps 1-5, the DDR storage address and image data corresponding to each row are calculated, and the data packets corresponding to the positions of each column of data are written into the FIFO in row order, and the corresponding storage coordinates are generated at the same time; according to the data and coordinates stored in the FIFO, they are sequentially written into the double-bit rate dynamic random access memory.

Images