A method for realizing efficient optimized rendering and fusion with pose anti-distortion
By combining the performance of CPU, VPU, and GPU, hardware decoding and map rendering of panoramic videos is solved, and the problem of low panoramic rendering performance in the existing technology is realized, efficient panoramic rendering and pose anti-distortion processing is achieved, and multiple projections are supported. The mode and high resolution have promoted the implementation of panoramic applications.
Patent Information
- Application Number
- CN201910218901.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-03-21
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-03-21
AI Technical Summary
The existing technology is difficult to achieve efficient panoramic rendering and pose anti-distortion processing. The traditional CPU instruction acceleration method is inefficient. Hardware acceleration is mainly limited to video decoding. It fails to effectively combine the performance of GPU, CPU, and VPU, resulting in poor VR rendering performance. .
The input data is collected through the CPU, the VPU performs hardware decoding, and the GPU performs map rendering, including projection modeling, chroma space conversion, and posture fusion. It uses multiple projection modes and anti-distortion processing, and combines the performance of CPU, VPU, and GPU for the overall hardware. Performance optimization.
It realizes efficient panoramic video rendering, supports multiple projection modes and high resolution, greatly reducing hardware requirements, promoting the implementation of panoramic applications, and is suitable for panoramic rendering with 2K, 4K and higher resolutions.
Smart Images

Figure CN111726594B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the embedded field, and in particular to a method for realizing efficient optimized rendering and fusion with posture anti-distortion. Background Art
[0002] With the rise of VR (virtual reality), how to make the most commonly used handheld devices support the generation and output of VR panoramas has become a hot research issue.
[0003] Most of the commonly used IVS (independent software vendor) solutions are based on CPU instruction acceleration, for example: decoding acceleration is achieved through assembly acceleration; however, for panoramic rendering, traditional instruction-based acceleration methods are basically not feasible because it involves too many matrix operations, and basic instruction acceleration is far from enough; although there are also hardware-based acceleration, they are basically limited to video decoding; and for application scenarios with more matrix operations such as panorama and anti-distortion, traditional CPUs are very inefficient; with the rise of VR, there are also GPU-based rendering, but the GPU (image processing unit), CPU (general computing processing unit), VPU (video processing unit) and VR-related common attributes are not combined, making it difficult to achieve overall hardware performance optimization and VR rendering. Summary of the invention
[0004] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a method for realizing efficient optimized rendering and fusion with posture anti-distortion, with multiple projection modes, utilizing the performance of VPU, CPU, and GPU to solve the optimization of overall hardware performance and VR rendering of high-resolution panoramic video.
[0005] The present invention provides a method for realizing efficient optimization rendering and fusion with posture anti-distortion, the method comprising the following steps:
[0006] S1: Using the CPU to collect external input data, the external input data includes four types, namely panoramic or 3D data source, posture information, FOV, and projection mode;
[0007] S2: Use VPU to perform hardware decoding and transfer the decoded output to GPU;
[0008] S3: Using GPU to perform texture rendering, the texture rendering includes projection modeling, color space down-conversion, FOV initialization and posture fusion;
[0009] S4: Perform loop iterations according to the requirements of the video or image.
[0010] Furthermore, the panoramic or 3D data source is a video or an image, and the video and the image are in a panoramic format with equal longitude and latitude, or in a 3D format; the posture information is output data of a device that can provide three-dimensional posture information; and FOV is a display field of view.
[0011] Furthermore, the projection modes include a plane projection mode, a spherical projection mode, and a cube projection mode.
[0012] Furthermore, anti-distortion is also a special projection mode.
[0013] Furthermore, the texture rendering includes the following steps:
[0014] S3.1: Projection modeling based on projection mode;
[0015] S3.2: Initialize the GPU according to external input data;
[0016] S3.3: stitching and fusion by custom vertex shader;
[0017] S3.4: Color space conversion via custom fragment shader;
[0018] S3.5: Posture fusion based on pose information and FOV;
[0019] S3.6: Use the GPU pipeline for real-time rendering and use the ping-pong buffer mechanism to control the output display.
[0020] As described above, the present invention provides a method for realizing efficient optimized rendering and fusion with posture anti-distortion, which has the following beneficial effects: the present invention makes full use of the common attributes related to VR such as the CPU, VPU, and GPU of handheld devices to render panoramic videos and pictures, and combines the requirements of large resolution, different projection modes, posture information, FOV, and other requirements of panoramic videos and images, and effectively utilizes hardware acceleration to complete rendering requirements, thereby providing effective technical guarantees for general embedded systems to efficiently complete panoramic rendering, making full use of the hardware capabilities of general systems, greatly reducing the hardware requirements for panoramic rendering, and greatly promoting the implementation of panoramic applications, for decoding and processing all 2K, 4K, 6K and future higher resolution panoramas. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 Shown is an overall flow chart of the implementation method disclosed in the embodiment of the present invention;
[0022] Figure 2 Shown is a relationship diagram among the CPU, GPU, and VPU disclosed in the embodiment of the present invention;
[0023] Figure 3 Shown is a flow chart of the splicing and fusion steps disclosed in an embodiment of the present invention;
[0024] Figure 4 Shown is a flow chart of the color space down-conversion steps disclosed in an embodiment of the present invention;
[0025] Figure 5 Shown is a flowchart of the posture fusion steps disclosed in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0027] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0028] like Figure 1 and Figure 2 As shown, the present invention provides a method for realizing efficient optimization rendering and fusion with posture anti-distortion, and the method comprises the following steps:
[0029] S1: Using the CPU to collect external input data, the external input data includes four types, namely panoramic or 3D data source, posture information, FOV (field of view), and projection mode;
[0030] The panoramic or 3D data source may be a video or an image, and the video and the image are in a panoramic format with equal longitude and latitude, and generally have an aspect ratio of 2:1. If it is stereoscopic, the left-right ratio is 4:1, and the top-bottom ratio is 1:1; the posture information is output data of a device that can provide three-dimensional posture information, such as thetas / Phi / Gamma (rotation angles in three-dimensional space along the XYZ directions), which is generally the output data of a gyroscope, but is not limited to a gyroscope; FOV generally refers to the display field angle, and commonly used ones are 90°, 110°, 130°, etc.; the commonly used projection modes are plane projection mode, spherical projection mode, cubic projection mode, etc.;
[0031] In addition, anti-distortion is also a special projection mode;
[0032] S2: Use VPU to perform hardware decoding and transfer the decoded output to GPU;
[0033] In traditional video rendering, video decoding is of utmost importance. For panoramic rendering, video decoding is not the most important, but it is also an important link. The VPU is used for hardware decoding to obtain texture input, and the texture input is passed to the GPU for high-performance computing.
[0034] In addition to the common video decoding data, the texture input also includes watermark or logo data;
[0035] S3: Using GPU to perform texture rendering, the texture rendering includes projection modeling, color space down-conversion, FOV initialization and posture fusion;
[0036] The texture rendering comprises the following steps:
[0037] S3.1: Projection modeling based on projection mode;
[0038] S3.2: Initialize the GPU according to external input data;
[0039] S3.3: stitching and fusion are performed through a customized vertex shader. Taking spherical projection as an example, the spherical XYZ coordinates, UV (texture) coordinates, weights, and vertex order coordinates of the panoramic or 3D data source are calculated according to the customized vertex shader, and the panoramic or 3D data source is stitched and fused; Figure 3 As shown, the number of cells in each row and column is obtained from the original image of the panoramic or 3D data source through LUT (lookup table). By configuring the number of rows and columns, the quality and efficiency of rendering can be balanced;
[0040] The LUT can be the output data after self-calibration, or it can be generated by a third-party tool such as PT-GUI, which is a lookup table used to expand the specific position after feature matching;
[0041] S3.4: Use custom fragment shaders to downconvert the color space, such as Figure 4 As shown, the transformation matrix from YUV space to RGB space is configured according to the custom fragment shader, and the YUV data of each frame is converted into RGB information suitable for LCD display;
[0042] S3.5: Perform posture fusion based on pose information and FOV, such as Figure 5 As shown, first, according to the FOV input, the view projection based on the FOV is obtained, then the fusion projection is performed based on the pose matrix, and finally the final projection is obtained according to the initialized magnification attribute;
[0043] S3.6: Use the GPU pipeline for real-time rendering and use the ping-pong buffer mechanism to control the output display.
[0044] S4: loop iteration according to the requirements of video or image;
[0045] Anti-distortion, logos, and watermarks can be seen as a special form of common rendering, and the same logic applies. Through the customization of vertex shaders and fragment shaders, anti-distortion processing is performed while common rendering is performed, and watermarks and logos are rendered and output.
[0046] In summary, the present invention makes full use of hardware acceleration, provides a comprehensive method for real-time video pipeline rendering with multiple projection modes, supports FOV (field of view), posture information, etc., and solves the VR rendering of high-resolution panoramic videos by utilizing the performance of VPU, CPU, and GPU. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.
[0047] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A method for realizing efficient optimized rendering and fusion with posture anti-distortion, characterized in that: The method is completed by using common attributes related to VR on the handheld device, wherein the common attributes related to VR include CPU, VPU, and GPU. The method includes the following steps: S1: Using the CPU to collect external input data, the external input data includes four types, namely panoramic or 3D data source, posture information, FOV, and projection mode; S2: Use VPU to perform hardware decoding to obtain texture input, and transmit the decoded texture input output to GPU, where the texture input includes video decoding data, watermark and logo data; S3: Using GPU to perform texture rendering, the texture rendering includes projection modeling, color space down-conversion, FOV initialization and posture fusion; the texture rendering includes the following steps: S3.1: Projection modeling based on projection mode; S3.2: Initialize the GPU according to external input data; S3.3: stitching and fusing through a custom vertex shader; calculating the texture coordinates, weights, and vertex order coordinates of the panoramic or 3D data source according to the custom vertex shader, and stitching and fusing the panoramic or 3D data source; S3.4: Perform color space conversion through a custom fragment shader; configure the transformation matrix from YUV space to RGB space according to the custom fragment shader, and convert the YUV data of each frame into RGB information suitable for LCD display; S3.5: Perform posture fusion based on posture information and FOV; first, obtain the view projection based on FOV according to the input of FOV, then perform fusion projection based on the posture matrix, and finally obtain the final projection according to the initialized magnification attribute; S3.6: Use the GPU pipeline to perform real-time rendering and use the ping-pong buffer mechanism to control the output display; Through the customization of the vertex shader and the fragment shader, anti-distortion processing is performed while common rendering is performed, and watermarks and logos are rendered and output; S4: Perform loop iterations according to the requirements of the video or image.
2. The method for realizing efficient optimized rendering and fusion with posture anti-distortion according to claim 1, characterized in that: The panoramic or 3D data source is a video or an image, and the video and the image are in a panoramic format or a 3D format with equal longitude and latitude; the posture information is the output data of a device that can provide three-dimensional posture information; and FOV is a display field of view.
3. The method for realizing efficient optimized rendering and fusion with posture anti-distortion according to claim 1, characterized in that: The projection modes include a plane projection mode, a spherical projection mode, and a cube projection mode.
Citation Information
Patent Citations
Virtual reality VR equipment-based image display method and device
CN107844190A
360 degree video system with coordinate compression
US20170339391A1
Interface for application-specified playback of panoramic video
US20180174619A1
