OpenGL ES panoramic and distant view splicing method based on Hiisis SoC

Abstract

The invention provides an OpenGL ES panoramic and distant view splicing method based on Hisilicon SoC, and the method comprises the specific steps: a preprocessing stage: completing the collection calibration, splicing simulation, parameter preprocessing, homography matrix calculation, coordinate mapping and generation of a pixel mapping table required by distant view splicing, and building an OpenGL pipeline; and a rendering stage: performing frame-by-frame pixel mapping and abutted seam fusion in the GPU by using an OpenGL pipeline, so as to improve the processing speed and reduce the burden of the CPU. Compared with the prior art, the real-time splicing of multiple paths of distant view input images is realized on a full-localization embedded platform by using simpler camera calibration and splicing parameter acquisition steps according to the requirements of various vehicle operators on real-time 360-degree panoramic perception of the external environment; and the vacancy that more cameras are used for real-time splicing at a longer distance is made up.

Description

Technical Field

[0001] The invention belongs to the field of image stitching, and in particular to an OpenGLES panoramic view and distant view stitching method based on HiSilicon SoC. Background Art

[0002] The observation requirements of various special-purpose vehicles for external scenes are different from those in daily scenes. The driver and the vehicle commander are located in a closed cockpit or inside the vehicle body and need to obtain continuous scene perception within a 360° range of the vehicle body so that the people on board can complete basic driving or other more complex operational tasks.

[0003] Existing technologies mainly study the stitching of large-field-of-view fisheye cameras. The number of cameras that need to be processed is 4 to 6 to meet the observation requirements around the vehicle body, but it is only applicable to observation close to the vehicle body and cannot cover long-distance observation. Long-distance observation requires cameras with longer focal lengths and smaller viewing angles, and the need for 360° scene perception will inevitably require the use of more cameras with smaller viewing angles and longer observation distances, which greatly increases the processing pressure of real-time stitching.

[0004] However, the number of real-time stitching input sources mentioned in the existing technology cannot meet the needs of long-distance 360° scene perception, and there is a risk that the processing speed cannot meet the real-time requirements. At the same time, this type of camera with a smaller viewing angle and a longer observation distance does not have serious distortion like fisheye cameras. Compared with the existing technology, the calibration method and stitching parameter acquisition can be simplified to a certain extent, which is more convenient for engineering use. In addition, in this special application context, we hope to deploy a purely domestic embedded hardware platform on various vehicles to complete panoramic stitching and other video processing needs, but most of the platforms used in the existing technology do not take into account the requirements of localization.

[0005] In order to speed up the stitching process of multiple video sources, GPU can be used. Common frameworks include OpenGL, OpenCL, CUDA, etc. OpenGL itself is not an API, but a specification developed and maintained by the Khronos organization, which contains a series of functions that can operate graphics and images. OpenGLES (Embedded System) is a 3D graphics library in the embedded field. It is tailored by the Khronos organization based on the desktop OpenGL standard.

[0006] The hardware platform to be used is the HiSilicon HI3559AV100 chip, which is a high-performance, low-power application processor chip integrated with 2 Cortex-A73 and 2 Cortex-A53 and an independent NEON coprocessor. The HI3559A has a built-in ARMMaliG71@900MHz GPU, which is fully compatible with OpenGLES 3.0 / 3.1 / 3.2 and provides EGL1.4, OpenGLES1.1 / 2.0 / 3.0 / 3.1 / 3.2 standard interfaces that fully comply with industry standards. OpenGLES is actually a state machine for a graphics rendering pipeline, while EGL (Embedded-System Graphics Library) is an interface between drawing interfaces such as OpenGL ES and the underlying window system, responsible for maintaining the state of the graphics rendering pipeline and sending the rendering results to the corresponding window or surface. The HI3559A also has a variety of powerful embedded hardware engines built in, providing excellent performance for high-end applications. At the same time, it uses advanced low-power technology and low-power architecture design to provide excellent image processing capabilities. Summary of the invention

[0007] The purpose of the present invention is to provide an OpenGL ES panoramic view stitching method based on HiSilicon SoC, which realizes the real-time stitching of multi-channel panoramic view input images on a domestic embedded platform to meet the needs of various vehicle operators for real-time 360° panoramic perception of the external environment, and makes up for the vacancy of real-time stitching using more cameras at a longer distance.

[0008] The technical solution to achieve the purpose of the present invention is: an OpenGL ES panoramic view stitching method based on HiSilicon SoC, comprising: step 1: preprocessing stage: completing the acquisition calibration, stitching simulation, parameter preprocessing, homography matrix calculation, coordinate mapping and pixel mapping table generation required for the stitching of the view, and building the OpenGL pipeline;

[0009] Step 2: Rendering stage: Use the OpenGL pipeline to perform frame-by-frame pixel mapping and seam blending in the GPU to increase processing speed and reduce the burden on the CPU.

[0010] Compared with the prior art, the present invention has the following significant advantages:

[0011] (1) In the panoramic long-range stitching method of the present invention, different methods are used in multiple links to achieve the real-time requirements. For multiple high-definition cameras as video source input, the steps of pixel mapping table generation and the establishment of vertex coordinates, texture coordinates, index coordinates, and texture units in OpenGL only need to be performed once in the preprocessing stage without repeating, thereby improving processing efficiency; making full use of the characteristics of OpenGL, combined with the requirements of the picture effects required in actual application scenarios, sampling vertex coordinates in alternate rows and columns, compressing the input textures of multiple high-definition cameras, and writing the image fusion steps into the shader for pixel-level processing, so as to break through the bottleneck that limits the rendering speed in the rendering stage, and improve rendering efficiency without affecting the stitching effect.

[0012] (2) In the panoramic view stitching method of the present invention, the localization requirements of the vehicle-mounted mobile platform are fully considered, and the HiSilicon HI3559AV100 chip is selected to implement the present invention. In addition to real-time video stitching, this chip can also realize other functions that may be required in more actual scenarios, such as recognition, fisheye camera correction, graphics drawing, display mode switching control, etc., which meets the requirements of localization while also having scalability.

[0013] (3) The panoramic view stitching method of the present invention combines the characteristics of the adopted camera and fully utilizes the existing mature software, thus simplifying the calibration and stitching parameter acquisition steps, bringing convenience to engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the implementation of the OpenGL ES panoramic view stitching method based on HiSilicon SoC. DETAILED DESCRIPTION

[0015] The terms used in the present invention are only for the purpose of illustrating the embodiments of the present invention and are not intended to limit the present invention. Figure 1 , some embodiments of the present invention are described in detail.

[0016] The scenarios to which the method proposed in this paper is applicable generally use 6 to 8 or even more than a dozen high-definition cameras as video source inputs. Whether it is the generation of pixel mapping tables or the establishment of various OpenGL coordinates or pixel mapping, a large amount of calculations are involved. Therefore, the generation of pixel mapping tables, which takes a lot of time, and the establishment of vertex coordinates, texture coordinates, index coordinates, and texture units in OpenGL only need to be performed once in the preprocessing stage. The real-time rendering stage is handed over to the GPU for processing, and only texture unit refresh and pixel mapping need to be completed. That is, the splicing of video frames only needs to render the continuously refreshed texture pixels frame by frame in the OpenGL pipeline. At the same time, in the implementation process, grid sampling, image compression and other processing are performed for some steps with large computational volume and high resource consumption. The specific operations can be seen in the implementation steps 1.5 and 2.1. The splicing method based on the above ideas can improve the processing speed of multi-channel camera splicing.

[0017] In the above situation, the OpenGL ES panoramic view and distant view stitching method based on HiSilicon SoC disclosed in the present invention designs video stitching into two stages: preprocessing stage and rendering stage. The preprocessing stage completes the acquisition calibration, stitching simulation, parameter preprocessing, homography matrix calculation, coordinate mapping and pixel mapping table generation required for distant view stitching, and builds the OpenGL pipeline (Graphics Pipeline). The rendering stage uses the OpenGL pipeline to perform frame-by-frame pixel mapping and stitching fusion in the GPU to improve the processing speed and reduce the burden on the CPU.

[0018] The specific implementation steps for using multiple cameras to achieve 360° long-range stitching on the HiSilicon HI3559AV100 platform are as follows:

[0019] Step 1: Preprocessing stage:

[0020] Step 1.1: Collect images: Build a calibration environment, place multiple cameras in a similar way to actual usage scenarios, and ensure that there is an overlapping area between them. Collect multiple images in different directions within a 360° observation range.

[0021] Step 1.2: Stitching simulation: Import the collected multiple images from different perspectives into the PTGui tool for stitching simulation, including adjusting the registration point, perspective transformation, distance adjustment, display FOV adjustment, etc., and finally obtain a pto file containing stitching parameters such as resolution, field of view, lens distortion parameters, yaw angle, pitch angle, roll angle, etc.

[0022] Step 1.3: Generate homography matrix H: Read the pto file, obtain the intrinsic parameter matrix and rotation matrix of each camera, and calculate the projection matrix, that is, the homography matrix H.

[0023] Step 1.4: Generate a pixel mapping table LUT (Look-Up-Table): According to the projection matrix H, the mapping coordinates of each pixel are calculated and stored as a pixel mapping table LUT.

[0024] Step 1.5: Initialize the OpenGL pipeline: Sample and mesh the target panoramic image pixels in alternate rows and columns, and determine the mesh vertices as OpenGL vertices. Although the drawing vertices for generating the panorama are not sampled pixel by pixel but in alternate rows and columns, the fragment shader will generate more fragments than the original specified vertices in the rasterization stage, and perform fragment interpolation, which improves rendering efficiency without affecting the observation effect. Establish vertex buffer objects (Vertex Buffer Objects, VBO) to store a large number of established vertices in the video memory. Establish vertex array objects (VertexArray Objects, VAO) to store vertex attribute calls and bind. Establish index coordinates according to the vertex connection drawing order and establish element buffer objects (Element BufferObjects, EBO) to store this index;

[0025] Each vertex can be associated with a texture coordinate to indicate which part of the texture image to sample from, and the pixel map table just records the mapping relationship from a single image to the target panoramic image, so the texture coordinate is established according to the pixel source address and target address stored in the pixel map table.

[0026] Step 1.6: Write OpenGL shader using GLSL (OpenGL Shading Language): Write vertex shader GLSL file to transmit and specify the vertex position to be rendered. Write fragment shader to render the pixel RGB value described by texture coordinates to the correct vertex coordinates. Considering that there are obvious seams in the spliced ​​images that need to be fused, the double distance weighted method is used in the fragment shader to smooth the seams. The double distances are the distances from the two adjacent image pixels on the left and right of the seam to the seam. While mapping, the adjacent image pixels within the seam fusion feathering range are multiplied by the weights for fusion.

[0027] Step 1.7: Create multiple texture units to store and update real-time video frames.

[0028] Step 1.8: Build the video path: According to the API in the HiSilicon GPU Development User Guide and HiMPPV4.0 Media Processing Software Development Reference, initialize the EGL (Embedded-System Graphics Library), VI (Video Iutput, video input), and VO (Video Output, video output) modules to build the video input and output paths.

[0029] Step 2: Rendering phase:

[0030] Step 2.1: Convert the YUV raw video frame obtained by the HiSilicon VI module into RGB format and pass it to the texture unit after compression.

[0031] The update of the texture unit is the process of mapping CPU data to the GPU. The method mentioned in this article involves a large number of camera video sources, which will directly affect the processing time of each frame. The generated long strip target panorama will also be compressed when displayed on a standard monitor. Therefore, compressing the images collected by each camera and then passing them to the texture unit will not only not affect the display effect, but also greatly improve the processing performance.

[0032] Step 2.2: OpenGL ES rendering: Use the vertex shader and fragment shader, vertex attribute configuration, vertex data and index coordinates prepared in the preprocessing stage, assemble the primitives according to the vertex coordinates and index coordinates in the pipeline, and map the textures corresponding to the multiple video frames to the corresponding vertices according to the texture coordinates to obtain the target panoramic image.

[0033] Step 2.3: EGL directly outputs the rendered stitching results to the HiSilicon VO module. This completes the video stitching.

[0034] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. An OpenGL ES panoramic view stitching method based on HiSilicon SoC, characterized in that: The specific steps include: Step 1: Preprocessing stage: complete the acquisition calibration, stitching simulation, parameter preprocessing, homography matrix calculation, coordinate mapping and pixel mapping table generation required for distant view stitching, and build the OpenGL pipeline; Step 2: Rendering stage: Use the OpenGL pipeline to perform frame-by-frame pixel mapping and seam blending in the GPU to increase processing speed and reduce the burden on the CPU.

2. The OpenGL ES panoramic view and distant view stitching method based on HiSilicon SoC according to claim 1, characterized in that: Specific steps of the preprocessing stage: Step 1.1: Collect images: Build a calibration environment, place multiple cameras in a similar way to real-life scenarios, and ensure that there is an overlap between them. Collect multiple images in different directions within a 360° observation range. Step 1.2: Stitching simulation: Import the collected multiple images from different perspectives into the PTGui tool for stitching simulation to obtain the pto file of stitching parameters; Step 1.3: Generate homography matrix: read the pto file, get the intrinsic matrix and rotation matrix of each camera, and calculate the projection matrix, i.e. the homography matrix; Step 1.4: Generate pixel mapping table: According to the projection matrix, calculate the mapping coordinates of each pixel and store them as a pixel mapping table; Step 1.5: Initialize the OpenGL pipeline: sample and mesh the target panoramic image pixels in alternate rows and columns, determine the mesh vertices as OpenGL vertices, and establish index coordinates and texture coordinates; Step 1.6: Write OpenGL shader using GLSL language; Step 1.7: Establish multiple texture units to store and update real-time video frames; Step 1.8: Build the video path: According to the API in the HiSilicon GPU Development User Guide and HiMPPV4.0 Media Processing Software Development Reference, initialize the EGL, VI, and VO modules, and build the video input and output paths.

3. The OpenGL ES panoramic view and distant view stitching method based on HiSilicon SoC according to claim 3, characterized in that: In step 1.5, initialize the OpenGL pipeline. The specific steps are: Create a vertex buffer object to store the created vertices in the video memory; Create a vertex array object to store vertex attribute calls and bind; Create index coordinates according to the vertex connection drawing order and create an element buffer object to store this index; Texture coordinates are established based on the pixel source address and target address stored in the pixel mapping table, and each vertex is associated with a texture coordinate to indicate the initial sampling position from the texture image.

4. The OpenGL ES panoramic view and distant view stitching method based on HiSilicon SoC according to claim 3, characterized in that: In step 1.6: Write a vertex shader GLSL file to transfer and specify the vertex positions to be rendered; Write a fragment shader GLSL file to render the pixel RGB value described by the texture coordinates to the correct vertex coordinates; Considering that there are obvious seams in the stitched images that need to be merged, the double distance weighted method is used in the fragment shader to smooth the seams.

5. The OpenGL ES panoramic view and distant view stitching method based on HiSilicon SoC according to claim 4, characterized in that: The double distances in the double distance weighted method are the distances from the pixels of the two adjacent images on the left and right sides of the seam to the seam. While mapping, the pixels of the adjacent images within the feathering range of the seam are multiplied by the weights for fusion.

6. The OpenGL ES panoramic view and distant view stitching method based on HiSilicon SoC according to claim 1, characterized in that: Step 2: Rendering phase includes: Step 2.1: Convert the YUV raw video frame obtained by the HiSilicon VI module into RGB format, compress it and pass it to the texture unit; Step 2.2: OpenGL ES rendering: Use the vertex shader and fragment shader, vertex attribute configuration, vertex data and index coordinates prepared in the preprocessing stage, assemble the primitives according to the vertex coordinates and index coordinates in the pipeline, and map the textures corresponding to the multiple video frames to the corresponding vertices according to the texture coordinates to obtain the target panoramic image; Step 2.3: EGL directly outputs the rendered stitching results to the HiSilicon VO module, thus completing the video stitching.

Images