A multi-eye 3D panoramic surround viewing system and stitching method

Abstract

The present invention discloses a multi-eye 3D panoramic surround view system and a stitching method, the system comprising: a plurality of cameras, respectively arranged at the front, rear, left and right sides of a vehicle, wherein at least one side is provided with two or more cameras to eliminate blind spots; an image stitching module, respectively connected to each of the cameras, for stitching images collected by each camera based on a pre-built virtual 3D model to form a virtual 3D panoramic surround view image, when the virtual 3D model is constructed, if more than two cameras are provided on the same side, the texture vertices overlapped by each camera on the same side are segmented using the Y axis as a fusion segmentation curve. The present invention can be applied to various types of vehicles to realize multi-eye 3D panoramic surround view, while effectively eliminating blind spots.

Description

Technical Field

[0001] The present invention relates to the technical field of assisted driving vision systems, and in particular to a multi-eye 3D panoramic surround view system and a splicing method. Background Art

[0002] 3D-360° panoramic view is a key assisted driving function in the assisted driving system. In the existing technology, a 3D-360° panoramic view system usually installs a fisheye camera in the front, rear, left and right directions of the vehicle body, and then uses image processing and computer vision technology to place the panoramic view of the vehicle body's surrounding environment and a three-dimensional model car in a three-dimensional environment, and virtually forms a 3D-360° panoramic view image of the vehicle's surrounding environment from a virtual perspective.

[0003] However, if only one camera is arranged on each side of the vehicle, for special-shaped engineering machinery equipment or long-body, long-box and other engineering machinery equipment, a blind spot will be created, making it impossible to obtain a 360° panoramic image. For example, for special-shaped engineering machinery equipment, if only one camera is arranged in the front, rear, left and right directions of the vehicle, the engineering arm and other devices on the carrier will block the field of view of the camera installed on the carrier body. Figure 1 As shown, it is impossible to complete the 360° surround stitching effect; for engineering machinery equipment with long bodies and long boxes, such as Figure 2 As shown, since the carrier body is too long, if there is only one camera on the left and right sides, the field of view is often insufficient, and it is impossible to cover the overlapping areas in the front and rear directions, resulting in blind spots, and thus it is impossible to completely and clearly form a surround stitching image around the carrier body.

[0004] Since the 3D-360° surround view system of the traditional 4-eye camera is usually based on OpenGL to build a virtual 3D model, such as a flat-bottomed bowl model, a boat-shaped model, etc., the real environment images of the 4 cameras are mapped and rendered onto the virtual 3D model to form a 3D-360° surround view display around the carrier body. The 4 cameras generate corresponding 3D texture vertex Grids for OpenGL rendering, and rotate 0°, 90°, 180°, and 270° in the front, back, left, and right directions based on the same coordinate origin (0,0,0) to achieve that the grids corresponding to the two cameras are perpendicular to each other. If it is considered to directly add cameras to the 3D-360° surround view system of the traditional 4-eye camera to eliminate the blind spot problem, when more than two cameras are installed on the same side, there will be overlapping areas between the images of the cameras. According to the image stitching method of the traditional 4-eye camera, it is impossible to segment the overlapping areas between the cameras on the same side. Therefore, directly adding cameras on the basis of the traditional 4-eye camera 3D-360° surround view method cannot achieve the 3D-360 surround view effect.

[0005] like Figure 3 As shown in the figure, in the 3D-360° surround view system of the traditional four-camera, the 3D texture vertex Grid corresponding to each camera defines 8 boundary points p1~p8 according to the same standard; the 3D texture vertex Grid corresponding to each camera forms a virtual 3D flat-bottomed bowl model after rotation, but the Grid points in the overlapping area exist in the Grids of two different cameras, which need to be divided into the Grid parts of the corresponding cameras for texture rendering, that is, the Grids of different cameras need to be divided. If the fusion segmentation line of the form y=ax+b is directly used for segmentation, that is, the fusion segmentation line is composed of S 1 and S 2 Two points determine the equation of a line, S 1 For line segment P 1 P 8 and line segment P 1 'P 8 'The intersection point, S 2 For line segment P 2 With P 7 ', since the 3D texture vertex Grids generated by the cameras installed on the same side are completely consistent and have the same rotation angle, the line segment P of the camera on the same side 1 P 8 and line segment P 1 'P 8 'Overlap, infinite intersections make it impossible to determine point S 1 ,Therefore, it is impossible to determine the linear equation of the dividing line, that is, it is impossible to achieve the segmentation of the overlapping area. Summary of the invention

[0006] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a multi-eye 3D panoramic view system and splicing method which has a simple structure, low cost, can be applied to various types of vehicles to achieve 3D panoramic view, and eliminate blind spots.

[0007] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0008] A multi-eye 3D panoramic surround viewing system, characterized by comprising:

[0009] A plurality of cameras are arranged at the front, rear, left and right sides of the vehicle, wherein at least two or more cameras are arranged on at least one side to eliminate blind spots;

[0010] The image stitching module is connected to each of the cameras respectively, and is used to stitch the images captured by each of the cameras based on a pre-built virtual 3D model to form a virtual 3D panoramic image. When the virtual 3D model is constructed, if more than two cameras are set on the same side, the texture vertices that overlap between the two cameras on the same side are segmented using the Y-axis as a fusion segmentation curve.

[0011] Furthermore, when applied to special-shaped engineering machinery vehicles, more than two cameras are set on the side with the special-shaped structure; when applied to vehicles whose left and right sides of the vehicle body are longer than a preset threshold, more than two of the cameras are set on the left and right sides of the vehicle respectively.

[0012] Furthermore, the image stitching module includes:

[0013] A model building unit, used for building a virtual 3D model for each camera based on the 3D flat-bottomed bowl model, and using the Y axis as a fusion segmentation curve to segment the overlapping area, and then rendering the texture grid points to build the virtual 3D model;

[0014] The real-time model rendering unit is used to acquire the images collected by each of the cameras in real time, and to render the virtual 3D model in real time according to the acquired images to obtain a real-time virtual 3D panoramic surround image.

[0015] Furthermore, the model construction unit comprises an internal and external parameter calibration subunit, a 3D model construction subunit, a fusion segmentation subunit and a texture network point rendering subunit which are connected in sequence. The internal and external parameter calibration subunit is used to calibrate the internal and external parameters of the camera (1); the 3D model construction subunit is used to construct a half-flat-bottomed bowl model for each camera; the fusion segmentation subunit is used to segment the texture vertices that overlap between two cameras on the same side in the half-flat-bottomed bowl model of each camera using the Y axis as a fusion segmentation curve; and the texture network point rendering subunit is used to perform texture grid point rendering on the half-flat-bottomed bowl model of each camera according to the calibrated internal and external parameters to obtain the virtual 3D model.

[0016] A multi-eye 3D panoramic view stitching method, the steps comprising:

[0017] S01. Pre-arrange cameras on the front, rear, left and right sides of the vehicle, wherein two or more cameras are provided on at least one side to eliminate blind spots;

[0018] S02. The images captured by each of the cameras are stitched together based on a pre-built virtual 3D model to form a virtual 3D panoramic surround image. When the virtual 3D model is constructed, if more than two cameras are set on the same side, the overlapping texture vertices of the two cameras on the same side are segmented using the Y-axis as the fusion segmentation curve.

[0019] Furthermore, the step S02 includes:

[0020] S201. Read local parameter files, including triangular texture files and 3D vehicle models;

[0021] S202. Receive images captured by each of the cameras;

[0022] S203. Render the vehicle body surrounding environment image onto a pre-built virtual 3D model according to the received image and the read file to form the virtual 3D panoramic surround image.

[0023] Furthermore, the steps of constructing the virtual 3D model include:

[0024] SA1. Calibrate the intrinsic and extrinsic parameters of the camera in advance to obtain the intrinsic and extrinsic parameter matrices;

[0025] SA2. Generate a half-sided flat-bottomed bowl model spanning one or two quadrants for each camera;

[0026] SA3. In the half-side flat-bottomed bowl model of each camera (, the texture vertices that overlap between the two cameras on the same side are segmented using the Y axis as the fusion segmentation curve;

[0027] SA4. Perform texture grid point rendering on the half-flat-bottomed bowl model of each camera formed after segmentation in step SA3 according to the intrinsic and extrinsic parameter matrices to obtain the virtual 3D model.

[0028] Furthermore, step SA4 includes:

[0029] SA41. According to the internal and external parameter matrices, the 3D grid points corresponding to each camera are projected into the 2D image, and the pixel coordinates of the image corresponding to the 3D grid points are calculated;

[0030] SA42. Set the segmentation boundary of each camera corresponding to the half-flat-bottomed bowl model to classify each half-flat-bottomed bowl model into a fusion area and a non-fusion area, wherein the fusion area is the overlapping area photographed by adjacent cameras, and the non-fusion area is the area photographed independently by each camera, and generate a binary image for segmenting the fusion area and the non-fusion area;

[0031] SA43. Rotate each of the half-flat-bottom models by a specified angle, wherein the 3D grid point models corresponding to each camera arranged on the same side are rotated by the same angle, and save the grid points and corresponding image coordinates of each half-flat-bottom bowl model in the form of a triangular texture file.

[0032] Furthermore, when calibrating the internal parameters of the camera, a distortion correction processing step is also included, including: generating a dedistorted mapping file according to the internal parameter matrix K, the distortion system matrix D and the scaling factor sf.

[0033] Furthermore, the step of calibrating the external parameters of the camera includes:

[0034] Arrange external calibration objects around the vehicle body and capture the corner points of each external calibration object: convert the camera dedistorted image containing the external calibration object image into a binary image, locate the position of the external calibration object, and calculate the image coordinates of each corner point of the external calibration;

[0035] According to the image coordinates and world coordinates of the corner points of the extrinsic calibration object captured by each camera, the extrinsic rotation matrix R and extrinsic translation matrix T of each camera are calculated.

[0036] Compared with the prior art, the advantages of the present invention are:

[0037] 1. The present invention installs one camera in each of the four directions of the front, rear, left and right of the carrier vehicle body, and installs two or more cameras on at least one side according to the actual structural characteristics of the carrier vehicle to eliminate blind spots, and then stitches the images collected by each camera based on a virtual 3D model, places a panoramic view of the vehicle body surrounding environment and a three-dimensional model car in a three-dimensional environment, and virtualizes a virtual perspective whose position, angle and direction can be continuously adjusted, so as to obtain a 3D-360° panoramic surround stitching image of the vehicle surrounding environment.

[0038] 2. The present invention uses the Y-axis as the fusion segmentation curve to segment the overlapping texture vertices of the two cameras on the same side when constructing a virtual 3D model. This can effectively segment the overlapping area of ​​the cameras on the same side relatively evenly without missing any vertices, thereby solving the problem in the traditional method that the overlapping texture vertex Grids corresponding to the two cameras installed on the same side cannot be segmented, thereby effectively eliminating blind spots and achieving a multi-eye 3D-360° panoramic viewing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a structural schematic diagram of a special-shaped engineering machinery.

[0040] Figure 2 It is a schematic diagram showing the principle of blind spots in the surround view system of a traditional long vehicle model.

[0041] Figure 3 This is a schematic diagram of the principle of implementing overlapping area segmentation in a traditional 3D panoramic viewing system with four cameras.

[0042] Figure 4 Schematic diagram of the structure of the multi-eye 3D panoramic surround viewing system of this embodiment.

[0043] Figure 5 It is a schematic diagram of the principle of implementing the segmentation of overlapping areas between same-side camera images in this embodiment.

[0044] Figure 6 Schematic diagram of the working process of the multi-eye 3D panoramic surround viewing system in this embodiment.

[0045] Figure 7 Schematic diagram of the external parameter calibration object pattern used in this embodiment.

[0046] Figure 8 This is a schematic diagram of the first type of external parameter calibration object arrangement principle in this embodiment.

[0047] Fig. 9 Schematic diagram of the second arrangement principle of external parameter calibration objects in this embodiment.

[0048] Fig.10 Schematic diagram of the automatic corner point capture effect in this embodiment.

[0049] Fig.11 3D flat-bottomed bowl model used in this embodiment.

[0050] Fig.12 It is a schematic diagram of the 3D texture mesh vertex effect of generating a half-flat-bottomed bowl model in this embodiment.

[0051] Fig.13 It is a schematic diagram of the effect of 3D texture mesh vertex segmentation of the camera on the same side in this embodiment.

[0052] Legend: 1. Camera; 2. Image stitching module; 21. Model building unit; 211. Internal and external parameter calibration subunit; 212. 3D model building subunit; 213. Fusion segmentation subunit; 214. Texture network point rendering subunit; 22. Real-time model rendering unit. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0054] like Figure 4 As shown, the multi-eye 3D panoramic surround viewing system of this embodiment includes:

[0055] A plurality of cameras 1 are arranged at the front, rear, left and right sides of the vehicle, wherein at least two cameras 1 are arranged on at least one side to eliminate blind spots;

[0056] The image stitching module 2 is connected to each camera 1 respectively, and is used to stitch the images captured by each camera 1 based on a pre-built virtual 3D model to form a virtual 3D panoramic image. When the virtual 3D model is constructed, if there are more than two cameras 1 on the same side, the overlapping texture vertices of the cameras 1 on the same side are segmented using the Y-axis as the fusion segmentation curve.

[0057] In this embodiment, one camera 1 (specifically a fisheye camera) is installed in each of the four directions of the front, rear, left and right of the carrier vehicle body. At the same time, according to the actual structural characteristics of the carrier vehicle, two or more cameras 1 are installed on the side where a blind spot will be generated when one camera is used to eliminate the blind spot. Then, 5 to 8 cameras of different numbers can be arranged on the entire carrier vehicle, or even a larger number of cameras to form a multi-eye 3D panoramic surround view system. Then, the images collected by each camera 1 are stitched based on a pre-built virtual 3D model, and the panoramic view of the surrounding environment of the vehicle body and a three-dimensional model car are placed in a three-dimensional environment, which is virtualized into a virtual perspective whose position, angle and direction can be continuously adjusted, to obtain a 3D-360° panoramic surround view stitched image of the vehicle surrounding environment.

[0058] When more than two cameras 1 are arranged on the same side, there will be overlapping areas in the images collected by the cameras arranged on the same side. If the overlapping areas are to be segmented, the following requirements need to be met:

[0059] 1. Texture vertex Grid segmentation needs to be performed in the overlapping area of ​​two cameras;

[0060] 2. The texture vertex Grids of the two cameras in the overlapping area need to be segmented, and no texture vertex Grids should be omitted;

[0061] 3. The texture vertex grid needs to be segmented relatively evenly. Uneven segmentation will cause the 3D-360° surround image to be misaligned and deformed in the camera overlap area.

[0062] Considering that for the camera 1 installed on the same side, after the external parameters are determined in advance by the same set of external parameter calibration objects on the ground, the texture vertex Grid generated based on the same coordinate system and the rotation angle are exactly the same, then the Y axis in the coordinate system can meet the above three segmentation requirements to perfectly segment the texture vertex Grids of the two cameras on the same side, such as Figure 5 As shown. In this embodiment, when constructing a virtual 3D model, the texture vertices overlapped by the two cameras 1 on the same side are segmented using the Y axis as the fusion segmentation curve, that is, the straight line equation of the fusion segmentation line is: x=0, which can meet the above three segmentation requirements and effectively segment the overlapping area of ​​the cameras 1 on the same side relatively evenly without missing vertices, solving the problem that the texture vertex Grids corresponding to the two cameras installed on the same side cannot be segmented in the traditional method, so that the blind spots can be effectively eliminated while achieving the multi-eye 3D-360° panoramic viewing effect.

[0063] This embodiment can be specifically applied to special-shaped engineering machinery vehicles. Since the special-shaped structures such as the engineering machinery arm in the special-shaped engineering machinery vehicles will block the field of view of the camera 1, two or more cameras 1 are arranged on the side where the special-shaped structures such as the engineering machinery arm are arranged, so as to avoid the blockage of the special-shaped structure device and eliminate the blind spot of the field of view, thereby forming a surround stitching image of the blocked field of view area; it can also be applied to vehicles whose left and right sides of the body are longer than a preset threshold, such as engineering machinery vehicles with long bodies and long compartments. Since the body is long, arranging one camera will also cause a blind spot of the field of view, so two or more cameras 1 are arranged on the left and right sides of the vehicle respectively to eliminate the blind spot of the field of view. It can be understood that in addition to the above-mentioned special-shaped engineering machinery vehicles, engineering machinery vehicles with bodies and long compartments, it can also be applied to other types of vehicles.

[0064] In this embodiment, the image stitching module 2 includes:

[0065] The model building unit 21 is used to build a virtual 3D model for each camera 1 based on the 3D flat-bottomed bowl model, and use the Y axis as a fusion segmentation curve to segment the overlapping area, and then render the texture grid points to build a virtual 3D model;

[0066] The real-time model rendering unit 22 is used to obtain the images collected by each camera 1 in real time, and render the virtual 3D model according to the obtained images to obtain a real-time virtual 3D panoramic surround image.

[0067] The above-mentioned model construction unit 21 is specifically configured to be executed offline, that is, a virtual 3D model is constructed offline based on the calibrated internal and external parameters for storage, and the overlapping areas in the model are divided using the Y-axis. The real-time model rendering unit 32 is configured to be executed in real time, that is, after the images captured by each camera 1 are acquired in real time, the virtual 3D model is rendered and displayed in real time using the images captured in real time.

[0068] In this embodiment, the model construction unit 21 includes an internal and external parameter calibration subunit 211, a 3D model construction subunit 212, a fusion segmentation subunit 213 and a texture network point rendering subunit 224 which are connected in sequence. The internal and external parameter calibration subunit 211 is used to calibrate the internal and external parameters of the camera 1; the 3D model construction subunit 212 is used to construct a half-flat-bottomed bowl model for each camera 1; the fusion segmentation subunit 213 is used to segment the texture vertices that overlap between two cameras 1 on the same side in the half-flat-bottomed bowl model of each camera 1 using the Y-axis as the fusion segmentation curve; the texture network point rendering subunit 224 is used to perform texture grid point rendering on the half-flat-bottomed bowl model of each camera 1 according to the calibrated internal and external parameters to obtain a virtual 3D model.

[0069] The internal and external parameter calibration subunit 211 also generates a dedistortion mapping file according to the internal parameter matrix K and the distortion system matrix D during the internal parameter calibration.

[0070] In this embodiment, the real-time model rendering unit 22 includes a local parameter reading unit, an image data acquisition unit, a model texture rendering unit and a display unit connected in sequence. The local parameter reading unit reads the local parameter file, specifically including a triangle texture file and a 3D vehicle model, as well as a camera dedistortion mapping file, etc.; the image data acquisition unit receives the images captured by each camera 1 through multi-threading; the model texture rendering unit renders the image of the vehicle body surrounding environment onto a pre-built virtual 3D model based on the received image and the read file to form a virtual 3D panoramic surround image, and the display unit is used for image display.

[0071] like Figure 6 As shown, the multi-eye 3D panoramic view system of this embodiment arranges multiple fisheye cameras around the vehicle, wherein more than two fisheye cameras are arranged on one side or more to eliminate blind spots, and performs camera intrinsic parameter calibration, extrinsic parameter calibration, 3D flat-bottomed bowl model construction and rendering texture grid points in an offline manner in advance to generate a virtual 3D model; after real-time image acquisition, real-time rendering is performed, local parameter files are read in sequence, camera image data is acquired through multi-threading, OpenGL is used to render the virtual 3D model texture, and then OSD display is performed to complete real-time 3D panoramic view.

[0072] like Figure 7 As shown, this embodiment also includes a multi-eye 3D panoramic surround stitching method, the steps include:

[0073] S01. Pre-arrange cameras 1 at the front, rear, left and right sides of the vehicle, wherein at least two cameras 1 are provided on at least one side to eliminate blind spots;

[0074] S02. The images captured by each camera 1 are stitched together based on a pre-built virtual 3D model to form a virtual 3D panoramic surround image. When the virtual 3D model is constructed, if there are more than two cameras 1 on the same side, the overlapping texture vertices of the two cameras 1 on the same side are segmented using the Y-axis as the fusion segmentation curve.

[0075] In this embodiment, the construction of the virtual 3D model corresponds to the model construction unit 21, and the steps of constructing the virtual 3D model include:

[0076] SA1. Calibration of internal and external parameters: Calibrate the internal and external parameters of camera 1 in advance to obtain the internal and external parameter matrices;

[0077] SA2.3D model construction: Generate a half-sided flat-bottomed bowl model spanning quadrants 1 and 2 for each camera 1;

[0078] SA3. Fusion segmentation: In the half-side flat-bottomed bowl model of each camera 1, the texture vertices that overlap between the two cameras 1 on the same side are segmented using the Y axis as the fusion segmentation curve;

[0079] SA4. Texture grid point rendering: Perform texture grid point rendering on the half-side flat-bottomed bowl model of each camera 1 formed after segmentation in step SA3 according to the internal and external parameter matrices to obtain a virtual 3D model.

[0080] In this embodiment, when calibrating the internal parameters of camera 1, a distortion correction processing step is also included. The distortion correction processing step specifically includes: generating a dedistorted mapping file according to the internal parameter matrix K and the distortion system matrix D and the scaling factor sf. In this embodiment, camera 1 specifically adopts a fisheye camera. The fisheye camera has an ultra-large field of view and is accompanied by a large barrel distortion. The fisheye camera needs to be subjected to distortion correction processing. This embodiment adopts Zhang Zhengyou's calibration method. By collecting 10 to 20 (determined according to actual needs) chessboard calibration pictures, the calculation of the internal parameter matrix K and the distortion coefficient matrix D of camera 1 is completed, and the dedistorted mapping file is generated and saved according to the scaling factor sf.

[0081] The above internal parameter matrix K is specifically: The distortion coefficient matrix is ​​specifically D = [k 1 k 2 p 1 p 2 k 3 ], and the scaling factor sf can be set according to the actual calibration scenario.

[0082] In this embodiment, the step of calibrating the external parameters of the camera 1 includes:

[0083] Arrange external calibration objects around the vehicle body and capture the corner points of each external calibration object: convert the camera dedistorted image containing the external calibration object image into a binary image, locate the position of the external calibration object, and calculate the image coordinates of each corner point of the external calibration;

[0084] Then, the image coordinates and world coordinates of the corner points of the extrinsic calibration objects captured by each camera 1 are used to calculate the extrinsic rotation matrix R and the extrinsic translation matrix T of each camera 1.

[0085] In a specific application embodiment, the detailed steps for calibrating the external parameters of the camera 1 are as follows:

[0086] a. Use external parameters to calibrate the object pattern, such as Figure 8 Outer black border shown.

[0087] b. Take the geometric center of the carrier body as the origin (0,0), and lay 4 to 8 (the specific number can be determined according to actual needs) external parameter calibration objects around the body in an equidistant manner according to the origin position; when two cameras are installed on one side of the carrier body, three external parameter calibration objects are placed on the ground on that side, such as Fig. 9 As shown in (a); if only one camera is installed on that side, then only two external parameter calibration objects need to be placed on the ground on that side for the external parameter calibration of the system, as shown in Fig. 9 As shown in (b), Figure 7 The rectangular frame in the geometric center is the carrier body, and the circle symbol represents the camera.

[0088] c. Automatically capture the corner points of the external calibration object: Based on the Otsu method adaptive threshold algorithm, the camera dedistorted image containing the external calibration object pattern is converted into a binary image. The position of the external calibration object is located through the quadrilateral contour approximation algorithm, and the image coordinates of each corner point of the external calibration are calculated, such as Fig.10 The above-mentioned adaptive threshold algorithm based on Otsu's method and quadrilateral contour approximation algorithm can adopt other algorithms to achieve similar functions.

[0089] d. Use the image coordinates and world coordinates of the corner points of the extrinsic calibration objects captured by each camera to calculate the extrinsic rotation matrix R and extrinsic translation matrix T of each camera.

[0090] After the calibration is completed, the model is constructed. In this embodiment, a virtual 3D model is constructed based on a 3D flat-bottomed bowl model. The bottom model of the bowl is the ground plane of the near field around the vehicle body, and the edge of the bowl is the height space of the far field around the vehicle body. Fig.11 As shown in the figure, the model is rendered by OpenGL. The points in the figure represent the 3D texture vertex grid, that is, the coordinates of the triangle texture vertices. Then, a half-sided flat-bottomed bowl model spanning one and two quadrants is generated for each camera. Each half-sided flat-bottomed bowl model is saved as a text file in the form of texture vertex grid points. The texture vertex grid of the half-sided flat-bottomed bowl model is as follows Fig.12 When two fisheye cameras are installed on the same side of the vehicle body, the vertical Y axis is used as the dividing boundary to divide the two camera texture vertex grids that overlap 100% on the same side into two groups of texture vertex grids, as shown in FIG. Fig.13 shown.

[0091] After the above model construction is completed, the texture grid points are further rendered. The specific steps of rendering the texture grid points in the above step SA4 include:

[0092] SA41. According to the internal and external parameter matrices, the 3D grid points corresponding to each camera 1 are projected into the 2D image, and the pixel coordinates of the image corresponding to the 3D grid points are calculated;

[0093] SA42. Set the segmentation boundary of each camera 1 corresponding to the half-flat-bottomed bowl model to classify each half-flat-bottomed bowl model into a fusion area and a non-fusion area, wherein the fusion area is the overlapping area jointly photographed by adjacent cameras 1, and the non-fusion area is the area independently photographed by each camera (1), and generate a binary image for segmenting the fusion area and the non-fusion area;

[0094] SA43. Rotate each half-flat-bottom model by a specified angle, wherein the 3D grid point models corresponding to each camera 1 arranged on the same side are rotated by the same angle, and save the grid points and corresponding image coordinates of each half-flat-bottom bowl model in the form of a triangular texture file.

[0095] In a specific application embodiment, when rendering texture grid points, first, according to the pre-calculated intrinsic and extrinsic parameter matrices of each camera: K, D, R, T, the 3D grid points corresponding to each camera are projected into the 2D image of each camera, and the pixel coordinates of the fisheye image corresponding to the 3D grid points are calculated; then, the segmentation boundary of the half-side flat-bottomed bowl model of each camera is set to classify each half-side flat-bottomed bowl model into a fusion area and a non-fusion area, and a binary mask image for segmenting the fusion area and the non-fusion area is generated; each half-side flat-bottomed model is rotated by a certain angle, and when two cameras are installed on the same side, the 3D grid point model generated by this pair of cameras is rotated by the same angle, and the grid points and corresponding image coordinates of each half-side flat-bottomed bowl model are saved as a text file in the form of triangular texture, where the triangular textures are V1-V2-V4 and V4-V2-V3:

[0096]

[0097] Among them, x, y, z are 3D grid points, and u, v are the corresponding image coordinates.

[0098] In this embodiment, step S02 real-time rendering specifically includes:

[0099] S201. Read local parameter files, including triangular texture files, 3D vehicle models, and camera distortion mapping files;

[0100] S202. Receive images captured by each camera 1;

[0101] S203. Render the vehicle body surrounding environment image onto a pre-built virtual 3D model according to the received image and the read file to form a virtual 3D panoramic surround image.

[0102] In a specific application embodiment, when performing real-time rendering, first read the local parameter file: triangular texture file, 3D vehicle model, camera dedistortion mapping file, etc.; then, multi-threading is used to simultaneously collect image frames of various fisheye cameras installed around the vehicle body to improve the system calculation efficiency and ensure the real-time frame rate of the image; OpenGL computer graphics library programming is used to perform calculations on the GPU based on the fisheye camera image and the triangular texture file, and the image of the vehicle body surrounding environment is rendered onto the 3D flat bowl model, and finally the surround image display is realized on the OSD.

[0103] Those skilled in the art will appreciate that the above description of the computer device is merely an example and does not constitute a limitation on the computer device. The computer device may include more or fewer components than described above, or may combine certain components, or may include different components, such as input and output devices, network access devices, buses, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect the various parts of the entire computer device.

[0104] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0105] If the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned distributed crawler method embodiments based on template annotation can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, electrical signal and software distribution medium, etc.

[0106] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A multi-eye 3D panoramic view system, characterized in that: include: A plurality of cameras (1) are arranged at the front, rear, left and right sides of the vehicle, respectively, wherein at least two or more cameras (1) are arranged on at least one side to eliminate blind spots, and there is a spacing between the cameras (1) arranged on the same side; An image stitching module (2) is connected to each of the cameras (1) and is used to stitch the images captured by each of the cameras (1) based on a pre-built virtual 3D model to form a virtual 3D panoramic image. When the virtual 3D model is built, if more than two cameras (1) are arranged on the same side, the texture vertices that overlap between the two cameras (1) on the same side are segmented using the Y axis as a fusion segmentation curve; The model construction unit (21) comprises an internal and external parameter calibration subunit (211), a 3D model construction subunit (212), a fusion segmentation subunit (213) and a texture network point rendering subunit (214) which are connected in sequence. The internal and external parameter calibration subunit (211) is used to calibrate the internal and external parameters of the camera (1); the 3D model construction subunit (212) is used to construct and generate a half-flat-bottomed bowl model spanning one or two quadrants for each camera (1); the fusion segmentation subunit (213) is used to segment the texture vertices that overlap between two cameras (1) on the same side in the half-flat-bottomed bowl model of each camera (1) using the Y axis as a fusion segmentation curve; and the texture network point rendering subunit (214) is used to perform texture grid point rendering on the half-flat-bottomed bowl model of each camera (1) according to the calibrated internal and external parameters to obtain the virtual 3D model.

2. The multi-eye 3D panoramic viewing system according to claim 1, characterized in that: When applied to a special-shaped engineering machinery vehicle, more than two cameras (1) are arranged on one side with the special-shaped structure; when applied to a vehicle whose left and right sides of the vehicle body are longer than a preset threshold, more than two cameras (1) are arranged on the left and right sides of the vehicle respectively.

3. The multi-eye 3D panoramic viewing system according to claim 1 or 2, characterized in that: The image stitching module (2) comprises: A model building unit (21) is used to build a virtual 3D model for each camera (1) based on the 3D flat-bottomed bowl model, and use the Y axis as a fusion segmentation curve to segment the overlapping area, and then render the texture grid points to build the virtual 3D model; A real-time model rendering unit (22) is used to acquire the images collected by each of the cameras (1) in real time, and to render the virtual 3D model in real time according to the acquired images, so as to obtain a real-time virtual 3D panoramic surround image.

4. A multi-eye 3D panoramic view stitching method, characterized in that the steps include: S01. Arrange cameras (1) in advance on the front, rear, left and right sides of the vehicle, wherein two cameras (1) are arranged on at least one side to eliminate blind spots, and there is a spacing between the cameras (1) arranged on the same side; S02. stitching the images captured by each of the cameras (1) based on a pre-constructed virtual 3D model to form a virtual 3D panoramic image. When constructing the virtual 3D model, if more than two cameras (1) are arranged on the same side, the texture vertices that overlap between the two cameras (1) on the same side are segmented using the Y axis as a fusion segmentation curve; The steps of constructing the virtual 3D model include: SA1. Pre-calibrate the intrinsic and extrinsic parameters of the camera (1) to obtain the intrinsic and extrinsic parameter matrices; SA2. Generate a half-flat bowl model spanning one or two quadrants for each camera (1); SA3. Segment the texture vertices that overlap between the two cameras (1) on the same side in the half-flat-bottomed bowl model of each camera (1) using the Y axis as the fusion segmentation curve; SA4. Perform texture grid point rendering on the half-side flat-bottomed bowl model of each camera (1) formed after segmentation in step SA3 according to the internal and external parameter matrices to obtain the virtual 3D model.

5. The multi-eye 3D panoramic view stitching method according to claim 4, characterized in that: The step S02 comprises: S201. Read local parameter files, including triangular texture files and 3D vehicle models; S202. Receive images captured by each of the cameras (1); S203. Render the vehicle body surrounding environment image onto a pre-built virtual 3D model according to the received image and the read file to form the virtual 3D panoramic surround image.

6. The multi-eye 3D panoramic view stitching method according to claim 4 or 5, characterized in that: The steps of step SA4 include: SA41. According to the internal and external parameter matrices, the 3D grid points corresponding to each camera (1) are projected into the 2D image, and the pixel coordinates of the image corresponding to the 3D grid points are calculated; SA42. Set the segmentation boundary of each camera (1) corresponding to the half-flat-bottomed bowl model to classify each half-flat-bottomed bowl model into a fusion area and a non-fusion area, wherein the fusion area is an overlapping area photographed by adjacent cameras (1) and the non-fusion area is an area photographed independently by each camera (1), and generate a binary image for segmenting the fusion area and the non-fusion area; SA43. Rotate each of the half-sided flat-bottomed bowl models by a specified angle, wherein the 3D grid point models corresponding to each camera (1) arranged on the same side are rotated by the same angle, and save the grid points and corresponding image coordinates of each of the half-sided flat-bottomed bowl models in the form of a triangular texture file.

7. The multi-eye 3D panoramic view stitching method according to claim 5, characterized in that: When calibrating the internal parameters of the camera (1), a distortion correction processing step is also included, including: generating a dedistortion mapping file according to the internal parameter matrix K, the distortion system matrix D and the scaling factor sf.

8. The multi-eye 3D panoramic view stitching method according to claim 5, characterized in that: The steps of calibrating the external parameters of the camera (1) include: Arrange external calibration objects around the vehicle body and capture the corner points of each external calibration object: convert the camera dedistorted image containing the external calibration object image into a binary image, locate the position of the external calibration object, and calculate the image coordinates of each corner point of the external calibration; According to the image coordinates and world coordinates of the corner points of the extrinsic calibration object captured by each camera (1), the extrinsic rotation matrix R and the extrinsic translation matrix T of each camera (1) are calculated.

Images