A vehicle-mounted 3D surround image display method

Abstract

The present invention relates to a vehicle-mounted 3D surround image display method, which first obtains vehicle size parameters, calibrates the internal reference matrix and the external matrix of the surround camera, then calculates the length and width of the 3D projection model according to the vehicle size, and generates the optimal size 3D projection model according to the surround camera calibration parameters; then establishes a gridded vertex coordinate set of the 3D projection model, calculates the image coordinate system coordinates after each vertex coordinate of the 3D projection model in the world coordinate system is projected to the corresponding surround camera, and establishes a 3D projection model vertex mapping table; then, according to the virtual viewpoint position and virtual imaging parameters, the fusion angle and area of ​​the surround image are dynamically adjusted through the imaging position of the overlapping area in the imaging picture, and the fusion weights of each point in the fusion area are calculated; finally, the image is fused according to the vertex mapping table and the fusion weight, and the image is displayed. The method can reduce the system calculation complexity while ensuring the surround display effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a vehicle-mounted 3D surround image display method. Background Art

[0002] With the rapid development of computer vision technology and the continuous improvement of mobile hardware processing capabilities, computer vision has been continuously applied in autonomous driving and has gradually played an increasingly important role. Fisheye cameras are widely used in the automotive camera market due to their ultra-wide angle advantages. The most widely used application is to use multiple fisheye cameras to construct surround panoramic images for assisted driving. At present, the main ways to obtain surround panoramic images include the following:

[0003] (1) Chinese patent CN 109741455 A discloses a method, computer-readable storage medium and system for displaying stereoscopic panoramic images on a vehicle. The patent establishes a three-dimensional grid model, which includes a plane part, a curved connection part and a three-dimensional cylindrical part. The plane part is the bottom surface of the three-dimensional grid model, which can be a circular plane, an elliptical plane, a rounded rectangle, etc. The curved connection part can be a spherical surface. The three-dimensional cylindrical part is the annular part of the three-dimensional grid model. The intersection of the longitude and latitude lines in the three-dimensional grid model is the vertex of the three-dimensional grid model. The three-dimensional grid model is enlarged to the world coordinate system. The fisheye image of the vehicle body captured by the camera is obtained and distortion correction is performed to obtain the corrected image and the coordinates of the imaging center. The camera internal parameter matrix is ​​calculated based on the coordinates of the imaging center. The external parameter matrix of each camera in the vehicle body coordinate system is determined. The mapping relationship between each point in the three-dimensional space and the pixel coordinates of the plane image is determined. The content of the corrected image is rendered into the three-dimensional grid model. For the overlapping area formed by the two cameras, the splicing traces are eliminated by weighted fusion to achieve a stereoscopic panoramic surround view. Although the patent proposes to establish a three-dimensional grid model, it does not provide a method for calculating the model size, and does not consider the actual installation position and imaging range of the surround-view camera. In actual applications, it can only adapt to different camera installation conditions through empirical values ​​or manual adjustment of parameters. It is not universal and is prone to problems such as mismatch between model size and camera imaging range. In addition, the image fusion area and fusion weight proposed in the patent are fixed and do not consider the impact of changes in virtual observation viewpoints on the fusion effect. This will lead to large differences in the fusion effect of overlapping areas under different observation viewpoints, such as ghosting, splicing misalignment, etc., which will affect the surround-view display effect. This will not only have a great impact on the user experience, but may even affect the safety of passengers.

[0004] (2) Chinese patent CN 111462172 A discloses a method for adaptively generating three-dimensional panoramic images based on driving scene estimation. The patent pre-designs a five-plane projection model with multiple sets of parameters and a bowl-shaped projection model with fixed parameters, obtains multiple sets of mapping tables, acquires images at fixed intervals, calculates the distance between the fisheye camera and the main plane of the environment, determines the projection model, matches the lookup table according to the environmental parameters, and performs texture mapping on the four fisheye video images according to the lookup table to obtain a three-dimensional panoramic image of the vehicle driving scene. However, the patent does not mention how to determine the size of each model. Different camera installation conditions and scenes require manual adjustment of model parameters, and adaptive adjustment cannot be achieved. In addition, the method of sensing the surrounding environment and changing different projection models in real time will increase the system's resource consumption and computing burden, affecting the real-time performance of the surround view.

[0005] (3) The “surface projection method and device for a multi-camera panoramic system” disclosed in Chinese patent CN 103065318 A first obtains a distortion-free image by distortion correction, then establishes a three-dimensional surface model, and then performs a unified coordinate transformation based on the Z-axis coordinate value Zc in the three-dimensional surface model. Finally, the distortion-free image is projected onto the three-dimensional surface model to achieve panoramic imaging on the surface. The model is generated according to the formula However, the β, a, and b in the formula are not clear. Although the patent gives the generation formula of the model, it does not give the calculation method of each parameter in the formula. In actual operation, it is still necessary to rely on engineering experience to continuously adjust the parameters, and it is impossible to adaptively calculate the optimal parameters, so it cannot adapt to different camera installation conditions and scenes.

[0006] (4) Chinese patent CN 111080519 A discloses a "car panoramic surround image fusion method". This patent divides the surround model into fusion zones based on the world coordinate system, and the fusion zones include the right front fusion zone, the left front fusion zone, the left rear fusion zone and the right rear fusion zone; respectively determine the reference origins of the right front fusion zone, the left front fusion zone, the left rear fusion zone and the right rear fusion zone; establish a comparison table, the query conditions of the comparison table are the radar number and the data of the corresponding radar, and the query result of the comparison table is the corrected fusion angle and corrected fusion range of at least one fusion zone; this car panoramic surround image fusion method divides the surround model into four fusion zones, corrects the fusion angle and fusion range of the fusion zone according to the collected vehicle data, thereby determining the fusion coefficient of the pixel points in the fusion zone, and then calculating the color of the pixel points, and finally generating a panoramic surround image. However, this patent needs to rely on the radar sensor to perceive the surrounding environment to correct the fusion angle and fusion range, which increases the calculation complexity of the system. Summary of the invention

[0007] The purpose of the present invention is to provide a vehicle-mounted 3D surround image display method, which does not rely on additional sensor assistance, but only uses the basic camera imaging principle to dynamically calculate the fusion angle and weight according to the projection difference of the 3D projection model under different virtual viewpoints, thereby reducing the system calculation complexity while ensuring the surround display effect.

[0008] To achieve the above object, the present invention adopts the following technical solution:

[0009] A method for displaying a 3D surround view image on a vehicle comprises the following steps:

[0010] Step S1, obtain the vehicle size parameters, and calibrate the internal parameter matrix and the external matrix of the surround view camera; suppose the vehicle size is [l, w, h], l is the length of the vehicle, w is the width of the vehicle, and h is the height of the vehicle; the internal parameter matrix calibrated by the i-th (i=0,1,2,3) surround view camera is M i1 , the external parameter matrix is ​​M i2 ,

[0011] Step S2, calculating the length and width of the 3D projection model according to the vehicle size; calculating the maximum imaging height of each surround view camera according to the surround view camera calibration parameters, taking the minimum value to obtain the optimal 3D projection model height, and the calculated 3D projection model size is [L, W, H], where L is the length of the 3D projection model, W is the width of the 3D projection model, and H is the height of the 3D projection model;

[0012] Step S3: According to the calculated 3D projection model size [L, W, H], a grid vertex coordinate set {(X n , Y n , Z n )|n=1,2,3...N},according to the imaging rules of the camera Calculate the coordinates of each vertex of the 3D projection model in the world coordinate system (X n , Y n , Z n ) is projected to the image coordinate system coordinates [x n ,y n ], establish a vertex mapping table of world coordinates and image coordinates of meshed vertices of the 3D projection model;

[0013] Step S4: Set the virtual viewpoint, the spatial position coordinate of the virtual viewpoint E is (X E , Y E , Z E ), the projection point in the XY plane is E', according to the virtual viewpoint position and virtual imaging parameters, the fusion angle and area of ​​the surround image are dynamically adjusted through the imaging position of the overlapping area in the imaging picture, and the fusion weight of each point in the fusion area is calculated;

[0014] Step S5: fuse the images according to the vertex mapping table and the fusion weights, generate textures through the GPU, project the textures into the 3D projection model, and display the images.

[0015] As a preferred embodiment of the present invention, the calculation method of the 3D projection model size [L, W, H] in step S2 is as follows:

[0016] Step S2.1, first determine the display range [L, W] outside the 3D projection model. The display area of ​​this part can be manually specified according to different functional requirements. If it is not specified, a default value can be set according to the vehicle size: such as W = k1*w (1.5≤k1≤4,), the default k1 = 3, then

[0017] Step S2.2, calculate the optimal height H of the 3D projection model, that is, the height of the edge of the model; let the limit projection height of the i-th surround camera in the model be H i , let H i The initial height is H0, which is not optimal. For example, H0 = k2*h (1.5 ≤ k2 ≤ 8), and k2 = 3 by default. According to the camera imaging principle, All points on the edge of the 3D projection model can be calculated {(X i，j , Y i，j , H0)|i=0,1,2,3;j=1,2,3...N} The point set in the image coordinate system of this surround-view camera is {(x ij ,y ij )|i=0, 1, 2, 3; j=1, 2, 3...N};

[0018] Step S2.3, let (x min ,y min )=min{(x ij ,y ij )|i=0,1,2,3;j=1,2,3...N}, if y min <0, then H i =H i -step; if y min >0, then H i =H i +step; if y min = 0, then the maximum imaging height H of the 3D projection model under this surround camera is obtained i ; Among them, step represents H i The amplitude of each adjustment, and H i The unit remains consistent, and the value range is 0≤step≤H i ;

[0019] Step S2.4: All surround cameras calculate the maximum imaging height H of their own cameras respectively. i , then the optimal height of the 3D projection model is H = min i＝0，1，2，3 {H i}.

[0020] As a preferred embodiment of the present invention, the gridded vertex coordinate set {(X n， Y n , Z n )|n=1,2,3...N},where Z n The calculation formula is as follows:

[0021]

[0022] In the formula, (a, b) represents the model surface control point, and its value range is By changing the values ​​of a and b, the shape and range of the model surface can be adjusted.

[0023] As a further preferred embodiment of the present invention, the fusion weight calculation method of each point in the fusion area in step S4 is as follows:

[0024] Step S4.1 Determination of fusion boundary vector

[0025] Step S4.1.1, obtain the 3D projection model mapping lookup table, establish a world coordinate system with the midpoint of the 3D projection model as the center, and construct the circumscribed rectangle of the 3D projection model in the top view, where Q i (i=1,2,3,4) represents the vertices of the circumscribed rectangle, and the coordinate values ​​can be obtained from the maximum values ​​of the coordinates of all vertices generated in the corresponding quadrant; P i (i=1,2,3,4) represents the vertex of the bottom edge of the vehicle in the top view; determine the basic fusion boundary vector of each fusion area The acute angle with the horizontal direction is βi;

[0026] Step S4.1.2, according to the size of the vehicle and the position of the virtual viewpoint E, calculate the blind area of ​​the 3D projection model blocked by the vehicle model under the viewpoint E. If a certain overlapping area is within the blind area and the proportion of the area blocked on the virtual imaging image exceeds the threshold η, the overlapping area is fused according to the default angle and range;

[0027] The calculation method of the occlusion ratio is as follows: first, the coordinates of the projection points of the boundary points of the fusion area and the eight vertices of the vehicle's circumscribed cuboid in the virtual imaging plane are calculated, and the area of ​​the overlapping part of the imaging area formed by the projection points of the fusion area and the eight vertices of the vehicle's circumscribed cuboid is calculated. The occlusion ratio = the ratio of the area of ​​the overlapping part to the area of ​​the polygon formed by the projection points of the fusion area;

[0028] Step S4.1.3: If the percentage of an overlapping area on the virtual image that is blocked does not exceed the threshold η, the fusion angle and weight are dynamically calculated according to the position of E'

[0029] Step S4.1.4, let k be the fusion boundary vector rotation factor (0≤k≤1), then

[0030] When E' is in the Q2Q3P3P2 region, Rotate (1-k)β1 clockwise around point P1, Rotate counterclockwise around point P4 by (1-k)β4, where

[0031] When E' is in the Q1Q4P4P1 region, Rotate (1-k)β3 clockwise around point P3, Rotate counterclockwise around point P2 by (1-k)β2, where

[0032] When E' is in the Q1Q2P2P1 region, Rotate clockwise around point P4 (1-k)(π / 2-β4), Rotate counterclockwise around point P3 by (1-k)(π / 2-β3), where

[0033] When E' is in the Q3Q4P4P3 region, Rotate clockwise around point P2 (1-k)(π / 2-β2), Rotate counterclockwise around point P1 by (1-k)(π / 2-β1), where

[0034] Step S4.2 Fusion weight calculation

[0035] Step S4.2.1, based on the calculated fusion boundary vector, calculate the fusion angle: When the fusion weight w of each point P(X, Y, Z) in the fusion area is (x，y，z) ;

[0036] Step S4.2.2: Calculate Direction vector

[0037] Step S4.2.3: Calculate and The vector angle k indicates polarity. When P is In the counterclockwise direction, k = -1; when P is In clockwise direction, k = 1;

[0038] Step S4.2.4: Calculate the fusion weight of the pixel at the image coordinate of the surround camera projected to the counterclockwise direction of point P(x,y)

[0039]

[0040] Then the fusion weight of the pixel at the image coordinate of the P(x,y) point projected to its clockwise surround camera is 1-w (x，y，z) ;

[0041] Step S4.2.5: Repeat steps S4.2.2 to S4.2.4 for all points in the 3D projection model to generate a fusion weight lookup table.

[0042] As a further preferred embodiment of the present invention, the steps of displaying and generating the 3D surround view image in step S5 are as follows:

[0043] Step S5.1, converting the original image collected by each surround camera in real time into a texture, and mapping the texture onto the 3D projection model according to the 3D projection model mapping lookup table to obtain texture coordinates;

[0044] Step S5.2, according to the position of the virtual viewpoint and the camera imaging rule, calculate the image coordinate set projected from the world coordinate vertex set in the 3D projection model mapping lookup table to the virtual imaging surface, and establish a virtual imaging mapping table;

[0045] Step S5.3, obtaining the texture in the texture object through the texture coordinates, remapping the texture according to the virtual imaging mapping table and the fusion weight lookup table in step S5.2, and finally displaying it on the screen area.

[0046] The advantages and positive effects of the present invention are:

[0047] (1) The method provided by the present invention can calculate the maximum imaging height of the surround-view camera at different installation positions according to the camera calibration parameters and the vehicle size through the surround-view camera imaging model, obtain the optimal size of the 3D projection model, and generate a gridded 3D projection model online, thereby reducing the system calculation complexity and reducing system resource consumption while ensuring the surround-view display effect.

[0048] (2) The method provided by the present invention does not rely on the assistance of additional sensors, but only uses the basic camera imaging principle. According to the projection difference of the surround view model under different virtual viewpoints, the fusion angle (fusion area) and fusion weight of the surround view image are dynamically calculated to avoid ghosting, splicing misalignment and the like in the fusion area, thereby greatly improving the display effect of the surround view panoramic image.

[0049] (3) The display method provided by the present invention is fully automated and can be integrated with the camera calibration process to achieve standard factory assembly line operations, improve efficiency, and have engineering practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a 3D surround image display method of the present invention;

[0051] Figure 2 A schematic diagram of the installation position of the surround view camera in the car;

[0052] Figure 3 Schematic diagram of different limit projection heights of surround-view cameras in projection models of different sizes;

[0053] Figure 4 Flowchart for generating the optimal size 3D projection model;

[0054] Figure 5 It is a schematic diagram of the external display range of the 3D projection model;

[0055] Figure 6 The process of generating a fusion weight lookup table;

[0056] Figure 7 It is a schematic diagram of the circumscribed rectangle of the 3D projection model in the top view.

[0057] Figure 8 Schematic diagram of virtual viewpoint E

[0058] Fig. 9 Schematic diagram of the blind area in the overlapping area. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] See also Figure 1 The present invention provides a method for displaying a 3D surround image on a vehicle, which specifically includes the following steps:

[0061] Step S1, obtain the vehicle size parameters, and calibrate the internal parameter matrix and the external matrix of the surround view camera; suppose the vehicle size is [l, w, h], l is the length of the vehicle, w is the width of the vehicle, and h is the height of the vehicle; the internal parameter matrix calibrated by the i-th (i=0,1,2,3) surround view camera is M i1 , the external parameter matrix is ​​M i2 ,

[0062] Step S2, calculating the length and width of the 3D projection model according to the vehicle size; calculating the maximum imaging height of each surround view camera according to the surround view camera calibration parameters, taking the minimum value to obtain the optimal 3D projection model height, and the calculated 3D projection model size is [L, W, H], where L is the length of the 3D projection model, W is the width of the 3D projection model, and H is the height of the 3D projection model;

[0063] Step S3: According to the calculated 3D projection model size [L, W, H], a grid vertex coordinate set {(X n , Y n , Z n )|n=1,2,3...N},according to the imaging rules of the camera Calculate the coordinates of each vertex of the 3D projection model in the world coordinate system (X n , Y n , Z n ) is projected to the image coordinate system coordinates [x n ,y n ], establish a vertex mapping table of world coordinates and image coordinates of meshed vertices of the 3D projection model;

[0064] Step S4: Set the virtual viewpoint, the spatial position coordinate of the virtual viewpoint E is (X E , Y E , Z E ), the projection point in the XY plane is E', according to the virtual viewpoint position and virtual imaging parameters, the fusion angle and area of ​​the surround image are dynamically adjusted through the imaging position of the overlapping area in the imaging picture, and the fusion weight of each point in the fusion area is calculated;

[0065] Step S5: fuse the images according to the vertex mapping table and the fusion weights, generate textures through the GPU, project the textures into the 3D projection model, and display the images.

[0066] See also Figures 2 to 5 The specific process of generating the optimal size 3D projection model in step S2 and step S3 of the present invention is as follows:

[0067] Generally, the surround view camera is installed in the car. Figure 2 As shown in the figure, due to the structural limitations of the car itself, the installation heights and angles of surround view cameras at different positions on the car body vary greatly, which results in different maximum projection heights of different surround view cameras in projection models of different sizes, such as Figure 3 shown.

[0068] Assume that the vehicle size is [l, w, h], and the intrinsic parameter matrix of the calibrated surround camera No. i (i = 0, 1, 2, 3) is M i1 , the external parameter matrix is ​​Mi2 , then the calculation method of the 3D projection model size [L, W, H] is as follows:

[0069] 1) First, you need to determine the display range [L, W] outside the 3D projection model (surround view model). Usually, the display area of ​​this part can be manually specified according to different functional requirements. If it is not specified, a default value can be set according to the vehicle size: such as W = k1*w (1.5≤k1≤4,), the default k1 = 3, then

[0070] 2) Calculate the optimal height H of the 3D projection model (the height of the model’s edge); let the limit projection height of the i-th surround camera in the model be H i , let H i The initial height is H0 (non-optimal), such as H0 = k2*h (1.5≤k2≤8), the default k2 = 3; according to the camera imaging principle All points on the edge of the 3D model can be calculated {(X i，j , Y i，j , H0)|i=0,1,2,3;j=1,2,3...N} The point set in the image coordinate system of this camera is {(x ij ,y ij )|i=0, 1, 2, 3; j=1, 2, 3...N};

[0071] 3) Let (x min ,y min )=min{(x ij ,y ij )|i=0,1,2,3;j=1,2,3...N}, if y min <0, then H i =H i -step; if y min >0, then H i =H i +step; if y min = 0, then the maximum imaging height H of the model under this camera is obtained i ; Among them, step represents H i The amplitude of each adjustment, and H i The unit remains consistent, and the value range is 0≤step≤H i ;

[0072] 4) All surround cameras calculate the maximum imaging height H of their own cameras respectively i , then the optimal height of the 3D model is H = min i＝0，1，2，3 {H i};

[0073] 5) The size of the 3D projection model calculated by the above method is [L, W, H], and the grid vertex coordinate set of the projection model {(X n , Y n , Z n )|n=1,2,3...N},where Z n The calculation formula is as follows:

[0074]

[0075] Where (a, b) represents the model surface control point, and its value range is By changing the values ​​of a and b, the shape and range of the model surface can be adjusted. n The calculation formula includes but is not limited to this formula.

[0076] 6) Finally, according to the camera's imaging rules Calculate the coordinates of each vertex of the 3D projection model in the world coordinate system (X n , Y n , Z n ) is projected to the image coordinate system coordinates [x n ,y n ], establish the vertex mapping table of the 3D projection model.

[0077] See also Figures 6 to 9 The adaptive fusion method based on virtual viewpoint in step S4 of the present invention is as follows:

[0078] Establish a world coordinate system with the midpoint of the 3D projection model as the center. Figure 7 is the circumscribed rectangle of the top view of the 3D projection model, where Q i (i=1,2,3,4) represents the vertices of the rectangle, and the coordinate values ​​can be obtained from the maximum values ​​of the coordinates of all vertices generated in the corresponding quadrant; P i (i=1,2,3,4) represents the vertex of the bottom edge of the vehicle in the top view. Usually, the overlapping area of ​​the imaging of adjacent surround view cameras is as follows: Figure 7 The position shown by the shadow; let the virtual viewpoint be E (that is, the virtual camera position, such as Figure 8 The spatial coordinates of the point are (X E , Y E , Z E ), the projection point in the XY plane is E', let the vector The acute angle with the model's X-axis is

[0079] Step S4.1: Determination of fusion boundary vector

[0080] 1) First determine the basic fusion boundary vector of each fusion area The acute angle with the horizontal direction is β i ;

[0081] 2) According to the size of the car (vehicle) and the position of the E viewpoint, the blind area of ​​the 3D projection model (surround view model) blocked by the car model under the E viewpoint is calculated. If a certain overlapping area is within the blind area and the proportion of the area blocked on the virtual imaging image exceeds the threshold η, the overlapping area is fused according to the default angle and range;

[0082] Calculation method of occlusion ratio:

[0083] First, the projection point coordinates of the boundary points of the fusion area and the eight vertices of the car's circumscribed cuboid in the virtual imaging plane are calculated, and the overlapping area of ​​the polygon formed by the projection point set of the fusion area and the imaging area formed by the projection points of the eight vertices of the car's circumscribed cuboid is calculated. The occlusion ratio = the ratio of the overlapping area to the polygon area formed by the projection point set of the fusion area;

[0084] 3) If the proportion of an overlapping area on the image being blocked does not exceed the threshold η, the fusion angle and weight are dynamically calculated according to the position of E';

[0085] 4) Let k be the fusion boundary vector rotation factor (0≤k≤1), then

[0086] When E' is in the Q2Q3P3P2 region, Rotate (1-k)β1 clockwise around point P1, Rotate counterclockwise around point P4 by (1-k)β4, where

[0087] When E' is in the Q1Q4P4P1 region, Rotate (1-k)β3 clockwise around point P3, Rotate counterclockwise around point P2 by (1-k)β2, where

[0088] When E' is in the Q1Q2P2P1 region, Rotate clockwise around point P4 (1-k)(π / 2-β4), Rotate counterclockwise around point P3 by (1-k)(π / 2-β3), where

[0089] When E' is in the Q3Q4P4P3 region, Rotate clockwise around point P2 (1-k)(π / 2-β2), Rotate counterclockwise around point P1 by (1-k)(π / 2-β1), where

[0090] Step S4.2 Fusion weight calculation

[0091] 1) According to the fusion boundary vector calculated previously, the fusion angle is calculated as σ When the fusion weight w of each point P(X, Y, Z) in the fusion area is (x，y，z) ;

[0092] 2) Calculation Direction vector

[0093] 3) Calculation and The vector angle k indicates polarity. When P is In the counterclockwise direction, k = -1; when P is In clockwise direction, k = 1;

[0094] 4) Calculate the fusion weight of the pixel at the image coordinates of the P(x,y) point projected to the anti-clockwise surround camera

[0095]

[0096] Then the fusion weight of the pixel at the image coordinate of the P(x,y) point projected to its clockwise surround camera is 1-w (x，y，z) ;

[0097] 5) Repeat steps 2) to 4) for all points in the 3D projection model to generate a fusion weight lookup table.

[0098] The steps of displaying and generating the 3D surround view image in step 5 of the present invention are as follows:

[0099] 1) Convert the original image collected by each camera in real time into texture, and map the texture onto the 3D projection model according to the 3D projection model projection mapping lookup table obtained above to obtain texture coordinates;

[0100] 2) According to the position of the virtual viewpoint and the camera imaging rule, the image coordinate set projected from the world coordinate vertex set in the 3D projection model mapping lookup table to the virtual imaging surface is calculated to establish a virtual imaging mapping table;

[0101] 3) The texture in the texture object is obtained through the texture coordinates, and the texture is remapped according to the virtual imaging mapping table and the fusion weight lookup table in 2), and finally displayed on the screen area.

Claims

1. A vehicle-mounted 3D surround image display method, characterized in that: The following steps are involved: Step S1, obtain the vehicle size parameters, and calibrate the internal parameter matrix and external matrix of the surround view camera; suppose the vehicle size is [l,w,h], l is the length of the vehicle, w is the width of the vehicle, and h is the height of the vehicle; the internal parameter matrix of the calibrated surround view camera No. i is M i1 , the external parameter matrix is ​​M i2 ,i=0,1,2,3; Step S2, calculating the length and width of the 3D projection model according to the vehicle size; calculating the maximum imaging height of each surround view camera according to the surround view camera calibration parameters, taking the minimum value to obtain the optimal 3D projection model height, and the calculated 3D projection model size is [L, W, H], where L is the length of the 3D projection model, W is the width of the 3D projection model, and H is the height of the 3D projection model; Step S3: According to the calculated 3D projection model size [L, W, H], a grid vertex coordinate set {(X n ,Y n ,Z n )|n=1,2,3…N}, according to the imaging rules of the camera Calculate the coordinates of each vertex of the 3D projection model in the world coordinate system (X n ,Y n ,Z n ) is projected to the image coordinate system coordinates [x n ,y n ], establish the vertex mapping table of the world coordinates and image coordinates of the meshed vertices of the 3D projection model, M1M2 represents the intrinsic parameter matrix and extrinsic parameter matrix of the camera calibration respectively; Step S4: Set the virtual viewpoint, the spatial position coordinate of the virtual viewpoint E is (X E ,Y E ,Z E ), the projection point in the XY plane is E', according to the virtual viewpoint position and virtual imaging parameters, the fusion angle and area of ​​the surround image are dynamically adjusted through the imaging position of the overlapping area in the imaging picture, and the fusion weight of each point in the fusion area is calculated; Step S5, fusing the images according to the vertex mapping table and the fusion weights, generating textures through the GPU, projecting the textures into the 3D projection model, and displaying the images; The fusion weight calculation method of each point in step S4 is as follows: Step S4.1 Determination of fusion boundary vector Step S4.1.1, obtain the 3D projection model mapping lookup table, establish a world coordinate system with the midpoint of the 3D projection model as the center, and construct the circumscribed rectangle of the 3D projection model in the top view, where Q i Represents the vertices of the circumscribed rectangle, i = 1, 2, 3, 4, and the coordinate values ​​can be obtained from the maximum values ​​of the coordinates of all vertices generated in the corresponding quadrant; P i Represents the vertices of the bottom edge of the vehicle in the top view, i = 1, 2, 3, 4; determines the basic fusion boundary vector of each fusion area The acute angle with the horizontal direction is β i ; Step S4.1.2, according to the size of the vehicle and the position of the virtual viewpoint E, calculate the blind area of ​​the 3D projection model blocked by the vehicle model under the viewpoint E. If a certain overlapping area is within the blind area and the proportion of the area blocked on the virtual imaging image exceeds the threshold η, the overlapping area is fused according to the default angle and range; The calculation method of the occlusion ratio is as follows: first, the coordinates of the projection points of the boundary points of the fusion area and the eight vertices of the vehicle's circumscribed cuboid in the virtual imaging plane are calculated, and the area of ​​the overlapping part of the imaging area formed by the projection points of the fusion area's projection point set and the projection points of the eight vertices of the vehicle's circumscribed cuboid is calculated, and the occlusion ratio = the ratio of the area of ​​the overlapping part to the area of ​​the polygon formed by the projection points of the fusion area's projection point set; Step S4.1.3, if the proportion of an overlapping area on the virtual imaging image that is blocked does not exceed the threshold η, the fusion angle and weight are dynamically calculated according to the position of E'; Step S4.1.4, let k be the fusion boundary vector rotation factor, 0≤k≤1, then When E' is in the Q2Q3P3P2 region, Rotate (1-k)β1 clockwise around point P1, Rotate counterclockwise around point P4 by (1-k)β4, where When E' is in the Q1Q4P4P1 region, Rotate (1-k)β3 clockwise around point P3, Rotate counterclockwise around point P2 by (1-k)β2, where When E' is in the Q1Q2P2P1 region, Rotate clockwise around point P4 (1-k)(π / 2-β4), Rotate counterclockwise around point P3 by (1-k)(π / 2-β3), where When E' is in the Q3Q4P4P3 region, Rotate clockwise around point P2 (1-k)(π / 2-β2), Rotate counterclockwise around point P1 by (1-k)(π / 2-β1), where Step S4.2 Fusion weight calculation Step S4.2.1, based on the calculated fusion boundary vector, calculate the fusion weight w of each point P (X, Y, Z) in the fusion area when the fusion angle is σ (x,y,z) , Step S4.2.2: Calculate Direction vector Step S4.2.3: Calculate and The vector angle k indicates polarity. When P is In the counterclockwise direction, k = -1; when P is In clockwise direction, k = 1; Step S4.2.4: Calculate the fusion weight of the pixel at the image coordinate of the surround camera projected to the counterclockwise direction of point P(x,y) Then the fusion weight of the pixel at the image coordinate of the P(x,y) point projected to its clockwise surround camera is 1-w (x,y,z) ; Step S4.2.5: Repeat steps S4.2.2 to S4.2.4 for all points in the 3D projection model to generate a fusion weight lookup table.

2. The 3D surround image display method according to claim 1, characterized in that: The calculation method of the 3D projection model size [L, W, H] in step S2 is as follows: Step S2.1, first determine the display range [L, W] outside the 3D projection model. The display area is manually specified according to different functional requirements. If it is not specified, the default value is set according to the vehicle size: W = k1*w, the default k1 = 3, then Step S2.2, calculate the optimal height H of the 3D projection model, that is, the height of the edge of the model; let the limit projection height of the i-th surround camera in the model be H i , let H i The initial height is H0, which is not optimal. For example, H0 = k2*h, and k2 = 3 by default. According to the camera imaging principle, Calculate all the points on the edge of the 3D projection model {(X i,j ,Y i,j ,H0)|i=0,1,2,3;j=1,2,3…N} The point set in the image coordinate system of this surround camera is {(x ij ,y ij )|i=0,1,2,3;j=1,2,3...N}; Step S2.3, let (x min ,y min )=min{(x ij ,y ij )|i=0,1,2,3;j=1,2,3…N}, if y min <0, then H i =H i -step; if y min >0, then H i =H i +step; if y min = 0, then the maximum imaging height H of the 3D projection model under this surround camera is obtained i ; Among them, step represents H i The amplitude of each adjustment, and H i The unit remains consistent, and the value range is 0≤step≤H i ; Step S2.4: All surround cameras calculate the maximum imaging height H of their own cameras respectively. i , then the optimal height of the 3D projection model is H = min i＝0,1,2,3 {H i }.

3. The 3D surround image display method according to claim 1, characterized in that: The meshed vertex coordinate set of the 3D projection model in step S3 {(X n ,Y n ,Z n )|n=1,2,3…N}, where Z n The calculation formula is as follows: In the formula, (a, b) represents the model surface control point, and its value range is The shape and range of the model surface can be adjusted by changing the values ​​of a and b.

4. The 3D surround image display method according to claim 1, characterized in that: In step S5, the texture is projected into the 3D projection model, and the steps of displaying the image are as follows: Step S5.1, converting the original image collected by each surround camera in real time into a texture, and mapping the texture onto the 3D projection model according to the 3D projection model mapping lookup table to obtain texture coordinates; Step S5.2, according to the position of the virtual viewpoint and the camera imaging rule, calculate the image coordinate set projected from the world coordinate vertex set in the 3D projection model mapping lookup table to the virtual imaging surface, and establish a virtual imaging mapping table; Step S5.3, obtaining the texture in the texture object through the texture coordinates, remapping the texture according to the virtual imaging mapping table and the fusion weight lookup table in step S5.2, and finally displaying it on the screen area.

Images