A pure vision transparent chassis implementation method and system based on a four-way camera on a vehicle

Abstract

The application discloses a kind of based on four-way camera of vehicle-mounted pure vision transparent chassis implementation method and system, comprising: (1) the original picture data of four-way camera is obtained and is deformed;(2) the image after deforming is projected according to mapping to bird's eye view, obtain four-way bird's eye view;(3) color and luminance of four-way bird's eye view are balanced, and the integral bird's eye view of current time is obtained by image splicing;(4) the feature points of adjacent two frames are calculated by LK optical flow method, filter feature points, and the homography matrix of two groups of feature points is calculated;(5) the homography matrix is applied to the bird's eye view P t‑1 of last frame, the transformed extended chassis area is taken out by the mask of extended chassis area, and the bird's eye view of current frame and the transformed extended chassis area are fused;(6) the fused picture is displayed in real time.Using the application, the bird's eye view with the blind area of vehicle chassis can be obtained, which is conducive to improving the safety and reliability of driving.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpits, and in particular to a method and system for realizing a purely visual transparent chassis based on four in-vehicle cameras. Background Technology

[0002] In recent years, with the development of algorithms in image processing and computer vision, the application of cameras in the automotive field has become more widespread, and the tasks that can be accomplished through automotive cameras have become more diverse, such as assisted parking, dashcams, and holographic imaging. More applications for automotive cameras are being explored, and the transparent chassis is one of them. A transparent chassis uses cameras around the vehicle to transform the scenes captured by each camera into a top-down (bird's-eye view) perspective, simultaneously displaying image data under the vehicle's chassis. Instead of directly mounting cameras under the vehicle, the previously captured image data is plotted on the chassis—a blind spot—as the vehicle moves. The advantage of this is that it helps drivers capture more comprehensive information about the road surface, such as lane lines, parking lines, and obstacles like potholes and stones, allowing for timely avoidance.

[0003] For example, Chinese patent document CN115891836A discloses a method for displaying a transparent chassis of an off-road vehicle based on a camera. The method uses the built-in algorithm of the surround vehicle image controller to cache the images captured by the front / rear cameras on the vehicle and stack the captured images frame by frame according to the vehicle speed when the images are captured to obtain the chassis blind spot image. The chassis blind spot is then displayed on the vehicle display screen to form a transparent chassis effect.

[0004] Chinese patent document CN114872631A discloses a method for implementing a transparent chassis function, including: when the vehicle is at location A, acquiring a road surface image in the direction the vehicle is about to travel; combining vehicle speed and steering wheel steering, when the vehicle travels to the corresponding road surface area location B, stitching together the road surface image acquired in advance at location A and the images in the four directions of front, back, left, and right at location B to obtain a 360° panoramic view of the vehicle in all directions and a 180° chassis perspective image of the ground.

[0005] However, current research primarily relies on acquiring vehicle motion signals, such as current speed, angular velocity, acceleration, and angular acceleration, to calculate motion changes in the previous frame or several frames and fill in blind spots in the current top-down view. This approach has the drawback of hindering mass production. Due to the difficulty for different manufacturers to obtain vehicle motion signals and the nonlinear control systems of gasoline-powered vehicles, this method has significant limitations. Therefore, a purely vision-based transparent chassis algorithm has high practical and mass production value. Summary of the Invention

[0006] This invention provides a method and system for achieving a purely visual transparent chassis based on a vehicle-mounted four-way camera. By generating a top-down view from the original images captured by the four-way camera, and by processing the images of historical frames, the road conditions in the blind spots of the current vehicle chassis can be obtained, which helps to improve driving safety.

[0007] A method for implementing a purely visually transparent chassis based on a four-channel vehicle camera includes the following steps:

[0008] (1) Acquire the raw image data from the four cameras and perform distortion correction to obtain the distorted image {I i ,i=1,2,3,4};

[0009] (2) Project the distorted image onto the top view according to the mapping to obtain four-way top views {B}. i ,i=1,2,3,4};

[0010] (3) Perform color and brightness equalization on the four top-view images and then stitch the images together. During the stitching process, construct a fusion weight matrix through the overlapping areas to perform pixel fusion in the overlapping areas, and finally obtain the overall top-view P at the current moment. t ;

[0011] (4) Calculate the feature points of two adjacent frames using the LK optical flow method, filter the feature points, and calculate the homography matrix M for the two sets of feature points;

[0012] (5) Apply the homography matrix M to the top view P of the previous frame. t-1 P t-1 The transformed extended chassis region is extracted by using the mask of the extended chassis region, and the top view P of the current frame is then used. t Fusion is performed using the chassis fusion weight matrix and the transformed extended chassis region;

[0013] (6) Display the overall top view including the transparent chassis at the current time t in real time.

[0014] The specific process of step (2) is as follows:

[0015] In the projection onto the top view, for each camera, a mapping matrix needs to be initialized using the camera's intrinsic and extrinsic parameters to store the pixel coordinates (x, y, y) of the top view. w ,y w The mapping relationship between the image coordinates (u,v) and the image coordinates after distortion removal. Subsequently, only this mapping relationship is needed to generate the top view B of the camera. i ;

[0016] Assume point p:(x) in the image camera coordinate system c ,y c,z c The imaging point in the image coordinate system is q:(x,y). Based on similar triangles, we obtain the following equation:

[0017]

[0018] Where f is the camera's focal length and z is the normalization coefficient, the transformation from image coordinates to pixel coordinates is then obtained as follows:

[0019]

[0020] Where dx and dy represent the size of each pixel in the image coordinate system, u0 and v0 represent the translation from the image coordinate system to the x-axis and y-axis of the pixel coordinate system, respectively, and are parameters inherent to each camera, just like the focal length f. Represents a rotation matrix. Let z represent the translation vector, which constitutes the extrinsic parameters of the camera; in calculating the top view of each camera, let z w =1, for every point (x w ,y w The corresponding pixel coordinates (u, v) are obtained through the above formula, and the pixel values ​​are extracted and assigned to the top view, thus obtaining the top view for each path. Therefore, we obtain

[0021] In step (3), the specific process of performing color equalization on the four top views is as follows:

[0022] There are already 4 top views B i i = 1, 2, 3, 4, assume B i-1 It is B i This is the previous top view in a clockwise direction. Here, we set the image above the first camera's image in a clockwise direction to be the image from the fourth camera. The color correction coefficient α for the c channel of the i-th camera is... i,c We obtain it using the following formula:

[0023]

[0024] Among them, B i-1 ∪B i It is picture B. i-1 and B i The overlapping part, B i,c (u,v) is the top view B i The pixel value of channel c at pixel coordinates (u,v), where c∈{R,G,B}, and λ represents the enhancement coefficient; the mean values ​​of the RGB three channels in the overlapping region need to be consistent; the resulting color correction coefficient α i,c Then, the image from the i-th camera is corrected using the following formula:

[0025] B i,c ←α i,c B i,c .

[0026] In step (3), the specific process of image stitching is as follows:

[0027] For weighted fusion of overlapping areas from various cameras, the projected image B on the top view... i-1 and B i The overlapping region, where the coordinates are (x,y)∈B i-1 ∪B i The distances from the point to the boundaries of the two cameras' projection areas in the top view are d1(x,y) and d2(x,y). The weights w1(x,y) and w2(x,y) are calculated. Since the image is an RGB three-channel image, weights are fused for each channel of the image. The formula for calculating the weights is:

[0028]

[0029] For top view image B i-1 and B i The overlapping areas are corrected as follows:

[0030] B i,c (x,y)←w 1,c (x,y)·B i,c (x,y)+w 2,c (x,y)·B i-1,c (x,y)

[0031] Finally, by fusing the four top views, we obtain the overall top view result P at time t, with a width of W and a height of H. t .

[0032] The specific process of step (4) is as follows:

[0033] First, the Harris corner detection algorithm is used to obtain a set of corner points in the image, and then the Lucas-Kanade optical flow method is used to calculate the feature points of two adjacent frames.

[0034] For each detected feature point, after obtaining the corresponding feature point matching pairs using a filtering strategy, the homography matrix is ​​calculated using the findHomography method provided by OpenCV.

[0035] The filtering strategies include:

[0036] Considering the direction of motion, vehicles can reverse or move forward. The incorrect directions are filtered out based on the mode of the direction.

[0037] Considering the direction of motion speed, points whose motion distance exceeds a threshold are filtered out; and the motion distance of points in each camera area is limited to [0.5μ, 1.5μ], where μ represents the average motion distance of all feature points;

[0038] The RANSAC algorithm is used for final overall filtering.

[0039] The specific process of step (5) is as follows:

[0040] Assuming the current time is t, the overall top view P at time t-1 is shown below. t-1 The OpenCV-provided warpPerspective function is called to apply the obtained homography matrix M to P. t-1 The above will yield P′ t-1 and P t Fusion is performed using the extended chassis fusion weight matrix Q;

[0041] The generation of the expanded chassis fusion weight matrix Q consists of two parts: the original chassis region, with a value of 1; and the expanded region, which comprises four quarter circles and four rectangles. For the rectangular region, the width is r, and its value is inversely proportional to the distance to the rectangular side of the original chassis region. For the quarter circle region, the radius is r, and its value is inversely proportional to the distance to the vertices of the original chassis region. The values ​​for other regions are 0. The pixel values ​​of the fused new image are:

[0042] P t =QP′ t-1 +(1-Q)P t .

[0043] A system for achieving a purely visual transparent chassis based on a four-channel vehicle-mounted camera includes:

[0044] The FakeCamera module is used to bridge with cameras and periodically capture image data from four cameras.

[0045] The Transform module is used to remove distortion from images from four cameras and to calculate the top view.

[0046] The four-way top view fusion module is used to merge four top view images into one top view;

[0047] The transparent chassis module is used to perform feature point matching using the LK optical flow method. It calculates feature point matching pairs between the previous frame top view and the current top view, calculates the homography matrix, obtains the transformed previous frame top view, and extracts the expanded chassis area and performs weighted fusion with the current top view.

[0048] The image output module is used to display the generated top view, which includes the transparent chassis, to the vehicle terminal in real time.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This invention is a pure vision solution for a transparent chassis. Compared to the need to obtain vehicle speed and angular velocity information, this invention has better portability. It only needs to obtain data from four cameras to complete the transparent chassis task, and it can even be deployed on traditional fuel vehicles where chassis information is difficult to obtain.

[0051] 2. The chassis fusion strategy proposed in this invention can effectively solve the problem of severely textured chassis images, resulting in a more aesthetically pleasing image.

[0052] 3. The color correction algorithm proposed in this invention can effectively balance the overall color and brightness of the top view under the condition that the exposure and scene brightness of the four cameras are inconsistent.

[0053] 4. The present invention utilizes relevant parameter initialization and matrix calculation, and through thread optimization strategies, it can achieve good performance on mobile embedded devices;

[0054] 5. Offers different resolution sizes to choose from to meet real-time requirements. Attached Figure Description

[0055] Figure 1 This is a flowchart of the method of the present invention;

[0056] Figure 2 A top-view diagram illustrating weighted fusion of the overlapping camera area;

[0057] Figure 3 For the extended chassis schematic diagram;

[0058] Figure 4 This is a schematic diagram of the motion changes of corner points obtained by LK optical flow detection;

[0059] Figure 5 This is a diagram showing the completion of the missing chassis image portion in the top view during a turning scene.

[0060] Figure 6 A visual diagram illustrating the weights of the front-view and left-view cameras, and the fusion effect of the overlapping parts;

[0061] Figure 7 A comparison of the transparent chassis effect obtained without using a color balance algorithm and with a color balance algorithm;

[0062] Figure 8 A visual diagram illustrating the integration weights of the chassis;

[0063] Figure 9A comparison chart showing the effects of using chassis weight fusion and not using chassis weight fusion. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0065] like Figure 1 As shown, a method for implementing a purely visual transparent chassis based on a vehicle-mounted four-channel camera includes the following steps:

[0066] (1) Acquire the raw image data from the four cameras and perform distortion correction to obtain the distortion-corrected image {I i ,i=1,2,3,4}.

[0067] (2) The distortion-removed image is then processed according to the mapping... Projecting onto the top view, we get four top views {B} i ,i=1,2,3,4}.

[0068] (3) Perform color and brightness equalization on the four top-view images, and stitch the images together. Since the top-view images of the front left, front right, left rear, and right rear of the four images will overlap, a fusion weight matrix is ​​constructed through the overlapping areas to perform pixel fusion in the overlapping areas, so that the final stitching will not be abrupt, and the overall top-view P at the current moment is obtained. t .

[0069] (4) Calculate the feature points of two adjacent frames using the LK optical flow method, filter the feature points, and calculate the homography matrix M for the two sets of feature points.

[0070] (5) Apply the homography matrix to the top view P of the previous frame. t-1 The transformed extended chassis region is extracted by using the mask of the extended chassis region, and the top view of the current frame is fused using the chassis fusion weight matrix.

[0071] (6) Display the overall top view including the transparent chassis at the current time t in real time.

[0072] Before proceeding with this process, it is necessary to obtain the intrinsic and extrinsic parameters and distortion parameters of the four cameras through camera calibration. In step (1), the raw image data of the four cameras is acquired and distortion is removed to obtain the distortion-removed image {I}. i ,i=1,2,3,4}.

[0073] In step (2), the distortion-corrected image is obtained and projected onto the top view. During the projection onto the top view, for each camera, a mapping matrix needs to be initialized using the camera's intrinsic and extrinsic parameters to store the pixel coordinates (x, y, y) of the top view.w ,y w The mapping relationship between the image coordinates (u,v) and the image coordinates after distortion removal. This mapping can also be written as Subsequently, the top-down view B of this camera can be generated simply by using this mapping relationship. i For each camera, with fixed intrinsic and extrinsic parameters, this matrix is ​​unique and does not need to be generated repeatedly, thus saving computation time. Assume the point p in the image camera coordinate system is: (z... c ,y c ,z c The imaging point in the image coordinate system is q:(x,y). Based on similar triangles, we can obtain Equation 1.1:

[0074]

[0075] Where f is the camera's focal length and z is the normalization coefficient. The complete transformation from the image coordinate system to the pixel coordinate system is shown in Equation 1.2:

[0076]

[0077] Where dx and dy represent the size of each pixel in the image coordinate system, u0 and v0 represent the translation amount from the image coordinate system to the x-axis and y-axis of the pixel coordinate system, respectively, and are inherent parameters of each camera, just like the focal length f. Represents a rotation matrix. Let z represent the translation vector, which constitutes the extrinsic parameters of the camera. In calculating the top-down view for each camera, let z... w =1, for every point (x w ,y w Each pixel coordinate (u, v) can be obtained using Equation 1.2. The pixel value is then extracted and assigned to the top view to obtain the top view for each path. From this, we can obtain

[0078] In step (3), the top-view images from the four cameras have already been obtained, so it is necessary to merge the top-view images from the four cameras into one image. Since the exposure intensity and lighting conditions of the different cameras vary in the actual captured images, there will be some differences even at the same time. Therefore, the following strategy is used for color correction. Because the top-view images from the front left, front right, left rear, and right rear of the four cameras will overlap, the following color correction algorithm for the four cameras is designed to perform color balance and improve the overall effect after stitching. Four top-view images B are already available. i i = 1, 2, 3, 4, assume B i-1 It is B iThis is the previous top view in a clockwise direction (here, when i=1, i-1=4, meaning the previous view clockwise from the first camera is the fourth camera's image), and the color correction coefficient α of the c channel (c∈{R,G,B}) of the i-th camera. i,c It can be obtained using formula 1.3:

[0079]

[0080] Among them, B i-1 ∪B i It is picture B. i-1 and B i The overlapping part, B i,c (u,v) is the top view B i The pixel coordinates are (u, v) representing the pixel value of channel c, and λ represents the enhancement coefficient, which can be set to 1.3. Equation 1.3 means that the mean values ​​of the RGB three channels representing the overlapping area need to be consistent. The resulting color correction coefficient α i,c Then, the image of the i-th camera is corrected as shown in Equation 1.4.

[0081] B i,c ←α i,c B i,c #1.4

[0082] The pseudocode for the color correction algorithm is as follows:

[0083]

[0084]

[0085] Starting from i=1, the subsequent color correction coefficients are calculated sequentially. After calculating one round of subsequent color correction coefficients, a global correction is required to ensure that the overall mean of the color correction coefficients for each channel is 1. This prevents color oversaturation. The final four-way top view is then corrected by iteratively calculating and updating the color correction coefficients.

[0086] For weighted fusion of overlapping areas of various cameras, such as Figure 2 As shown, the projected image B on the top view i-1 and B i Consider the overlapping region where the coordinates are (x,y)∈B. i-1 ∪B i Considering the distances from this point to the boundaries of the projection areas of the two cameras in the top view are d1(x,y) and d2(x,y), the weights w1(x,y) and w2(x,y) are calculated. Since the image is an RGB three-channel image, weight fusion is performed for each channel of the image. The formula for calculating the weights is shown in Equation 1.5.

[0087]

[0088] After obtaining the weights w1(x,y) and w2(x,y), the top view image B is processed. i-1 and B i The overlapping areas are corrected as follows:

[0089] B i,c (x,y)←w 1,c (x,y)·B i,c (x,y)+w 2,c (x,y)·B i-1,c (x,y)

[0090] Finally, by fusing the four top views, we obtain the overall top view result P at time t, with a width of W and a height of H. t .

[0091] In step (4), the Harris corner detection algorithm is first used to obtain a set of corner points on the image, and then the Lucas-Kanade optical flow method is used to calculate the feature points of adjacent frames.

[0092] In the LK optical flow method, the image from the camera is considered to change over time. Therefore, at time t, the pixel gray level at position (x,y) can be written as:

[0093] G(x,y,t)

[0094] Assume that the pixels within a certain window have the same motion. Consider a window surrounding a corner point; the window is a square with side length w, and w pixels are within the window. 2 The number of pixels, since the pixels within the window all move in the same way, is w in total. 2 The equation is as shown in Equation 1.6:

[0095]

[0096] Among them, v x v is the velocity of a pixel on the x-axis. y G represents the velocity of a pixel on the y-axis. t G represents the change in image grayscale over time. x and G y Let represent the gradients of the image at that point with respect to the x and y directions, respectively.

[0097]

[0098] For this overdetermined linear equation in terms of u and v, a closed-form solution can be obtained using the least squares method:

[0099]

[0100] For the detected feature points, considering the vehicle's motion characteristics and the actual effect, the following filtering strategy is used:

[0101] 1. Considering the direction of motion, vehicles can reverse or move forward. Filter out incorrect directions based on the mode of the direction (forward and backward).

[0102] 2. Considering the direction of motion speed, filter out points whose motion distance exceeds the threshold; and ensure that the motion distance of points in each camera area is limited to [0.5μ, 1.5μ], where μ represents the average motion distance of all feature points;

[0103] 3. The RANSAC algorithm is used to finally filter the entire system.

[0104] Once the corresponding feature point matching pairs are obtained, the homography matrix can be calculated using the findHomography method provided by OpenCV.

[0105] In step (5), assuming the current time is t, the overall top view P at time t-1 is... t-1 The OpenCV-provided warpPerspective function is called to apply the obtained homography matrix M to P. t-1 The above will yield P. ′ t-1 and P t The fusion is performed using the expanded chassis weight matrix. The generation of the expanded chassis weight fusion matrix Q consists of two parts: one part is the original chassis region, with a value of 1; the other part is the expanded region, such as... Figure 3 As shown. The extended region consists of four quarter circles and four rectangles. For the rectangular region, the width is r, and its value is inversely proportional to the distance from the rectangular side of the original chassis region; for the quarter circle region, the radius is r, and its value is inversely proportional to the distance from the vertex of the original chassis region; the values ​​for other regions are 0. The pixel values ​​of the merged new image are shown in Equation 1.8.

[0106] P t =QP′ t-1 +(1-Q)P t #1.8

[0107] This invention also provides a pure visual transparent chassis implementation system based on a vehicle-mounted four-channel fisheye camera. This system converts fisheye images captured by the four fisheye cameras into a top-down view in real time, and allows for selective activation of the transparent chassis function. Specific modules include a FakeCamera module, a Transform module, a four-channel top-down view fusion module, a transparent chassis module, and an image output module. The data transfer types between the modules are mainly ImageFrame and ImageFrameBev. The ImageFrame type ensures data synchronization; when generating the original ImageFrame image, the current time's original images from the four cameras and the timestamp are obtained. ImageFrameBev stores the ImageFrame type and the generated overall top-down view data.

[0108] The FakeCamera module is used to bridge with cameras and periodically capture data from four cameras. The ImageFrame type stores the image data from the four cameras. By maintaining a queue of these ImageFrames, the captured data from the four cameras at the current moment, along with the current timestamp, is packaged into an ImageFrame and added to the queue.

[0109] The Transform module is used for image distortion correction and top-view calculations. It retrieves data from the ImageFrame queue in the FakeCamera module, performs distortion correction and top-view calculations, and then enqueues the data into a new ImageFrame queue.

[0110] The four-view top-down fusion module receives the ImageFrame queue data after image processing from the previous module. It merges the four top-down images in the ImageFrame into a single top-down view. The processed top-down view is then added to the ImageFrameBev queue.

[0111] The transparent chassis module maintains the top view of the previous frame and receives data from the ImageFrameBev queue to obtain the current top view. Feature point matching is performed using the LK optical flow method, calculating feature point matching pairs between the previous and current top views, and calculating the homography matrix to obtain the transformed top view of the previous frame. The expanded chassis region is then extracted and weighted with the current top view for fusion.

[0112] The image output module displays the generated top-down view, including the transparent chassis, to the vehicle terminal in real time.

[0113] Since the system of this invention runs on an embedded device, high real-time performance is required. Therefore, the system of this invention provides three different resolution solutions: output transparent chassis image sizes of 200×280, 400×500, and 480×960, to meet the real-time requirements of embedded devices with different processing speeds.

[0114] In the specific implementation process, the first step is to remove distortion from the original fisheye image, and then project it onto the top view.

[0115] After obtaining the four top-view images, a color correction algorithm is used to process the four images. Figure 7 To compare the effects of using and not using the color correction algorithm, the left image shows the transparent chassis effect obtained without applying the color balance algorithm, while the right image shows the effect after applying the color correction algorithm. It can be clearly seen that in scenes with obvious brightness imbalance, the overall color and brightness of the top view have a better balance effect after using the color correction algorithm.

[0116] The four images are fused using a weighted fusion method for overlapping regions, such as... Figure 6 As shown, the left image is a visualization of the weights of the front-view and left-view cameras, and the right image shows the fusion effect of the overlapping parts. Since the cameras are fixed, the overlap between the cameras does not change during movement. Therefore, only the weight matrices of the pairwise overlapping areas need to be initialized once. Furthermore, this fusion step can be transformed into matrix multiplication operations, further improving performance and obtaining an overall top-down view.

[0117] Subsequently, on the overall top view, the Harris corner detection algorithm and LK optical flow feature point motion prediction were used, such as... Figure 4 As shown, the homography matrix is ​​obtained by performing corresponding filtering operations. Figure 4 In the diagram, the red dots represent the corners detected by the Harris corner detection algorithm, serving as feature points; each green arrow represents the direction of movement for each feature point.

[0118] Through homography matrix transformation and chassis fusion weighting, such as Figure 8 The diagram shows the weight matrix of the chassis area. For a more intuitive presentation, the original weight range [0,1] has been linearly extended to the range [0,255].

[0119] Complete the chassis details in the current top view. The final schematic diagram with the completed chassis area is shown below. Figure 5 As shown. Figure 9 The image shown is a comparison of the effects of using fusion without chassis weights and using fusion with chassis weights.

[0120] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for realizing a pure vision transparent chassis based on a four-lens camera on a vehicle, characterized in that, Includes the following steps: (1) Acquire the raw image data from the four cameras and perform distortion correction to obtain the distorted image. ; (2) Project the distorted image onto the top view according to the mapping to obtain four top views. ; (3) Perform color and brightness equalization on the four top views and stitch the images together; during the stitching process, construct a fusion weight matrix through the overlapping areas to perform pixel fusion in the overlapping areas, and finally obtain the overall top view at the current moment. ; The specific process of performing color equalization on the four top views is as follows: There are already 4 top-down views Assuming yes This is the previous top view in a clockwise direction. Here, we set the image above the first camera's image in a clockwise direction to be the image from the fourth camera. One camera Color correction factor of the channel We obtain it using the following formula: in, It's a picture. and The overlapping part, Top view The pixel coordinates are The passage pixel values, , The enhancement factor is represented by the average values ​​of the RGB three channels in the overlapping region, which need to be consistent; the resulting color correction factor is... Afterwards, regarding the first The images from each camera are corrected using the following formula: ； (4) Calculate the feature points of two adjacent frames using the LK optical flow method, filter the feature points, and calculate the homography matrix for the two sets of feature points. The specific process is as follows: First, the Harris corner detection algorithm is used to obtain a set of corner points on the image. Then, the Lucas-Kanade optical flow method is used to calculate feature points between adjacent frames. For the detected feature points, a filtering strategy is used to obtain corresponding feature point matching pairs. Finally, the homography matrix is ​​calculated using the findHomography method provided by OpenCV. ; (5) The homography matrix Top view applied to the previous frame The transformed extended chassis region is extracted using the mask of the extended chassis region, and the top view of the current frame is obtained. Fusion is performed using the chassis fusion weight matrix and the transformed extended chassis region; the specific process is as follows: Assume the current time is , Overall top view of the moment Call the warpPerspective function provided by OpenCV to obtain the homography matrix. Effect on Up, will get and Through the expanded chassis fusion weight matrix To merge; Regarding the expanded chassis fusion weight matrix The generation consists of two parts: the original chassis area, with a value of 1; and an extended area, which comprises four quarter circles and four rectangles; the width of the rectangular area is... Its value is inversely proportional to the distance from the rectangular side of the original chassis area; for a 1 / 4 circular area with a radius of... r Its value is inversely proportional to the vertex distance of the original chassis region, and 0 in other regions; the pixel values ​​of the fused new image are: ； (6) Real-time display of current status A top-down view that includes the transparent chassis at all times.

2. The method for achieving a purely visually transparent chassis based on a vehicle-mounted four-channel camera according to claim 1, characterized in that, The specific process of step (2) is as follows: In the projection onto the top view, for each camera, a mapping matrix needs to be initialized using the camera's intrinsic and extrinsic parameters to store the pixel coordinates of the top view. and the coordinates of the image after distortion removal mapping relationship Subsequently, a top-down view of the camera can be generated simply by using this mapping relationship. ; Assuming the point in the image camera coordinate system The imaging point in the image coordinate system is Based on similar triangles, we obtain the following equation: in, For the camera's focal length, The normalization coefficients are then transformed from the image coordinate system to the pixel coordinate system, resulting in the complete transformation as follows: in, and These represent the dimensions of each pixel in the image coordinate system. and Represents the image coordinate system to the pixel coordinate system shaft and The amount of translation of the axis and the focal length These are parameters inherent to each camera; Represents a rotation matrix. Let represent the translation vector, which constitutes the extrinsic parameters of the camera; in calculating the top view of each camera, let For each point The corresponding pixel coordinates are obtained through the above formula. Then, extract the pixel values ​​and assign them to the top view, thus obtaining the top view for each path. Thus we obtain .

3. The method for achieving a purely visually transparent chassis based on a vehicle-mounted four-channel camera according to claim 1, characterized in that, In step (3), the specific process of image stitching is as follows: Weighted fusion of overlapping areas from various cameras, projected image on top view and The overlapping region, where the coordinates are ( The distance from the point to the boundary of the projection area of ​​the two cameras in the top view is and Calculate weights and Since the image is an RGB three-channel image, weighted fusion is performed on each channel of the image. The formula for calculating the weights is: Top view image and The overlapping areas are corrected as follows: Finally, by merging the four top views, we obtain Time width is Gao Wei Overall top view results .

4. The method for achieving a purely visually transparent chassis based on a vehicle-mounted four-channel camera according to claim 1, characterized in that, The filtering strategy includes: Considering the direction of motion, vehicles can reverse or move forward. The incorrect directions are filtered out based on the mode of the direction. Considering the direction of motion speed, points whose motion distance exceeds a threshold are filtered out; and the motion distance of points within each camera area is limited. Between, among This represents the mean distance traveled by all feature points. The RANSAC algorithm is used for final overall filtering.

5. A system for implementing a purely visually transparent chassis based on a vehicle-mounted four-channel camera, used to execute the purely visually transparent chassis implementation method according to any one of claims 1 to 4, characterized in that, include: The FakeCamera module is used to bridge with cameras and periodically capture image data from four cameras. The Transform module is used to remove distortion from images from four cameras and to calculate the top view. The four-way top view fusion module is used to merge four top view images into one top view; The transparent chassis module is used to perform feature point matching using the LK optical flow method. It calculates feature point matching pairs between the previous frame top view and the current top view, calculates the homography matrix, obtains the transformed previous frame top view, and extracts the expanded chassis area and performs weighted fusion with the current top view. The image output module is used to display the generated top view, which includes the transparent chassis, to the vehicle terminal in real time.

Images