Vehicle surround view image stitching method and device, computer device, and storage medium

Abstract

The embodiment of the application belongs to the field of intelligent networked vehicles, and relates to a vehicle surround view image splicing method, which comprises the following steps: acquiring a to-be-corrected image; performing a checkerboard calibration processing on the to-be-corrected image to obtain a first corrected image; performing a top-down transformation processing on the first corrected image to obtain a second corrected image; and performing a correction splicing processing on the second corrected image to obtain a target corrected image. The application also provides a vehicle surround view image splicing device, a computer device and a storage medium. The application can significantly improve the geometric accuracy and visual continuity of the surround view image, reduce the splicing traces and blind area, provide a driver with a higher-quality and more stable panoramic surround view image, and improve the driving safety.

Description

Technical Field

[0001] This application relates to the field of intelligent connected vehicle technology, and in particular to a method, apparatus, computer equipment, and storage medium for stitching vehicle surround view images. Background Technology

[0002] Existing vehicle surround view systems acquire images of the vehicle's surroundings through multiple cameras, but suffer from issues such as noticeable stitching artifacts and unnatural transitions in the surround view images. Current mainstream technologies mostly employ traditional correction and stitching algorithms to stitch together images from multiple cameras.

[0003] However, traditional correction stitching algorithms, when processing overlapping areas of images, are still prone to leaving seams in the overlapping areas due to camera pixel limitations and low image quality. These seams are often obvious and difficult to achieve a natural and seamless transition, resulting in unsatisfactory fusion effects. Summary of the Invention

[0004] The purpose of this application is to provide a vehicle surround view image stitching method, device, computer equipment, and storage medium to solve the problems of obvious stitching marks and unnatural transitions in existing vehicle surround view systems.

[0005] To address the aforementioned technical problems, this application provides a method for stitching together vehicle surround view images, employing the following technical solution:

[0006] Obtain the image to be corrected;

[0007] The image to be corrected is subjected to checkerboard calibration to obtain the first corrected image;

[0008] The first corrected image is subjected to a top-view transformation to obtain a second corrected image;

[0009] The second corrected image is then subjected to corrective stitching processing to obtain the target corrected image. Further, the step of performing checkerboard calibration processing on the image to be corrected to obtain the first corrected image specifically includes:

[0010] Obtain the camera coordinate system of the chessboard calibration version;

[0011] The camera correction parameters are obtained by performing distortion processing on the camera coordinate system.

[0012] The image to be corrected is processed based on the camera correction parameters to obtain the first corrected image.

[0013] Furthermore, the camera coordinate system includes camera corner pixel coordinates and physical coordinates, and the camera correction parameters include camera intrinsic parameter matrix and camera distortion coefficients. The step of performing checkerboard distortion processing based on the camera coordinate system to obtain the camera correction parameters specifically includes:

[0014] The physical coordinates are subjected to matrix transformation to obtain the camera intrinsic parameter matrix;

[0015] The camera distortion coefficients are obtained by calculating the pixel coordinates of the camera corner points using the radial distortion formula.

[0016] Furthermore, the step of performing a top-view transformation on the first corrected image to obtain the second corrected image specifically includes:

[0017] Vehicle feature points are determined based on the first corrected image;

[0018] The vehicle feature points are subjected to a direct linear transformation to obtain the second corrected image.

[0019] Furthermore, the vehicle feature points include original points and target projection points, and the step of performing a direct linear transformation on the vehicle feature points to obtain the second corrected image further includes:

[0020] Based on the original points and the target projection points, homography matrix transformation is performed to obtain the target transformed image;

[0021] The target transformed image is processed by position planning based on the direct linear transformation algorithm to obtain the second corrected image.

[0022] Furthermore, the step of performing correction and stitching processing on the second corrected image to obtain the target corrected image further includes:

[0023] The second corrected image is divided into regions to determine the target overlapping image and the non-target overlapping image;

[0024] The overlapping target images are subjected to fade-in / fade-out fusion processing to obtain a third corrected image;

[0025] The third corrected image is stitched together with the non-target overlapping image to obtain the target corrected image.

[0026] Furthermore, the step of performing the fade-in / fade-out fusion process on the target overlapping images to obtain the third corrected image includes:

[0027] Obtain the two-dimensional coordinate information of each target pixel in the overlapping image;

[0028] The non-target overlapping image is dilated to obtain its contour information.

[0029] Based on the two-dimensional coordinate information and the contour information, the weight matrix and inverse weight matrix of the target pixel are calculated to obtain the target pixel.

[0030] Based on the weight matrix and the inverse weight matrix, the RGB channels corresponding to the target pixel are weighted to obtain the third corrected image.

[0031] To address the aforementioned technical problems, this application also provides a vehicle surround view image stitching device, which employs the following technical solution:

[0032] The acquisition module is used to acquire the image to be corrected.

[0033] The first processing module is used to perform checkerboard calibration processing on the image to be corrected to obtain the first corrected image;

[0034] The second processing module is used to perform top-view transformation processing on the first corrected image to obtain the second corrected image;

[0035] The third processing module is used to perform correction and stitching processing on the second corrected image to obtain the target corrected image.

[0036] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0037] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the vehicle surround view image stitching method.

[0038] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0039] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the vehicle surround view image stitching method.

[0040] Compared with the prior art, the embodiments of this application have the following main advantages:

[0041] This application embodiment obtains an image to be corrected; performs checkerboard calibration on the image to be corrected to obtain a first corrected image; performs top-view transformation on the first corrected image to obtain a second corrected image; and performs correction and stitching on the second corrected image to obtain a target corrected image. By sequentially performing checkerboard calibration, top-view transformation, and correction and stitching on the image to be corrected, this method can significantly improve the geometric accuracy and visual coherence of the panoramic image, reduce stitching marks and blind spots, provide the driver with a higher quality and more stable panoramic view, and improve driving safety. Attached Figure Description

[0042] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is an exemplary architecture diagram to which this application can be applied;

[0044] Figure 2 This is a flowchart of an embodiment of the vehicle surround view image stitching method according to this application;

[0045] Figure 3 This is a schematic diagram of a structure of an embodiment of the vehicle surround view image stitching device according to this application;

[0046] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application;

[0047] Figure 5 This is a flowchart of the checkerboard calibration process according to the vehicle surround view image stitching method of this application;

[0048] Figure 6 This is a view stitching area division diagram based on the vehicle surround view image stitching method of this application;

[0049] Figure 7 This is a top-view transformation diagram of the vehicle surround view image stitching method according to this application; Detailed Implementation

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0051] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, the system architecture 100 of the vehicle surround view image stitching system may include a terminal device 101, a network 102, and a server 103. The terminal device 101 may be an in-vehicle computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is used as a medium to provide a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0054] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0055] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to in-vehicle computer 1011, tablet computer 1012 or mobile phone 1013, terminal device 101 can also be e-book reader, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer and desktop computer, etc.

[0056] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0057] It should be noted that the vehicle surround view image stitching method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the vehicle surround view image stitching device is generally set in the server / terminal device.

[0058] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0059] Continue to refer to Figure 2 A flowchart illustrating an embodiment of a vehicle surround view image stitching method according to this application is shown. The vehicle surround view image stitching method includes the following steps:

[0060] Step S201: Obtain the image to be corrected.

[0061] In this embodiment, the above-described vehicle surround view image stitching method can be deployed in a vehicle surround view image stitching platform. This platform can be constructed using a server or server cluster. The server or server cluster can be any electronic device with data transmission and data storage functions. The electronic device on which the vehicle surround view image stitching method runs (e.g., Figure 1 The server / terminal device shown can acquire the image to be corrected via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods.

[0062] In this embodiment, the image to be corrected can be the original vehicle surround view image captured in real time by fisheye cameras installed in the front, rear, left, right or multiple directions of the vehicle, or it can be a test image or calibration image pre-stored in the vehicle surround view image stitching system, used for subsequent distortion correction, top view transformation and stitching fusion processing to obtain a seamless and natural panoramic surround view.

[0063] Step S202: Perform checkerboard calibration on the image to be corrected to obtain the first corrected image.

[0064] In this embodiment, the first corrected image can be a distortion-free image obtained by processing the image to be corrected through checkerboard calibration and camera intrinsic parameters and distortion coefficient correction. That is, a standardized image that has completed fisheye distortion correction, geometric correction and pixel remapping processing, which is used for subsequent top-view transformation and stitching fusion steps.

[0065] Specifically, the detailed implementation process of performing checkerboard calibration on the image to be corrected to obtain the first corrected image will be described in further detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0066] Step S203: Perform top-view transformation on the first corrected image to obtain the second corrected image.

[0067] In this embodiment, the second corrected image can be a bird's-eye view image obtained by top-down transformation of the first corrected image. That is, by performing direct linear transformation and homography matrix calculation on the feature points around the vehicle, the original tilted view image is projected into a top-down plane image for subsequent correction stitching and fade-in / fade-out fusion processing.

[0068] Specifically, the detailed implementation process of performing top-view transformation on the first corrected image to obtain the second corrected image will be further described in subsequent specific embodiments of this application, and will not be elaborated on here.

[0069] Step S204: Perform correction and stitching processing on the second corrected image to obtain the target corrected image.

[0070] In this embodiment, the target corrected image can be the final surround view stitched image obtained by processing the second corrected image through region division, weight matrix calculation and improved fade-in and fade-out fusion algorithm. That is, the panoramic surround view image obtained by seamlessly and naturally fusing the bird's-eye view images generated by multiple cameras after distortion correction and top-down transformation in the overlapping area. It is used to provide a high-precision, wide-field-of-view environmental perception image for the driver or vehicle surround view image stitching system.

[0071] Specifically, the detailed implementation process of performing correction and stitching on the second corrected image to obtain the target corrected image will be further described in subsequent specific embodiments of this application, and will not be elaborated on here.

[0072] This application obtains an image to be corrected; performs checkerboard calibration on the image to be corrected to obtain a first corrected image; performs top-view transformation on the first corrected image to obtain a second corrected image; and performs correction and stitching on the second corrected image to obtain the target corrected image. By sequentially performing checkerboard calibration, top-view transformation, and correction and stitching on the image to be corrected, this method can significantly improve the geometric accuracy and visual coherence of the panoramic image, reduce stitching marks and blind spots, provide drivers with a higher quality and more stable panoramic view, and improve driving safety.

[0073] In some alternative implementations, step S202 includes the following steps:

[0074] Obtain the camera coordinate system of the chessboard calibration version;

[0075] The camera correction parameters are obtained by performing distortion processing using the camera coordinate system;

[0076] The image to be corrected is processed based on the camera correction parameters to obtain the first corrected image.

[0077] In this embodiment, the aforementioned checkerboard calibration plate can be a standardized checkerboard calibration plate for camera calibration, which consists of regularly arranged black and white squares. Each square has a known size and is used to extract the corner coordinates and physical coordinates of the camera imaging system, thereby establishing a camera coordinate system.

[0078] In this embodiment, the camera correction parameters may include a camera intrinsic parameter matrix and camera distortion coefficients. Specifically, the camera intrinsic parameter matrix may include camera focal length parameters, principal point coordinates, pixel ratio coefficients, and the horizontal and vertical angles of the photosensitive plate, etc., to describe the geometric projection relationship of the camera imaging system; the camera distortion coefficients may include radial distortion coefficients and tangential distortion coefficients, to characterize the degree of barrel or pincushion distortion produced by the fisheye camera lens on the image at different radii.

[0079] By using both the camera intrinsic matrix and the camera distortion coefficients, the true projection position of each pixel can be accurately calculated in the subsequent distortion correction process, achieving high-precision geometric correction of the image to be corrected and laying an accurate image foundation for subsequent top-down transformation and stitching fusion.

[0080] In this embodiment, the camera coordinate system may include the camera corner pixel coordinates and physical coordinates (the camera's attitude and position parameters in the world coordinate system) extracted during the calibration process, which are used to subsequently calculate the camera intrinsic parameter matrix and distortion coefficients, thereby achieving accurate distortion removal processing of the image to be corrected.

[0081] Specifically, the aforementioned camera corner pixel coordinates can be the horizontal and vertical coordinates of each chessboard intersection (corner point) in the image pixel plane, obtained by the image detection algorithm when the camera captures the image of the chessboard calibration board, and are used to characterize the two-dimensional position of the corner point on the sensor imaging surface; the aforementioned physical coordinates can be the three-dimensional spatial position coordinates of each corner point in the world coordinate system, calculated based on the known actual size of each square of the chessboard calibration board, and are used to reflect the geometric position of the corner point in real space.

[0082] In one possible embodiment, the process of processing the image to be corrected based on camera correction parameters may include: after the vehicle is calibrated, the vehicle surround view image stitching system loads the intrinsic parameter matrices and distortion coefficients of the four fisheye cameras (front, rear, left, and right) respectively, and inputs the real-time acquired image to be corrected into the image processing module; the processing module calls image processing libraries such as OpenCV to remap each pixel of the image according to the distortion model, converting the distorted pixel coordinates into ideal distortion-free coordinates, and generating the corrected image; the remapped output is the first corrected image, which can be directly used for subsequent top-down transformation and stitching fusion.

[0083] Specifically, the detailed implementation process of obtaining camera correction parameters through distortion processing using the camera coordinate system will be further described in subsequent specific embodiments of this application, and will not be elaborated upon here.

[0084] In some alternative implementations, the step "perform distortion correction through the camera coordinate system to obtain camera correction parameters" includes the following steps:

[0085] The physical coordinates are transformed by matrix transformation to obtain the camera intrinsic parameter matrix;

[0086] The camera distortion coefficients are obtained by calculating the pixel coordinates of the camera corners using the radial distortion formula.

[0087] In this embodiment, the pixel coordinates and physical coordinates of all corner points are obtained through a checkerboard calibration board, resulting in an intrinsic parameter matrix and distortion coefficients, which are then used to perform distortion correction on the image.

[0088] In one possible embodiment, the camera's imaging system comprises four main coordinate systems: the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. These systems are sequentially linked through rigid body transformation, perspective projection, and affine transformation, as shown below:

[0089] World coordinate system → Camera coordinate system (rigid body transformation): This means transforming a point in the 3D real world into the camera's own coordinate system using camera extrinsic parameters (rotation matrix and translation vector).

[0090] Camera coordinate system → Image coordinate system (perspective transformation): Using camera intrinsic parameters (focal length, principal point position, pixel scale factor), three-dimensional points are projected onto the imaging plane to obtain two-dimensional image coordinates;

[0091] Image coordinate system → pixel coordinate system (affine transformation): Based on the pixel size, pixel spacing and origin offset, the image plane coordinates are converted into pixel coordinates, thus obtaining the specific pixel position of the image on the sensor.

[0092] The transformation relationships between the four coordinate systems can be defined as follows:

[0093] ;

[0094] Where (I,J,K) are the physical coordinates of a point in the world coordinate system; (i,j) are the pixel coordinates of the corresponding pixel in the pixel coordinate system; and Z is the scale factor. dX and dY are the physical lengths of a unit pixel on the camera's image sensor in the X and Y directions, respectively; i0 and j0 represent the coordinates of the center of the camera's image sensor in the pixel coordinate system. R is the angle between the horizontal and vertical edges of the photosensitive plate; R is the rotation matrix, used to describe the attitude change between the world coordinate system and the camera coordinate system, that is, R is responsible for "rotating" the world coordinate axis to align with the camera coordinate axis; T is the translation vector, used to describe the translation relationship (i.e., position difference) between the origin of the world coordinate system and the origin of the camera coordinate system, that is, T is responsible for "translating" the origin of the world coordinate system to the origin of the camera coordinate system.

[0095] In this embodiment, the camera's intrinsic parameter matrix is ​​as follows:

[0096] ;

[0097] In addition, photos taken with a fisheye camera exhibit barrel distortion, the radial distortion formula of which is as follows:

[0098] ;

[0099] ;

[0100] in 、( , ) represent the coordinates of the ideal, distortion-free, normalized image and the coordinates of the distorted, normalized image, respectively; r is the distance from the image pixel to the image center, i.e. The second purpose of camera calibration is to obtain the camera's distortion coefficients, as shown in the formula above. , , Then, distortion correction is performed on the image.

[0101] In one possible embodiment, such as Figure 5 The flowchart shown illustrates the checkerboard calibration process. First, an 11×15 black and white checkerboard is used as the calibration board, with each square checkerboard unit having a side length of 42mm. Multiple calibration images are captured from different angles. Feature points in each image are detected through image processing. Then, under the assumption of no distortion, the camera's intrinsic and extrinsic parameters are solved and optimized using the maximum likelihood method to improve accuracy. Subsequently, the actual distortion coefficients are calculated using the least squares method. The intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients are combined and optimized again using the maximum likelihood method. Finally, high-precision camera intrinsic, extrinsic, and distortion parameters are obtained, providing reliable calibration data for subsequent image correction and top-view transformation.

[0102] By adopting the aforementioned checkerboard calibration process, not only can the camera imaging characteristics be comprehensively acquired under multi-angle shooting conditions, but feature points can also be automatically extracted and the intrinsic parameters, extrinsic parameters, and distortion coefficients can be optimized multiple times using maximum likelihood estimation and least squares method, significantly improving calibration accuracy and stability. At the same time, it reduces manual measurement and debugging steps, simplifies calibration operations, reduces dependence on high-precision equipment, and provides high-precision basic data for subsequent image distortion correction, top-view transformation, and multi-camera surround view stitching, thereby significantly improving the overall stitching effect and real-time performance of the system, and enhancing the adaptability and safety of the vehicle surround view system in different environments.

[0103] In some alternative implementations, step S203 includes the following steps:

[0104] Vehicle feature points are determined based on the first corrected image;

[0105] The vehicle feature points are subjected to a direct linear transformation to obtain the second corrected image.

[0106] In some alternative implementations, the step "perform a direct linear transformation on the vehicle feature points to obtain a second corrected image" includes the following steps:

[0107] Based on the original points and the target projection points, homography matrix transformation is performed to obtain the target transformed image;

[0108] The target transformed image is processed by position planning based on the direct linear transformation algorithm to obtain the second corrected image.

[0109] In this embodiment, vehicle feature points can be key location points extracted from the first corrected image for top-down transformation calculation. They can include fixed and easily identifiable reference points around the vehicle, such as corner points of a checkerboard calibration board, four corners of the vehicle body, wheel center points, vehicle body edges, or other feature markers with known coordinates. They can also include environmental feature points such as lane lines and parking space corner points. These are used to construct the correspondence between the original points and the target projection points, which facilitates the calculation of the homography matrix and enables accurate transformation from an oblique viewpoint to a bird's-eye viewpoint.

[0110] In this embodiment, the target transformation image can be an intermediate image calculated based on vehicle feature points through homography matrix transformation. That is, the bird's-eye view projection image obtained by perspective mapping the original feature points in the first corrected image according to the correspondence of the target projection points is used as a transition or correction result in the top-view transformation process, which facilitates subsequent position planning and further generation of the second corrected image, so as to achieve accurate conversion from the tilted view to the standardized top-view.

[0111] In one possible embodiment, based on the direct linear transformation method, by selecting the original points and calculating the target points for projection, the homography transformation matrix is ​​calculated to achieve the top-view transformation of the image. The calculation formula for the top-view transformation is as follows:

[0112] ;

[0113] Wherein: , These are the coordinates of the original point and the transformed target point, respectively.

[0114] In one possible embodiment, feature points around the vehicle (including the corner points of the checkerboard calibration board) are selected as the origin and target points. Image transformation from an oblique view to a bird's-eye view is achieved by calculating the homography matrix. Furthermore, a direct linear transformation method combined with position planning is used to rationally determine the display range and viewing angle of the surround view. And based on the actual vehicle dimensions and parameters such as the edge distance of the selected checkerboard points, the inner and outer adjustment parameters are dynamically adjusted to optimize the surround view effect (e.g., ...). Figure 7 (As shown) the field of view and blind spot coverage, improving the accuracy and visual continuity of the bird's-eye view.

[0115] In some alternative implementations, step S204 includes the following steps:

[0116] The second corrected image is divided into regions to determine the target overlapping image and the non-target overlapping image.

[0117] A fade-in / fade-out fusion process is performed on the overlapping target images to obtain a third corrected image;

[0118] The third corrected image is stitched together with the non-target overlapping image to obtain the target corrected image.

[0119] In some alternative implementations, the step "perform a fade-in / fade-out fusion process on the target overlapping images to obtain a third corrected image" includes the following steps:

[0120] Obtain the two-dimensional coordinate information of each target pixel in the overlapping image;

[0121] Dilation processing is performed on non-target overlapping images to obtain the contour information of the non-target overlapping images;

[0122] The weight matrix and inverse weight matrix of the target pixel are calculated based on the two-dimensional coordinate information and contour information.

[0123] Based on the weight matrix and the inverse weight matrix, the RGB channels corresponding to the target pixel are weighted to obtain the third corrected image.

[0124] In this embodiment, the target overlapping image and the non-target overlapping image can be different region images obtained by dividing the second corrected image into regions. The target overlapping image is the intersection region where multiple top-view transformed images overlap during the stitching process, and is used to calculate the weight matrix and the inverse weight matrix. The non-target overlapping image is an independent region in each top-view image that does not overlap with other images, and is used for direct stitching.

[0125] The third corrected image mentioned above can be a seamless fused image obtained by weighting the target overlapping image using an improved fade-in and fade-out fusion algorithm. This image is then stitched together with the non-target overlapping image to form the final target corrected image, thereby significantly reducing stitching marks and improving the continuity and naturalness of the panoramic view image.

[0126] In this embodiment, the two-dimensional coordinate information and contour information can be the horizontal and vertical coordinate data of the pixel points on the image plane obtained by binarizing, extracting contours and dilating the target overlapping images, as well as the corresponding boundary contours of the left and right image regions, used to determine the spatial position and distance relationship of each pixel point in the overlapping region; the weight matrix and inverse weight matrix of the target pixel points can be weighted coefficient matrices calculated according to the above two-dimensional coordinate information and contour information based on the straight-line distance from the pixel point to each image contour, and the sum of the two is 1, which correspond to the weight distribution of the left and right images in the overlapping region, respectively, and are used to realize pixel-by-pixel weighted processing in each RGB channel, thereby obtaining a third corrected image with natural transition and seamless stitching.

[0127] In one possible embodiment, the traditional fade-in / fade-out blending algorithm mainly achieves the purpose of removing seams by processing the gray values ​​of pixels in the overlapping areas of the image, and its calculation formula is as follows.

[0128] ;

[0129] Wherein: The grayscale values ​​of the pixels in the merged image; The grayscale value of the pixel in the left image to be merged; The grayscale value of the pixel in the right image to be merged; For the corresponding weight value and , , .

[0130] Specifically, , The calculation formula is shown below.

[0131] ;

[0132] ;

[0133] in, is the x-coordinate of any pixel in the overlapping region; , These are the x-coordinates of the corresponding left and right boundary points.

[0134] An improved fade-in / fade-out fusion algorithm is used to calculate the distance from each pixel in the image to the left and right boundary contours, and obtain the calculated weight matrix to achieve natural image fusion.

[0135] The improved fade-in / fade-out algorithm fully utilizes the two-dimensional coordinate information of pixels in the overlapping region to calculate the straight-line distance from each pixel to the contours of the left and right images, and then calculates the weight value of that point in each image. The weight matrix obtained after calculation is multiplied by the channel-separated left image to be fused and then combined. The corresponding right image to be fused is multiplied by the corresponding inverse weight matrix. The left and right images are then superimposed to obtain the final fused image. and The calculation formula is shown below.

[0136] ;

[0137] ;

[0138] in, Let be the straight-line distance from a point in the overlapping region to the contour of the left image to be fused. This is the straight-line distance from the pixel to the outline of the right image to be merged.

[0139] In one possible embodiment, the specific implementation steps of the algorithm are as follows:

[0140] 1) Obtain the overlapping region image of the images to be stitched.

[0141] 2) Binarize the overlapping area image. Set the grayscale value of each pixel to 0 or 255 to give the entire image a clear black and white effect, thus obtaining the mask array image. At the same time, obtain the inverse mask array image.

[0142] 3) Finding the contour. Obtain the two images after removing the overlapping parts, then dilate the resulting images and iterate through them to find the contour with the largest area. At this point, the contours of the left and right images (after removing the overlapping parts) and the overlapping areas of the images are clearly identified.

[0143] 4) Calculate the weight matrix and inverse weight matrix. To check if a pixel is within the contour, iterate through all pixels in the left image. If a pixel is within the left image region, set its weight to 1; if it's within the right image region, set its weight to 0. If a pixel is within the common region of both images, calculate the divisor of the sum of the straight-line distance from the pixel to the left image region and the sum of the straight-line distances from the pixel to both the left and right image regions. The sum of this divisor and the pixel's weight value is 1. Within the common region of both images, the sum of the weight value and the inverse weight value of pixels at the same location is 1.

[0144] Since the default color space of an image is RGB, the two images need to be separated into channels first during image fusion. Each channel of the corresponding image is multiplied by a weight matrix and an inverse weight matrix, and then the channels are combined. The images processed by the weight matrix are then superimposed to obtain the final fused image.

[0145] In one possible embodiment, to Figure 6 Taking the surround view stitching area division diagram shown as an example, the surround view is divided into eight regions: L, F, R, B, LF, RF, RB, and LB. Image registration is performed on the overlapping regions. An improved fade-in / fade-out fusion algorithm is used to calculate the two-dimensional coordinate information and weight values ​​of the pixels in the overlapping regions. The images of the non-overlapping regions are copied to the surround view planning diagram. The images of the overlapping regions are fused to eliminate seams and obtain the final vehicle surround view.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0147] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0148] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a vehicle surround view image stitching device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0149] like Figure 3 As shown, the vehicle surround view image stitching device 300 described in this embodiment includes: an acquisition module 301, a first processing module 302, a second processing module 303, and a third processing module 304. Wherein:

[0150] The acquisition module 301 is used to acquire the image to be corrected;

[0151] The first processing module 302 is used to perform checkerboard calibration processing on the image to be corrected to obtain a first corrected image;

[0152] The second processing module 303 is used to perform a top-view transformation on the first corrected image to obtain a second corrected image;

[0153] The third processing module 304 is used to perform correction and stitching processing on the second corrected image to obtain the target corrected image.

[0154] The first processing module 302 includes:

[0155] The acquisition submodule is used to obtain the camera coordinate system of the chessboard calibration plate;

[0156] The distortion processing submodule is used to perform distortion processing through the camera coordinate system to obtain camera correction parameters;

[0157] The correction processing submodule is used to process the image to be corrected based on the camera correction parameters to obtain the first corrected image.

[0158] The distortion processing includes:

[0159] The first processing unit is used to perform matrix transformation processing on the physical coordinates to obtain the camera intrinsic parameter matrix;

[0160] The first calculation unit is used to calculate the camera corner pixel coordinates using the radial distortion formula to obtain the camera distortion coefficients.

[0161] The second processing module 303 includes:

[0162] A determination submodule is used to determine vehicle feature points based on the first corrected image;

[0163] The direct linear transformation processing submodule is used to perform direct linear transformation processing on the vehicle feature points to obtain the second corrected image.

[0164] The direct linear transformation processing submodule includes:

[0165] The second processing unit is used to perform homography matrix transformation processing based on the original point and the target projection point to obtain the target transformed image;

[0166] The third processing unit is used to perform position planning processing on the target transformed image based on the direct linear transformation algorithm to obtain the second corrected image.

[0167] The third processing module 304 includes:

[0168] The segmentation submodule is used to perform region segmentation processing on the second corrected image to determine the target overlapping image and the non-target overlapping image;

[0169] The fusion submodule is used to perform fade-in and fade-out fusion processing on the target overlapping images to obtain a third corrected image;

[0170] The stitching submodule is used to stitch the third corrected image with the non-target overlapping image to obtain the target corrected image.

[0171] The fusion submodule includes:

[0172] The acquisition unit is used to acquire the two-dimensional coordinate information of each target pixel in the target overlapping image;

[0173] A dilation unit is used to dilate the non-target overlapping image to obtain the contour information of the non-target overlapping image.

[0174] The second calculation unit is used to calculate, based on the two-dimensional coordinate information and the contour information, the weight matrix and the inverse weight matrix of the target pixel;

[0175] The weighted processing unit is used to perform weighted processing on the RGB channels corresponding to the target pixel based on the weight matrix and the inverse weight matrix to obtain the third corrected image.

[0176] In this embodiment, the method involves acquiring an image to be corrected; performing checkerboard calibration on the image to be corrected to obtain a first corrected image; performing a top-view transformation on the first corrected image to obtain a second corrected image; and performing correction and stitching on the second corrected image to obtain the target corrected image. By sequentially performing checkerboard calibration, top-view transformation, and correction and stitching on the image to be corrected, this method can significantly improve the geometric accuracy and visual coherence of the panoramic image, reduce stitching marks and blind spots, provide the driver with a higher quality and more stable panoramic view, and improve driving safety.

[0177] In this embodiment, the operations performed by the above-mentioned units or modules correspond one-to-one with the steps of the vehicle surround view image stitching method of the above-described embodiments, and will not be repeated here.

[0178] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0179] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0180] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0181] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for a vehicle surround view image stitching method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0182] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the vehicle surround view image stitching method.

[0183] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0184] This embodiment provides a computer device that acquires an image to be corrected; performs checkerboard calibration on the image to be corrected to obtain a first corrected image; performs top-view transformation on the first corrected image to obtain a second corrected image; and performs correction and stitching on the second corrected image to obtain a target corrected image. By sequentially performing checkerboard calibration, top-view transformation, and correction and stitching on the image to be corrected, this method can significantly improve the geometric accuracy and visual coherence of the panoramic image, reduce stitching marks and blind spots, provide the driver with a higher quality and more stable panoramic view, and improve driving safety.

[0185] This application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the vehicle surround view image stitching method described above.

[0186] This embodiment provides a computer-readable storage medium that acquires an image to be corrected; performs checkerboard calibration on the image to be corrected to obtain a first corrected image; performs top-view transformation on the first corrected image to obtain a second corrected image; and performs correction and stitching on the second corrected image to obtain a target corrected image. By sequentially performing checkerboard calibration, top-view transformation, and correction and stitching on the image to be corrected, this method can significantly improve the geometric accuracy and visual coherence of the panoramic image, reduce stitching marks and blind spots, provide the driver with a higher quality and more stable panoramic view, and improve driving safety.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for stitching together vehicle surround view images, characterized in that, Includes the following steps: Obtain the image to be corrected; The image to be corrected is subjected to checkerboard calibration to obtain the first corrected image; The first corrected image is subjected to a top-view transformation to obtain a second corrected image; The second corrected image is then corrected and stitched together to obtain the target corrected image. The step of performing correction and stitching processing on the second corrected image to obtain the target corrected image further includes: The second corrected image is divided into regions to determine the target overlapping image and the non-target overlapping image; The overlapping target images are subjected to fade-in / fade-out fusion processing to obtain a third corrected image; The third corrected image is stitched together with the non-target overlapping image to obtain the target corrected image; The step of performing the fade-in / fade-out fusion process on the target overlapping image to obtain the third corrected image includes: Obtain the two-dimensional coordinate information of each target pixel in the overlapping image; The non-target overlapping image is dilated to obtain its contour information. Based on the two-dimensional coordinate information and the contour information, the weight matrix and inverse weight matrix of the target pixel are calculated to obtain the target pixel. Based on the weight matrix and the inverse weight matrix, the RGB channels corresponding to the target pixel are weighted to obtain the third corrected image; The step of weighting the RGB channels corresponding to the target pixel based on the weight matrix and the inverse weight matrix to obtain the third corrected image includes: Multiply the left image to be fused with the weight matrix, and the left image to be fused is the contour image of the left image region of the target overlapping image after removing the overlap; Multiply the right image to be fused with the inverse weight matrix, and the right image to be fused is the contour image of the right image region of the target overlapping image after removing the overlap; The third corrected image is obtained by overlaying the left image, which incorporates the weight matrix, with the right image, which incorporates the inverse weight matrix.

2. The vehicle surround view image stitching method according to claim 1, characterized in that, The step of performing checkerboard calibration on the image to be corrected to obtain the first corrected image specifically includes: Obtain the camera coordinate system of the chessboard calibration version; The camera correction parameters are obtained by performing distortion processing on the camera coordinate system. The image to be corrected is processed based on the camera correction parameters to obtain the first corrected image.

3. The vehicle surround view image stitching method according to claim 2, characterized in that, The camera coordinate system includes camera corner pixel coordinates and physical coordinates. The camera correction parameters include a camera intrinsic parameter matrix and camera distortion coefficients. The step of performing checkerboard distortion processing based on the camera coordinate system to obtain the camera correction parameters specifically includes: The physical coordinates are subjected to matrix transformation to obtain the camera intrinsic parameter matrix; The camera distortion coefficients are obtained by calculating the pixel coordinates of the camera corner points using the radial distortion formula.

4. The vehicle surround view image stitching method according to claim 1, characterized in that, The step of performing a top-view transformation on the first corrected image to obtain the second corrected image specifically includes: Vehicle feature points are determined based on the first corrected image; The vehicle feature points are subjected to a direct linear transformation to obtain the second corrected image.

5. The vehicle surround view image stitching method according to claim 4, characterized in that, The vehicle feature points include original points and target projection points. The step of performing a direct linear transformation on the vehicle feature points to obtain the second corrected image further includes: Based on the original points and the target projection points, homography matrix transformation is performed to obtain the target transformed image; The target transformed image is processed by position planning based on the direct linear transformation algorithm to obtain the second corrected image.

6. A vehicle surround view image stitching device, characterized in that, When the vehicle surround view image stitching device is executed, it implements the steps of the vehicle surround view image stitching method as described in any one of claims 1 to 5, including: The acquisition module is used to acquire the image to be corrected. The first processing module is used to perform checkerboard calibration processing on the image to be corrected to obtain the first corrected image; The second processing module is used to perform top-view transformation processing on the first corrected image to obtain the second corrected image; The third processing module is used to perform correction and stitching processing on the second corrected image to obtain the target corrected image.

7. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the vehicle surround view image stitching method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the vehicle surround view image stitching method as described in any one of claims 1 to 5.

Images