360-degree looking-around automatic calibration method, device and system for engineering vehicle

Abstract

The invention discloses a 360-degree looking-around automatic calibration method, device and system for an engineering vehicle, and the method comprises the steps: carrying out the iterative calculation of homography matrixes corresponding to all cameras through employing a feature matching algorithm and an RANSAC algorithm for vehicle surrounding images at different angles obtained by multiple cameras; calculating a rotation matrix based on the IMU data; calculating a translation vector based on the GPS data; based on the rotation matrix and the translation vector, correcting each calculated homography matrix in combination with an adaptive weight fusion algorithm to obtain a corrected homography matrix; performing projection transformation on the corresponding vehicle surrounding image by using each corrected homography matrix to obtain a corresponding projection image; and processing each projection image by adopting an image splicing algorithm to obtain a 360-degree look-around image of the engineering vehicle. According to the invention, automatic calibration of a multi-camera system can be realized, the precision and consistency of image splicing are improved, and the method is especially suitable for engineering vehicles and other scenes requiring high precision and real-time performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of engineering machinery, and in particular relates to a 360-degree surround automatic calibration method, device and system for an engineering vehicle. Background Art

[0002] As the automation and intelligence level of engineering vehicles continues to improve, drivers' need to perceive the surrounding environment is increasing, and 360-degree surround view systems have become a key assisted driving technology. However, existing 360-degree surround view systems generally rely on manual calibration, which has problems such as complex operation, low efficiency, and insufficient accuracy. Especially under multi-camera installation conditions, the relative position, angle differences, and lens distortion between cameras often lead to deviations in image stitching. Manual calibration is not only time-consuming and labor-intensive, but also difficult to meet the real-time adjustment needs of engineering vehicles in complex environments.

[0003] In the Chinese invention patent application with publication number CN109089086A, a panoramic camera system is disclosed, including multiple camera devices, each of which is connected to a surround view processing host for processing images through a connecting line, and the surround view processing host is connected to a vehicle-mounted video terminal through a connecting line. Each camera device is respectively set at a fixed position on the fuselage of an engineering machinery vehicle through a corresponding pan-tilt head. The camera device shoots the surroundings of the engineering machinery vehicle to obtain a video image, and sends the video image to the surround view processing host; the surround view processing host and the vehicle-mounted video terminal are both set in the vehicle compartment, and the surround view processing host receives the video image and processes it to obtain a panoramic video image, and sends the panoramic video image to the vehicle-mounted video terminal, which displays a 360-degree panoramic video image. This solution cannot be properly applied to loaders, because most loaders have an articulated chassis, and the front and rear bodies are not rigid bodies, so there will be a problem that the front loading attachment jumps out of the surround view area.

[0004] The Chinese utility model patent with publication number CN220165657U discloses a lifting sling with a panoramic view function, including a lifting sling, an ultra-wide-angle camera, a PLC controller, and a remote control display terminal. The cross section of the lifting sling is fish-belly-shaped, and a spacer ring and a pin are provided below the end of the frame to complete the hanging component; the ultra-wide-angle camera is installed on the lifting sling, and is used to perform 360-degree panoramic monitoring of the environment of the suspended component, including the front, rear, left, and right directions. The ultra-wide-angle camera is connected to the PLC controller, and the PLC controller is connected to the remote control display terminal. This solution uses the traditional manual calibration form to complete the calibration of the 360-degree panoramic view system. Summary of the invention

[0005] In response to the above problems, the present invention proposes a 360-degree surround view automatic calibration method, device and system for engineering vehicles, which can realize automatic calibration of multi-camera systems and improve the accuracy and consistency of image stitching. It is particularly suitable for scenes such as engineering vehicles that require high precision and real-time performance.

[0006] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, the present invention provides a 360-degree surround view automatic calibration method for an engineering vehicle, characterized by comprising:

[0008] For the vehicle surrounding images acquired by multiple cameras at different angles, the feature matching algorithm and RANSAC algorithm are used to iteratively calculate the homography matrix corresponding to each camera;

[0009] Calculate the rotation matrix based on the IMU data;

[0010] Calculate the translation vector based on GPS data;

[0011] Based on the rotation matrix and the translation vector, the calculated homography matrices are corrected in combination with an adaptive weight fusion algorithm to obtain a corrected homography matrix;

[0012] Using each corrected homography matrix to perform a projection transformation on the corresponding vehicle surrounding image to obtain a corresponding projection image;

[0013] The image stitching algorithm is used to process each projection image to obtain a 360-degree surround view image of the engineering vehicle.

[0014] In combination with the first aspect, optionally, the method for calculating the homography matrix includes:

[0015] Perform feature operator extraction on two vehicle surrounding images at the same angle, and identify matching points in two disparity images based on the feature operator extraction results;

[0016] Based on the two-dimensional coordinates of the selected matching points and the intrinsic parameter matrix of the camera, the corresponding initial homography matrix is ​​calculated;

[0017] Using the initial homography matrix, all matching points are projected and the reprojection error of each matching point is calculated;

[0018] If the reprojection error of a matching point meets the preset error threshold, the matching point is defined as an inlier point.

[0019] With the goal of maximizing the number of internal points, the RANSAC algorithm is used to iteratively calculate the final homography matrix.

[0020] In combination with the first aspect, optionally, the calculation formula of the reprojection error is:

[0021]

[0022] Where i is the index of the matching point, indicating the i-th matching point, (x″, y″) is the coordinate of the point obtained by projection of the homography matrix, and (x′, y′) is the coordinate of the matching point in the original image;

[0023] The objective function of the RANSAC algorithm is:

[0024]

[0025] Where n is the total number of matching points and H is the homography matrix.

[0026] In combination with the first aspect, optionally, the expression of the homography matrix before correction is:

[0027]

[0028] The expression of the rotation matrix is:

[0029]

[0030] The expression of the translation vector is:

[0031]

[0032] The expression of the corrected homography matrix is:

[0033]

[0034] Among them, H is the homography matrix before correction, H′ is the homography matrix after correction, α and β are dynamic weight coefficients used to balance the influence of sensor data on the homography matrix, h mn is an element in the homography matrix H, m = 1 to 3, n = 1 to 3, representing the transformation relationship between image planes; r mn is the element in the rotation matrix R, which represents the rotation of the camera on each axis; t x , t y , t z are the components of the translation vector t, representing the translation of the camera in each direction.

[0035] In combination with the first aspect, optionally, the dynamic weight coefficients α and β are adjusted in the following manner:

[0036] When the noise of IMU data is lower than the set noise threshold, the value of α is increased;

[0037] If the satellite signal strength is greater than the set threshold and the number of satellites is greater than the set number threshold, the value of β is increased.

[0038] In combination with the first aspect, optionally, the homography matrix before correction is calculated by the following formula:

[0039]

[0040] in, and are the corresponding coordinates of the corresponding matching points in the two disparity images, j = 1, 2, ... 8, are the elements in the homography matrix H, m = 1 ~ 3, n = 1 ~ 3.

[0041] In combination with the first aspect, optionally, the matching point recognition method includes:

[0042] Preprocess the images around the vehicle, including noise reduction and contrast enhancement;

[0043] Use the FAST algorithm to detect feature points, calculate Harris response values, and select stable feature points through non-maximum suppression;

[0044] Calculate the main direction of the feature point and extract the rotation BRIEF descriptor;

[0045] The Hamming distance is used to measure the similarity of BRIEF descriptors and obtain matching points.

[0046] In combination with the first aspect, optionally, the method for generating a 360-degree surround view image of the engineering vehicle includes:

[0047] According to the projection image obtained after the projective transformation using the corrected homography matrix, the brightness and color matrix corresponding to each overlapping part is calculated;

[0048] The spliced ​​images are corrected and rendered using the brightness and color matrix to obtain a final panoramic image.

[0049] In a second aspect, the present invention provides a 360-degree surround automatic calibration device for an engineering vehicle, comprising:

[0050] The homography matrix calculation module is used to iteratively calculate the homography matrix corresponding to each camera using the feature matching algorithm and the RANSAC algorithm for the vehicle surrounding images at different angles acquired by multiple cameras;

[0051] Rotation matrix calculation module, used to calculate the rotation matrix based on IMU data;

[0052] A translation vector calculation module, used to calculate the translation vector based on GPS data;

[0053] A homography matrix correction module, used to correct each calculated homography matrix based on the rotation matrix and the translation vector in combination with an adaptive weight fusion algorithm to obtain a corrected homography matrix;

[0054] A projection transformation module, used to perform projection transformation on the corresponding vehicle surrounding images using each corrected homography matrix to obtain a corresponding projection image;

[0055] The image stitching module is used to process each projection image using an image stitching algorithm to obtain a 360-degree surround image of the engineering vehicle.

[0056] In a third aspect, the present invention provides a 360-degree surround view automatic calibration system for an engineering vehicle, including a storage medium and a processor;

[0057] The storage medium is used to store instructions;

[0058] The processor is configured to operate according to the instructions to execute the method according to any one of the first aspects.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] The present invention captures images around the vehicle by installing wide-angle cameras around the vehicle, and stitches these images around the vehicle into a seamless panoramic overhead view through image processing technology, thereby providing the driver with an all-round view. Specifically, the present invention solves the problems of low calibration accuracy, poor efficiency and insufficient robustness in the prior art by combining feature matching algorithm, RANSAC algorithm, multi-sensor data fusion and adaptive weight fusion algorithm. The present invention can realize automatic calibration of multi-camera systems, improve the accuracy and consistency of image stitching, and is particularly suitable for scenes that require high precision and real-time performance, such as engineering vehicles. This technology greatly enhances the driver's perception of the vehicle's surroundings, allowing the driver to respond more calmly in situations such as parking, meeting in narrow roads, avoiding obstacles, etc., effectively reducing the occurrence of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor, among which:

[0062] Figure 1 A schematic diagram of a flow chart of a method for automatic calibration of a 360-degree surround view of an engineering vehicle according to an embodiment of the present invention;

[0063] Figure 2The figure is a flowchart of a feature matching algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0066] Example 1

[0067] An embodiment of the present invention provides a 360-degree surround view automatic calibration method for an engineering vehicle, comprising the following steps:

[0068] (1) For the vehicle surrounding images acquired by multiple cameras at different angles, the feature matching algorithm and the RANSAC algorithm are used to iteratively calculate the homography matrix corresponding to each camera;

[0069] (2) Calculate the rotation matrix based on IMU data;

[0070] (3) Calculate the translation vector based on GPS data;

[0071] (4) Based on the rotation matrix and the translation vector, the calculated homography matrices are corrected in combination with an adaptive weight fusion algorithm to obtain a corrected homography matrix;

[0072] (5) using each corrected homography matrix to perform a projection transformation on the corresponding vehicle surrounding image to obtain a corresponding projection image;

[0073] (6) Use an image stitching algorithm to process each projection image to obtain a 360-degree surround view image of the engineering vehicle.

[0074] In a specific implementation of the embodiment of the present invention, the method for calculating the homography matrix includes:

[0075] Perform feature operator extraction on two vehicle surrounding images at the same angle, and identify matching points in two disparity images based on the feature operator extraction results;

[0076] Based on the two-dimensional coordinates of the selected matching points and the intrinsic parameter matrix of the camera, the corresponding initial homography matrix is ​​calculated;

[0077] Using the initial homography matrix, all matching points are projected and the reprojection error of each matching point is calculated;

[0078] If the reprojection error of a matching point meets the preset error threshold, the matching point is defined as an inlier point.

[0079] With the goal of maximizing the number of internal points, the RANSAC algorithm is used to iteratively calculate the final homography matrix.

[0080] In a specific implementation of the embodiment of the present invention, the calculation formula of the reprojection error is:

[0081]

[0082] Where i is the index of the matching point, indicating the i-th matching point, (x″, y″) is the coordinate of the point obtained by projection of the homography matrix, and (x′, y′) is the coordinate of the matching point in the original image;

[0083] The objective function of the RANSAC algorithm is:

[0084]

[0085] Where n is the total number of matching points.

[0086] In a specific implementation of the embodiment of the present invention, the expression of the homography matrix before correction is:

[0087]

[0088] The expression of the rotation matrix is:

[0089]

[0090] The expression of the translation vector is:

[0091]

[0092] The expression of the corrected homography matrix is:

[0093]

[0094] Among them, H is the homography matrix before correction, H′ is the homography matrix after correction, α and β are dynamic weight coefficients used to balance the influence of sensor data on the homography matrix, h mn is an element in the homography matrix H, m = 1 to 3, n = 1 to 3, representing the transformation relationship between image planes; r mn is the element in the rotation matrix R, which represents the rotation of the camera on each axis; t x , t y , t z are the components of the translation vector t, representing the translation of the camera in each direction.

[0095] In a specific implementation of the embodiment of the present invention, the dynamic weight coefficients α and β are adjusted in the following manner:

[0096] When the noise of IMU data is lower than the set noise threshold, the value of α is increased;

[0097] If the satellite signal strength is greater than the set threshold and the number of satellites is greater than the set number threshold, the value of β is increased.

[0098] In a specific implementation of the embodiment of the present invention, the homography matrix before correction is calculated by the following formula:

[0099]

[0100] in, and are the corresponding coordinates of the corresponding matching points in the two disparity images, j = 1, 2, ... 8, h mn It is an element in the homography matrix H, m=1~3, n=1~3.

[0101] In a specific implementation of the embodiment of the present invention, the matching point recognition method includes:

[0102] Preprocess the images around the vehicle, including noise reduction and contrast enhancement;

[0103] Use the FAST algorithm to detect feature points, calculate Harris response values, and select stable feature points through non-maximum suppression;

[0104] Calculate the main direction of the feature point and extract the rotation BRIEF descriptor;

[0105] The Hamming distance is used to measure the similarity of BRIEF descriptors and obtain matching points.

[0106] In a specific implementation of the embodiment of the present invention, the method for generating a 360-degree surround view image of the engineering vehicle includes:

[0107] Calculate the brightness and color matrix corresponding to each overlapping part according to the projected image obtained after applying the homography matrix to perform the projection transformation;

[0108] The stitched images are corrected and rendered based on the brightness and color matrices to obtain the final panoramic image. In the correction stage, the grid deformation technology is used for geometric correction to eliminate local distortions, the pixels are resampled using bilinear interpolation, and the photometric correction is performed through histogram matching and color migration algorithms. In the rendering stage, the system sequentially performs adaptive contrast enhancement, edge-preserving smoothing filtering, and local tone mapping to improve image quality and maintain details. Correction can repair the distortion introduced by the stitching process, and rendering can enhance the visual effect of the image.

[0109] The 360-degree surround view automatic calibration method for an engineering vehicle in an embodiment of the present invention is described in detail below in conjunction with a specific implementation.

[0110] like Figure 1 As shown, the 360-degree surround automatic calibration method of the engineering vehicle includes the following steps:

[0111] Step (1) Image acquisition

[0112] Calibration of image acquisition is a key step in machine vision systems, which involves determining the internal parameters (intrinsic parameters) and external parameters (external parameters) of the camera in order to accurately map the image coordinates to the real-world coordinate system. This process is crucial to improving the accuracy and stability of image processing.

[0113] In the embodiment of the present invention, multiple cameras are installed according to the size of the vehicle to collect images of the surrounding environment. The image can be a specific calibration cloth or a non-specific environment image.

[0114] Step (2) Image preprocessing

[0115] Image preprocessing is a key step in image processing and computer vision tasks. It involves processing the original image data (after dedistortion) to improve image quality, enhance image features, or reduce the complexity of subsequent processing steps. The purpose of image preprocessing is to provide a higher quality image input for subsequent image analysis tasks (such as feature extraction, object recognition, and image segmentation). In the embodiment of the present invention, a grayscale solution is adopted, and the subsequent process performs binary processing on the image.

[0116] Grayscale: If it is a color image, it needs to be converted to a grayscale image. Each pixel value in a grayscale image represents the brightness information of the pixel, usually ranging from 0 (black) to 255 (white). Grayscale processing can effectively reduce the data dimension while retaining the main structural features of the image, laying the foundation for subsequent processing.

[0117] Threshold processing: The core of binarization is to select a threshold value to distinguish the foreground and background in the image. Pixel values ​​above the selected threshold will be classified into one category (usually white, representing the foreground), while those below the threshold will be classified into another category (usually black, representing the background). The binary data matrix (containing only 0 and 1) obtained in this step has two key functions: highlighting the difference between the target area and the background, eliminating non-critical grayscale changes and noise interference; providing clear edge and contour information for the matching of feature operators, which is helpful for subsequent edge detection and morphological processing. Appropriate threshold selection (such as the OTSU algorithm) can ensure that the binarization result retains the integrity of the target while suppressing background interference.

[0118] Step (3) using a feature matching algorithm to obtain matching points;

[0119] Feature operator processing is a key step in image processing and computer vision, which is used to extract key information from images, which is helpful for subsequent image analysis tasks such as image recognition, classification, tracking and 3D reconstruction. Feature operators are usually used to detect specific types of features in images, such as edges, corners, spots or textures. This system performs correspondence matching on feature points in images collected by multiple cameras.

[0120] ORB (Oriented FAST and Rotated BRIEF) is a popular feature point extraction and description algorithm that combines the advantages of FAST key point detection and BRIEF descriptor, and introduces directionality, making the feature point descriptor rotationally invariant. The ORB algorithm has the characteristics of fast calculation speed, small memory usage, and open source and free, which is suitable for real-time application scenarios.

[0121] The process of feature matching algorithm mainly includes the following steps: first, preprocess the input vehicle image, including noise reduction and contrast enhancement; then use FAST algorithm to detect feature points and calculate Harris response value, and select stable feature points through non-maximum suppression; then calculate the main direction of feature points and extract rotation BRIEF descriptor; finally, when matching feature points, use Hamming distance to measure the similarity of descriptors and remove mismatched points. The matching results include the pixel coordinates of the corresponding feature points and the descriptor distance. These matching data can be used to calculate camera calibration parameters, estimate camera motion, 3D reconstruction, image stitching and other application scenarios. For details, see Figure 2 .

[0122] Step (4) projection matrix calculation;

[0123] The projection matrix, or homography matrix, is a 3x3 non-singular matrix used to describe the mapping relationship between two planes. It is usually used for the correspondence between points in images, especially in the case of perspective transformation. In computer vision, homography matrices are often used for tasks such as image registration, 3D reconstruction, and motion estimation. For each pair of matching points, a system of linear equations can be constructed. The RANSAC algorithm is used to randomly select a set of sample points from the matched feature point pairs. If the coordinates of a point in one of the disparity images are (x', y') and the coordinates in the other disparity image are (x, y), then the homography matrix H satisfies the following relationship:

[0124]

[0125] Taking four groups of corresponding points that are not on the same straight line can generate 8 groups of equations, forming the following set of equations. Since the H matrix has 8 unknowns, H can be calculated through this set of equations.

[0126]

[0127] This system of equations can be solved using linear least squares or other numerical methods to obtain the homography matrix H.

[0128] Using the calculated homography matrix H, all matching points are projected and the reprojection error of each point is calculated. The reprojection error is defined as the distance between the actual point and the mapped point, and the formula is as follows:

[0129]

[0130] Among them, i is the index of the matching point, indicating the i-th matching point, (x″, y″) is the coordinate of the point obtained by projection of the homography matrix, and (x′, y′) is the coordinate of the matching point in the original image.

[0131] According to the set threshold (for example, the reprojection error is less than a certain value), the matching points are determined as inliers. By counting the number of inliers, the validity of the homography matrix can be evaluated.

[0132] In order to further improve the accuracy of the homography matrix, the RANSAC algorithm is used to minimize the reprojection error to obtain the optimal homography matrix. The objective function of the RANSAC algorithm is:

[0133]

[0134] Where n is the total number of matching points.

[0135] After multiple iterations, the RANSAC algorithm will retain the homography matrix with the largest number of inliers as the final output homography matrix. This matrix will be used in subsequent image correction and stitching operations to ensure the generation of high-quality panoramic images.

[0136] Step (5) Multi-sensor data fusion

[0137] In automatic calibration, sensor data fusion is achieved through an extended Kalman filter (EKF) that combines IMU data, GPS data, and camera data to improve the accuracy and robustness of the system.

[0138] IMU data: Provides angular velocity and acceleration information for estimating the vehicle’s attitude change and motion state. The high frequency of IMU data updates helps provide accurate attitude estimation in a short time.

[0139] GPS data: Provides global position coordinates to help determine the absolute position of the vehicle. GPS data provides position correction during long-term movement, reducing cumulative errors.

[0140] Camera data: The images captured by the camera are used for feature point extraction and matching and are the core input for image calibration.

[0141] In order to achieve efficient data fusion, the system uses an adaptive weight fusion algorithm to dynamically adjust the weight according to the quality of the sensor signal. IMU weight: based on the noise level of acceleration and angular velocity. When the noise is low, the weight of IMU data increases; GPS weight: based on the number of satellites and signal strength. When the signal strength is high and the number of satellites is large, the weight of GPS data increases; Visual weight: based on the number of feature points and matching quality. When there are many feature points and the matching quality is high, the weight of visual data increases.

[0142] Combining data from multiple sensors such as the inertial measurement unit (IMU) and global positioning system (GPS), the extended Kalman filter (EKF) is used to predict and update the state during the calibrated image stitching and fusion process. The EKF uses IMU data for state prediction and combines GPS and visual data for update. The sensor data fusion method can effectively improve the position accuracy and stability of the calibration system. Through the adaptive weight fusion algorithm, the number of image feature points matching and sensor signal strength, the weights are dynamically adjusted to obtain the corrected homography matrix.

[0143] The angular velocity provided by the IMU data can be used to calculate the rotation matrix R, which is used to compensate for the rotation in the image and adjust the rotation component of the homography matrix. The position information provided by the GPS data can be used to adjust the translation component of the homography matrix. The translation vector t is calculated through the GPS data to adjust the translation component of the homography matrix.

[0144] The homography matrix H is usually expressed as:

[0145]

[0146] Rotation matrix R and translation vector t:

[0147]

[0148]

[0149] The fused homography matrix can be expressed as H′:

[0150]

[0151] Where α and β are weight coefficients used to balance the impact of visual data and sensor data.

[0152] Step (6) image projection transformation;

[0153] Image projection transformation is a common concept in image processing and computer vision. It involves the process of mapping an image from one plane to another. In mathematics, projection transformation can be achieved through a linear transformation matrix. This transformation can maintain the straight line properties in the image, that is, the straight line before and after the transformation remains a straight line. The projection transformation can be represented by a 3x3 matrix, which defines the new position of the point in the image. In order to obtain the projected image, two key steps are required: first, obtain the projection transformation matrix through camera calibration, and then use this projection transformation matrix to achieve the perspective transformation of the image. During the calibration process, the mapping relationship between the camera coordinate system and the ground coordinate system is established according to the camera's intrinsic parameter matrix and installation position, so as to calculate the transformation parameters from the original perspective to the bird's-eye view.

[0154] As a special case of projection transformation, the core task of perspective transformation is to project points in three-dimensional space onto a two-dimensional plane. This transformation can simulate the perspective effect of human eyes when observing the world, that is, objects in the distance appear smaller. In the specific implementation, the system uses homogeneous coordinates to simplify the calculation and completes the transformation through the calibrated 3x3 perspective transformation matrix. During the conversion process, the system first predefines the range and resolution of the target projection image, and then maps each pixel in the original image to the virtual ground plane through matrix operations. This mapping process takes into account parameters such as the pitch angle and installation height of the camera, and calculates the new pixel value through bilinear interpolation, and finally generates a projection image of the ground viewed from directly above.

[0155] Step (7) image stitching;

[0156] Because there are overlapping parts between the cameras, after the image is projected, the surround view images must be stitched based on the overlapping parts. First, the overlapping areas in the adjacent camera images are identified and the feature points of the overlapping areas are matched. Then, the optimal stitching position and stitching line are calculated based on the matched feature points. Next, in order to eliminate the visible seams at the stitching, the system uses a gradient blending method to smoothly transition pixel by pixel in the overlapping area. This module stitches the images captured by multiple cameras around the vehicle into a panoramic image (i.e., a bird's-eye view image) through image correction and rendering.

[0157] According to the projection image obtained after the homography matrix is ​​applied for projection transformation, the corresponding brightness and color matrix of each overlapping part is calculated, and the spliced ​​image is corrected and rendered using the brightness and color matrix to obtain the final panoramic image. In the correction stage, the system first uses a geometric correction method to eliminate the local distortion caused by splicing, including using a grid deformation technique to correct the distorted area and using a bilinear interpolation method to resample the pixel value. Then, photometric correction is performed to unify the brightness distribution of each area through a histogram matching technique, and a color migration algorithm is used to correct the chromatic aberration. In the rendering stage, the system first performs adaptive contrast enhancement on the image to improve the visibility of dark details; then an edge-preserving smoothing filter is applied to reduce noise while retaining image details; finally, the overall visual effect of the picture is optimized through local tone mapping, making the image clearer and more natural while maintaining authenticity. Correction can repair the distortion introduced by the splicing process, and rendering can enhance the visual effect of the image.

[0158] Example 2

[0159] Based on the same inventive concept as that of Example 1, an embodiment of the present invention provides a 360-degree surround automatic calibration device for an engineering vehicle, comprising:

[0160] The homography matrix calculation module is used to iteratively calculate the homography matrix corresponding to each vehicle image using the feature matching algorithm and the RANSAC algorithm for the vehicle images acquired at different angles;

[0161] Rotation matrix calculation module, used to calculate the rotation matrix based on IMU data;

[0162] A translation vector calculation module, used to calculate the translation vector based on GPS data;

[0163] A homography matrix correction module, used to correct each calculated homography matrix based on the rotation matrix and the translation vector in combination with an adaptive weight fusion algorithm to obtain a corrected homography matrix;

[0164] A projection transformation module, used to perform projection transformation on the vehicle image corresponding to each corrected homography matrix to obtain a corresponding projection image;

[0165] The image stitching module is used to process each projection image using an image stitching algorithm to obtain a 360-degree surround image of the engineering vehicle.

[0166] The rest are the same as in Example 1.

[0167] Example 3

[0168] An embodiment of the present invention provides a 360-degree surround view automatic calibration system for an engineering vehicle, including a storage medium and a processor;

[0169] The storage medium is used to store instructions;

[0170] The processor is configured to operate according to the instructions to execute the method according to any one of the embodiments 1.

[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

[0176] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A 360-degree surround automatic calibration method for an engineering vehicle, characterized in that: include: For the vehicle surrounding images acquired by multiple cameras at different angles, the feature matching algorithm and RANSAC algorithm are used to iteratively calculate the homography matrix corresponding to each camera; Calculate the rotation matrix based on the IMU data; Calculate the translation vector based on GPS data; Based on the rotation matrix and the translation vector, the calculated homography matrices are corrected in combination with an adaptive weight fusion algorithm to obtain a corrected homography matrix; Using each corrected homography matrix to perform a projection transformation on the corresponding vehicle surrounding image to obtain a corresponding projection image; The image stitching algorithm is used to process each projection image to obtain a 360-degree surround view image of the engineering vehicle.

2. The 360-degree surround automatic calibration method for an engineering vehicle according to claim 1, characterized in that: The calculation method of the homography matrix includes: Perform feature operator extraction on two vehicle surrounding images at the same angle, and identify matching points in two disparity images based on the feature operator extraction results; Based on the two-dimensional coordinates of the selected matching points and the intrinsic parameter matrix of the camera, the corresponding initial homography matrix is ​​calculated; Using the initial homography matrix, all matching points are projected and the reprojection error of each matching point is calculated; If the reprojection error of a matching point meets the preset error threshold, the matching point is defined as an inlier point. With the goal of maximizing the number of internal points, the RANSAC algorithm is used to iteratively calculate the final homography matrix.

3. The 360-degree surround automatic calibration method for an engineering vehicle according to claim 2, characterized in that: The calculation formula of the reprojection error is: Among them, i is the index of the matching point, indicating the i-th matching point, (x ″ ,y ″ ) are the coordinates of the point obtained by projection of the homography matrix, and (x′, y′) are the coordinates of the matching point in the original image; The objective function of the RANSAC algorithm is: Where n is the total number of matching points and H is the homography matrix.

4. The 360-degree surround automatic calibration method for an engineering vehicle according to claim 2, characterized in that: The expression of the homography matrix before correction is: The expression of the rotation matrix is: The expression of the translation vector is: The expression of the corrected homography matrix is: Among them, H is the homography matrix before correction, H ′ is the corrected homography matrix, α and β are dynamic weight coefficients used to balance the impact of sensor data on the homography matrix, h mn is an element in the homography matrix H, m = 1 to 3, n = 1 to 3, representing the transformation relationship between image planes; r mn is the element in the rotation matrix R, which represents the rotation of the camera on each axis; t x , t y , t z are the components of the translation vector t, representing the translation of the camera in each direction.

5. The 360-degree surround automatic calibration method for an engineering vehicle according to claim 4, characterized in that: The dynamic weight coefficients α and β are adjusted in the following way: When the noise of IMU data is lower than the set noise threshold, the value of α is increased; If the satellite signal strength is greater than the set threshold and the number of satellites is greater than the set number threshold, the value of β is increased.

6. The 360-degree surround automatic calibration method for an engineering vehicle according to claim 4, characterized in that: The homography matrix before correction is calculated by the following formula: in, and are the corresponding coordinates of the corresponding matching points in the two disparity images, j = 1, 2, ... 8, h mn It is an element in the homography matrix H, m=1~3, n=1~3.

7. The 360-degree surround automatic calibration method for an engineering vehicle according to claim 2, characterized in that: The matching point recognition method comprises: Preprocess the images around the vehicle, including noise reduction and contrast enhancement; Use the FAST algorithm to detect feature points, calculate Harris response values, and select stable feature points through non-maximum suppression; Calculate the main direction of the feature point and extract the rotation BRIEF descriptor; The Hamming distance is used to measure the similarity of BRIEF descriptors and obtain matching points.

8. The 360-degree surround automatic calibration method for an engineering vehicle according to claim 2, characterized in that: The method for generating the 360-degree surround view image of the engineering vehicle comprises: According to the projection image obtained after the projective transformation using the corrected homography matrix, the brightness and color matrix corresponding to each overlapping part is calculated; The spliced ​​images are corrected and rendered using the brightness and color matrix to obtain a final panoramic image.

9. A 360-degree surround automatic calibration device for an engineering vehicle, characterized in that: include: The homography matrix calculation module is used to iteratively calculate the homography matrix corresponding to each camera using the feature matching algorithm and the RANSAC algorithm for the vehicle surrounding images at different angles acquired by multiple cameras; Rotation matrix calculation module, used to calculate the rotation matrix based on IMU data; A translation vector calculation module, used to calculate the translation vector based on GPS data; A homography matrix correction module, used to correct each calculated homography matrix based on the rotation matrix and the translation vector in combination with an adaptive weight fusion algorithm to obtain a corrected homography matrix; A projection transformation module, used to perform projection transformation on the corresponding vehicle surrounding images using each corrected homography matrix to obtain a corresponding projection image; The image stitching module is used to process each projection image using an image stitching algorithm to obtain a 360-degree surround image of the engineering vehicle.

10. A 360-degree surround automatic calibration system for an engineering vehicle, characterized in that: including storage media and processors; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the method according to any one of claims 1-8.

Images