Extrinsic calibration method and device of perception device, equipment and storage medium

Abstract

The application provides an external parameter calibration method and device of a perception device, equipment and a storage medium. The method comprises: determining a camera image mask set corresponding to a camera image collected by a camera in a vehicle at a current time, and a front view mask set corresponding to point cloud data collected by a laser radar at the current time; evaluating the scene quality at the current time based on the camera image mask set and the front view mask set; in the case where the scene quality score is greater than or equal to a preset score threshold, performing mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set, and determining an initial value of the external parameter of the camera and the laser radar based on the successfully matched mask pairs; inputting the initial value of the external parameter into an online calibration algorithm to obtain an external parameter calibration result, and outputting the external parameter calibration result in the case where the external parameter calibration result meets a convergence condition. In this way, the external parameter calibration accuracy of the perception device can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for calibrating the external parameters of a sensing device. Background Technology

[0002] In autonomous driving systems, cameras and LiDAR are two core sensors that provide complementary environmental perception information. Cameras provide rich texture and color information, while LiDAR accurately measures the distance and spatial position of objects. To effectively fuse data from these two sensors, their relative positions and orientations must be precisely calibrated—this is known as extrinsic parameter calibration. The accuracy of extrinsic parameter calibration directly impacts the environmental perception capabilities and decision-making accuracy of the autonomous driving system. Although autonomous vehicles undergo extrinsic parameter calibration of their cameras and LiDAR at the factory, during long-term driving, vehicle vibrations can cause slight displacements in both sensors; in the event of a collision, significant pose shifts can occur, directly causing the camera-LiDAR fusion perception system to fail. Therefore, current extrinsic parameter calibration technology in the field of autonomous driving is gradually developing towards targetless automatic calibration, meaning that extrinsic parameters can be updated and calibrated online without relying on manual calibration targets.

[0003] Currently, methods for online updating and calibration of extrinsic parameters without relying on manual target calibration mainly include feature-based online calibration methods for cameras and LiDAR without targets, and deep learning-based online calibration methods for cameras and LiDAR without targets. However, all of these methods rely on rich environmental features, which can easily lead to low calibration accuracy when there are few environmental features.

[0004] Therefore, how to improve the calibration accuracy of the external parameters of sensing devices has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for calibrating the external parameters of a sensing device, which can improve the calibration accuracy of the external parameters of the sensing device.

[0006] In a first aspect, embodiments of this application provide a method for calibrating the external parameters of a sensing device, the method comprising:

[0007] Determine the set of camera image masks corresponding to the camera images captured by the cameras in the vehicle at the current moment, and determine the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment.

[0008] Based on the set of camera image masks and the set of front view masks, the scene quality at the current moment is evaluated to obtain a scene quality score;

[0009] If the scene quality score is greater than or equal to the preset score threshold, perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set, and determine the initial values ​​of the external parameters of the camera and the LiDAR based on the successfully matched mask pairs.

[0010] The initial values ​​of the extrinsic parameters are input into the online calibration algorithm to obtain the extrinsic parameter calibration results. If the extrinsic parameter calibration results meet the convergence conditions, the extrinsic parameter calibration results are output.

[0011] In one embodiment, the scene quality at the current moment is evaluated based on a set of camera image masks and a set of front view masks to obtain a scene quality score. This includes: determining the camera image mask sufficiency index value corresponding to the set of camera image masks and the front view mask sufficiency index value corresponding to the set of front view masks; determining a first score corresponding to the scene quality at the current moment based on the camera image mask sufficiency index value and the front view mask sufficiency index value; determining the camera image mask distribution entropy based on the center coordinates corresponding to each camera image mask in the set of camera image masks, and determining the front view mask distribution entropy based on the center coordinates corresponding to each front view mask in the set of front view masks; determining a second score corresponding to the scene quality at the current moment based on the camera image mask distribution entropy and the front view mask distribution entropy; determining the camera image edge intensity based on the set of camera image masks, and determining the front view edge intensity based on the set of front view masks; determining a third score corresponding to the scene quality at the current moment based on the camera image edge intensity and the front view edge intensity; and evaluating the scene quality at the current moment based on at least two of the first score, the second score, and the third score to obtain a scene quality score.

[0012] In one embodiment, determining the edge intensity of a camera image based on a set of camera image masks, and determining the edge intensity of a front view based on a set of front view masks, includes: extracting edge pixels of each camera image mask in the set of camera image masks to obtain a first edge pixel set corresponding to the set of camera image masks; determining the edge intensity of a camera image based on the first edge pixel set, the total number of camera image masks, and the perimeter of the minimum bounding rectangle of each camera image mask; and extracting edge pixels of each front view mask in the set of front view masks to obtain a second edge pixel set corresponding to the set of front view masks; determining the edge intensity of a front view based on the second edge pixel set, the total number of front view masks, and the perimeter of the minimum bounding rectangle of each front view mask.

[0013] In one embodiment, mask matching is performed on each camera image mask in the camera image mask set and each front view mask in the front view mask set, including: determining a first feature vector corresponding to each camera image mask in the camera image mask set, and determining a second feature vector corresponding to each front view mask in the front view mask set; determining the similarity between the i-th camera image mask and the j-th front view mask based on each first feature vector and each second feature vector; 1≤i≤N, 1≤j≤M, where i, j, N, and M are all integers; N is the total number of camera image masks, and M is the total number of front view masks; constructing a cost matrix based on the similarity, and constructing a Hungarian objective function based on the cost matrix and binary decision variables; and performing mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set with the objective of minimizing the Hungarian objective function.

[0014] In one embodiment, determining the first feature vector corresponding to each camera image mask in the camera image mask set, and determining the second feature vector corresponding to each front view mask in the front view mask set, includes: for each camera image mask in the camera image mask set, determining a first normalized area and a first moment corresponding to the camera image mask, and determining the first feature vector corresponding to the camera image mask based on the first normalized area and the first moment; and for each front view mask in the front view mask set, determining a second normalized area and a second moment corresponding to the front view mask, and determining the second feature vector corresponding to the front view mask based on the second normalized area and the second moment.

[0015] In one embodiment, the initial values ​​of the extrinsic parameters of the camera and the LiDAR are determined based on the successfully matched mask pairs, including: determining the two-dimensional centroid coordinates of the target camera image mask based on the pixel values ​​of the target camera image mask in the pixel coordinate system in the successfully matched mask pairs; determining the three-dimensional centroid coordinates of the target front view mask based on the coordinate values ​​of the target front view mask in the LiDAR coordinate system and the total number of three-dimensional points of the target front view mask; and determining the initial values ​​of the extrinsic parameters of the camera and the LiDAR based on the two-dimensional centroid coordinates and the three-dimensional centroid coordinates.

[0016] In one embodiment, determining the set of camera image masks corresponding to the camera image captured by the camera in the vehicle at the current moment, and determining the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment, includes: performing image distortion correction and normalization processing on the camera image captured by the camera in the vehicle at the current moment to obtain a processed camera image, and inputting the processed camera image into an image segmentation model to obtain the set of camera image masks; and projecting the point cloud data captured by the lidar in the vehicle at the current moment to obtain a front view corresponding to the point cloud data, performing normalization processing on the front view to obtain a processed front view, and inputting the processed front view into an image segmentation model to obtain the set of front view masks.

[0017] Secondly, embodiments of this application provide an external parameter calibration device for a sensing device, the device comprising:

[0018] The mask determination module is used to determine the set of camera image masks corresponding to the camera images captured by the camera in the vehicle at the current moment, and to determine the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment.

[0019] The scene evaluation module is used to evaluate the scene quality at the current moment based on the camera image mask set and the front view mask set, and obtain a scene quality score.

[0020] The mask matching module is used to perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set when the scene quality score is greater than or equal to a preset score threshold.

[0021] The calibration module is used to determine the initial values ​​of the extrinsic parameters of the camera and LiDAR based on the successfully matched mask pairs, input the initial values ​​of the extrinsic parameters into the online calibration algorithm to obtain the extrinsic parameter calibration results, and output the extrinsic parameter calibration results if the extrinsic parameter calibration results meet the convergence conditions.

[0022] Thirdly, embodiments of this application provide an apparatus including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method provided in the first aspect.

[0023] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect above.

[0024] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method provided in the first aspect above.

[0025] The aforementioned method, apparatus, device, and storage medium for extrinsic parameter calibration of sensing devices determine the set of camera image masks corresponding to the camera images acquired by the camera in the vehicle at the current moment, and determine the set of front view masks corresponding to the point cloud data acquired by the lidar in the vehicle at the current moment; based on the set of camera image masks and the set of front view masks, the scene quality at the current moment is evaluated to obtain a scene quality score; if the scene quality score is greater than or equal to a preset score threshold, mask matching is performed on each camera image mask in the set of camera image masks and each front view mask in the set of front view masks, and based on the successfully matched mask pairs, the initial values ​​of the extrinsic parameters of the camera and lidar are determined; the initial values ​​of the extrinsic parameters are input into an online calibration algorithm to obtain the extrinsic parameter calibration result, and if the extrinsic parameter calibration result meets the convergence condition, the extrinsic parameter calibration result is output. This method introduces a scene quality prediction mechanism and performs extrinsic parameter calibration when the scene quality score is greater than or equal to a preset score threshold. This proactively identifies and filters environmental conditions unsuitable for extrinsic parameter calibration before the calibration process begins, fundamentally avoiding the problem of decreased algorithm accuracy caused by forced calibration in scenarios with insufficient features. This improves the accuracy of the determined initial values ​​of extrinsic parameters, and consequently, improves the extrinsic parameter calibration accuracy of the sensing device. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating a method for calibrating the external parameters of a sensing device according to an embodiment of this application;

[0028] Figure 2 This is a schematic diagram of a scene quality assessment process provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram illustrating a process for determining initial values ​​of external parameters provided in an embodiment of this application;

[0030] Figure 4 This is a schematic diagram illustrating the process of determining a camera image mask set and a front view mask set according to an embodiment of this application;

[0031] Figure 5 This is a flowchart illustrating another method for calibrating the external parameters of a sensing device provided in an embodiment of this application;

[0032] Figure 6This is a schematic diagram of the structure of an external parameter calibration device for a sensing device provided in an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] In autonomous driving systems, cameras and LiDAR are two core sensors that provide complementary environmental perception information. Cameras provide rich texture and color information, while LiDAR accurately measures the distance and spatial position of objects. To effectively fuse data from these two sensors, their relative positions and orientations must be precisely calibrated—this is known as extrinsic parameter calibration. The accuracy of extrinsic parameter calibration directly impacts the environmental perception capabilities and decision-making accuracy of the autonomous driving system. Although autonomous vehicles undergo extrinsic parameter calibration of their cameras and LiDAR at the factory, during long-term driving, vehicle vibrations can cause slight displacements in both sensors; in the event of a collision, significant pose shifts can occur, directly causing the camera-LiDAR fusion perception system to fail. Therefore, current extrinsic parameter calibration technology in the field of autonomous driving is gradually developing towards targetless automatic calibration, meaning that extrinsic parameters can be updated and calibrated online without relying on manual calibration targets.

[0036] Currently, methods for online updating and calibration of extrinsic parameters without relying on manually calibrated targets mainly include feature-based online calibration methods for cameras and LiDAR without targets, and deep learning-based online calibration methods for cameras and LiDAR without targets. Feature-based online calibration methods for cameras and LiDAR without targets aim to calculate extrinsic parameters online by automatically extracting and associating natural features (such as edges, planes, and point features) in the environment without relying on manual targets. They typically utilize geometric constraints (such as collinearity and coplanarity) or mutual information maximization to establish the correspondence between image and point cloud data, and solve the transformation matrix through optimization algorithms. However, this type of method has significant technical drawbacks: First, the calibration performance is highly dependent on the richness and distinguishability of environmental features. In unstructured scenes such as open areas without obvious edges or planes, insufficient feature extraction or matching errors can easily lead to calibration failure. Second, the extrinsic parameter optimization process relies on relatively accurate initial estimates—initial values ​​are used to narrow the search space for feature matching and ensure that the optimization algorithm converges to the global optimum. If the initial values ​​deviate significantly from the true extrinsic parameters, the calibration accuracy will decrease significantly. A deep learning-based online calibration method for cameras and LiDAR without targets involves designing a two-branch neural network to extract high-level feature representations from image and point cloud data, respectively. A specific feature matching module then uses these features to learn the implicit correspondence between the two data patterns, ultimately outputting 6-DOF extrinsic parameters for the camera and LiDAR. However, this method also has technical limitations: First, before inputting the LiDAR point cloud into the neural network, it needs to be mapped onto the image plane using perspective projection based on preset initial extrinsic parameter values. If the preset initial values ​​deviate significantly from the actual true extrinsic parameter values, subsequent feature matching and extrinsic parameter solving will fail. Second, although the neural network has strong feature extraction capabilities, its calibration accuracy is still affected by the density of environmental features. In scenarios with sparse environmental features, the calibration accuracy decreases significantly as the amount of feature information decreases.

[0037] Based on this, embodiments of this application provide a method, apparatus, device, and storage medium for extrinsic parameter calibration of a sensing device. The method includes: a computer device evaluating the scene quality at the current moment based on a set of camera image masks corresponding to camera images captured by a camera in the vehicle at the current moment and a set of front view masks corresponding to point cloud data captured by a lidar in the vehicle at the current moment, to obtain a scene quality score; if the scene quality score is greater than or equal to a preset score threshold, performing mask matching on each camera image mask in the set of camera image masks and each front view mask in the set of front view masks, and determining initial extrinsic parameter values ​​for the camera and lidar based on the successfully matched mask pairs; inputting the initial extrinsic parameter values ​​into an online calibration algorithm to obtain extrinsic parameter calibration results, and outputting the extrinsic parameter calibration results if the extrinsic parameter calibration results meet the convergence condition. This method introduces a scene quality prediction mechanism and performs extrinsic parameter calibration when the scene quality score is greater than or equal to a preset score threshold. This proactively identifies and filters environmental conditions unsuitable for extrinsic parameter calibration before the calibration process begins, fundamentally avoiding the problem of decreased algorithm accuracy caused by forced calibration in scenarios with insufficient features. This improves the accuracy of the determined initial values ​​of extrinsic parameters, and consequently, improves the extrinsic parameter calibration accuracy of the sensing device.

[0038] Optionally, the computer device can be a terminal or a server. The terminal may include, but is not limited to, controllers in a vehicle, such as controllers related to intelligent driving or assisted driving. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0039] The external parameter calibration method for sensing devices provided in the embodiments of this application is described below.

[0040] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for calibrating the external parameters of a sensing device according to an embodiment of this application. This method can be executed by a computer device. Figure 1 As shown, the external parameter calibration method for this sensing device may include, but is not limited to, the following steps:

[0041] S101. Determine the set of camera image masks corresponding to the camera images captured by the camera in the vehicle at the current moment, and determine the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment.

[0042] The camera image mask set is the product of structurally and finely decomposing the rich visual information contained in the camera image at the current moment. Each mask in the camera image mask set marks a specific region in the original image from different dimensions or for different purposes.

[0043] The front view mask set is the product of a structured and refined decomposition of the rich visual information contained in the front view at the current moment. Each mask in the front view mask set annotates a specific region in the original image from different dimensions or for different purposes. The front view refers to the view directly in front obtained by projecting point cloud data.

[0044] In one alternative implementation, the computer device determines the set of camera image masks corresponding to the camera images captured by the camera in the vehicle at the current moment, and determines the set of front view masks corresponding to the point cloud data captured by the LiDAR in the vehicle at the current moment. This can be achieved by calling an image segmentation model to obtain the set of camera image masks based on the camera images captured by the camera in the vehicle at the current moment, and by determining the front view based on the point cloud data captured by the LiDAR in the vehicle at the current moment, calling an image segmentation model to obtain the set of front view masks based on the front view.

[0045] S102. Based on the camera image mask set and the front view mask set, evaluate the scene quality at the current moment and obtain a scene quality score.

[0046] The current scene refers to a real-world environment with rich and distinct visual and geometric features selected specifically during the calibration of the camera and LiDAR extrinsic parameters in order to accurately calculate the relative position and attitude (i.e., extrinsic parameters) between the two. This real-world environment is the real-time environment in which the user is driving the vehicle.

[0047] In one alternative implementation, the computer device evaluates the scene quality at the current moment based on the camera image mask set and the front view mask set to obtain a scene quality score. This can be achieved by evaluating the scene quality at the current moment from one or more dimensions based on the camera image mask set and the front view mask set to obtain a scene quality score.

[0048] S103. When the scene quality score is greater than or equal to the preset score threshold, perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set, and determine the initial values ​​of the external parameters of the camera and the LiDAR based on the successfully matched mask pairs.

[0049] The initial extrinsic values ​​for the camera and LiDAR refer to an initial, approximately correct transformation matrix provided before starting precise, iterative, automated extrinsic calibration.

[0050] In one alternative implementation, the preset scoring threshold may be determined based on historical experiments or expert experience, etc., and is not limited here.

[0051] In an optional implementation, if the scene quality score is less than a preset score threshold, the computer device may also reacquire camera images captured by the camera in the vehicle and point cloud data collected by the LiDAR, and execute step S101. In this case, the computer device does not need to execute step S104.

[0052] S104. Input the initial values ​​of the external parameters into the online calibration algorithm to obtain the calibration results of the external parameters, and output the calibration results of the external parameters if the calibration results meet the convergence conditions.

[0053] In one optional implementation, after obtaining the extrinsic parameter calibration results, the computer device may determine whether the convergence condition is met by: determining the point cloud set corresponding to the target front view mask in the successfully matched mask pair; projecting the point cloud set onto the image plane using perspective projection based on the extrinsic parameter calibration results and camera intrinsic parameters to obtain the point cloud projection mask; determining the overlap value between the point cloud projection mask and the target camera image mask in the successfully matched mask pair; if the overlap value is greater than or equal to a preset overlap threshold, then the extrinsic parameter calibration results are determined to meet the convergence condition; if the overlap value is less than the preset overlap threshold, then the extrinsic parameter calibration results are determined not to meet the convergence condition.

[0054] If the extrinsic parameter calibration results meet the convergence condition, it indicates that the overlap between the point cloud projection and the camera image is high; if the extrinsic parameter calibration results do not meet the convergence condition, it indicates that the overlap between the point cloud projection and the camera image is low.

[0055] In some embodiments, when a computer device determines the overlap value between a point cloud projection mask and a target camera image mask in a successfully matched mask pair, it may use the following formula (1).

[0056] (1)

[0057] In formula (1), IoU represents the overlap value between the point cloud projection mask and the target camera image mask in the successfully matched mask pair; Area() represents the number of pixels in the calculated mask area; This represents the target camera image mask in a successfully matched mask pair; This represents a point cloud projection mask, where the point cloud projection mask is based on the target front view mask in a successfully matched mask pair. The corresponding point cloud set Sure.

[0058] In some embodiments, the preset overlap threshold can be 0.8. This preset overlap threshold can be determined based on historical experiments, expert experience, etc., and is not limited here.

[0059] In an optional implementation, the computer device may also reacquire camera images collected by the camera in the vehicle and point cloud data collected by the lidar if the external parameter calibration results do not meet the convergence conditions, and then execute step S101.

[0060] In this embodiment, by introducing a scene quality prediction mechanism and performing external parameter calibration when the scene quality score is greater than or equal to a preset score threshold, environmental conditions unsuitable for external parameter calibration can be actively identified and filtered before the calibration process starts. This fundamentally avoids the problem of decreased algorithm accuracy caused by forced calibration in scenarios with insufficient features, thereby improving the accuracy of the determined initial external parameter values ​​and, consequently, improving the external parameter calibration accuracy of the sensing device.

[0061] Furthermore, since the scene at that moment is the real-time environment of the user driving the vehicle, the external parameter calibration of the camera and LiDAR in the vehicle can be performed without the need for professional personnel, thereby improving the convenience of camera and LiDAR external parameter calibration.

[0062] In one alternative implementation, Figure 1 Step S102 in the extrinsic parameter calibration method of the sensing device shown, i.e., the computer device evaluates the scene quality at the current moment based on the camera image mask set and the front view mask set to obtain a scene quality score, can be as follows: determine the camera image mask sufficiency index value corresponding to the camera image mask set and the front view mask sufficiency index value corresponding to the front view mask set, and determine the first score corresponding to the scene quality at the current moment based on the camera image mask sufficiency index value and the front view mask sufficiency index value; determine the camera image mask distribution based on the center coordinates corresponding to each camera image mask in the camera image mask set. The scene quality is determined by: entropy, and, based on the center coordinates of each front view mask in the front view mask set, the front view mask distribution entropy is determined; based on the camera image mask distribution entropy and the front view mask distribution entropy, a second score corresponding to the scene quality at the current moment is determined; based on the camera image mask set, the camera image edge intensity is determined, and, based on the front view mask set, the front view edge intensity is determined; based on the camera image edge intensity and the front view edge intensity, a third score corresponding to the scene quality at the current moment is determined; based on at least two of the first score, the second score, and the third score, the scene quality at the current moment is evaluated to obtain a scene quality score.

[0063] The mask sufficiency index is a metric used to measure the richness of scene features in an image. In other words, the mask sufficiency index value characterizes the richness of scene features in an image. For example, images captured in scenes with high contrast between the target and background, and where the target object has a simple and regular structure (such as a passport photo of a person standing in front of a white background, or an image of windows arranged in a regular pattern) have higher mask sufficiency index values; conversely, images captured in scenes with low contrast, motion, or poor lighting conditions (such as an image of a snow leopard lying in the snow, an image taken by a vehicle-mounted camera in a fast-moving car, or an image taken under strong light) have lower mask sufficiency index values. The mask sufficiency index value ranges from [0,1].

[0064] In some embodiments, when a computer device determines the camera image mask sufficiency index value corresponding to a set of camera image masks, it may use the following formula (2).

[0065] (2)

[0066] In formula (2), This represents the camera image mask sufficiency index value; This represents the total number of camera image masks; This represents the number of valid camera image masks in the camera image mask; k represents the attenuation coefficient. The valid camera image mask is a binary image (usually a black and white image) used to identify which pixels in the camera image are reliable, usable, or distortion-free.

[0067] Formula (2) is essentially a sigmoid function. The sigmoid function is a continuous, smooth, monotonically increasing function. Its core characteristics are: ① The value range is strictly limited to [0,1], which can be directly used as a normalized score (no additional scaling is required, adapting to the quantitative requirements of the index); ② The function has the largest slope at x=0 (most sensitive to change), and F(x) tends to 1 (saturation) when x approaches +∞, and F(x) tends to 0 (saturation) when x approaches -∞. That is, the score changes slowly when the input value is near the threshold, and the score converges quickly to 0 or 1 when it is far from the threshold. It can naturally distinguish the boundary of sufficient / insufficient features, while avoiding abrupt changes. Formula (2) uses the characteristics of the sigmoid function to reduce the number of invalid camera image masks ( ) is mapped to a continuous fraction between 0 and 1, where what is “sufficient” is flexibly defined by a decay factor k.

[0068] In this application, the reason for choosing the Sigmoid function to determine the mask sufficiency index value is as follows: (1) Avoiding the sudden misjudgment of linear threshold: Traditional technology often uses linear threshold rules (such as N'≥10 gets 1 point, otherwise gets 0 points). However, in actual scenarios, the feature sufficiency difference between N'=9 and N'=10 is very small (only 1 target difference), but they are directly classified as unqualified / qualified by the linear threshold. The Sigmoid function, through its smooth transition characteristics, allows N'=9 to get 0.42 points and N'=10 to get 0.5 points, which not only reflects the difference but also does not deny the value of the scenario close to the threshold, which is more in line with the continuity of the actual scenario; (2) Naturally adapting to the quantitative scoring requirements: This index is a component of the comprehensive score F_total, and needs to output the normalized score in the [0,1] interval (for easy weighted summation). The Sigmoid function does not require additional normalization processing and directly outputs the required score. The value range boundary (0 and 1) can clearly distinguish between complete insufficiency and complete sufficiency, and the logic is closed loop; (3) Balance sensitivity and fault tolerance: By adjusting the attenuation coefficient k, the influence of quantity changes on the score can be precisely controlled; (4) Match the calibration constraint requirements: The requirement of the number of features for external parameter calibration is that the standard is met, and the excess gain decreases marginally.

[0069] In some embodiments, when a computer device determines the front view mask sufficiency index value corresponding to the front view mask set, it may use the following formula (3).

[0070] (3)

[0071] In formula (3), This represents the front view mask sufficiency index value; This indicates the total number of front view masks; This represents the number of valid front view masks in the front view mask; k represents the attenuation coefficient. The valid front view mask is a binary image (usually a black and white image) used to identify which pixels in the front view are reliable, usable, or distortion-free.

[0072] Formula (3) is essentially a sigmoid function, cleverly used to transform the proportion of valid masks into a smooth, normalized score. Formula (2) utilizes the properties of the sigmoid function to transform the number of invalid front view masks ( ) is mapped to a continuous fraction between 0 and 1, where what is “sufficient” is flexibly defined by a decay factor k.

[0073] In some embodiments, when a computer device determines the first score corresponding to the scene quality at the current moment based on the camera image mask sufficiency index value and the front view mask sufficiency index value, the following formula (4) may be used.

[0074] (4)

[0075] In formula (4), F1 represents the first score (also known as the mask quantity sufficiency score); F 1,img This represents the camera image mask sufficiency index value; F 1,lidar This represents the front view mask sufficiency index value.

[0076] In other words, the first score corresponding to the scene quality at the current moment is the smaller value between the camera image mask sufficiency index and the front view mask sufficiency index.

[0077] It should be noted that since the mask sufficiency index is used to measure the richness of scene features in an image, the first score determined based on the camera image mask sufficiency index value and the front view mask sufficiency index value can be used to evaluate scene quality.

[0078] The center coordinates of the camera image mask are the average of the coordinates of all pixels in the camera image mask. In some embodiments, the center coordinates of each camera image mask in the camera image mask set can be determined by a computer device using the following formula (5).

[0079] (5)

[0080] In formula (5), S represents the i-th camera image mask in the set of camera image masks M1; i The area of ​​the image mask of the i-th camera; The coordinates of the center of the i-th camera image mask are represented.

[0081] The core of formula (5) is to calculate the geometric centroid of the i-th camera image mask, rather than the simple arithmetic mean of pixel coordinates. The specific principle can be broken down into two steps: (1) Calculate the sum of coordinates of all pixels in the mask: Since the mask is a binary image (pixel value 1 indicates that it belongs to the mask area, and 0 indicates the background), it is necessary to first filter out all points with a pixel value of 1, and then accumulate the x and y coordinates of these points respectively. This step is to obtain the total position information of all pixels in the mask area and avoid interference from background pixels; (2) Normalize the mask area to obtain the centroid coordinates: Divide the sum of coordinates by the mask area (i.e., the number of pixels). In essence, it is to perform an area-weighted average of all pixel coordinates. Among them, the larger the area of ​​the mask, the more the centroid can represent the position of the target as a whole, rather than the position of local pixels.

[0082] In this application, the reason for choosing formula (5) to determine the center coordinates of the camera image mask is: (1) To adapt to irregular mask shapes and avoid misjudgment of position: The masks segmented by the image segmentation model are mostly irregular shapes (such as pedestrian masks and tree masks). If the position is represented by the pixel coordinates of the upper left corner of the mask or the midpoint between the maximum and minimum coordinates of the mask, it will be seriously deviated from the actual position of the target (such as the midpoint of the L-shaped mask may fall in the background area). Formula (5) uses full pixel coordinate weighting to calculate the centroid reflecting the overall position of the target regardless of whether the mask shape is regular, ensuring the accuracy of subsequent spatial distribution assessment; (2) It directly matches the target of spatial distribution uniformity assessment: the core of spatial distribution uniformity is to judge the degree of dispersion of feature targets in the field of view, and the centroid coordinate is the unique position anchor point of each target. Only by obtaining accurate centroid coordinates through this formula can we subsequently count which grids the centroid falls in, and then quantify the grid coverage. Without this formula, spatial distribution assessment will lose its meaning because the target position is not clearly represented; (3) It has low computational complexity and is suitable for engineering implementation: the calculation of this formula only involves pixel traversal, summation, and division, without the need for complex matrix operations or iterative optimization. It can run quickly in embedded devices, vehicle computing platforms and other scenarios (the centroid calculation time of a single frame image mask is usually <1ms), taking into account both accuracy and engineering practicality.

[0083] In some embodiments, the computer device may use the following formula (6) to determine the camera image mask distribution entropy.

[0084] (6)

[0085] In formula (6), K1 represents the entropy of the camera image mask distribution; K1 represents the side length of the grid (K1×K1) obtained by dividing the camera image resolution W1×H1. This represents the number of grid cells at the center of the camera image mask. After dividing the camera image into K1×K1 grid cells, each grid cell has a defined coordinate range. This refers to the number of grids containing the center coordinates of at least one camera image mask.

[0086] The evaluation of camera image mask distribution entropy essentially uses grid coverage quantification to objectively measure the spatial dispersion of feature targets (masks) in a camera image. Here, distribution entropy is not the entropy calculation in traditional information theory, but a simplified and engineered grid coverage entropy. By statistically analyzing the distribution density of mask centers within the grid, the uniformity of the distribution is indirectly reflected: the more uniform the distribution, the more grids the mask centers cover, the higher the grid coverage, and the higher the equivalent distribution entropy; conversely, when features are concentrated in a few grids, the coverage is low, and the equivalent distribution entropy is low.

[0087] The center coordinates of the front view mask are the average of the coordinates of all pixels in the front view mask. In some embodiments, the center coordinates of each front view mask in the front view mask set can be determined by a computer device using the following formula (7).

[0088] (7)

[0089] In formula (7), This represents the j-th front view mask in the front view mask set M2; S i The area of ​​the j-th front view mask; The coordinates of the center of the j-th front view mask are given. The essence of formula (7) is to calculate the average of the x-coordinates and the average of the y-coordinates of all pixels in the j-th front view mask.

[0090] In some embodiments, the computer device may use the following formula (8) to determine the front view mask distribution entropy.

[0091] (8)

[0092] In formula (8), This represents the entropy of the front view mask distribution; K1 represents the number of grids at the center of the front view mask; K2 represents the side length of the grid (K2×K2) obtained by dividing the front view resolution W2×H2.

[0093] Among them, the distribution entropy evaluation of the front view mask is essentially an objective measure of the spatial dispersion of feature targets (masks) in the front view through grid coverage quantification, which is adapted to the point cloud projection characteristics of the front view.

[0094] In this application, the core reason for choosing the grid-covered distribution entropy (for the characteristics of the front view) is: (1) To adapt to the point cloud projection characteristics of the front view and avoid misjudgment of distribution: The front view is generated by the spherical projection of the point cloud, which has two key characteristics: ① Uneven point cloud density (sparse point cloud in the far field and dense point cloud in the near field), which may result in more irregular mask shapes (such as the far field target mask being thin strips); ② The edges of the front view are easily affected by point cloud noise (such as the tiny mask formed by isolated points, which has been filtered through preprocessing). (2) To solve the pain point of traditional subjective judgment: no objective standard When traditional technology evaluates the distribution of front view features, it mostly relies on visual observation to see if they are dispersed, which has serious subjectivity: For example, 8 masks in the front view are concentrated in the left half (3 grids), which may be misjudged as uniformly distributed, but in reality, it will lead to calibration bias (the matching error of the right half field of view reaches 0.3m). (3) Balancing evaluation accuracy and engineering computation: Real-time processing of the front view is a core requirement for scenarios such as vehicle and robot (it is necessary to quickly screen qualified scenarios and avoid excessive calibration time): If more complex distribution evaluation methods (such as kernel density estimation and cluster analysis) are used, although the accuracy can be improved, the calculation time of a single frame of the front view will increase from <1ms to >10ms, which cannot meet the real-time requirements; The grid-covered distribution entropy of this scheme only involves the calculation of the center coordinate → grid ownership judgment → counting, which has low computational complexity and can be run quickly in embedded platforms and vehicle computing units, taking into account both accuracy and engineering practicality. (4) Directly and strongly correlated with calibration quality: The core requirement for ensuring the full field of view constraint balance of extrinsic parameter calibration is the matching consistency of bimodal features in the full field of view. If the front view mask is concentrated in a certain area (such as the near field), the calibration result will be biased towards that area, resulting in the amplification of the bimodal matching error of the far field target (angle error ≥0.5°). (5) In collaboration with image distribution entropy, a dual-modal distribution complementary front view (LiDAR) is good at capturing depth features in 3D space (such as the outlines of vehicles and obstacles), while the image is good at capturing texture features (such as the details of road signs and pedestrians). There may be slight differences in the distribution of the two (such as the front view having one more mask in the far field and the image having one more mask in the near field).

[0095] In some embodiments, when a computer device determines the second score corresponding to the scene quality at the current moment based on the camera image mask distribution entropy and the front view mask distribution entropy, the following formula (9) may be used.

[0096] (9)

[0097] In formula (9), F2 represents the second score (also known as the spatial distribution uniformity score) camera image mask distribution entropy; This represents the entropy of the camera image mask distribution; This represents the entropy of the front view mask distribution.

[0098] In other words, the second score corresponding to the scene quality at the current moment is the average of the camera image mask distribution entropy and the front view mask distribution entropy.

[0099] It should be noted that since image mask distribution entropy reflects the content complexity of the image scene, and content complexity is a key dimension for scene quality assessment, the second score determined based on camera image mask distribution entropy and front view mask distribution entropy can be used to assess scene quality.

[0100] In some embodiments, the computer device determines the edge intensity of a camera image based on a set of camera image masks, and determines the edge intensity of a front view based on a set of front view masks. This can be achieved by: extracting edge pixels from each camera image mask in the set of camera image masks to obtain a first edge pixel set corresponding to the set of camera image masks; determining the edge intensity of the camera image based on the first edge pixel set, the total number of camera image masks, and the perimeter of the minimum bounding rectangle of each camera image mask; and extracting edge pixels from each front view mask in the set of front view masks to obtain a second edge pixel set corresponding to the set of front view masks; and determining the edge intensity of the front view based on the second edge pixel set, the total number of front view masks, and the perimeter of the minimum bounding rectangle of each front view mask.

[0101] Optionally, when determining the edge intensity of a camera image based on the first set of edge pixels, the total number of camera image masks, and the perimeter of the minimum bounding rectangle of each camera image mask, the computer device may use the following formula (10).

[0102] (10)

[0103] In formula (10), This represents the edge intensity of the camera image; This indicates the number of camera image masks; This represents the number of edge pixels in the i-th camera image mask within the first set of edge pixels; This represents the perimeter of the minimum bounding rectangle of the i-th camera image mask. Edge pixels refer to pixels in a digital image located at positions where the color, brightness, or texture of the image content undergoes significant and drastic changes. They essentially constitute the boundaries between different regions in the image.

[0104] The principle of formula (10) can be summarized as follows: first, set an edge integrity benchmark for each feature target, then quantify the actual edge integrity, and finally obtain the edge quality assessment of the entire modality through global averaging, so as to ensure that the assessment results are both accurate and representative, and fully meet the edge matching accuracy requirements of dual-modal extrinsic calibration.

[0105] In this application, the reason for determining the edge intensity of the camera image by formula (10) is: (1) to eliminate the difference in mask size / shape and achieve fair quantification; (2) to accurately reflect the core requirements of clear and continuous edges; (3) the external parameter calibration depends on the edge alignment matching of the dual-modal mask. Clear and continuous edges are the premise of matching accuracy (edge ​​breakage will lead to misjudgment of matching points, and blurred edges will amplify coordinate deviation); (4) to adapt to the edge characteristic difference of dual-modality; (5) there is an essential difference in the edge characteristics of the image and the front view (the image edge is clear and has less noise; the front view edge is easily affected by sparse point cloud / noise, and has a high risk of breakage).

[0106] Optionally, when determining the edge strength of the front view based on the second edge pixel set, the total number of front view masks, and the minimum perimeter of the bounding rectangle of each front view mask, the computer device may use the following formula (11).

[0107] (11)

[0108] In formula (11), This indicates the edge strength of the front view; This indicates the number of front view masks; This represents the number of edge pixels of the j-th front view mask; This represents the perimeter of the minimum bounding rectangle of the j-th front view mask. Edge pixels refer to pixels in a digital image located at positions where the color, brightness, or texture of the image content undergoes significant and drastic changes. They essentially constitute the boundaries between different regions in the image.

[0109] The principle of formula (11) and the reason for determining the edge strength of the front view through formula (11) can be found in the previous description of formula (10), which will not be repeated here.

[0110] In some embodiments, when a computer device determines a third score corresponding to the scene quality at the current moment based on the edge intensity of the camera image and the edge intensity of the front view, the following formula (12) may be used.

[0111] (12)

[0112] In formula (12), F3 represents the third score (also known as the edge sharpness score); F 3,img This represents the edge intensity of the camera image; F 3,lidar This indicates the edge intensity of the front view.

[0113] In other words, the third score corresponding to the scene quality at the current moment is the smaller value between the edge intensity of the camera image and the edge intensity of the front view.

[0114] It should be noted that since images in high-quality scenes have high edge strength and images in low-quality scenes have low edge strength, the third score determined based on the edge strength of the camera image and the edge strength of the front view can be used to evaluate the scene quality.

[0115] In this application, by using formula (12) to take the minimum value between the edge intensity of the camera image and the edge intensity of the front view, the problem of decreased accuracy of dual-modal matching due to poor edge quality of single-modal model can be avoided (e.g., if the edge of the front view is blurry, even if the image edge is clear, there will still be a deviation during matching), thereby ensuring that the dual-modal edges meet the calibration requirements.

[0116] In some embodiments, the computer device evaluates the scene quality at the current moment based on at least two of the first score, the second score, and the third score to obtain a scene quality score. This can be achieved by weighted summation of the first score, the second score, and the third score. Optionally, the weighted summation of the first score, the second score, and the third score to obtain the scene quality score can be expressed as the following formula (13).

[0117] (13)

[0118] In formula (13), F total F1 represents the scene quality score; F2 represents the first score; F3 represents the second score; w1 represents the weight corresponding to the first score (also known as the mask quantity sufficiency weight); w2 represents the weight corresponding to the second score (also known as the spatial distribution uniformity weight); w3 represents the weight corresponding to the third score (also known as the edge sharpness weight).

[0119] Optionally, w1, w2, and w3 can be determined based on expert experience or through multiple experiments; no limitation is imposed here. For example, w1, w2, and w3 can be 0.3, 0.5, and 0.2, respectively.

[0120] The essence of formula (13) is based on the difference in contribution of indicators to the calibration quality. By assigning different weights, it achieves accurate aggregation of multi-dimensional quality. The core principle follows the general logic of the weighted average method, but it is deeply adapted for the dual-modal calibration scenario. The specific breakdown is as follows: (1) The essential logic of weighted calculation: The scenario selection for dual-modal extrinsic parameter calibration needs to comprehensively evaluate the four dimensions of feature quantity, distribution, edge, and overlap. However, the impact of each dimension on the final calibration accuracy is significantly different. For example, insufficient feature quantity will directly lead to missing matching points, while slightly blurred edges will only slightly affect the matching accuracy. Weighted calculation assigns higher weights to high-impact indicators, so that the comprehensive score can accurately reflect the actual support capability of the scenario for the calibration task, and avoids the key defects caused by the simple average method (equal weights of each indicator) being covered up or secondary problems being amplified.

[0121] In this application, the threshold can be flexibly adjusted by adopting formula (13) to adapt to the calibration requirements of multiple scenarios; different calibration scenarios (such as urban roads, highways, and parking lots) have different tolerances for indicators, and the fixed weight architecture can achieve scenario adaptation by adjusting the threshold of a single indicator; it is compatible with industry general standards, which facilitates the implementation of the technology.

[0122] In some embodiments, the computer device may also determine that the acquired camera images and point cloud data are qualified if the scene quality score is greater than or equal to a preset scene quality score, and perform subsequent steps to solve for the initial values ​​of external parameters; if the scene quality score is less than the preset scene quality score, it may determine that the acquired camera images and point cloud data are unqualified, and re-acquire camera images and re-acquire point cloud data using the LiDAR.

[0123] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating a scene quality assessment process provided in an embodiment of this application. For example... Figure 2 As shown, the computer device can determine the mask quantity sufficiency index value (or mask quantity sufficiency score), spatial distribution uniformity score, and edge sharpness score based on the camera image mask set and the front view mask set. Then, the scores of the above three dimensions are weighted and summed to obtain the scene quality score. After that, it is determined whether the scene quality score is greater than or equal to the preset score threshold. If it is, the scene quality is determined to be qualified, and the subsequent steps of solving the initial values ​​of external parameters are executed. If not, the scene quality is determined to be unqualified, and camera images are re-acquired through the camera, and point cloud data is re-acquired through the LiDAR.

[0124] This implementation method, by evaluating scene quality from the dimension of mask quantity sufficiency, not only helps avoid insufficient calibration constraints due to insufficient feature quantity, but also allows for a direct assessment of the richness of environmental features. Evaluating scene quality from the dimension of spatial distribution uniformity helps avoid calibration shifts caused by local features. Evaluating scene quality from the dimension of edge sharpness helps avoid directly amplifying matching errors due to blurred edges, thus preventing inaccurate calibration results. Therefore, by evaluating scene quality from at least two of these three dimensions, accurate scene quality evaluation results can be obtained, which is beneficial for accurately distinguishing between feature-rich and feature-scarce scenes.

[0125] In one alternative implementation, Figure 1 In step S103 of the extrinsic parameter calibration method for the sensing device shown, the computer device performs mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set. This includes: determining the first feature vector corresponding to each camera image mask in the camera image mask set, and determining the second feature vector corresponding to each front view mask in the front view mask set; determining the similarity between the i-th camera image mask and the j-th front view mask based on each first feature vector and each second feature vector; 1≤i≤N, 1≤j≤M, where i, j, N, and M are all integers; N is the total number of camera image masks, and M is the total number of front view masks; constructing a cost matrix based on the similarity, and constructing a Hungarian objective function based on the cost matrix and binary decision variables; and performing mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set with the objective of minimizing the Hungarian objective function. In this way, a front view mask can be quickly matched for each camera image mask.

[0126] In some embodiments, the computer device determines the first feature vector corresponding to each camera image mask in the camera image mask set, and determines the second feature vector corresponding to each front view mask in the front view mask set. This can be achieved by: for each camera image mask in the camera image mask set, determining the first normalized area and the first Hu moment corresponding to the camera image mask, and determining the first feature vector corresponding to the camera image mask based on the first normalized area and the first Hu moment; and for each front view mask in the front view mask set, determining the second normalized area and the second Hu moment corresponding to the front view mask, and determining the second feature vector corresponding to the front view mask based on the second normalized area and the second Hu moment. Thus, through the "normalized area + Hu moment" multi-feature fusion strategy, the normalized area eliminates the influence of resolution inconsistency, and the Hu moment provides rotation / translation invariance to cope with viewpoint scale deviations, forming a dual-modal exclusive matching logic. This improves the accuracy of the determined feature vectors, thereby improving the accuracy of subsequent mask matching.

[0127] The normalized area refers to the ratio of the number of foreground pixels of the mask to the total number of pixels in the image. This metric represents the relative size of the mask in the image.

[0128] Optionally, when determining the first normalized area corresponding to the camera image mask, the computer device may use the following formula (14).

[0129] (14)

[0130] In formula (14), This represents the normalized area corresponding to the image mask of the i-th camera; This represents the number of foreground pixels in the i-th camera image mask; These represent the width and height of the camera image, respectively. The foreground pixels refer to all pixels marked as the target object in the binary mask. In the binary mask image, these pixels typically have a value of 1 (white). They represent the isolated regions of interest segmented from the image.

[0131] Formula (14) is based on proportional calculation, which aims to transform an absolute quantity related to the absolute size of the image into a relative quantity unrelated to the absolute size of the image. The numerator is the sum of all foreground pixels (pixels with a value of 1) in the mask, directly representing the absolute size of the mask area in the image; the denominator is the total number of pixels in the entire image, representing the total canvas area of ​​the image, which is a natural benchmark for measuring the image scale; by dividing the absolute area by the total area, a ratio is obtained, which describes the proportion of the image occupied by the mask (object) in the current image, or in other words, describes the visual salience of the mask (object) in the entire image or its relative scale, i.e., the normalized area. The normalized area theoretically ranges from 0 to 1, with a value of 0 indicating no foreground (the object is infinitely small or does not exist); a value of 1 indicates that the entire image is foreground (the object fills the entire screen). In practical applications, it is usually a decimal much smaller than 1.

[0132] Among them, Hu moments serve as the primary shape descriptor in the field of computer vision, exhibiting excellent robustness to geometric transformations and being suitable for handling perspective differences caused by varying sensor positions.

[0133] Optionally, when determining the first Hu moment corresponding to the camera image mask, the computer device may include the following steps:

[0134] Step 1: Determine the centroid coordinates of the camera image mask.

[0135] Optionally, the centroid coordinates of the camera image mask can be determined by a computer device using the following formula (15).

[0136] (15)

[0137] In formula (15), m(x,y) represents the value of the camera image mask at pixel position (x,y); These represent the x-coordinate and y-coordinate of the centroid, respectively.

[0138] Step 2: Determine the central moments based on the centroid coordinates.

[0139] Optionally, when determining the central moment based on the centroid coordinates, the computer device may use the following formula (16).

[0140] (16)

[0141] In formula (16), represents the central moment, where p and q both represent the order of the central moment, p represents the order of the moment in the x-direction, and q represents the order of the moment in the y-direction; m(x,y) represents the value of the camera image mask at pixel (x,y); These represent the x-coordinate and y-coordinate of the centroid, respectively. The centroid moment is a mathematical tool used in image processing and object recognition to describe the shape of an object. Its core idea is to calculate the distribution of image features (such as brightness and mask values) relative to its centroid (i.e., the "center point" of the object).

[0142] Step 3: Normalize the central moments to obtain the normalized central moments.

[0143] Optionally, the computer device can normalize the central moments to obtain the normalized central moments using the following formula (17).

[0144] (17)

[0145] In formula (17), This represents the normalized central moments; γ represents the central moment, where p and q both represent the order of the central moment, p represents the order of the moment in the x-direction, and q represents the order of the moment in the y-direction; γ represents the normalization exponent, which is dynamically determined based on the order of the central moment. This represents a scale normalization factor specifically designed to eliminate scale effects, where... It is the absolute area of ​​the mask, that is, the total number of all foreground pixels (pixels with a value of 1) in the mask.

[0146] in, This represents the absolute size of the pixels occupied by the object described by the mask in the image, where γ is an exponent dynamically calculated based on the order of the moments to be normalized. The fundamental purpose of this design is to enable the normalization of objects by dividing by a factor of γ when the object's scale changes uniformly in the image. This mechanism precisely counteracts the scaling effect caused by the central moments. It ensures that the subsequently calculated normalized central moments and the resulting Hu moments are invariant, depending only on the geometry of the mask and not on the absolute size of the object in the image.

[0147] Step 4: Based on the normalized central moments, determine the first Hu moment corresponding to the camera image mask.

[0148] Optionally, the computer device can determine seven first Hu moments based on the normalized central moments, wherein each first Hu moment can be represented by the following formula (18).

[0149] (18)

[0150] In some embodiments, the computer device determines the first feature vector corresponding to the camera image mask based on the first normalized area and the first Hu moment, which may be done using the following formula (19).

[0151] (19)

[0152] In formula (19), The first feature vector corresponding to the mask of the i-th camera image; This represents the normalized area corresponding to the image mask of the i-th camera; This represents the 7 first Hu moments corresponding to the image mask of the i-th camera.

[0153] In some embodiments, the computer device determines the second normalized area corresponding to the front view mask in a manner that can be referred to the aforementioned method for determining the first normalized area corresponding to the camera image mask, i.e., referring to the aforementioned formula (14), which will not be repeated here.

[0154] In some embodiments, the computer device determines the second Hu moment corresponding to the front view mask in a manner similar to the aforementioned method of determining the first Hu moment corresponding to the camera image mask, i.e., by referring to the aforementioned formulas (15) to (18), which will not be repeated here.

[0155] In some embodiments, the computer device determines the second feature vector corresponding to the front view based on the second normalized area and the second Hu moment, which can be referred to the aforementioned method of the computer device determining the first feature vector corresponding to the camera image mask based on the first normalized area and the first Hu moment, i.e., referring to the aforementioned formula (18), and will not be repeated here. That is to say, the second feature vector corresponding to the j-th front view mask can be expressed as .

[0156] In some embodiments, the computer device determines the similarity between the i-th camera image mask and the j-th front view mask based on each first feature vector and each second feature vector. This can be achieved by: determining a first maximum value and a first minimum value among each first feature vector, and normalizing each first feature vector based on the first maximum value and the first minimum value to obtain normalized first feature vectors; determining a second maximum value and a second minimum value among each second feature vector, and normalizing each second feature vector based on the second maximum value and the second minimum value to obtain normalized second feature vectors; and determining the similarity between the i-th camera image mask and the j-th front view mask based on the normalized first feature vectors and the normalized second feature vectors. By normalizing the feature vectors, differences in feature dimensions can be eliminated, thereby allowing for a more accurate determination of the similarity between the i-th camera image mask and the j-th front view mask.

[0157] Optionally, the computer device normalizes each first feature vector based on the first maximum value and the first minimum value to obtain each normalized first feature vector, which can be expressed as the following formula (20).

[0158] (20)

[0159] In formula (20), The normalized first feature vector corresponding to the i-th camera image mask; The first feature vector corresponding to the mask of the i-th camera image; These represent the first maximum and first minimum values ​​in each of the first eigenvectors.

[0160] Based on the aforementioned formulas (18) and (19), the normalized first feature vector corresponding to the i-th camera image mask can be expressed as: .

[0161] Optionally, the computer device normalizes each second feature vector based on the second maximum and second minimum values ​​to obtain the normalized second feature vectors. This process can be referred to the aforementioned formula (20), and will not be elaborated here. The normalized second feature vector corresponding to the j-th front view mask can be expressed as: .

[0162] Optionally, when determining the similarity between the i-th camera image mask and the j-th front view mask based on the normalized first feature vector and the normalized second feature vector, the computer device may do so by: determining the Euclidean distance between the i-th normalized first feature vector and the j-th normalized second feature vector; normalizing the Euclidean distance to obtain the normalized Euclidean distance; and determining the similarity between the i-th camera image mask and the j-th front view mask based on the normalized Euclidean distance. Using this method, the Euclidean distance is simple and effective, and after normalization, it has a unified interpretation standard. Furthermore, compared to other metrics such as cosine similarity, the Euclidean distance is more sensitive to the absolute numerical differences of the feature vectors, making it more suitable for shape matching tasks. Therefore, based on the normalized Euclidean distance, the accurate similarity between the i-th camera image mask and the j-th front view mask can be determined simply and quickly.

[0163] When determining the Euclidean distance between the i-th normalized first eigenvector and the j-th normalized second eigenvector, the computer device can use the following formula (21).

[0164] (twenty one)

[0165] In formula (21), d(i,j) represents the Euclidean distance between the i-th normalized first eigenvector and the j-th normalized second eigenvector; The normalized first feature vector corresponding to the i-th camera image mask; The normalized second eigenvector corresponding to the j-th front view mask.

[0166] Since both the normalized first eigenvector and the normalized second eigenvector contain 8 components, the maximum possible distance is... Therefore, when computer equipment normalizes the Euclidean distance to obtain the normalized Euclidean distance, the following formula (22) can be used.

[0167] (twenty two)

[0168] In formula (22), d(i,j) represents the Euclidean distance between the i-th normalized first eigenvector and the j-th normalized second eigenvector; This represents the normalized Euclidean distance between the i-th normalized first eigenvector and the j-th normalized second eigenvector.

[0169] When determining the similarity between the i-th camera image mask and the j-th front view mask based on the normalized Euclidean distance, the computer device can use the following formula (23).

[0170] (twenty three)

[0171] In formula (23), S(i,j) represents the similarity between the i-th camera image mask and the j-th front view mask; This represents the normalized Euclidean distance between the i-th normalized first eigenvector and the j-th normalized second eigenvector.

[0172] In some embodiments, when a computer device constructs a cost matrix based on similarity, it may use the following formula (24).

[0173] (twenty four)

[0174] In formula (24), C(i,j) represents the cost matrix; S(i,j) represents the similarity between the i-th camera image mask and the j-th front view mask; This represents the normalized Euclidean distance between the i-th normalized first eigenvector and the j-th normalized second eigenvector.

[0175] In some embodiments, when a computer device constructs a Hungarian objective function based on a cost matrix and binary decision variables, the Hungarian objective function can be represented by the following formula (25).

[0176] (25)

[0177] In formula (25), E represents the Hungarian objective function; C(i,j) represents the cost matrix; N represents the total number of camera image masks; M represents the total number of front view masks; and X(i,j) represents the binary decision variable, which can be determined by the following formula (26).

[0178] (26)

[0179] Using this implementation method, mask matching can be performed quickly and accurately on each camera image mask in the camera image mask set and each front view mask in the front view mask set.

[0180] In one alternative implementation, Figure 1Step S103 of the extrinsic parameter calibration method for the sensing device shown, namely, the way in which the computer device determines the initial values ​​of the extrinsic parameters of the camera and the LiDAR based on the successfully matched mask pairs, may be as follows: Based on the pixel values ​​of the target camera image mask in the pixel coordinate system in the successfully matched mask pair, determine the two-dimensional centroid coordinates corresponding to the target camera image mask; Based on the coordinate values ​​of the target front view mask in the LiDAR coordinate system and the total number of three-dimensional points of the target front view mask, determine the three-dimensional centroid coordinates corresponding to the target front view mask; Based on the two-dimensional centroid coordinates and the three-dimensional centroid coordinates, determine the initial values ​​of the extrinsic parameters of the camera and the LiDAR.

[0181] For example, suppose the target camera image mask in a successfully matched mask pair is The target front view mask is The computer equipment is based on When determining the two-dimensional centroid coordinates corresponding to the target camera image mask based on the pixel value in the pixel coordinate system, the following formula (27) can be used.

[0182] (27)

[0183] In formula (27), (x,y) represents the image pixel coordinates, where x represents the horizontal coordinate and y represents the vertical coordinate; It means (i.e., the pixel value of the k-th camera image mask in the (x,y) pixel coordinate system; It means The two-dimensional centroid coordinates corresponding to the k-th camera image mask.

[0184] In image processing and computer vision, the two-dimensional centroid coordinates refer to a two-dimensional shape or region (such as...). The coordinates of the center of mass or geometric center of ) are as follows: The weighted average position calculated based on the image pixel values ​​of the camera image.

[0185] When a computer device determines the three-dimensional centroid coordinates corresponding to the front view mask of a target based on the coordinates of the front view mask of the target in the lidar coordinate system and the total number of three-dimensional points of the front view mask of the target, the following formula (28) can be used.

[0186] (28)

[0187] In formula (28), (x n ,y n ,z n ) represents the coordinates of the nth 3D lidar coordinate system; N represents the coordinates of the nth lidar coordinate system. (i.e., the total number of 3D point clouds corresponding to the s-th front view mask); It means The three-dimensional centroid coordinates of the three-dimensional point cloud corresponding to the s-th front view mask. The three-dimensional centroid coordinates are the arithmetic mean position of all points in the three-dimensional point cloud, representing the geometric center of the point cloud in three-dimensional space.

[0188] In some embodiments, the computer device determines the initial values ​​of the extrinsic parameters of the camera and the lidar based on the two-dimensional centroid coordinates and the three-dimensional centroid coordinates. This can be achieved by: determining a set P composed of the two-dimensional centroid coordinates and the three-dimensional centroid coordinates; and using the Efficient Perspective-n-Point (EPNP) algorithm to determine the initial values ​​of the extrinsic parameters of the camera and the lidar based on the set P.

[0189] The set P can be represented by the following formula (29).

[0190] (29)

[0191] In formula (29), It means (i.e., the two-dimensional centroid coordinates corresponding to the k-th camera image mask); It means The three-dimensional centroid coordinates corresponding to the s-th front view mask.

[0192] Using this implementation method, the initial values ​​of the external parameters of the camera and lidar can be accurately determined.

[0193] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating a process for determining initial values ​​of external parameters provided in an embodiment of this application. For example... Figure 3 As shown, the computer device first performs normalized area calculation and Hu moment calculation on each mask in the camera image mask set and the front view mask set, and determines the feature vector corresponding to each mask based on the normalized area and Hu moment corresponding to each mask; normalizes the feature vector corresponding to each mask to obtain the processed feature vector corresponding to each mask; determines the similarity between each camera image mask and each front view mask based on the processed feature vector corresponding to each camera image mask and each front view mask; performs mask matching using the Hungarian algorithm based on the similarity between each camera image mask and each front view mask to obtain successfully matched mask pairs; for the successfully matched mask pairs, determines the two-dimensional centroid coordinates corresponding to the camera image mask and the three-dimensional centroid coordinates corresponding to the front view mask in the mask pair; and uses the EPNP algorithm to determine the initial values ​​of the extrinsic parameters of the camera and the LiDAR based on the two-dimensional centroid coordinates and the three-dimensional centroid coordinates.

[0194] In one alternative implementation, Figure 1 Step S101 of the external parameter calibration method for the sensing device shown, namely, the computer device determining the set of camera image masks corresponding to the camera image acquired by the camera in the vehicle at the current moment, and determining the set of front view masks corresponding to the point cloud data acquired by the lidar in the vehicle at the current moment, can be as follows: performing image distortion correction and normalization processing on the camera image acquired by the camera in the vehicle at the current moment to obtain the processed camera image, and inputting the processed camera image into the image segmentation model to obtain the set of camera image masks; and projecting the point cloud data acquired by the lidar in the vehicle at the current moment to obtain the front view corresponding to the point cloud data, and performing normalization processing on the front view to obtain the processed front view, and inputting the processed front view into the image segmentation model to obtain the set of front view masks.

[0195] In particular, due to optical manufacturing errors, camera lenses exhibit radial and tangential distortion, causing straight lines at image edges to become curved. Uncorrected image feature point positions are inaccurate, severely impacting extrinsic parameter calibration accuracy. Geometric distortion can also affect the accuracy of segmentation boundaries. Therefore, computer equipment can perform distortion correction processing on camera images to restore them to distortion-free, ideal perspective projection images, thereby ensuring the accuracy of subsequent feature extraction.

[0196] In some embodiments, the computer device performs image distortion correction and normalization processing on the camera image captured by the camera in the vehicle at the current moment to obtain a processed camera image. This can be done by first performing image distortion correction processing on the camera image captured by the camera in the vehicle at the current moment to obtain a distorted camera image, and then performing normalization processing on the distorted camera image to obtain a processed camera image.

[0197] Optionally, the computer device can use the OpenCV image processing library to perform image distortion correction processing on the camera image captured by the camera in the vehicle at the current moment, based on the two parameters K and D provided by the camera manufacturer, to obtain the distortion-corrected camera image.

[0198] Optionally, the computer device can use the image processing library OpenCV to normalize the distorted camera image to obtain the processed camera image.

[0199] Since the front view generated by the spherical projection of the point cloud is similar to the camera image perspective, it can eliminate the differences between different modes between 2D images and 3D point clouds. Therefore, the computer equipment can project the point cloud data collected by the lidar in the vehicle at the current moment to obtain the front view corresponding to the point cloud data.

[0200] In some embodiments, when a computer device projects the point cloud data collected by the lidar in the vehicle at the current moment to obtain a front view corresponding to the point cloud data, each point cloud data P i =(x i ,y i ,z i The corresponding front view pixel (u) i ,v i ) can be determined by the following formula (30).

[0201] (30)

[0202] In formula (30), W and H represent the width and height of the front view, respectively; These represent the maximum and minimum vertical angles of the lidar, respectively. This indicates rounding down to the nearest integer.

[0203] Formula (30) essentially follows the perspective projection geometry principle of the pinhole camera model, but a key simplification is made to adapt to the requirements of extrinsic parameter calibration. This projection transforms each 3D lidar point cloud data P(x,y,z) from its Cartesian coordinate system to a spherical coordinate system centered on the lidar. The position of the point on the sphere is determined by calculating the pitch angle α=arcsin(z / r) and the azimuth angle φ=arctan2(y,x). Then, the angular coordinates (α,φ) are converted into image pixel coordinates (u,v) through linear mapping.

[0204] Since the pixel value of the point cloud front view is the distance from the point to the radar origin, in order to unify the data scale, avoid numerical overflow, and make the image and the front view comparable in numerical range, the computer device can perform normalization processing on the front view.

[0205] In some embodiments, the computer device normalizes the front view to obtain a processed front view. This can be achieved by copying the point cloud channel front view into a three-channel image and using the image processing library OpenCV to normalize the three-channel image to obtain the processed front view.

[0206] One example of an image segmentation model is the Mobile Segment Anything Model (MobileSAM). MobileSAM is a lightweight version of the Segment Anything Model (SAM), with the primary goal of significantly improving running speed and reducing resource requirements while maintaining powerful performance. While maintaining similar zero-shot segmentation capabilities to SAM, MobileSAM is approximately four times faster than the original model, enabling real-time interaction on mobile devices or edge computing devices. As an image segmentation model, MobileSAM performs excellently in image and point cloud front view mask segmentation without requiring training. Its high speed ensures efficient completion of dual-path segmentation of images and point cloud front views, meeting the timeliness requirements of the calibration process. Its lightweight nature also reduces the difficulty of deployment on resource-constrained embedded platforms (such as the computing units of autonomous vehicles). Despite having fewer model parameters, it still possesses strong zero-shot generalization capabilities and the ability to generate high-quality, well-defined masks, accurately segmenting various objects and structures in images and front views.

[0207] Optionally, the computer device can be based on the PyTorch framework, load the official MobileSAM code and pre-trained model weights, load the pre-trained model through the provided mobile_sam module, and use the OpenCV image processing library to segment the processed camera image to obtain a set of camera image masks, and segment the processed front view to obtain a set of front view masks.

[0208] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating a process for determining a camera image mask set and a front view mask set, as provided in an embodiment of this application. Figure 4 As shown, the computer equipment can determine the current camera image captured by the camera in the vehicle and the current point cloud data captured by the lidar; perform image distortion correction processing on the camera image to obtain a distorted camera image, and then normalize the distorted camera image to obtain a processed camera image; project the point cloud data to obtain a front view, and then normalize the front view to obtain a processed front view; input both the processed camera image and the processed front view into the MobileSAM model for mask segmentation inference to obtain the camera image mask set corresponding to the camera image and the front view mask set corresponding to the front view.

[0209] Using this implementation method, the camera image mask set and the front view mask set can be accurately determined, thereby providing a data foundation for subsequent steps.

[0210] The following is combined Figure 5 This paper provides an overall description of the external parameter calibration method for the sensing device provided in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating another method for calibrating the external parameters of a sensing device provided in an embodiment of this application. For example... Figure 5 As shown, the external parameter calibration method for this sensing device may include, but is not limited to, the following steps:

[0211] S501. Acquire the current camera image captured by the camera in the vehicle and the current point cloud data captured by the lidar.

[0212] S502. Preprocess the camera image and point cloud data respectively to obtain the processed camera image and the processed front view.

[0213] In one optional implementation, the computer device preprocesses the camera image and point cloud data respectively to obtain a front view corresponding to the processed camera image and the processed point cloud data. This can be done by: performing image distortion correction and normalization processing on the camera image to obtain a processed camera image; and projecting the point cloud data to obtain a front view corresponding to the point cloud data, and performing normalization processing on the front view to obtain a processed front view.

[0214] S503. Input the processed camera image and the processed front view into the ModelSAM model to obtain the camera image mask set and the front view mask set, respectively.

[0215] S504. Determine the camera image mask sufficiency index value corresponding to the camera image mask set and the front view mask sufficiency index value corresponding to the front view mask set, and determine the first score corresponding to the scene quality at the current moment based on the camera image mask sufficiency index value and the front view mask sufficiency index value.

[0216] In an alternative implementation, when the computer device determines the first score corresponding to the scene quality at the current moment based on the camera image mask sufficiency index value and the front view mask sufficiency index value, the aforementioned formula (4) may be used.

[0217] S505. Based on the center coordinates corresponding to each camera image mask in the camera image mask set, determine the camera image mask distribution entropy; and based on the center coordinates corresponding to each front view mask in the front view mask set, determine the front view mask distribution entropy; and based on the camera image mask distribution entropy and the front view mask distribution entropy, determine the second score corresponding to the scene quality at the current moment.

[0218] In an alternative implementation, when the computer device determines the second score corresponding to the scene quality at the current moment based on the camera image mask distribution entropy and the front view mask distribution entropy, the aforementioned formula (9) may be used.

[0219] S506. Based on the set of camera image masks, determine the edge intensity of the camera image, and based on the set of front view masks, determine the edge intensity of the front view, and based on the edge intensity of the camera image and the edge intensity of the front view, determine the third score corresponding to the scene quality at the current moment.

[0220] In an alternative implementation, when the computer device determines the third score corresponding to the scene quality at the current moment based on the edge intensity of the camera image and the edge intensity of the front view, the aforementioned formula (12) may be used.

[0221] It should be noted that this application does not limit the execution order of steps S504 to S506. For example, the computer device may execute step S504 first, then step S505, and finally step S506; or it may execute step S504 first, then step S506 first, and finally step S505; or it may execute at least two of steps S504 to S506 simultaneously, for example, executing steps S504 and S505 first, then executing step S506, or executing steps S505 to S506 simultaneously.

[0222] S507. Based on at least two of the first score, the second score, and the third score, evaluate the scene quality at the current moment to obtain a scene quality score.

[0223] In one alternative implementation, the computer device evaluates the scene quality at the current moment based on at least two of the first score, the second score, and the third score, and the aforementioned formula (13) can be used to obtain the scene quality score.

[0224] S508. Determine whether the scene quality score is greater than or equal to the preset score threshold. If yes, proceed to step S509; otherwise, proceed to step S501.

[0225] S509. Determine the first feature vector corresponding to each camera image mask in the camera image mask set, and determine the second feature vector corresponding to each front view mask in the front view mask set.

[0226] In one optional implementation, the computer device determines a first feature vector corresponding to each camera image mask in the camera image mask set, and a second feature vector corresponding to each front view mask in the front view mask set. This can be achieved by: for each camera image mask in the camera image mask set, determining a first normalized area and a first Hu moment corresponding to the camera image mask, and determining a first feature vector corresponding to the camera image mask based on the first normalized area and the first Hu moment; and for each front view mask in the front view mask set, determining a second normalized area and a second Hu moment corresponding to the front view mask, and determining a second feature vector corresponding to the front view mask based on the second normalized area and the second Hu moment. Thus, through a multi-feature fusion strategy of "normalized area + Hu moment," the normalized area eliminates the influence of resolution inconsistency, and the Hu moment provides rotation / translation invariance to cope with viewpoint scale deviations, forming a dual-modal exclusive matching logic. This improves the accuracy of the determined feature vectors, thereby improving the accuracy of subsequent mask matching.

[0227] S510. Based on each first feature vector and each second feature vector, determine the similarity between the i-th camera image mask and the j-th front view mask.

[0228] Where 1≤i≤N, 1≤j≤M, and i, j, N, and M are all integers; N is the total number of camera image masks, and M is the total number of front view masks.

[0229] In one optional implementation, the computer device determines the similarity between the i-th camera image mask and the j-th front view mask based on each first feature vector and each second feature vector. This can be achieved by: determining a first maximum and a first minimum value among each first feature vector, and normalizing each first feature vector based on the first maximum and the first minimum value to obtain normalized first feature vectors; determining a second maximum and a second minimum value among each second feature vector, and normalizing each second feature vector based on the second maximum and the second minimum value to obtain normalized second feature vectors; and determining the similarity between the i-th camera image mask and the j-th front view mask based on the normalized first feature vectors and the normalized second feature vectors. By normalizing the feature vectors, differences in feature dimensions can be eliminated, thereby allowing for a more accurate determination of the similarity between the i-th camera image mask and the j-th front view mask.

[0230] S511. Construct a cost matrix based on similarity, and construct a Hungarian objective function based on the cost matrix and binary decision variables.

[0231] In an alternative implementation, the Hungarian objective function may be as described in the aforementioned company (25).

[0232] S512, with the objective of determining the minimum value of the Hungarian objective function, perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set.

[0233] S513. Determine the two-dimensional centroid coordinates of the target camera image mask and the three-dimensional centroid coordinates of the target front view mask in the successfully matched mask pair.

[0234] In one alternative implementation, the computer device may use the aforementioned formula (27) to determine the two-dimensional centroid coordinates corresponding to the target camera image mask in the matched mask pair; and use the aforementioned formula (28) to determine the three-dimensional centroid coordinates corresponding to the target front view mask.

[0235] S514. Using the EPNP algorithm, determine the initial values ​​of the external parameters of the camera and LiDAR based on the two-dimensional and three-dimensional centroid coordinates.

[0236] S515. Input the initial values ​​of the external parameters into the online calibration algorithm to obtain the calibration results of the external parameters.

[0237] S516. Determine whether the external parameter calibration results meet the convergence conditions. If yes, proceed to step S517; otherwise, proceed to step S501.

[0238] S517, Output the external parameter calibration results.

[0239] In this embodiment, on the one hand, by employing a segmentation model based on a large image model, the quality prediction of scene features of the input camera image and the front view of the LiDAR is achieved: by quantitatively evaluating key indicators such as the number of targets in the vehicle environment and the clarity of target boundaries, the feature-rich scene and the feature-scarce scene are accurately distinguished. This mechanism can proactively identify and filter environmental conditions unsuitable for extrinsic parameter calibration before the calibration process starts, fundamentally avoiding the problem of decreased algorithm accuracy caused by forced calibration in scenes with insufficient features, reducing the waste of computing resources caused by environmental incompatibility, and enhancing the practicality and robustness of the system. On the other hand, the large image segmentation model used does not require any additional training for specific scenes and can directly segment images of different resolutions and the front view of the LiDAR point cloud, achieving plug-and-play functionality. This completely avoids the time and economic costs of collecting a large amount of scene data and performing tedious model training, greatly simplifying the deployment process. The same system can be adapted to different sensor configurations (such as cameras with different intrinsic parameters and LiDARs with different line counts) without adjustment, reducing the marginal cost of technology migration and large-scale application.

[0240] Furthermore, by adopting the embodiments of this application, it becomes possible to frequently perform online extrinsic parameter calibration of the vehicle throughout its life cycle, providing a foundation for continuous calibration and health monitoring capabilities for high-level autonomous driving systems. The system can periodically (e.g., each time it is powered on or during driving) automatically detect whether the extrinsic parameters have deviated due to bumps, collisions, or other reasons, and immediately provide a new high-precision initial value for system self-calibration, thereby improving the safety redundancy and long-term operational reliability of the autonomous driving system.

[0241] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0242] Based on the same inventive concept, this application also provides an external parameter calibration device for implementing the external parameter calibration method of the sensing device described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations in one or more embodiments of the external parameter calibration device for sensing devices provided below can be found in the limitations of the external parameter calibration method for sensing devices described above, and will not be repeated here.

[0243] Please see Figure 6 , Figure 6 This is a schematic diagram of the external parameter calibration device for a sensing device provided in an embodiment of this application. Figure 6 As shown, the external parameter calibration device of the sensing device may include, but is not limited to:

[0244] The mask determination module 601 is used to determine the set of camera image masks corresponding to the camera images collected by the camera in the vehicle at the current moment, and to determine the set of front view masks corresponding to the point cloud data collected by the lidar in the vehicle at the current moment.

[0245] The scene evaluation module 602 is used to evaluate the scene quality at the current moment based on the camera image mask set and the front view mask set, and obtain a scene quality score.

[0246] The mask matching module 603 is used to perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set when the scene quality score is greater than or equal to a preset score threshold.

[0247] The calibration module 604 is used to determine the initial values ​​of the extrinsic parameters of the camera and the lidar based on the successfully matched mask pairs, input the initial values ​​of the extrinsic parameters into the online calibration algorithm to obtain the extrinsic parameter calibration results, and output the extrinsic parameter calibration results when the extrinsic parameter calibration results meet the convergence conditions.

[0248] In some embodiments, when the scene evaluation module 602 evaluates the scene quality at the current moment based on the camera image mask set and the front view mask set to obtain a scene quality score, it specifically performs the following steps: determining the camera image mask sufficiency index value corresponding to the camera image mask set and the front view mask sufficiency index value corresponding to the front view mask set; and determining a first score corresponding to the scene quality at the current moment based on the camera image mask sufficiency index value and the front view mask sufficiency index value; determining the camera image mask distribution entropy based on the center coordinates corresponding to each camera image mask in the camera image mask set; and... Based on the center coordinates of each front view mask in the front view mask set, determine the front view mask distribution entropy; based on the camera image mask distribution entropy and the front view mask distribution entropy, determine the second score corresponding to the scene quality at the current moment; based on the camera image mask set, determine the camera image edge intensity, and based on the front view mask set, determine the front view edge intensity; based on the camera image edge intensity and the front view edge intensity, determine the third score corresponding to the scene quality at the current moment; based on at least two of the first score, the second score, and the third score, evaluate the scene quality at the current moment to obtain a scene quality score.

[0249] In some embodiments, when the scene evaluation module 602 is used to determine the edge intensity of a camera image based on a set of camera image masks and to determine the edge intensity of a front view based on a set of front view masks, it specifically performs the following steps: for each camera image mask in the set of camera image masks, extract the edge pixels of the camera image mask to obtain a first edge pixel set corresponding to the set of camera image masks; determine the edge intensity of a camera image based on the first edge pixel set, the total number of camera image masks, and the perimeter of the minimum bounding rectangle of each camera image mask; and for each front view mask in the set of front view masks, extract the edge pixels of the front view mask to obtain a second edge pixel set corresponding to the set of front view masks; determine the edge intensity of a front view based on the second edge pixel set, the total number of front view masks, and the perimeter of the minimum bounding rectangle of each front view mask.

[0250] In some embodiments, when the mask matching module 603 performs mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set, it specifically performs the following steps: determining the first feature vector corresponding to each camera image mask in the camera image mask set, and determining the second feature vector corresponding to each front view mask in the front view mask set; determining the similarity between the i-th camera image mask and the j-th front view mask based on each first feature vector and each second feature vector; 1≤i≤N, 1≤j≤M, where i, j, N, and M are all integers; N is the total number of camera image masks, and M is the total number of front view masks; constructing a cost matrix based on the similarity, and constructing a Hungarian objective function based on the cost matrix and binary decision variables; and performing mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set with the objective of minimizing the Hungarian objective function.

[0251] In some embodiments, when the mask matching module 603 determines the first feature vector corresponding to each camera image mask in the camera image mask set and the second feature vector corresponding to each front view mask in the front view mask set, it specifically performs the following steps: for each camera image mask in the camera image mask set, it determines the first normalized area and the first moment corresponding to the camera image mask, and determines the first feature vector corresponding to the camera image mask based on the first normalized area and the first moment; and for each front view mask in the front view mask set, it determines the second normalized area and the second moment corresponding to the front view mask, and determines the second feature vector corresponding to the front view mask based on the second normalized area and the second moment.

[0252] In some embodiments, when the mask matching module 603 determines the initial values ​​of the extrinsic parameters of the camera and the LiDAR based on the successfully matched mask pairs, it specifically performs the following steps: determining the two-dimensional centroid coordinates corresponding to the target camera image mask based on the pixel values ​​of the target camera image mask in the pixel coordinate system within the successfully matched mask pairs; determining the three-dimensional centroid coordinates corresponding to the target front view mask based on the coordinate values ​​of the target front view mask in the LiDAR coordinate system within the mask pairs and the total number of three-dimensional points of the target front view mask; and determining the initial values ​​of the extrinsic parameters of the camera and the LiDAR based on the two-dimensional and three-dimensional centroid coordinates.

[0253] In some embodiments, when the mask determination module 601 determines the set of camera image masks corresponding to the camera image captured by the camera in the vehicle at the current moment, and the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment, it specifically performs the following: performs image distortion correction and normalization processing on the camera image captured by the camera in the vehicle at the current moment to obtain a processed camera image, and inputs the processed camera image into an image segmentation model to obtain a set of camera image masks; and projects the point cloud data captured by the lidar in the vehicle at the current moment to obtain a front view corresponding to the point cloud data, performs normalization processing on the front view to obtain a processed front view, and inputs the processed front view into an image segmentation model to obtain a set of front view masks.

[0254] It is understood that the specific implementation of each module in the external parameter calibration device of the sensing device provided in this application embodiment and the beneficial effects that can be achieved can be referred to the description of the aforementioned external parameter calibration method embodiment of the sensing device, and will not be repeated here.

[0255] The modules in the external parameter calibration device of the aforementioned sensing device and the camera focal length calibration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of the vehicle control device as software, so that the processor can call and execute the corresponding operations of each module.

[0256] In one exemplary embodiment, a computer device is provided, the internal structure of which can be as shown in the figure. Figure 7As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for calibrating the external parameters of a sensing device or a method for determining the focal length of a camera. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads installed inside the computer device.

[0257] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0258] In one exemplary embodiment, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program; when the processor executes the computer program, it implements the steps in the above-described method for calibrating the extrinsic parameters of a sensing device.

[0259] In one exemplary embodiment, this application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the steps in the above-described method for calibrating the extrinsic parameters of a sensing device.

[0260] In one exemplary embodiment, this application provides a computer program product, including a computer program. When executed by a processor, the computer program implements the steps in the above-described method for calibrating the extrinsic parameters of a sensing device.

[0261] It should be noted that the data involved in this application (including but not limited to camera images, point cloud data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0262] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0263] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0264] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for calibrating the external parameters of a sensing device, characterized in that, The method includes: Determine the set of camera image masks corresponding to the camera images captured by the cameras in the vehicle at the current moment, and determine the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment. Based on the set of camera image masks, a camera image mask sufficiency index, a camera image mask distribution entropy, and a camera image edge intensity are determined. Similarly, based on the set of front view masks, a front view mask sufficiency index, a front view mask distribution entropy, and a front view edge intensity are determined. The mask sufficiency index measures the richness of scene features in the image; the mask distribution entropy measures the complexity of content within the scene of the image; and the edge intensity measures the integrity of the image edges. Based on the camera image mask sufficiency index, the camera image mask distribution entropy, the camera image edge intensity, the front view mask sufficiency index, the front view mask distribution entropy, and the front view edge intensity, the scene quality at the current moment is evaluated to obtain a scene quality score. If the scene quality score is greater than or equal to a preset score threshold, perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set, and determine the initial values ​​of the extrinsic parameters of the camera and the lidar based on the successfully matched mask pairs. The initial values ​​of the extrinsic parameters are input into the online calibration algorithm to obtain the extrinsic parameter calibration results. If the extrinsic parameter calibration results meet the convergence conditions, the extrinsic parameter calibration results are output.

2. The method of claim 1, wherein, Based on the camera image mask sufficiency index, the camera image mask distribution entropy, the camera image edge intensity, the foreground view mask sufficiency index, the foreground view mask distribution entropy, and the foreground view edge intensity, the scene quality at the current moment is evaluated to obtain a scene quality score, including: Based on the camera image mask sufficiency index value and the front view mask sufficiency index value, determine the first score corresponding to the scene quality at the current moment; Based on the camera image mask distribution entropy and the front view mask distribution entropy, a second score corresponding to the scene quality at the current moment is determined; Based on the edge intensity of the camera image and the edge intensity of the front view, a third score corresponding to the scene quality at the current moment is determined; The scene quality at the current moment is evaluated based on at least two of the first score, the second score, and the third score to obtain a scene quality score.

3. The method of claim 1, wherein, Based on the set of camera image masks, the edge intensity of the camera image is determined, including: For each camera image mask in the camera image mask set, the edge pixels of the camera image mask are extracted to obtain the first edge pixel set corresponding to the camera image mask set; based on the first edge pixel set, the total number of camera image masks, and the perimeter of the minimum bounding rectangle of each camera image mask, the edge strength of the camera image is determined. Based on the aforementioned front view mask set, the front view edge strength is determined, including: For each front view mask in the front view mask set, extract the edge pixels of the front view mask to obtain the second edge pixel set corresponding to the front view mask set; based on the second edge pixel set, the total number of the front view masks, and the perimeter of the minimum bounding rectangle of each front view mask, determine the front view edge intensity.

4. The method of claim 1, wherein, Perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set, including: Determine the first feature vector corresponding to each camera image mask in the camera image mask set, and determine the second feature vector corresponding to each front view mask in the front view mask set; Based on each of the first feature vectors and each of the second feature vectors, the similarity between the i-th camera image mask and the j-th front view mask is determined; 1≤i≤N, 1≤j≤M, where i, j, N, and M are all integers; N is the total number of camera image masks, and M is the total number of front view masks; A cost matrix is ​​constructed based on the similarity, and a Hungarian objective function is constructed based on the cost matrix and binary decision variables; To determine the minimum value of the Hungarian objective function, mask matching is performed on each camera image mask in the camera image mask set and each front view mask in the front view mask set.

5. The method of claim 4, wherein, Determining the first feature vector corresponding to each camera image mask in the camera image mask set, and determining the second feature vector corresponding to each front view mask in the front view mask set, includes: For each camera image mask in the set of camera image masks, determine the first normalized area and the first Hu moment corresponding to the camera image mask, and based on the first normalized area and the first Hu moment, determine the first feature vector corresponding to the camera image mask; and, For each front view mask in the set of front view masks, determine the second normalized area and the second Hu moment corresponding to the front view mask, and determine the second feature vector corresponding to the front view mask based on the second normalized area and the second Hu moment.

6. The method of claim 1, wherein, Based on the successfully matched mask pairs, the initial extrinsic values ​​of the camera and the lidar are determined, including: Based on the pixel values ​​of the target camera image mask in the pixel coordinate system in the successfully matched mask pair, the two-dimensional centroid coordinates of the target camera image mask are determined. Based on the coordinate values ​​of the target front view mask in the lidar coordinate system in the mask pair, and the total number of three-dimensional points of the target front view mask, the three-dimensional centroid coordinates corresponding to the target front view mask are determined. Based on the two-dimensional centroid coordinates and the three-dimensional centroid coordinates, the initial values ​​of the extrinsic parameters of the camera and the lidar are determined.

7. The method according to any one of claims 1 to 6, characterized in that, Determining the set of camera image masks corresponding to the camera images captured by the cameras in the vehicle at the current moment, and determining the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment, including: The camera image captured by the camera in the vehicle at the current moment undergoes image distortion correction and normalization processing to obtain a processed camera image. This processed camera image is then input into an image segmentation model to obtain a set of camera image masks; and... The point cloud data collected by the lidar in the vehicle at the current moment is projected to obtain the front view corresponding to the point cloud data. The front view is then normalized to obtain the processed front view. The processed front view is then input into the image segmentation model to obtain the front view mask set.

8. An external parameter calibration device for a sensing device, characterized in that, The device includes: The mask determination module is used to determine the set of camera image masks corresponding to the camera image captured by the camera in the vehicle at the current moment, and to determine the set of front view masks corresponding to the point cloud data captured by the lidar in the vehicle at the current moment. The scene evaluation module is used to determine, based on the camera image mask set, a camera image mask sufficiency index, a camera image mask distribution entropy, and a camera image edge intensity; and, based on the front view mask set, to determine, a front view mask sufficiency index, a front view mask distribution entropy, and a front view edge intensity. The mask sufficiency index measures the richness of scene features in the image; the mask distribution entropy measures the complexity of content within the scene; and the edge intensity measures the completeness of image edges. Based on the camera image mask sufficiency index, the camera image mask distribution entropy, the camera image edge intensity, the front view mask sufficiency index, the front view mask distribution entropy, and the front view edge intensity, the scene quality at the current moment is evaluated to obtain a scene quality score. The mask matching module is used to perform mask matching on each camera image mask in the camera image mask set and each front view mask in the front view mask set when the scene quality score is greater than or equal to a preset score threshold. The calibration module is used to determine the initial values ​​of the extrinsic parameters of the camera and the lidar based on the successfully matched mask pairs, input the initial values ​​of the extrinsic parameters into the online calibration algorithm to obtain the extrinsic parameter calibration results, and output the extrinsic parameter calibration results if the extrinsic parameter calibration results meet the convergence conditions.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program; when the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images