Radar-camera extrinsic parameter calibration method, electronic device and storage medium

Abstract

The application discloses a radar-camera extrinsic parameter calibration method, comprising: acquiring time-aligned camera visible light images and radar point cloud data; the fields of view of the camera and the radar overlap; generating a depth estimation image and a grayscale image of the visible light images, and generating a depth projection image and a reflection intensity projection image of the point cloud data based on a preset initial extrinsic parameter matrix; the initial extrinsic parameter matrix is used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system; calculating the structural loss between the depth estimation image and the depth projection image, and calculating the texture loss between the grayscale image and the reflection intensity projection image; iteratively optimizing the initial extrinsic parameter matrix based on the comprehensive loss until the iteration termination condition is met, to obtain an extrinsic parameter matrix meeting the iteration termination condition; the comprehensive loss is composed of the structural loss and the texture loss; the extrinsic parameter matrix meeting the iteration termination condition is determined as a target extrinsic parameter matrix; and the target extrinsic parameter matrix is used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a radar-camera extrinsic parameter calibration method, electronic device and storage medium. Background Technology

[0002] Currently, intelligent driving vehicles typically use radar and cameras as their primary sensors. Radar can acquire spatial information about the vehicle's environment, and while it is independent of ambient light, it struggles to obtain color information, and the points it captures are relatively sparse. Optical cameras, on the other hand, are highly dependent on ambient light, but they acquire relatively dense information, including color information, resulting in a larger overall data volume. Therefore, the output data from these two sensors is usually fused to achieve a comprehensive understanding of the environment. The fusion process typically involves using the 3D point cloud data acquired by the radar as points in the world coordinate system. This is then adjusted to the camera coordinate system using a radar-camera extrinsic parameter matrix, and finally transformed to the image coordinate system using a camera intrinsic parameter matrix. This ultimately achieves pixel-level alignment between the radar point cloud data and multi-source data, including camera visible light images.

[0003] In related technologies, special markers, such as checkerboard patterns or QR codes, are needed to calibrate radar-camera extrinsic parameters. The extrinsic parameter matrix is ​​solved by using the corresponding features of these markers in radar point cloud data and camera visible light images. However, this calibration method suffers from problems such as low accuracy, poor robustness, and reliance on manual intervention. Summary of the Invention

[0004] This application provides a radar-camera extrinsic parameter calibration method, electronic device, and storage medium, which can solve the problems of low accuracy, poor robustness, and reliance on manual intervention in the calibration of radar-camera extrinsic parameters in related technologies.

[0005] According to a first aspect of this application, a radar-camera extrinsic parameter calibration method is disclosed, the method comprising:

[0006] Acquire time-aligned camera visible light images and radar point cloud data; wherein the fields of view of the camera and the radar overlap.

[0007] Generate a depth estimation image and a grayscale image of the visible light image, and generate a depth projection image and a reflection intensity projection image of the point cloud data based on a preset initial extrinsic parameter matrix; wherein, the initial extrinsic parameter matrix is ​​used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system;

[0008] Calculate the structural loss between the depth estimation image and the depth projection image, and calculate the texture loss between the grayscale image and the reflection intensity projection image;

[0009] Based on the comprehensive loss, the initial extrinsic parameter matrix is ​​iteratively optimized until the iteration termination condition is met, resulting in an extrinsic parameter matrix that satisfies the iteration termination condition; wherein, the comprehensive loss consists of the structural loss and the texture loss, and the iteration termination condition includes: the comprehensive loss converges to a preset threshold or reaches a preset number of iterations;

[0010] The extrinsic parameter matrix that satisfies the iteration termination condition is determined as the target extrinsic parameter matrix; wherein, the target extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

[0011] According to a second aspect of this application, an electronic device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the radar-camera extrinsic parameter calibration method as described in the first aspect.

[0012] According to a third aspect of this application, a computer-readable storage medium is disclosed having a computer program / instructions stored thereon, which, when executed by a processor, implements the radar-camera extrinsic parameter calibration method as described in the first aspect.

[0013] According to a fourth aspect of this application, a computer program product is disclosed, comprising a computer program / instructions that, when executed by a processor, implement the radar-camera extrinsic parameter calibration method as described in the first aspect.

[0014] In this embodiment, time-aligned camera visible light images and radar point cloud data are acquired; wherein the fields of view of the camera and radar overlap; a depth estimation image and a grayscale image of the visible light image are generated, and a depth projection image and a reflection intensity projection image of the point cloud data are generated based on a preset initial extrinsic parameter matrix; wherein the initial extrinsic parameter matrix is ​​used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system; the structural loss between the depth estimation image and the depth projection image is calculated, and the texture loss between the grayscale image and the reflection intensity projection image is calculated; based on the comprehensive loss, the initial extrinsic parameter matrix is ​​iteratively optimized until the iteration termination condition is met, resulting in an extrinsic parameter matrix that meets the iteration termination condition; wherein the comprehensive loss consists of structural loss and texture loss, and the iteration termination condition includes: the comprehensive loss converges to a preset threshold or reaches a preset number of iterations; the extrinsic parameter matrix that meets the iteration termination condition is determined as the target extrinsic parameter matrix; wherein the target extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

[0015] As can be seen, in this embodiment, firstly, during the extrinsic parameter calibration process, a strict time synchronization mechanism ensures that the visible light image acquired by the camera and the point cloud data acquired by the radar are precisely aligned in the time dimension, effectively eliminating spatial misalignment caused by sensor sampling time differences and providing a reliable data foundation for subsequent calibration. Then, based on the raw data such as the visible light image and point cloud data, depth estimation images, grayscale images, and point cloud projection images are generated, and geometric and texture information are extracted respectively, establishing a direct correlation between radar and camera data. This provides a structured and comparable data format for loss calculation. Structural loss focuses on geometric alignment accuracy, ensuring the consistency of spatial structure, while texture loss strengthens the matching of materials and details, improving feature similarity. The comprehensive loss function achieves synergistic optimization of geometry and texture by balancing the weights of both, avoiding the limitations of a single loss. Finally, an iterative optimization algorithm is used to gradually adjust the extrinsic parameter matrix to minimize the comprehensive loss until the iteration termination condition is met. The extrinsic parameter matrix when the iteration termination condition is met is used as the final calibration result, ensuring that the calibration accuracy meets the high-requirement requirements of intelligent driving applications. Through techniques such as time alignment, multimodal data generation, joint optimization of structure and texture loss, and iterative optimization, high-precision and robust calibration of radar and cameras has been achieved, while reducing manual intervention and significantly improving the automation level of the calibration process. Attached Figure Description

[0016] Figure 1 This is a flowchart of a radar-camera extrinsic parameter calibration method provided in an embodiment of this application;

[0017] Figure 2 This is an example diagram of a radar-camera extrinsic parameter calibration method provided in an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of the structure of a radar-camera extrinsic parameter calibration device provided in an embodiment of this application;

[0019] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] It should be noted that all data obtained in this application, including but not limited to image data, are accessed, collected, stored, and used for subsequent analysis and processing after the user or relevant data owner has been clearly informed of the content of the data collection, the purpose of the data, the processing method, etc., and with the consent and authorization of the user or relevant data owner. Furthermore, the application can provide the user or relevant data owner with the means to access, correct, or delete the data, as well as the method to revoke consent or authorization.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0023] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, intelligent driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0024] This application provides a radar-camera extrinsic parameter calibration method, an electronic device, and a storage medium. For ease of understanding, some related concepts involved in the embodiments of this application are first introduced.

[0025] An extrinsic parameter matrix is ​​a mathematical tool used to describe spatial transformations between different coordinate systems. It is used to transform a point or vector in one coordinate system to another. In multi-sensor systems (such as radar and cameras) or robot localization, the extrinsic parameter matrix typically represents the relative position and orientation (rotation and translation) between sensors, or the robot's pose relative to the world coordinate system.

[0026] Radar-camera extrinsic calibration refers to determining the extrinsic parameter matrix between the two sensors. This extrinsic parameter matrix records the spatial transformation relationship from the radar coordinate system to the camera coordinate system, so that the data they collect can be aligned and fused in the same coordinate system. This is a key step in multimodal perception systems such as intelligent driving and robot navigation.

[0027] The extrinsic parameter matrix equals the rotation matrix (R) plus the translation vector (t), where the rotation matrix (R) is a 3×3 matrix describing the rotation relationship between the two coordinate systems, and the translation vector (t) is a 3×1 vector describing the translation relationship between the two coordinate systems. In practical applications, the extrinsic parameter matrix is ​​usually represented as a 4×4 homogeneous transformation matrix (including rotation and translation) for unified handling of three-dimensional transformations.

[0028] In this embodiment, the main focus is on calibrating the extrinsic parameter matrices of the radar and camera.

[0029] The following description, in conjunction with the accompanying drawings, introduces a radar-camera extrinsic parameter calibration method provided in an embodiment of this application.

[0030] Figure 1 This is a flowchart of a radar-camera extrinsic parameter calibration method provided in an embodiment of this application, such as... Figure 1 As shown, the method may include the following steps: step 101, step 102, step 103, step 104, step 105, step 106 and step 107.

[0031] In step 101, time-aligned camera visible light images and radar point cloud data are acquired; wherein the fields of view of the camera and radar overlap.

[0032] In this embodiment, the overlap of the camera and radar fields of view means that both the radar and the camera must be able to observe the same scene simultaneously. Field of view overlap reduces the difference in viewing angles, thus minimizing the impact of parallax on the fusion algorithm, making the spatial alignment of the camera image and the radar point cloud more accurate, and reducing calibration errors.

[0033] For example, in intelligent driving vehicles, extrinsic parameter calibration of radar and camera located on the same side is a key step to ensure accurate data fusion between the two and achieve high-precision environmental perception.

[0034] In this embodiment, the visible light image originates from a camera, and is typically an RGB image. A visible light image is a projection of a three-dimensional scene onto a two-dimensional plane, recording visual information such as the color, texture, and shape of objects in the scene.

[0035] In this embodiment, point cloud data, also known as 3D point cloud data, originates from radar, which can be lidar or millimeter-wave radar. Point cloud data is a collection of discrete points in three-dimensional space, each point containing spatial coordinates (usually X, Y, and Z coordinates) and possible other attributes such as color, reflection intensity, and timestamp. Point cloud data is used to describe the geometric shape and surface features of objects or scenes.

[0036] In this embodiment, time alignment refers to synchronizing data collected from different sensors or data sources in the time dimension to ensure that they correspond to the scene state at the same moment or within the same time window. Its core purpose is to eliminate the spatiotemporal inconsistency of data caused by differences in sensor sampling times, thereby providing a reliable foundation for subsequent fusion processing (such as calibration, perception, and modeling).

[0037] In this embodiment, by acquiring time-aligned camera visible light images and radar point cloud data, spatial registration errors caused by differences in sensor sampling times are eliminated, ensuring the accuracy of subsequent projection and loss calculation. In dynamic scenarios such as intelligent driving and robot navigation, time alignment can avoid calibration deviations caused by target movement and improve the reliability of calibration results.

[0038] In step 102, a depth estimation image and a grayscale image corresponding to the visible light image are generated.

[0039] In this embodiment, a depth estimation image is a dense depth information map inferred from a monocular or multi-view visible light image using algorithms or models. Each pixel value represents the distance (depth value) of that location in three-dimensional space relative to the camera (or reference coordinate system). The essence of a depth estimation image is to map scene information from a two-dimensional image to three-dimensional space, providing a geometric basis for computer vision tasks.

[0040] In this embodiment of the application, the purpose of generating a depth estimation image corresponding to a visible light image is to extract the geometric information of the scene from the visible light image and perform structural alignment with the depth projection image obtained by projecting point cloud data.

[0041] In this embodiment, considering that point cloud data acquired by radar is usually sparse (especially in long-range or low-beamwidth devices), while depth estimation images can generate dense depth information covering the entire image plane, thereby compensating for the sparsity of point cloud data and providing a more complete geometric description of the scene. Furthermore, the depth values ​​of point cloud data are affected by the radar's measurement accuracy and incident angle (e.g., objects with low reflectivity may lack point cloud data), while depth estimation images generate depth based on image semantics (e.g., texture, edges). The complementarity of these two approaches improves calibration robustness. By comparing the depth estimation image with the depth projection image obtained from projecting point cloud data, the differences in their geometric structures can be quantified, providing a basis for calculating structural loss and thus optimizing the extrinsic parameter matrix.

[0042] In this embodiment of the application, a grayscale image is a digital image that contains only brightness information (no color information), and the value of each pixel represents the brightness intensity at that location. It is usually represented by a single channel (grayscale channel).

[0043] In this embodiment of the application, the purpose of generating a grayscale image corresponding to a visible light image is to extract the texture information of the visible light image and align the texture with the reflection intensity projection image obtained by projecting point cloud data.

[0044] In this embodiment, considering that relying solely on depth information may lead to insufficient accuracy in calibration results in texture-rich areas (such as buildings and road signs), the texture similarity between grayscale images and reflection intensity projection maps can provide additional constraints and improve the global consistency of calibration. For example, in indoor scenes, the reflection intensity of walls may vary depending on the material, but the texture of the grayscale image (such as wallpaper patterns) can provide a stable matching basis. Furthermore, depth estimation images may generate inaccurate depths in low-texture areas (such as solid-color walls), while the texture information of grayscale images can serve as an aid, providing a basis for texture loss calculation, thereby helping to optimize the extrinsic parameter matrix.

[0045] In this embodiment, the depth estimation image and the grayscale image describe the scene from both geometric and textural dimensions, respectively. Combining them provides more comprehensive information, offering a dense and complete scene description and avoiding the limitations of a single modality. For example, during calibration, the depth estimation image may not be able to distinguish between two objects at the same distance but different colors, while the texture information in the grayscale image can help differentiate them. Furthermore, in dynamic scenarios such as autonomous driving and robot navigation, the combination of the depth estimation image and the grayscale image can reduce calibration deviations caused by target motion or sensor noise, improving calibration robustness. Through joint optimization of structural loss (depth) and texture loss (grayscale), the adjustment direction of the extrinsic parameter matrix can be more comprehensively constrained, improving calibration accuracy.

[0046] In step 103, a depth projection image and a reflection intensity projection image of the point cloud data are generated based on a preset initial extrinsic parameter matrix.

[0047] In this embodiment, the initial extrinsic parameter matrix is ​​used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

[0048] In this embodiment of the application, the initial extrinsic parameter matrix can be obtained through coarse calibration or historical calibration results.

[0049] In this embodiment, the depth projection image is a dense depth map generated by projecting point cloud data onto a two-dimensional image plane through the camera's extrinsic matrix (rotation matrix R and translation vector t) and intrinsic matrix (focal length, optical center, etc.). The value of each pixel represents the distance (depth value) from that location to the camera plane in three-dimensional space, usually in meters or millimeters.

[0050] In this embodiment of the application, the purpose of generating a depth projection image corresponding to point cloud data is to project the three-dimensional spatial coordinates of the point cloud onto the camera image plane, and to convert sparse point cloud data into dense depth information that matches the image resolution, so as to perform pixel-level alignment and comparison with the depth estimation image of the visible light image.

[0051] In this embodiment, considering that the radar coordinate system and the camera coordinate system are usually different, and the initial extrinsic parameter matrix describes the spatial relationship between the two, point cloud data can be transformed into the camera coordinate system and then mapped onto the image plane through the initial extrinsic parameter matrix and projection algorithm to achieve spatial alignment. Although point cloud data is usually sparse, the depth projection image is a dense depth map that matches the image resolution, which can be directly compared with the dense depth estimation map, thereby quantifying the error of the extrinsic parameter matrix.

[0052] In this embodiment, the Reflectance Intensity Projection Image is a two-dimensional image generated by projecting the reflection intensity values ​​in radar point cloud data onto a two-dimensional image plane through the camera extrinsic and intrinsic parameter matrices. The value of each pixel in the image represents the reflectivity of the target surface to laser in three-dimensional space at that location, and is usually represented by a normalized gray value (0-255) to reflect the material, color, or surface roughness of the object.

[0053] In this embodiment of the application, the purpose of generating the reflection intensity projection image corresponding to the point cloud data is to convert the sparse reflection intensity data into a dense grayscale image that matches the image resolution by projecting the reflection intensity value of the point cloud data onto the camera image plane, so as to facilitate texture consistency comparison with the grayscale image of the visible light image and assist in multi-sensor calibration.

[0054] In this embodiment, considering that reflection intensity reflects the ability of an object's surface to reflect laser light, and is generally related to the object's material and color, a reflection intensity projection image can be projected from a point cloud image to perform texture matching with the grayscale image of the visible light image captured by the camera. For example, a white wall has high reflection intensity (projected as bright grayscale), while a black object has low reflection intensity (projected as dark grayscale), consistent with the brightness distribution of the grayscale image. Furthermore, relying solely on depth information may lead to insufficient accuracy in low-texture areas (such as solid-color walls), while the reflection intensity projection image provides additional texture constraints, improving the robustness of the calibration. By comparing the texture similarity between the reflection intensity projection image and the grayscale image, the differences in texture features between the two can be quantified, further optimizing the extrinsic parameter matrix.

[0055] In this embodiment, the depth projection image provides geometric constraints, and the reflection intensity projection image provides texture constraints. Combining the two allows for a more comprehensive evaluation of the accuracy of the extrinsic parameter matrix. For example, in calibration optimization, simultaneously minimizing the difference between the depth projection image and the depth estimation image, as well as the difference between the reflection intensity projection image and the grayscale image, can improve calibration accuracy. Furthermore, relying solely on depth information may lead to ambiguity in geometrically symmetrical scenes (such as two objects at the same distance), while reflection intensity information can help distinguish objects with different materials. Therefore, in dynamic scenes, the joint constraints of depth and reflection intensity can reduce calibration deviations caused by target motion or sensor noise, resulting in more accurate optimization of the extrinsic parameter matrix.

[0056] In step 104, the structural loss between the depth estimation image and the depth projection image is calculated.

[0057] In this embodiment, structural loss focuses on the spatial (geometric) structural consistency of the image, rather than simple pixel value differences.

[0058] In this embodiment, the structural loss between the depth estimation image and the depth projection image can capture the similarity of their spatial layout, avoiding ambiguity or incorrect estimation caused by relying solely on pixel-level errors. In radar and camera extrinsic parameter calibration, structural loss can be used as one of the optimization objectives to adjust the extrinsic parameter matrix, aligning the geometric structure of the depth projection image and the depth estimation image.

[0059] In this embodiment, the extrinsic parameter matrix can be optimized by minimizing the structural difference between the depth estimation image and the depth projection image. For example, in autonomous vehicles, structural loss helps ensure that the depth map projected by the radar point cloud is aligned with the depth map estimated by the camera at locations such as vehicle edges and road boundaries.

[0060] In step 105, the texture loss between the grayscale image and the reflected intensity projected image is calculated.

[0061] In this embodiment, texture loss focuses on local texture features of the image (such as edges, spots, and repetition patterns), rather than global pixel values ​​or geometric structures.

[0062] In this embodiment, the texture loss between the grayscale image and the reflection intensity projection image can capture the similarity of their local textures, avoiding errors caused by relying solely on direct matching of reflection intensity or grayscale values. In radar and camera extrinsic parameter calibration, texture loss can also be used as an optimization objective, adjusting the extrinsic parameter matrix to align the texture details of the reflection intensity projection image and the grayscale image. Furthermore, reflection intensity is sensitive to material properties (e.g., high reflection intensity in metals, low reflection intensity in vegetation), while grayscale images reflect color and lighting information. Texture loss can combine both information, thus improving the accuracy of material classification.

[0063] In this embodiment, the extrinsic parameter matrix can be optimized by minimizing the texture difference between the grayscale image and the reflection intensity projection image. For example, in intelligent driving vehicles, texture loss helps ensure that the reflection intensity map projected by the radar point cloud is aligned with the grayscale image acquired by the camera in terms of texture details such as vehicle markings and road markings.

[0064] In step 106, the initial extrinsic parameter matrix is ​​iteratively optimized based on the comprehensive loss until the iteration termination condition is met, thus obtaining the extrinsic parameter matrix that meets the iteration termination condition; wherein, the comprehensive loss consists of structural loss and texture loss, and the iteration termination condition includes: the comprehensive loss converges to a preset threshold or reaches a preset number of iterations.

[0065] In this embodiment, unlike a single loss function, the comprehensive loss function consists of structural loss and texture loss, which can simultaneously optimize geometric structure and texture details, thereby significantly improving the accuracy and robustness of the calibration method.

[0066] Considering that structural loss focuses on global geometric consistency, ensuring spatial alignment of depth, edges, or surface normals between the depth projection image and the depth estimation image, while texture loss focuses on local texture details, ensuring alignment of material, color, or texture mode between the reflection intensity projection image and the grayscale image, structural loss can be used to optimize geometric structure, and texture loss can be used to optimize material details. In this embodiment, in the extrinsic parameter calibration of radar and camera, structural loss and texture loss are combined to optimize the initial extrinsic parameter matrix, achieving more accurate geometric and material alignment. For example, in intelligent driving vehicles, structural loss aligns with vehicle edges, and texture loss aligns with the reflective properties of the paint, avoiding perception errors caused by calibration errors.

[0067] In this embodiment of the application, the comprehensive loss function can be a weighted sum of structural loss and texture loss.

[0068] For example, if the structural loss is denoted as L1, the texture loss as L2, and the overall loss as L3, then L3 = a1*L1 + a2*L2, where a1 and a2 are weight values ​​that can be flexibly set according to the actual scenario.

[0069] In the embodiments of this application, in the extrinsic parameter calibration of radar and camera, the overall loss is typically used to measure the accuracy of the calibration results. The smaller the overall loss, the more accurately the spatial transformation relationship between the radar and camera is estimated. This helps ensure that the radar point cloud can be accurately projected onto the camera image, or that the target in the camera image can be accurately mapped into the radar coordinate system.

[0070] In this embodiment, the preset number of iterations refers to the maximum number of iterations. The iteration termination condition limits computational resource consumption by using the maximum number of iterations, thus avoiding infinite iteration.

[0071] In this embodiment, the iteration termination condition uses a preset threshold to determine whether the optimization has reached the expected accuracy, thus ensuring convergence.

[0072] Considering that the initial values ​​of the initial extrinsic parameter matrix are usually obtained through manual measurement, sensor installation parameters, or coarse calibration methods, which contain significant errors, the initial extrinsic parameter matrix cannot directly meet the high-precision requirements. In this embodiment, by adjusting the initial extrinsic parameter matrix, the overall loss is gradually reduced, approaching the global minimum of the loss function. Ultimately, the extrinsic parameter matrix approaches the global optimum, aligning radar and camera data spatially and in terms of features, thus ensuring the accuracy of the extrinsic parameter matrix.

[0073] In this embodiment, since the comprehensive loss function ensures the consistency of radar and camera data in geometry, texture and semantics, the inaccurate initial extrinsic parameter matrix is ​​adjusted to a high-precision extrinsic parameter matrix that meets the iteration termination condition through iterative optimization. This high-precision extrinsic parameter matrix can adapt to complex scenarios, improve the robustness and generalization ability of the system, and provide reliable multi-sensor perception capabilities for applications such as intelligent driving and robot navigation.

[0074] In step 107, the extrinsic parameter matrix that satisfies the iteration termination condition is determined as the target extrinsic parameter matrix; wherein, the target extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

[0075] In this embodiment, the extrinsic parameter matrix that satisfies the iteration termination condition is the high-precision calibration result, which is used as the target extrinsic parameter matrix of the radar-camera system. This target extrinsic parameter matrix is ​​used to fuse the data subsequently acquired by the radar and camera, and the fusion result is used to guide intelligent driving.

[0076] As can be seen from the above embodiments, in this embodiment, time-aligned camera visible light images and radar point cloud data are acquired; wherein, the fields of view of the camera and radar overlap; a depth estimation image and a grayscale image of the visible light image are generated, and a depth projection image and a reflection intensity projection image of the point cloud data are generated based on a preset initial extrinsic parameter matrix; wherein, the initial extrinsic parameter matrix is ​​used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system; the structural loss between the depth estimation image and the depth projection image is calculated, and the texture loss between the grayscale image and the reflection intensity projection image is calculated; based on the comprehensive loss, the initial extrinsic parameter matrix is ​​iteratively optimized until the iteration termination condition is met, and an extrinsic parameter matrix that meets the iteration termination condition is obtained; wherein, the comprehensive loss consists of structural loss and texture loss, and the iteration termination condition includes: the comprehensive loss converges to a preset threshold or reaches a preset number of iterations; the extrinsic parameter matrix that meets the iteration termination condition is determined as the target extrinsic parameter matrix; wherein, the target extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

[0077] As can be seen, in this embodiment, firstly, during the extrinsic parameter calibration process, a strict time synchronization mechanism ensures that the visible light image acquired by the camera and the point cloud data acquired by the radar are precisely aligned in the time dimension, effectively eliminating spatial misalignment caused by sensor sampling time differences and providing a reliable data foundation for subsequent calibration. Then, based on the raw data such as the visible light image and point cloud data, depth estimation images, grayscale images, and point cloud projection images are generated, and geometric and texture information are extracted respectively, establishing a direct correlation between radar and camera data. This provides a structured and comparable data format for loss calculation. Structural loss focuses on geometric alignment accuracy, ensuring the consistency of spatial structure, while texture loss strengthens the matching of materials and details, improving feature similarity. The comprehensive loss function achieves synergistic optimization of geometry and texture by balancing the weights of both, avoiding the limitations of a single loss. Finally, an iterative optimization algorithm is used to gradually adjust the extrinsic parameter matrix to minimize the comprehensive loss until the iteration termination condition is met. The extrinsic parameter matrix when the iteration termination condition is met is used as the final calibration result, ensuring that the calibration accuracy meets the high-requirement requirements of intelligent driving applications. Through techniques such as time alignment, multimodal data generation, joint optimization of structure and texture loss, and iterative optimization, high-precision and robust calibration of radar and cameras has been achieved, while reducing manual intervention and significantly improving the automation level of the calibration process.

[0078] In some embodiments provided in this application, step 102 may include the following steps: step 1021 and step 1022.

[0079] In step 1021, the visible light image is input into the depth estimation model, the depth of the visible light image is estimated by the depth estimation model, and the depth estimated image is output.

[0080] In this embodiment, the visible light image captured by the camera can be input into a pre-trained depth estimation model (such as a monocular depth estimation model based on a convolutional neural network or Transformer architecture). This model learns the semantic, texture, geometric and other features of the image to predict the depth value of each pixel in the visible light image and outputs a depth estimation image with the same resolution as the visible light image.

[0081] In this embodiment of the application, the depth estimation model can be trained using a large-scale depth dataset (such as NYU DepthV2, KITTI, etc.) under supervision, or by combining it with a self-supervised method (such as photometric consistency loss). This embodiment of the application does not limit this.

[0082] In step 1022, the visible light image is sequentially subjected to grayscale conversion and histogram equalization to obtain a grayscale image.

[0083] In this embodiment, the visible light image can first be converted to grayscale, transforming it into single-channel grayscale information while preserving the image's brightness information. Then, histogram equalization is performed on the single-channel grayscale information to obtain a grayscale image. This method can enhance the contrast of the grayscale image and highlight detailed features.

[0084] In this embodiment of the application, when the visible light image is an RGB image, a weighted average method such as Gray = 0.299*R + 0.587*G + 0.114*B can be used to convert the RGB image into a grayscale image.

[0085] In this embodiment, pixel values ​​can be redistributed through nonlinear stretching to make the grayscale histogram of the image more uniform, thereby achieving histogram equalization and improving the visibility of details in both dark and bright areas.

[0086] As can be seen, in this embodiment of the application, since the depth estimation model can handle complex scenes such as changes in illumination and occlusion, generating a depth estimation image through the depth estimation model can improve the robustness of the calibration process; by using grayscale conversion and histogram equalization, the usability of the image in low-light and high dynamic range scenes is enhanced, and the generalization ability of the calibration system is improved.

[0087] In some embodiments provided in this application, step 103 may include the following steps: step 1031 and step 1032.

[0088] In step 1031, histogram equalization is performed on the reflection intensity values ​​of the point cloud data.

[0089] In this embodiment of the application, histogram equalization is performed on the reflection intensity value of each point in the point cloud data, which can enhance the contrast of reflection intensity and highlight the details of different materials.

[0090] In step 1032, based on the initial extrinsic parameter matrix, the equalized point cloud data is projected onto the image plane, and the depth value and reflection intensity value of the projected points are mapped to the corresponding pixel positions on the image plane to generate a depth projection image and a reflection intensity projection image.

[0091] In this embodiment, the point cloud data is first transformed from the radar coordinate system to the camera coordinate system based on the initial extrinsic parameter matrix. Then, based on the camera's intrinsic parameter matrix, the points in the camera coordinate system are projected onto the image plane. Finally, the depth values ​​of the projected points are mapped to the corresponding pixel positions on the image plane to generate a single-channel depth projection image, and the equalized reflection intensity values ​​of the projected points are mapped to the corresponding pixel positions on the image plane to generate a single-channel reflection intensity projection image.

[0092] As can be seen, in this embodiment, histogram equalization makes the difference between weak reflections (such as distant objects) and strong reflections (such as metal surfaces) more significant, facilitating subsequent texture matching with grayscale images. After reflection intensity equalization, the reflection characteristics of different materials (such as roads, vegetation, and vehicles) are clearer, which helps with feature alignment during the calibration process. Through equalization processing, the influence of sensor noise on reflection intensity values ​​can be reduced, improving the quality of the projected image.

[0093] In some embodiments provided in this application, step 104 may include the following steps: step 1041, step 1042 and step 1043.

[0094] In step 1041, the depth estimation image and the depth projection image are divided into S×S image blocks according to the same partitioning rules; where S is an integer greater than 1.

[0095] In this embodiment, the S×S block division method can effectively utilize local information of the image, reduce the impact of inconsistency in depth estimation scale, and improve the accuracy and robustness of the algorithm.

[0096] In this embodiment, the image can be divided into S×S non-overlapping rectangular image blocks in row-major order, with each block having a size of (H / S)×(W / S); where H and W are the height and width of the image, respectively. If the image size is not divisible by S, the size can be adjusted by padding or cropping.

[0097] In this embodiment, by segmenting the data, the global depth alignment problem is transformed into a local geometric matching problem, allowing for greater focus on the consistency of local spatial structures. After segmentation, the complexity of calculating the Pearson correlation coefficient is reduced from O(H×W) to O((H / S)×(W / S)), making it suitable for real-time calibration systems. Since the depth distribution varies significantly between different regions (e.g., foreground and background), segmentation can optimize local alignment errors separately.

[0098] In step 1042, the Pearson correlation coefficient Z between each effective image block in the depth projection image and the corresponding image block in the depth estimation image is calculated, and (1-Z) is used as the structural loss of the corresponding effective image block; wherein, the number of pixels with pixel values ​​greater than 0 in the effective image block is greater than Q, where Q is an integer greater than 1.

[0099] In this embodiment of the application, for each image block in the depth projection image, effective image blocks (i.e., the number of pixels with a pixel value greater than 0 exceeds Q) are selected, and the Pearson correlation coefficient Z between the effective image block and the corresponding image block in the depth estimation image is calculated. 1-Z is used as the structural loss of the effective image block.

[0100] In this embodiment of the application, since the actual point cloud data is sometimes sparse, it is difficult to generate dense depth projection images. In order to improve robustness, only those image blocks containing more radar points are selected as effective image blocks to participate in the calculation.

[0101] In this embodiment, by excluding invalid blocks (such as all-zero or noise blocks) from interfering with the loss calculation and retaining only valid blocks for loss calculation, the robustness of the loss function can be improved. The Pearson correlation coefficient measures linear correlation and is insensitive to the absolute scale of depth values, making it suitable for depth alignment between different sensors (radar and camera). 1-Z transforms similarity into a loss value; the larger Z (higher similarity), the smaller the loss, which aligns with optimization expectations.

[0102] In step 1043, the average loss of the structural loss of all valid image blocks is calculated, and this average loss is determined as the structural loss between the depth estimation image and the depth projection image.

[0103] In this embodiment, the average loss incorporates the structural errors of all valid blocks, ensuring the accuracy of the overall geometric alignment. By focusing on local structure through block-based processing and on global consistency through average loss, local optima are avoided.

[0104] As can be seen, in this embodiment, local geometric features are extracted from the depth estimation image and the depth projection image, and the alignment error between the two is quantified by structural loss, providing accurate feedback for the optimization of the extrinsic parameter matrix. The use of effective block selection and Pearson correlation coefficient makes the loss function more robust to noise, occlusion and depth loss. This process not only improves the accuracy and robustness of calibration, but also adapts to complex scenarios such as uneven depth distribution and object occlusion through block processing and average loss mechanism, providing reliable geometric constraints for radar-camera extrinsic parameter calibration and improving the generalization ability of the calibration system.

[0105] In some embodiments provided in this application, step 105 may include the following steps: step 1051, step 1052 and step 1053.

[0106] In step 1051, a first histogram of the grayscale image and a second histogram of the reflected intensity projection image are calculated.

[0107] In this embodiment, by calculating the texture similarity loss between the grayscale image and the reflection intensity projection image, the alignment error between the two in terms of material and texture features is quantified, providing a texture dimension constraint for radar-camera extrinsic parameter calibration.

[0108] In this embodiment of the application, the number of pixels at each gray level (0-255) in the grayscale image is counted to generate a first histogram.

[0109] In this embodiment of the application, the number of pixels for each reflection intensity value (typically normalized to 0-255) in the reflection intensity projection image is counted to generate a second histogram.

[0110] In this embodiment, the histogram statistically analyzes the pixel value distribution of the grayscale image and the reflectance intensity projection image, providing a basis for subsequent texture similarity analysis. The histograms of the grayscale image and the reflectance intensity projection image respectively reflect the texture features of the camera and radar, providing a basis for cross-modal calibration.

[0111] In step 1052, the mutual information and cross-entropy of the first histogram and the second histogram are calculated.

[0112] In this embodiment, the first and second histograms can be normalized to represent probability distributions. Then, based on the normalization results, their mutual information (MI) and cross-entropy (CE) are calculated to quantify the similarity and difference of texture features.

[0113] In this embodiment, the histogram is normalized to a probability distribution, eliminating the influence of image size on the statistical results and making the texture loss in different scenes comparable.

[0114] In this embodiment, mutual information measures the statistical dependence between grayscale images and reflected intensity projection images, and is robust to interference such as illumination changes and noise.

[0115] In this embodiment, cross-entropy directly quantifies the difference between the two distributions, is sensitive to minute texture changes, and helps to capture calibration errors.

[0116] In this embodiment, mutual information and cross-entropy describe the consistency of texture features from different perspectives, providing comprehensive information for the subsequent calculation of normalized information distance.

[0117] In step 1053, the normalized information distance is calculated based on mutual information and cross-entropy, and the normalized information distance is determined as the texture loss between the grayscale image and the reflection intensity projection image.

[0118] In this embodiment, the normalized information distance combines mutual information and cross-entropy, taking into account both texture similarity (mutual information) and difference (cross-entropy), thus quantifying texture loss more comprehensively. A larger normalized information distance value indicates greater texture difference; through normalization or thresholding, it can be limited to a reasonable range, facilitating optimization.

[0119] As can be seen, in this embodiment, texture features are extracted from grayscale images and reflection intensity projection images, and the texture alignment error between the two is quantified by normalized information distance. This process not only improves the accuracy and robustness of calibration, but also adapts to complex scenarios through the combined use of mutual information and cross-entropy, providing reliable texture constraints for radar-camera extrinsic parameter calibration.

[0120] In some embodiments provided in this application, step 106 above may include the following steps: step 1061.

[0121] In step 1061, the initial extrinsic parameter matrix is ​​iteratively optimized according to the comprehensive loss and parameter search strategy until the iteration termination condition is met, thus obtaining the extrinsic parameter matrix that satisfies the iteration termination condition.

[0122] In this embodiment of the application, the parameter search strategy may include at least one of the following: grid search, random search.

[0123] In this embodiment of the application, grid search refers to traversing all possible parameter combinations within a preset parameter range with a fixed step size, calculating the loss for each set of parameters, and selecting the parameter combination that minimizes the loss as the current optimal solution.

[0124] In this embodiment, grid search can find a local optimum by exhaustively searching the parameter space, and the fixed step size of grid search can guarantee search coverage.

[0125] In this embodiment of the application, random search refers to randomly sampling parameter combinations (such as uniform distribution or Gaussian distribution) within a preset parameter range, iteratively evaluating the loss of each parameter group, and can be combined with an early stopping strategy, such as terminating if there is no improvement after F consecutive iterations, where F is an integer greater than 1.

[0126] In this embodiment, random search efficiently explores in a high-dimensional space through random sampling, and the randomness of random search can avoid getting trapped in local optima.

[0127] In this embodiment of the application, random search may include at least one of the following: coarse-grained random search and fine-grained random search.

[0128] In this embodiment of the application, coarse-grained random search refers to randomly sampling within a large range of the parameter space with a large step size or interval to quickly explore the possible optimal solution region.

[0129] In this embodiment, coarse-grained random search has the following advantages: by sampling over a large area, it can quickly locate the possible optimal solution region and avoid getting trapped in local optima; coarse-grained search has fewer sampling points, which is suitable for quickly narrowing the search range when computing resources are limited.

[0130] In this embodiment of the application, fine-grained random search refers to performing more intensive random sampling with a smaller step size or interval within the candidate region determined by coarse-grained search, thereby refining the search for the optimal solution.

[0131] In this embodiment, fine-grained random search has the following advantages: it performs dense sampling within the candidate region, refines the search for the optimal solution, and improves calibration accuracy; within the region determined by coarse-grained search, fine-grained search can more accurately approximate the global optimal solution; combining coarse-grained and fine-grained search ensures both global exploration capability and improves local optimization accuracy.

[0132] In this embodiment, coarse-grained search quickly locates candidate regions, while fine-grained search refines and optimizes the process, significantly improving optimization efficiency. In radar-camera extrinsic parameter calibration, the two-stage search strategy balances calibration accuracy and computation time, adapting to real-time requirements.

[0133] As can be seen, in this embodiment, the extrinsic parameter matrix is ​​iteratively optimized using parameter search strategies (such as grid search and random search) to minimize the overall loss, achieving high-precision extrinsic parameter calibration of the radar and camera. This process not only improves the robustness and accuracy of the calibration but also increases efficiency through automated optimization, providing reliable extrinsic parameter constraints for multi-sensor fusion. Furthermore, by further subdividing the random search into coarse-grained and fine-grained stages, candidate regions can be quickly explored globally, while the optimal solution can be searched more precisely within local regions. This strategy not only improves optimization efficiency but also enhances the algorithm's robustness to complex loss functions, providing an efficient and accurate parameter optimization method for tasks such as radar-camera extrinsic parameter calibration.

[0134] In some embodiments provided in this application, the initial extrinsic parameter matrix includes an initial rotation matrix and an initial translation vector. The initial rotation matrix is ​​used to determine the rotation relationship between the radar coordinate system and the camera coordinate system, and the initial translation vector is used to determine the position of the origin of the radar coordinate system in the camera coordinate system. Specifically, a "coarse-to-fine search method can be used to iteratively optimize the extrinsic parameter matrix." Accordingly, step 1061 above can specifically include the following steps: step 10611, step 10612, and step 10613.

[0135] In step 10611, the initial rotation matrix is ​​optimized by mesh search based on the comprehensive loss to obtain the mesh search optimized rotation matrix.

[0136] In this embodiment, the translation vector is fixed in the initial extrinsic parameter matrix, and a grid search is performed only on the parameters of the rotation matrix (such as Euler angles) to quickly locate the candidate region of the rotation parameters.

[0137] In some embodiments, step 10611 may include the following step: step 106111;

[0138] In step 106111, centered on the Euler angles (α, β, γ) of the initial rotation matrix, enumeration is performed within the Euler angle ranges [α-A, α+A], [β-A, β+A], and [γ-A, γ+A] according to a preset step size to determine the first Euler angle combination that minimizes the overall loss; this first Euler angle combination is then determined as the rotation moment optimized by the mesh search; where A is a preset value. For example, A is 10°, and the preset step size is 1°.

[0139] As can be seen, in this embodiment, the grid search traverses the parameter space with a fixed step size to quickly locate the candidate region of the rotation matrix. By optimizing only the rotation matrix and fixing the translation vector, the computational complexity can be reduced.

[0140] In step 10612, based on the comprehensive loss, coarse-grained random search optimization is performed on the extrinsic parameter matrix after grid search optimization to obtain the coarse-grained random search optimized extrinsic parameter matrix.

[0141] In this embodiment, the rotation matrix and the initial translation vector optimized by the grid search constitute the extrinsic parameter matrix after the grid search optimization.

[0142] In this embodiment of the application, based on the comprehensive loss, a coarse-grained random search optimization is performed on the extrinsic parameter matrix after grid search optimization in order to quickly explore the parameter space.

[0143] In some embodiments, step 10612 may include the following step: step 106121;

[0144] In step 106121, for the i-th coarse-grained random search, M different i-th candidate extrinsic parameter matrices are generated, and the comprehensive loss corresponding to each i-th candidate extrinsic parameter matrix is ​​calculated. If there exists an i-th candidate extrinsic parameter matrix whose comprehensive loss is less than that of the (i-1)-th candidate extrinsic parameter matrix, then the i-th candidate extrinsic parameter matrix with the smallest comprehensive loss is determined as the extrinsic parameter matrix after the i-th coarse-grained random search. The initial value of i is 1, and i is incremented by 1 and this step is repeated until i = N. The extrinsic parameter matrix after the N-th coarse-grained random search is determined as the extrinsic parameter matrix optimized by the coarse-grained random search.

[0145] In this embodiment, the rotation matrices of the M different i-th candidate extrinsic matrices are obtained by perturbing the rotation matrices of the (i-1)-th candidate extrinsic matrices together with the first angle perturbation set.

[0146] In this embodiment, the translation vectors of the M different candidate extrinsic matrices are obtained by superimposing a three-dimensional vector perturbation randomly sampled from the interval [-B, B] onto the translation vector of the (i-1)th candidate extrinsic matrix.

[0147] In this embodiment of the application, M and N are both integers greater than 1, and B is a preset value.

[0148] For example, the first angular perturbation set is {-0.5°, -0.2°, -0.1°, 0.1°, 0.2°, 0.5°}, where M = 6. 3 =216. B is 0.2m, N is 150.

[0149] In this embodiment, if the overall loss corresponding to the i-th candidate extrinsic parameter matrix is ​​less than the overall loss corresponding to the (i-1)-th candidate extrinsic parameter matrix, it indicates that the accuracy of the i-th candidate extrinsic parameter matrix is ​​higher than that of the (i-1)-th candidate extrinsic parameter matrix. In this case, the i-th candidate extrinsic parameter matrix with the smallest overall loss is determined as the extrinsic parameter matrix after the i-th coarse-grained random search. If the overall loss corresponding to the i-th candidate extrinsic parameter matrix is ​​not less than the overall loss corresponding to the (i-1)-th candidate extrinsic parameter matrix, it indicates that the accuracy of the i-th candidate extrinsic parameter matrix is ​​higher than that of the (i-1)-th candidate extrinsic parameter matrix. In this case, the extrinsic parameter matrix after the (i-1)-th fine-grained random search is still determined as the extrinsic parameter matrix after the i-th coarse-grained random search.

[0150] As can be seen, in this embodiment, the coarse-grained random search randomly samples over a large range to further explore the parameter space. This randomness avoids the fixed step size limitation of grid search, allowing for the discovery of better candidate regions. Furthermore, coarse-grained search can quickly skip low-quality regions to avoid interference from noise or occlusion.

[0151] In step 10613, based on the comprehensive loss, the coarse-grained random search optimized extrinsic parameter matrix is ​​optimized by fine-grained random search until the iteration termination condition is met, thus obtaining the extrinsic parameter matrix that meets the iteration termination condition.

[0152] In this embodiment of the application, based on the comprehensive loss, a fine-grained random search is performed on the extrinsic parameter matrix optimized by the coarse-grained random search to approximate the optimal solution more precisely.

[0153] In some embodiments, step 10613 above may include the following step: step 106131;

[0154] In step 106121, for the j-th fine-grained random search, X different j-th candidate extrinsic matrices are generated. The comprehensive loss corresponding to each j-th candidate extrinsic matrix is ​​calculated. If there exists a j-th candidate extrinsic matrix whose comprehensive loss is less than that of the (j-1)-th candidate extrinsic matrix, then the j-th candidate extrinsic matrix with the smallest comprehensive loss is determined as the extrinsic matrix after the j-th fine-grained random search. The initial value of j is 1. This step is repeated cyclically after j is incremented by 1 until j = Y. The extrinsic matrix after the Y-th fine-grained random search is determined as the extrinsic matrix that satisfies the iteration termination condition.

[0155] In this embodiment, the rotation matrices of the X different j-th candidate extrinsic matrices are obtained by perturbing the rotation matrices of the (j-1)-th candidate extrinsic matrices together with the second angle perturbation set.

[0156] In this embodiment, the translation vectors of X different j-th candidate extrinsic matrices are obtained by superimposing a three-dimensional vector perturbation randomly sampled from the interval [-K,K] onto the translation vector of the (j-1)-th candidate extrinsic matrix.

[0157] In this embodiment of the application, the accuracy of the second angle perturbation set is higher than that of the first angle perturbation set.

[0158] In this embodiment of the application, X and Y are both integers greater than 1, and K is a preset value.

[0159] For example, the second angular perturbation set is {-0.1°, -0.04°, -0.02°, 0.02°, 0.04°, 0.1°}, where X = 6. 3 =216. K is 0.2m, Y is 150.

[0160] In this embodiment, if the comprehensive loss corresponding to the j-th candidate extrinsic matrix is ​​less than the comprehensive loss corresponding to the (j-1)-th candidate extrinsic matrix, it indicates that the accuracy of the j-th candidate extrinsic matrix is ​​higher than that of the (j-1)-th candidate extrinsic matrix. In this case, the j-th candidate extrinsic matrix with the smallest comprehensive loss is determined as the extrinsic matrix after the j-th fine-grained random search. If the comprehensive loss corresponding to the j-th candidate extrinsic matrix is ​​not less than the comprehensive loss corresponding to the (j-1)-th candidate extrinsic matrix, it indicates that the accuracy of the (j-1)-th candidate extrinsic matrix is ​​higher than that of the j-th candidate extrinsic matrix. In this case, the extrinsic matrix after the (j-1)-th fine-grained random search is still determined as the extrinsic matrix after the j-th fine-grained random search.

[0161] As can be seen, in this embodiment, the fine-grained random search performs dense sampling within the candidate region, significantly improving calibration accuracy. By narrowing the search range and step size, it reduces computational load while maintaining accuracy. Furthermore, by incorporating termination conditions, unnecessary iterations are avoided, making it suitable for real-time scenarios such as autonomous driving and robot navigation.

[0162] As can be seen, in this embodiment, the coarse-to-fine optimization strategy, consisting of three stages—grid search, coarse-grained random search, and fine-grained random search—enables rapid global exploration of candidate regions and refined approximation of the optimal solution within local regions. This method not only improves optimization efficiency but also significantly enhances the algorithm's robustness to complex loss functions, providing an efficient and accurate parameter optimization scheme for tasks such as radar-camera extrinsic parameter calibration.

[0163] To facilitate a comprehensive understanding of the radar-camera extrinsic parameter calibration method provided in the embodiments of this application, combined with... Figure 2 The example diagram shown is used to illustrate this point.

[0164] First, the light-capturing image 21 acquired by the camera is input into the monocular depth estimation model 22 for depth estimation, and the monocular depth estimation image 23 is output. Then, the light-capturing image 21 is input into the grayscale conversion and histogram equalization module 24 for processing, and the grayscale image 25 is output.

[0165] Meanwhile, for the 3D point cloud data 26 acquired by the radar and processed by histogram equalization of reflection intensity, each point in the point cloud data 26 is projected onto the image plane according to the given initial extrinsic parameter matrix, and the corresponding pixel values ​​are assigned to the depth value and reflection intensity of the radar point, respectively, to obtain the depth projection image 27 and the reflection intensity projection image 28.

[0166] Next, the monocular depth estimation image 23 and the depth projection image 27 are divided into densely packed small image blocks of size S×S. The Pearson correlation coefficient Z between the two images is calculated within each small image block, and (1-Z) is used as the structural loss value for that small image block. The final structural loss value is the average of the structural loss values ​​of all valid small blocks. Here, a valid small block refers to a small image block with more than a certain value (e.g., 15) of non-zero pixels. This is because actual point clouds are sometimes sparse, making it difficult to generate dense LDPs. To improve robustness, only blocks containing a large number of radar points are selected for calculation.

[0167] Simultaneously, histograms of grayscale image 25 and reflectance intensity projection image 28 are calculated, and their mutual information M and cross-entropy H are calculated. The normalized information distance between the two is then calculated using the formula (1-M / H) as the final texture loss value. The total loss value is a weighted average of structural loss and texture loss.

[0168] Finally, the extrinsic parameters that minimize the total loss are obtained through grid search, coarse-grained random search, and fine-grained random search. The specific search process is as follows:

[0169] 1) Grid Search: For a given initial extrinsic parameter matrix, a grid search is first performed on its rotation matrix, because the rotation matrix has a more significant impact on the loss function than the translation components. Specifically, for example, the Euler angles corresponding to the rotation matrix of the initial extrinsic parameter matrix can be represented as α, β, and γ. With a specific compensation (e.g., 1°), [α-A, α+A], [β-A, β+A], and [γ-A, γ+A] are enumerated to find the combination that minimizes the loss function as the initial extrinsic parameter rotation matrix for coarse-grained random search; where A can be set according to the precision of the initial value, for example, 10°.

[0170] 2) Coarse-grained random search: Each coarse-grained random search generates 216 candidate extrinsic parameter values. The Euler angles of their rotation matrices are the current optimal Euler angles (i.e., the Euler angles that have yielded the smallest loss function value so far). All combinations of perturbations using the set {-0.5°, -0.2°, -0.1°, 0.1°, 0.2°, 0.5°} are considered, totaling 6. 3 = 216. The translation component is obtained by randomly sampling 216 three-dimensional vectors from [-B, B] and perturbing them onto the initial translation component. Here, B can be set according to the initial precision, for example, 0.2m. The loss function values ​​generated by these 216 candidate extrinsic parameters are calculated. If the minimum value among them is less than the minimum function value obtained so far, the corresponding candidate is updated as the optimal extrinsic parameter.

[0171] 3) Fine-grained random search: After N (e.g., 150) coarse-grained random searches, a final fine-grained random search is performed to further improve accuracy. The search process is similar to the coarse-grained random search, except that the perturbation set for Euler angles is replaced with a finer perturbation set {-0.1°, -0.04°, -0.02°, 0.02°, 0.04°, 0.1°}. The optimal extrinsic parameters obtained from the search will be used as the final calibration result.

[0172] In summary, for a given pair of temporally synchronized radar point cloud data and camera visible light images, a loss function is first defined to measure the alignment quality between the point cloud and the image. This function includes a structural loss term (calculated using Pearson correlation coefficients from block-based depth projection maps and monocular depth estimation maps) and a texture loss term (calculated using normalized information distance from reflection intensity projection maps and grayscale images). These two terms measure the alignment consistency between the point cloud data and the visible light image from the perspectives of geometric structure and color texture, respectively, and can serve as a good criterion for alignment quality. A coarse-to-fine search method is then designed, progressively solving for the extrinsic parameter value that minimizes this loss function through grid search and random search, resulting in the final calibration result. Validation was performed on publicly available KITTI, Waymo, and MIAS-LCEC data, demonstrating that the proposed method achieves superior performance under different sensor configurations and scenarios, especially when fusing data from multiple frames of different scenes.

[0173] Figure 3 This is a schematic diagram of the structure of a radar-camera extrinsic parameter calibration device provided in an embodiment of this application, as shown below. Figure 3 As shown, the radar-camera extrinsic parameter calibration device 300 may include: an acquisition module 301, a first generation module 302, a second generation module 303, a first calculation module 304, a second calculation module 305, an iteration module 306, and a determination module 307.

[0174] The acquisition module 301 is used to acquire time-aligned camera visible light images and radar point cloud data; wherein the fields of view of the camera and the radar overlap.

[0175] The first generation module 302 is used to generate a depth estimation image and a grayscale image corresponding to the visible light image;

[0176] The second generation module 303 is used to generate a depth projection image and a reflection intensity projection image corresponding to the point cloud data based on a preset initial extrinsic parameter matrix; wherein, the initial extrinsic parameter matrix is ​​used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system;

[0177] The first calculation module 304 is used to calculate the structural loss between the depth estimation image and the depth projection image;

[0178] The second calculation module 305 is used to calculate the texture loss between the grayscale image and the reflection intensity projection image;

[0179] The iterative module 306 is used to iteratively optimize the initial extrinsic parameter matrix based on the comprehensive loss until the iteration termination condition is met, thereby obtaining an extrinsic parameter matrix that satisfies the iteration termination condition; wherein, the comprehensive loss consists of the structural loss and the texture loss, and the iteration termination condition includes: the comprehensive loss converges to a preset threshold or reaches a preset number of iterations;

[0180] The determining module 307 is used to determine the extrinsic parameter matrix that satisfies the iteration termination condition as the target extrinsic parameter matrix; wherein, the target extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

[0181] As can be seen from the above embodiments, in this embodiment, time-aligned camera visible light images and radar point cloud data are acquired; wherein, the fields of view of the camera and radar overlap; a depth estimation image and a grayscale image of the visible light image are generated, and a depth projection image and a reflection intensity projection image of the point cloud data are generated based on a preset initial extrinsic parameter matrix; wherein, the initial extrinsic parameter matrix is ​​used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system; the structural loss between the depth estimation image and the depth projection image is calculated, and the texture loss between the grayscale image and the reflection intensity projection image is calculated; based on the comprehensive loss, the initial extrinsic parameter matrix is ​​iteratively optimized until the iteration termination condition is met, and an extrinsic parameter matrix that meets the iteration termination condition is obtained; wherein, the comprehensive loss consists of structural loss and texture loss, and the iteration termination condition includes: the comprehensive loss converges to a preset threshold or reaches a preset number of iterations; the extrinsic parameter matrix that meets the iteration termination condition is determined as the target extrinsic parameter matrix; wherein, the target extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system.

[0182] As can be seen, in this embodiment, firstly, during the extrinsic parameter calibration process, a strict time synchronization mechanism ensures that the visible light image acquired by the camera and the point cloud data acquired by the radar are precisely aligned in the time dimension, effectively eliminating spatial misalignment caused by sensor sampling time differences and providing a reliable data foundation for subsequent calibration. Then, based on the raw data such as the visible light image and point cloud data, depth estimation images, grayscale images, and point cloud projection images are generated, and geometric and texture information are extracted respectively, establishing a direct correlation between radar and camera data. This provides a structured and comparable data format for loss calculation. Structural loss focuses on geometric alignment accuracy, ensuring the consistency of spatial structure, while texture loss strengthens the matching of materials and details, improving feature similarity. The comprehensive loss function achieves synergistic optimization of geometry and texture by balancing the weights of both, avoiding the limitations of a single loss. Finally, an iterative optimization algorithm is used to gradually adjust the extrinsic parameter matrix to minimize the comprehensive loss until the iteration termination condition is met. The extrinsic parameter matrix when the iteration termination condition is met is used as the final calibration result, ensuring that the calibration accuracy meets the high-requirement requirements of intelligent driving applications. Through techniques such as time alignment, multimodal data generation, joint optimization of structure and texture loss, and iterative optimization, high-precision and robust calibration of radar and cameras has been achieved, while reducing manual intervention and significantly improving the automation level of the calibration process.

[0183] Optionally, as an embodiment, the iteration module 306 may include:

[0184] The iterative submodule is used to iteratively optimize the initial extrinsic parameter matrix according to the comprehensive loss and parameter search strategy until the iteration termination condition is met, thereby obtaining an extrinsic parameter matrix that satisfies the iteration termination condition.

[0185] The parameter search strategy includes at least one of the following: grid search, random search;

[0186] The random search includes at least one of the following: coarse-grained random search and fine-grained random search.

[0187] Optionally, as an embodiment, the initial extrinsic parameter matrix includes an initial rotation matrix and an initial translation vector, wherein the initial rotation matrix is ​​used to determine the rotation relationship between the radar coordinate system and the camera coordinate system, and the initial translation vector is used to determine the position of the origin of the radar coordinate system in the camera coordinate system;

[0188] The iterative submodule may include:

[0189] The first search unit is used to perform grid search optimization on the initial rotation matrix based on the comprehensive loss to obtain the grid search optimized rotation matrix; wherein, the grid search optimized rotation matrix and the initial translation vector constitute the grid search optimized extrinsic parameter matrix;

[0190] The second search unit is used to perform coarse-grained random search optimization on the extrinsic parameter matrix after the grid search optimization based on the comprehensive loss, so as to obtain the extrinsic parameter matrix after coarse-grained random search optimization.

[0191] The third search unit is used to perform fine-grained random search optimization on the coarse-grained random search optimized extrinsic parameter matrix based on the comprehensive loss, until the iteration termination condition is met, and thus obtain the extrinsic parameter matrix that satisfies the iteration termination condition.

[0192] Optionally, as an embodiment, the first search unit is specifically used to enumerate within the Euler angle range [α-A, α+A], [β-A, β+A], [γ-A, γ+A] with a preset step size, centered on the Euler angles (α, β, γ) of the initial rotation matrix, to determine the first Euler angle combination that minimizes the overall loss; where A is a preset value; and the first Euler angle combination is determined as the rotation matrix optimized by the grid search.

[0193] Optionally, as an embodiment, the second search unit, specifically for the i-th coarse-grained random search, generates M different i-th candidate extrinsic parameter matrices, calculates the comprehensive loss corresponding to each i-th candidate extrinsic parameter matrix, and if there exists a candidate extrinsic parameter matrix whose comprehensive loss is less than that of the (i-1)-th candidate extrinsic parameter matrix, then the candidate extrinsic parameter matrix with the smallest comprehensive loss is determined as the extrinsic parameter matrix after the i-th coarse-grained random search. The initial value of i is 1, and this step is repeated cyclically after i is incremented by 1 until i = N; the extrinsic parameter matrix after the N-th coarse-grained random search is determined as the extrinsic parameter matrix optimized by the coarse-grained random search.

[0194] The rotation matrices of the M different i-th candidate extrinsic matrices are obtained by perturbing the rotation matrices of the (i-1)-th candidate extrinsic matrices together with the first angle perturbation set.

[0195] The translation vectors of the M different i-th candidate extrinsic parameters matrices are obtained by superimposing a three-dimensional vector perturbation randomly sampled from the interval [-B, B] onto the translation vector of the (i-1)-th candidate extrinsic parameters matrices; M and N are both integers greater than 1, and B is a preset value.

[0196] Optionally, as an embodiment, the third search unit is specifically used to generate X different candidate extrinsic matrices for the j-th fine-grained random search, calculate the comprehensive loss corresponding to each candidate extrinsic matrix, and if there exists a candidate extrinsic matrix whose comprehensive loss is less than that of the (j-1)-th candidate extrinsic matrix, then the candidate extrinsic matrix with the smallest comprehensive loss is determined as the extrinsic matrix after the j-th fine-grained random search. The initial value of j is 1, and this step is repeated cyclically after j is incremented by 1 until j = Y; the extrinsic matrix after the Y-th fine-grained random search is determined as the extrinsic matrix that satisfies the iteration termination condition.

[0197] The rotation matrices of the X different j-th candidate extrinsic matrices are obtained by perturbing the rotation matrices of the (j-1)-th candidate extrinsic matrices together with the second angle perturbation set.

[0198] The translation vectors of the X different j-th candidate extrinsic matrices are obtained by superimposing a three-dimensional vector perturbation randomly sampled from the interval [-K, K] onto the translation vector of the (j-1)-th candidate extrinsic matrix; the precision of the second angle perturbation set is higher than that of the first angle perturbation set, X and Y are both integers greater than 1, and K is a preset value.

[0199] Optionally, as an embodiment, the first computing module 304 may include:

[0200] The partitioning submodule is used to divide both the depth estimation image and the depth projection image into S×S image blocks according to the same partitioning rules; where S is an integer greater than 1.

[0201] The first calculation submodule is used to calculate the Pearson correlation coefficient Z between each effective image block in the depth projection image and the corresponding image block in the depth estimation image, and to use (1-Z) as the structural loss of the corresponding effective image block; wherein, the number of pixels with pixel values ​​greater than 0 in the effective image block is greater than Q, where Q is an integer greater than 1;

[0202] The second calculation submodule is used to calculate the average loss of the structural loss of all valid image blocks;

[0203] The first determining submodule is used to determine the average loss as the structural loss between the depth estimation image and the depth projection image.

[0204] Optionally, as an embodiment, the second computing module 305 may include:

[0205] The third calculation submodule is used to calculate the first histogram of the grayscale image and the second histogram of the reflection intensity projection image;

[0206] The fourth calculation submodule is used to calculate the mutual information and cross-entropy between the first histogram and the second histogram;

[0207] The fifth calculation submodule is used to calculate the normalized information distance based on the mutual information and cross-entropy;

[0208] The second determining submodule is used to determine the normalized information distance as the texture loss between the grayscale image and the reflection intensity projection image.

[0209] Any step and specific operation in the embodiment of the radar-camera extrinsic parameter calibration method provided in this application can be completed by the corresponding module in the radar-camera extrinsic parameter calibration device. The process of the corresponding operation completed by each module in the radar-camera extrinsic parameter calibration device is referred to the process of the corresponding operation described in the embodiment of the radar-camera extrinsic parameter calibration method.

[0210] As the device embodiment is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0211] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application, such as... Figure 4 As shown, the electronic device includes a processing component 422, which further includes one or more processors, and memory resources represented by memory 432 for storing instructions executable by the processing component 422, such as application programs. The application programs stored in memory 432 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 422 is configured to execute instructions to perform the aforementioned radar-camera extrinsic parameter calibration method.

[0212] The electronic device may also include a power supply component 426 configured to perform power management of the electronic device, a wired or wireless network interface 450 configured to connect the electronic device to a network, and an input / output (I / O) interface 458. The electronic device may operate on an operating system stored in memory 432, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0213] According to yet another embodiment provided in this application, this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the radar-camera extrinsic parameter calibration method as described in any of the above embodiments.

[0214] According to yet another embodiment provided in this application, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the radar-camera extrinsic parameter calibration method as described in any of the above embodiments.

[0215] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0216] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0217] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0218] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0219] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0220] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0221] The radar-camera extrinsic parameter calibration method, electronic device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A radar-camera extrinsic parameter calibration method, characterized in that, The method includes: Acquire time-aligned camera visible light images and radar point cloud data; wherein the fields of view of the camera and the radar overlap. Generate a depth estimation image and a grayscale image of the visible light image, and generate a depth projection image and a reflection intensity projection image of the point cloud data based on a preset initial extrinsic parameter matrix; wherein, the initial extrinsic parameter matrix is ​​used to determine the spatial transformation relationship between the radar coordinate system and the camera coordinate system; Calculate the structural loss between the depth estimation image and the depth projection image, and calculate the texture loss between the grayscale image and the reflection intensity projection image; Based on the comprehensive loss, the initial extrinsic parameter matrix is ​​iteratively optimized until the iteration termination condition is met, resulting in an extrinsic parameter matrix that satisfies the iteration termination condition; wherein, the comprehensive loss consists of the structural loss and the texture loss, and the iteration termination condition includes: the comprehensive loss converges to a preset threshold or reaches a preset number of iterations; The extrinsic parameter matrix that satisfies the iteration termination condition is determined as the target extrinsic parameter matrix; wherein, the target extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the camera coordinate system; The calculation of the structural loss between the depth estimation image and the depth projection image includes: dividing both the depth estimation image and the depth projection image into S×S image blocks according to the same partitioning rule; where S is an integer greater than 1; calculating the Pearson correlation coefficient Z between each effective image block in the depth projection image and the corresponding image block in the depth estimation image, and using (1–Z) as the structural loss of the corresponding effective image block; wherein the number of pixels with a pixel value greater than 0 in the effective image block is greater than Q, where Q is an integer greater than 1; calculating the average structural loss of all effective image blocks; and determining the average loss as the structural loss between the depth estimation image and the depth projection image. The step of calculating the texture loss between the grayscale image and the reflected intensity projection image includes: calculating a first histogram of the grayscale image and a second histogram of the reflected intensity projection image; calculating the mutual information and cross-entropy between the first histogram and the second histogram; calculating the normalized information distance based on the mutual information and cross-entropy; and determining the normalized information distance as the texture loss between the grayscale image and the reflected intensity projection image.

2. The method according to claim 1, characterized in that, The process of iteratively optimizing the initial extrinsic parameter matrix based on the comprehensive loss until the iteration termination condition is met, resulting in an extrinsic parameter matrix that satisfies the iteration termination condition, includes: Based on the comprehensive loss and parameter search strategy, the initial extrinsic parameter matrix is ​​iteratively optimized until the iteration termination condition is met, thus obtaining an extrinsic parameter matrix that satisfies the iteration termination condition. The parameter search strategy includes at least one of the following: grid search, random search; The random search includes at least one of the following: coarse-grained random search and fine-grained random search.

3. The method according to claim 2, characterized in that, The initial extrinsic parameter matrix includes an initial rotation matrix and an initial translation vector. The initial rotation matrix is ​​used to determine the rotation relationship between the radar coordinate system and the camera coordinate system, and the initial translation vector is used to determine the position of the origin of the radar coordinate system in the camera coordinate system. The step of iteratively optimizing the initial extrinsic parameter matrix according to the comprehensive loss and parameter search strategy until the iteration termination condition is met, and obtaining the extrinsic parameter matrix that satisfies the iteration termination condition, includes: Based on the comprehensive loss, the initial rotation matrix is ​​optimized by a grid search to obtain the grid-optimized rotation matrix; wherein, the grid-optimized rotation matrix and the initial translation vector constitute the grid-optimized extrinsic parameter matrix; Based on the comprehensive loss, coarse-grained random search optimization is performed on the extrinsic parameter matrix after grid search optimization to obtain the coarse-grained random search optimized extrinsic parameter matrix. Based on the comprehensive loss, the coarse-grained random search optimized extrinsic parameter matrix is ​​further optimized by fine-grained random search until the iteration termination condition is met, thus obtaining an extrinsic parameter matrix that satisfies the iteration termination condition.

4. The method according to claim 3, characterized in that, The step of performing grid search optimization on the initial rotation matrix based on the comprehensive loss to obtain the grid search optimized rotation matrix includes: Centered on the Euler angles (α, β, γ) of the initial rotation matrix, enumerate within the Euler angle range [α–A, α+A], [β–A, β+A], [γ–A, γ+A] according to a preset step size to determine the first Euler angle combination that minimizes the overall loss; where A is a preset value; The first Euler angle combination is determined as the rotation matrix after grid search optimization.

5. The method according to claim 3 or 4, characterized in that, The step of performing coarse-grained random search optimization on the extrinsic parameter matrix after grid search optimization based on the comprehensive loss, to obtain the coarse-grained random search optimized extrinsic parameter matrix, includes: For the i-th coarse-grained random search, M different i-th candidate extrinsic matrices are generated. The comprehensive loss corresponding to each i-th candidate extrinsic matrix is ​​calculated. If there is an i-th candidate extrinsic matrix whose comprehensive loss is less than that of the (i-1)-th candidate extrinsic matrix, then the i-th candidate extrinsic matrix with the smallest comprehensive loss is determined as the extrinsic matrix after the i-th coarse-grained random search. The initial value of i is 1. The step is repeated cyclically after i is incremented by 1 until i=N. The extrinsic parameter matrix after the Nth coarse-grained random search is determined as the extrinsic parameter matrix optimized by the coarse-grained random search. The rotation matrices of the M different i-th candidate extrinsic matrices are obtained by perturbing the rotation matrices of the (i-1)-th candidate extrinsic matrices together with the first angle perturbation set. The translation vectors of the M different i-th candidate extrinsic parameters matrices are obtained by superimposing a three-dimensional vector perturbation randomly sampled from the interval [-B, B] onto the translation vector of the (i-1)-th candidate extrinsic parameters matrices; M and N are both integers greater than 1, and B is a preset value.

6. The method according to claim 5, characterized in that, The step of performing fine-grained random search optimization on the coarse-grained random search optimized extrinsic parameter matrix based on the comprehensive loss until the iteration termination condition is met, thereby obtaining an extrinsic parameter matrix that satisfies the iteration termination condition, includes: For the j-th fine-grained random search, X different j-th candidate extrinsic matrices are generated. The comprehensive loss corresponding to each j-th candidate extrinsic matrix is ​​calculated. If there is a j-th candidate extrinsic matrix whose comprehensive loss is less than that of the (j-1)-th candidate extrinsic matrix, then the j-th candidate extrinsic matrix with the smallest comprehensive loss is determined as the extrinsic matrix after the j-th fine-grained random search. The initial value of j is 1. This step is repeated cyclically after j is incremented by 1 until j=Y. The extrinsic parameter matrix after the Yth fine-grained random search is determined as the extrinsic parameter matrix that satisfies the iteration termination condition; The rotation matrices of the X different j-th candidate extrinsic matrices are obtained by perturbing the rotation matrices of the (j-1)-th candidate extrinsic matrices together with the second angle perturbation set. The translation vectors of the X different j-th candidate extrinsic matrices are obtained by superimposing a three-dimensional vector perturbation randomly sampled from the interval [-K, K] onto the translation vector of the (j-1)-th candidate extrinsic matrix; the precision of the second angle perturbation set is higher than that of the first angle perturbation set, X and Y are both integers greater than 1, and K is a preset value.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the radar-camera extrinsic parameter calibration method according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the radar-camera extrinsic parameter calibration method according to any one of claims 1-6.

Images