Automatic calibration system and method for end-to-end thundersight all-in-one machine based on deep learning

Abstract

The invention belongs to the technical field of intelligent traffic sensor calibration, and particularly relates to an end-to-end thunder all-in-one machine automatic calibration system and method based on deep learning, and the system comprises a data collection module, an image target detection module and a calibration network module. The system collects images and three-dimensional coordinates of a target point through a camera and a radar. Identifying a target in the image by using a deep learning model and acquiring pixel coordinates; and the calibration network learns the mapping relation between the image and the radar point, and directly outputs a high-precision calibration matrix T. According to the method, end-to-end automatic calibration in the whole process is achieved, manual arrangement of calibration plates or selection of corresponding points is not needed, efficiency is remarkably improved, and stable operation can be achieved in complex scenes such as dynamic traffic flow, night and rainy days. Through deep fusion of the high-dimensional features of the Leiyu, error accumulation of step-by-step calibration is effectively avoided, the projection error can be controlled within 5 pixels, and the high-precision sensing requirement of intelligent traffic is met.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent traffic sensor calibration, and specifically relates to an end-to-end radar-vision integrated machine automatic calibration system and method based on deep learning. BACKGROUND

[0002] The radar-vision integrated machine combines the technical advantages of radar and vision sensors, and has broad application prospects in the fields of intelligent transportation and autonomous driving. Radar has strong anti-interference and long-distance detection capability, and vision sensors can provide rich scene semantic information. The data fusion of the two can significantly improve the comprehensiveness and reliability of environmental perception. However, there are essential differences between the working principles and physical characteristics of radar and vision sensors, resulting in significant deviations in space and time dimensions of the data collected by the two. If joint calibration is not performed, it will directly affect the data fusion accuracy, and thus threaten the reliability of the subsequent decision and control modules, so joint calibration is a core prerequisite for the practical application of the radar-vision integrated machine.

[0003] At present, the mainstream radar-vision joint calibration methods in the industry can be summarized into three technical paths: the first is a feature matching-based calibration technology, which extracts corner points, edges and other features from vision data through image feature extraction algorithms (such as SIFT, Harris corner detection), synchronously extracts corresponding geometric features from radar point clouds, and then establishes the correspondence relationship between the features of the two sensors through feature matching algorithms (such as RANSAC), and finally solves the transformation matrix between the radar and vision coordinate systems; the second is a calibration technology based on a checkerboard, which uses a checkerboard with a known size as a calibration object and places it in the common field of view of the two sensors. The vision sensor determines the pixel coordinates of the corner points in the image coordinate system through a checkerboard corner detection algorithm, and the radar obtains the three-dimensional space coordinates of the checkerboard corner points through point cloud clustering and fitting. Combined with the physical size constraint of the checkerboard, the relative position and attitude of the two sensors are calculated; the third is a calibration method based on target tracking, which uses the trajectory information of moving targets (such as vehicles and pedestrians) in consecutive frames to obtain the motion trajectories of the targets in the radar point cloud and the vision image through a multi-target tracking algorithm. Based on the spatiotemporal consistency constraint, the coordinate transformation relationship between the sensors is estimated.

[0004] Although the above method can realize basic calibration in a controllable laboratory scene, it is difficult to meet the core needs of rapid, efficient and accurate in actual application, mainly existing three key short boards: First, the operation complexity is high, the method based on feature matching and the chessboard is dependent on artificial deep intervention, which needs to manually arrange the calibration object, manually select the effective feature points, and repeatedly adjust the sensor posture to optimize the matching effect. The professional skill requirement of the operator is strict, and the time cost of single calibration is usually several hours, which is a significant human and time cost; second, the environmental adaptability is weak, the traditional method is highly sensitive to scene conditions, and the sharp change of light will lead to the failure of visual feature extraction, the shielding will destroy the feature correspondence of radar and vision, and the electromagnetic noise will cause the distortion of radar point cloud. The above factors will greatly reduce the calibration accuracy, and even lead to calibration failure.

[0005] In addition, the existing method generally cannot realize real-time online calibration, and is mostly offline calibration mode, which is only used for one-time calibration during system installation or debugging stage. In actual application, the sensor will slowly drift due to vibration, temperature change, component aging and other factors, and the offline calibration result will gradually fail over time. Moreover, the traditional method lacks real-time monitoring and dynamic correction capability, and cannot meet the long-term reliable operation demand.

[0006] In summary, the existing radar-vision integrated machine joint calibration method has obvious limitations in automation degree, environmental robustness and real-time performance. It is necessary to break through the traditional technical framework and develop a calibration scheme with automation, efficiency and high precision to support the large-scale application of radar-vision integrated machine in complex actual scenes. SUMMARY

[0007] The purpose of the present application is to provide an end-to-end radar-vision integrated machine automatic calibration system and method based on deep learning to solve the problem of high operation complexity and weak environmental adaptability in the prior art.

[0008] To solve the above technical problems, the technical solution adopted by the present application is: The end-to-end radar-vision integrated machine automatic calibration system based on deep learning comprises a data acquisition module, an image target detection module and a calibration network module. The data acquisition module comprises a camera data acquisition submodule and a radar data acquisition submodule. The camera data acquisition submodule is used to acquire and output image data collected by the camera. The radar data acquisition submodule is used to acquire and output radar target point information (x, y, z), x, y and z representing three-dimensional coordinates of the target point in the radar coordinate system, respectively. The image target detection module comprises an image target detection deep learning network model, the target detection deep learning network model is used for identifying target objects from images collected by the camera data collection sub-module and determining the pixel coordinates of the target objects in the images; the input is camera image data, and the output is image target point pixel coordinates; The calibration network module comprises a deep learning network model for calibration, which learns the mapping relationship between the input image target points and radar target points, so as to output an accurate calibration matrix T.

[0009] The end-to-end radar and camera integrated machine automatic calibration method based on deep learning comprises the following steps: Step S1, collecting radar target points Pr and camera image data I in the same scene through a data collection module; Step S2, inputting the image I collected by the camera into the target detection module to output image target points P I ; Step S3, training the calibration network to accurately learn the mapping relationship between the radar target points and the image target points; Step S4, after the network training is completed, inputting the radar target points and the image target data generated in steps S1 and S2 into the calibration network, so as to obtain an accurate calibration matrix T, and using the calibration matrix T, the target point information detected by the radar can be accurately mapped into the camera image, thereby realizing the joint calibration of the radar and camera integrated machine.

[0010] According to the above technical solution, in step S1, the radar target points Specifically,

[0011] wherein, N is the number of targets in the current scene, x i , y i , and z i represent the position coordinates of the i-th radar point in the radar coordinate system.

[0012] According to the above technical solution, in step S2, the output image target points P I Specifically,

[0013] wherein, u i and v i represent the pixel positions of the i-th target in the image.

[0014] According to the above technical solution, in step S3, the training of the calibration network comprises the following steps: Step S301, setting the calibration network, including setting an input layer, a feature extraction layer, a feature fusion layer and an output layer; Step S302, prepare training data, collect radar target points and image data at the same time through step S1, and after obtaining image target points through step S2, mark the one-to-one correspondence between the radar target points and the image target points for each set of data to form the training data of the calibration network; Step S303, construct a loss function, project the radar point Pr to the image to obtain the radar projection point P proj (u proj ,v proj ); and the radar point projection point P proj and the image target point P I mean square error as the loss function L of the training network; Step S304, train the calibration network, use the loss function L to perform back propagation on the network, adjust the parameters of the network to minimize the loss function; during the training process, an optimization algorithm is used to update the network parameters iteratively until the loss function converges or a preset number of training times is reached.

[0015] According to the above technical solution, in step S301, the input layer includes a radar input layer and an image input layer; the radar input layer is used to input the radar target points into the deep learning network, and the image input layer is used to input the image target points into the network; In the radar input layer, the radar target points contain three-dimensional coordinate (x, y, z) information of each target, and the radar input data is represented as a tensor with a shape of (N, 3), where N is the number of targets detected by the radar; The image input layer: the image target detection information is represented as M two-dimensional pixel points (u, v), and the image target input data is represented as a tensor with a shape of (M, 2), where M is the number of targets detected by the camera.

[0016] According to the above technical solution, in step S301, the feature extraction layer includes a radar feature extraction layer and an image feature extraction layer, which are used to extract the radar and image target information features input by the input layer; The feature extraction layer includes multiple fully connected layers and activation functions, and the input is the (N, 3) tensor transmitted by the radar input layer; a ReLU activation function is connected after each fully connected layer, the data is expanded by the fully connected layer to enrich the network features, and the output is a (N, 128) tensor; The image feature extraction layer includes multiple fully connected layers and activation functions, and the input is the (M, 2) tensor transmitted by the image input layer; a ReLU activation function is connected after each fully connected layer, and the output is a (M, 128) tensor.

[0017] According to the technical solution, in step S301, the feature fusion layer splices the data output by the feature extraction layer; tensors of (N, 128) output by the radar feature extraction layer and tensors of (M, 128) output by the image feature extraction layer are filled and spliced in the feature dimension, and a tensor of (max(N, M), 256) is output. The output layer uses several fully connected layers to further process the fused features, and reduces the dimension of the data, and finally outputs a tensor of (1, 12), that is, the calibration matrix T.

[0018] According to the technical solution, in step S303, the radar point Pr is projected to the image to obtain a radar projection point P proj (u proj , v proj ) is specifically:

[0019] Where K is the camera intrinsic matrix, R is the rotation matrix in the calibration matrix T, t is the translation vector in the calibration matrix T, and || represents the normalization symbol. The projection point P proj of the radar point to the image and the image target point P I The mean square error is used as the loss function L of the training network, and is specifically:

[0020] Where u proj，i , v proj，i respectively represent the projection point coordinates of the radar point projected to the image, N is the number of targets detected by the radar, u i , v i represent the pixel position of the i-th target on the image.

[0021] According to the technical solution, in step S304, the loss function L is used to perform back propagation on the network, which specifically includes the following steps: Step S3041, calculating the loss function L according to the calibration matrix T output by the network; Step S3042, calculating the gradient of the loss function L on each parameter in the deep learning network, including all parameters in each layer of the calibration network; Step S3043, updating each parameter in the deep learning network according to the gradient using the Adam optimization algorithm to obtain a new deep learning network; Step S3044, inputting the training data into the updated network again, and repeating steps S3041-S3043 until the loss function L converges, then the deep learning network at this time can be used to infer the calibration matrix T.

[0022] Compared with the prior art, the present invention has the following beneficial effects: In this invention, the entire process from data acquisition to calibration matrix T output is completed end-to-end, eliminating the need for manual deployment of calibration boards and selection of corresponding points, thus significantly improving calibration efficiency. Furthermore, the system in this invention does not depend on specific calibration objects and can work stably in complex traffic scenarios such as dynamic traffic flow, rainy days, and nighttime, exhibiting significantly better robustness than traditional methods. In addition, the method in this invention uses deep learning to fuse high-dimensional features of radar and images, avoiding the accumulation of errors in step-by-step calibration. The projection error can be controlled within 5 pixels, meeting the high-precision perception requirements of intelligent transportation. Attached Figure Description

[0023] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1 This deep learning-based end-to-end automatic calibration system for radar-visual integrated machines achieves an end-to-end closed loop of data acquisition, target detection, mapping learning, and calibration output through modular design. For example... Figure 1 As shown, it specifically includes a data acquisition module, an image target detection module, and a calibration network module. The functions and relationships of each module are as follows: The data acquisition module is used to synchronously acquire raw data from the radar and camera in the integrated radar-visual system. It includes two sub-modules: Camera data acquisition submodule: Uses an industrial camera (such as a 2-megapixel CMOS camera) to acquire RGB image data of the scene, and outputs the data in a standard image file format (such as JPG / PNG). The resolution can be configured according to the actual scene (such as 1920×1080). Radar data acquisition submodule: Employs millimeter-wave radar (such as 77GHz vehicle-mounted radar) to acquire the three-dimensional coordinate information of targets in the scene and outputs a set of radar target points. Each target point is represented by a triple (x i ,y i ,z i ) indicates that x, y, and z are the horizontal, vertical, and height coordinates of the target point in the radar coordinate system, respectively, in meters (m).

[0026] An image target detection module, which takes the accurate extraction of target pixel coordinates from images as the core and adopts a deep learning target detection network as the core algorithm: Input: raw image data output by the camera data acquisition submodule; Core network: a mature target detection model such as YOLO series (e.g. YOLOv5), Faster R-CNN, etc. (the present application preferably uses YOLOv5, which balances detection speed and accuracy) can be used to adapt to traffic scenes (such as vehicle and pedestrian targets) through pre-training + fine-tuning; Output: image target point set P I where each target point is represented by a binary tuple (u, v), and u and v are the horizontal and vertical pixel coordinates of the target in the image coordinate system, respectively, with units of pixels (pixel).

[0027] A calibration network module, which is the core of end-to-end calibration, learns the mapping relationship between radar target points and image target points through a deep learning network, and outputs a calibration matrix T. The network structure includes four layers, as follows: Input layer: divided into radar input layer and image input layer, which process two types of target data respectively; Radar input layer: receives radar target point set Convert it into a tensor with shape (N, 3), where N is the number of targets detected by the radar, and 3 corresponds to the three-dimensional coordinates (x, y, z); Image input layer: receives image target point set P I Convert it into a tensor with shape (M, 2), where M is the number of targets detected by the camera, and 2 corresponds to the two-dimensional pixel coordinates (u, v); Feature extraction layer: divided into radar feature extraction layer and image feature extraction layer, used to convert the original target data into high-dimensional feature vectors; Radar feature extraction layer: composed of 3 fully connected layers + ReLU activation function, input (N, 3) tensor, expand dimension through fully connected layer (e.g. from 3D to 64D, then to 128D), finally output radar feature tensor (N, 128); Image feature extraction layer: structure consistent with radar feature extraction layer (3 fully connected layers + ReLU), input (M, 2) tensor, finally output image feature tensor (M, 128); Feature fusion layer: solves the problem of inconsistent number of radar and camera targets N≠M, through zero padding to unify the two types of feature tensors to the same length max(N, M), then concatenate in feature dimension, finally output a fusion feature tensor with shape (max(N, M), 256) (256 = 128 + 128); Output layer: composed of 2 fully connected layers, dimension reduction processing is performed on the fusion feature tensor, and finally a tensor with a shape of (1, 12) is output, which corresponds to the calibration matrix T (including a 3x3 rotation matrix R and a 3x1 translation vector t, a total of 12 parameters).

[0028] The end-to-end radar and vision integrated machine automatic calibration method based on deep learning is realized based on the above system, and the automatic calibration is completed through four steps of data acquisition, target detection, network training and calibration output. The specific steps are as follows: Step S1: synchronously acquiring radar and vision data, controlling the data acquisition module, and acquiring two types of data in the scene at the same time (timestamp error ≤10ms): Radar data: radar target point set , wherein N is the number of current scene targets (such as the number of vehicles); Camera data: original image I, the resolution is consistent with the camera acquisition submodule configuration.

[0029] Step S2: detecting image target pixel coordinates, inputting the original image I acquired in step S1 into the image target detection module, identifying the target (such as a vehicle) in the image through a deep learning network (such as YOLOv5), and outputting an image target point set , wherein u i ,v i are the pixel coordinates of the i-th target.

[0030] Step S3: training the calibration network, training the calibration network to learn the mapping relationship between the radar target point and the image target point through the labeled training data, which is specifically divided into four sub-steps: S301: setting the calibration network structure, configuring the network according to the structure of the input layer, the feature extraction layer, the feature fusion layer and the output layer, wherein the number of neurons in the fully connected layer and the type of activation function can be adjusted according to the amount of training data (the present application defaults to ReLU activation function, and the number of neurons in the fully connected layer is 64, 128, 256 and 12 in turn).

[0031] S302: preparing training data, collecting at least 1000 sets of synchronous radar and vision data (covering different traffic scenes: such as peak traffic, rainy day, night), executing steps S1-S2 on each set of data to obtain the radar target point and the paired data of the image target point P I , as the training set of the calibration network (divided into training set and validation set according to 8:2).

[0032] S303: constructing a loss function, the core of the loss function is to minimize the error of the radar point projection to the image, and the specific calculation logic is as follows: Based on the calibration matrix T(R, t) output by the calibration network, and combined with the known camera intrinsic parameter matrix K (which is fixed and constant at the time of camera factory calibration), the radar target point is... Projecting onto the image yields the projection point. The projection formula is:

[0033] Where z' is the depth normalization factor after projection. This indicates a normalization operation; With projection point With image target point P I The mean squared error is used as the loss function L, and the formula is:

[0034] Where N is the number of radar target points. Let u be the projected pixel coordinates of the i-th radar point. i v i These are the pixel coordinates of the target point in the corresponding image.

[0035] S304: Iteratively train the network using the Adam optimization algorithm (initial learning rate set to 0.001, decaying to 0.5 every 100 rounds), minimizing the loss function L through backpropagation. Specific steps: Input the training set data into the calibration network, and calculate the output calibration matrix T and loss function L; Calculate the gradient of L with respect to all network parameters (such as fully connected layer weights and biases); Based on the gradient, update the network parameters using the Adam algorithm; Repeat the above steps until the loss function L converges on the validation set (e.g., L < 0.001) or the preset number of training rounds (e.g., 500 rounds) is reached. Then stop training and save the completed calibration network.

[0036] Step S4: Output calibration matrix T and perform joint calibration by synchronizing the data from the radar-visual integrated machine to be calibrated (from step S1). Step S2 P I The trained calibration network is input, and the network directly outputs a precise calibration matrix T (containing R and t). Using this calibration matrix T, any target point (x, y, z) detected by the radar can be mapped onto the camera image, realizing joint calibration of radar and camera and meeting the requirements of radar-visual fusion perception.

[0037] Example 2 To make the technical solution of this invention clearer, the implementation process is explained in detail below with reference to a specific application scenario (calibration of integrated radar-visual cameras at urban intersections): Step 1: Equipment and data preparation, including: Hardware device: Choose the radar-vision integrated machine (containing 200 million pixel CMOS camera, resolution 1920x1080; 77GHz millimeter wave radar, detection distance 0-200m), installed in the city intersection crossbar (height 6m, depression angle 15°); Camera intrinsic parameters: Obtain the intrinsic matrix K through the camera factory calibration; Training data collection: Collect 3 consecutive days of synchronous radar-vision data (including early peak 7:00-9:00, evening peak 17:00-19:00, rainy day, night), a total of 1200 groups, each group contains 1 frame image and corresponding radar point cloud.

[0038] Step two, image target detection module deployment, specifically: Choose YOLOv5s model as the target detection network, based on COCO dataset pre-training, using urban intersection vehicle dataset (containing 10000 labeled images) fine-tuning, so that the vehicle detection accuracy ≥95%; Deploy the fine-tuned YOLOv5s model to the image target detection module, input the camera image, and output the pixel coordinates P I of the vehicle, detection speed ≥30fps.

[0039] Step three, calibration network training, specifically: Network configuration: Build a calibration network using the PyTorch framework, the input layer receives (N,3) radar tensor and (M,2) image tensor; the feature extraction layer is 3 fully connected layers (neuron number 64, 128, 128); the feature fusion layer is spliced after zero padding to unify the length; the output layer is 2 fully connected layers (neuron number 256, 12); Training parameters: batch size is set to 16, training rounds are 500, Adam optimizer initial learning rate is 0.001, and every 100 rounds are attenuated by 0.5; Training results: After 500 rounds of training, the validation set loss function \( L = 0.0008 \), which meets the convergence condition, and the network model is saved.

[0040] Step four, calibration verification, specifically: select 100 groups of test data (sunny day, rainy day, night scene) that do not participate in training, input the trained calibration network, and output the calibration matrix T; use T to project the radar target point to the image, and calculate the average error between the projected point and the image target point: sunny scene error 2.1 pixels, rainy scene error 3.5 pixels, night scene error 3.8 pixels, all meet the accuracy requirements of intelligent traffic radar-vision fusion (≤5 pixels).

[0041] Example three This embodiment is a further refinement of Embodiment 2. This embodiment provides a detailed explanation of the loss function calculation module. The loss function is the core guiding principle for the calibration network to learn the mapping relationship between radar target points and image target points, and its calculation accuracy directly affects the accuracy of the calibration matrix T. In this invention, the calculation of the loss function needs to solve three key problems: projection of 3D radar points to 2D image points, target matching, and error quantization. The specific process is as follows: Step 1, firstly, the three-dimensional coordinates of the radar target point in the radar coordinate system are: =(x, y, z), which needs to be projected onto the image coordinate system through the calibration matrix T (rotation matrix R, translation vector t, and camera intrinsic parameter matrix K) output by the calibration network to obtain the projection points. Specific steps: Step 101: Transformation from radar coordinate system to camera coordinate system.

[0042] The core function of the calibration matrix T is to establish a rigid transformation relationship between the radar coordinate system and the camera coordinate system. This is achieved through the rotation matrix R (3×3) and the translation vector t (3×1), which transforms the radar points... Convert to 3D point P in camera coordinate system c :

[0043] Where, x c ,y c ,z c Let z be the three-dimensional coordinates of the target point in the camera coordinate system. c For the depth from the camera to the target, z must be guaranteed c >0, otherwise the target is outside the camera's field of view and is not included in the loss calculation.

[0044] Step 102: Perspective projection from camera coordinate system to image coordinate system The camera intrinsic parameter matrix K (3×3) is obtained from the camera's factory calibration and has the following form:

[0045] Where f x ,f y For camera focal length (in pixels), (c x ,c y (P) represents the coordinates of the image center point, and the 3D point (P) in the camera coordinate system is... c Projection to image pixel coordinates:

[0046] Divide both sides by z c (Normalization operation) yields:

[0047]

[0048] Step 2, target matching, solve the problem (N≠M): the number of radar detected targets N and the number of camera detected targets M are often not equal (such as radar missed detection, camera occlusion), which needs to be determined through target matching which radar points and image points are the same physical target, and only the error of the matched target pair is calculated. The present application adopts the "distance priority matching" strategy: Step 201: Calculate the distance of the candidate matching pair. For each radar point , calculate the Euclidean distance between its projection point and all image target points :

[0049] Step 202: Screen effective matching pairs. Set the distance threshold d th (such as 10 pixels, adjusted according to the scene dynamics), if d i,j ≤d th , it is considered that and are the same target; for unmatched radar points / image points (such as distance exceeding d th ), do not count into loss calculation (avoid noise interference).

[0050] Step 203: De-duplication processing. If a radar point matches multiple image points (or vice versa), keep the pair with the smallest distance as the effective matching pair, and ensure that each target participates in error calculation only once.

[0051] Step 3, mean square error (MSE) calculation. For all effective matching pairs, calculate the pixel coordinate error between the projection point and the image target point, and take the average value as the loss function L:

[0052] Where K is the number of effective matching pairs (K≤min(N,M), is the kth matched radar projection point, (u k ,v k ) is the corresponding image target point.

[0053] The embodiment also provides a specific implementation of back propagation. Back propagation is an iterative process of calibrating the network from random parameters to optimal parameters, which needs to go through a closed loop of data loading, forward propagation, loss calculation, reverse optimization, and verification monitoring to realize the convergence of the loss function. The back propagation of the present application is divided into two parts: training preparation and iterative training process, which are as follows: Step 1, training preparation. Data preprocessing: preprocess the training data set (radar point set , image point set PI Standardize the paired data: Radar point coordinates: Subtract the mean of all radar points in the training set Divide by standard deviation This avoids bias in network learning caused by differences in coordinate value ranges (e.g., (x,y) is in meters and has a larger value; z may be smaller).

[0054] Image point coordinates: Normalize the pixel coordinates (u, v) to the [0,1] interval (e.g., u'=u / W, v'=v / H, where (W,H) are the image width and height) to reduce the influence of the absolute value of the coordinates on the network parameters.

[0055] Step 2, Network Initialization; A calibration network is built using the PyTorch framework. The parameters for each layer are initialized as follows: Fully connected layer weights: Kaiming normal initialization is used (to adapt to the characteristics of the ReLU activation function); bias terms are initialized to 0.

[0056] Step 3, optimizer and learning rate configuration.

[0057] Optimizer: Adam optimizer was selected (fast convergence speed, robust to noisy data), initial learning rate α = 0.001, momentum parameters β1 = 0.9, β2 = 0.999, weight decay coefficient λ = 1e-5 (to prevent overfitting).

[0058] Learning rate scheduling: A cosine annealing strategy is used, and the learning rate is decayed to 0.5 of the current value (i.e., α) every 100 epochs. epoch =0.5·α epoch This avoids excessively high learning rates that could cause oscillations or excessively low learning rates that could lead to convergence stagnation.

[0059] Step 4, Iterative Training Process (Operations within a Single Epoch). Let the total number of samples in the training set be S, and the batch size be B (in this invention, B = 16). Each epoch contains S / B iteration steps. The specific process is as follows: Step 401, Data Loading and Target Matching. B groups of samples are randomly selected from the training set, each group containing a radar point set. and image point set Perform target matching on each group of samples to obtain K valid matching pairs.

[0060] Step 402, forward propagation (calculate calibration matrix T). The radar point set... Convert to an (N, 3) tensor, image point set P I Convert to an (M,2) tensor and input it into the calibration network: The feature extraction layer outputs (N, 128) and (M, 128) feature tensors; the feature fusion layer fills zeros to unify the tensors into (max(N, M), 256) tensors; The output layer outputs (1, 12) tensors, which are parsed into a rotation matrix R (first 9 parameters) and a translation vector t (last 3 parameters).

[0061] Step 403: loss calculation and back propagation. For the output R and t, the projection point P of each valid matching pair is calculated according to the radar point projection step proj ; The average loss L of the current batch is calculated by substituting the loss function formula batch .

[0062] Perform back propagation: call L batch .backward(), calculate the gradient of the loss function to all network parameters (fully connected layer weights, bias) ( ) for the network parameter set.

[0063] Step 404: parameter update and gradient zeroing.

[0064] The optimizer updates the parameters according to the gradient: ; Zero the gradient: call optimizer.zero_grad() to avoid gradient accumulation affecting the calculation of the next batch.

[0065] Step 405: validation set monitoring (performed once every 10 epochs). Extract samples from the validation set (independent of the training set), repeat steps 401-403 (without back propagation), and calculate the validation loss L val : If L val has not decreased (by less than 1e-4) for 3 consecutive validations (30 epochs), trigger the early stopping strategy and stop training (to prevent overfitting); If L val is the current minimum value, save the current network parameters as the "optimal model".

[0066] Step 5, training termination condition.

[0067] The training loop terminates when one of the following conditions is met: The validation loss L val ≤1e-3 (satisfies the calibration accuracy requirement, with a projection error ≤3 pixels); The training round reaches the preset maximum value (such as 500 epochs), and L valConverge (loss difference of adjacent 10 epochs <1e-5).

[0068] Through the above loss function calculation and training cycle process, the mapping relationship between the radar and the camera can be learned stably, and a high-precision calibration matrix T can be output, thereby meeting the actual needs of the joint calibration of the radar and camera.

[0069] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between or among such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0070] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not intended to limit the present application, although the above embodiments of the present application are described in detail, those skilled in the art can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based end-to-end radar and vision integrated machine automatic calibration system, characterized in that: The system comprises a data acquisition module, an image target detection module, and a calibration network module. The data acquisition module comprises a camera data acquisition submodule and a radar data acquisition submodule. The camera data acquisition submodule is configured to acquire and output image data collected by a camera. The radar data acquisition submodule is configured to acquire and output radar target point information (x, y, z), wherein x, y, and z represent three-dimensional coordinates of a target point in a radar coordinate system.

2. The end-to-end radar and vision integrated machine automatic calibration method based on deep learning, characterized in that: The image target detection module comprises an image target detection deep learning network model. Step S1, collect radar target points in the same time scene through a data collection module and camera image data I; Step S2, input the image I collected by the camera into the target detection module, and output the image target point P I ; The target detection deep learning network model is configured to identify a target object from an image collected by the camera data acquisition submodule and determine pixel coordinates of the target object in the image. The input is camera image data, and the output is image target point pixel coordinates.

3. The deep learning-based end-to-end radar and vision all-in-one automatic calibration method according to claim 2, characterized in that: In step S1, the radar target points collected Specifically: where N is the number of targets in the current scene, x i ,y i ,z i denotes the position coordinates of the i-th radar point in the radar coordinate system.

4. The deep learning-based end-to-end radar-camera integrated machine automatic calibration method according to claim 2, characterized in that: In step S2, the output image target point P is outputted I Specifically: where u i ,v i denotes the pixel position of the i-th object on the image.

5. The deep learning-based end-to-end radar-camera integrated machine automatic calibration method according to claim 2, characterized in that: The calibration network module comprises a deep learning network model for calibration. The calibration network model is configured to learn a mapping relationship between the input image target point and radar target point information, and output an accurate calibration matrix T. The joint calibration method comprises the following steps: Step S303, constructing a loss function, projecting the radar points to the image to obtain radar projection points P proj (u proj , v proj ); the projection points P proj of the radar points projected to the image and the image target points P I are used as the loss function L for training the network; Step S3: training the calibration network to accurately learn the mapping relationship between the radar target point and the image target point; 6. The deep learning-based end-to-end radar-camera integrated machine automatic calibration method according to claim 5, characterized in that: Step S4: after the network training is completed, inputting the radar target point and image target data generated in steps S1 and S2 into the calibration network to obtain an accurate calibration matrix T. The calibration matrix T is used to accurately map the target point information detected by the radar to the camera image, thereby realizing the joint calibration of the radar-camera integrated machine. Step S3 comprises the following steps:

7. The deep learning-based end-to-end radar-camera integrated machine automatic calibration method according to claim 6, characterized in that: Step S301: setting the calibration network, including setting an input layer, a feature extraction layer, a feature fusion layer, and an output layer; Step S302: preparing training data, acquiring radar target points and image data at the same time through step S1, obtaining image target points through step S2, and labeling a one-to-one correspondence between the radar target points and the image target points for each set of data to form training data for the calibration network; Step S304: training the calibration network, using a loss function L to perform back propagation on the network, adjusting the parameters of the network to minimize the loss function, and using an optimization algorithm to iteratively update the network parameters until the loss function converges or a preset number of training times is reached. In step S301, the input layer comprises a radar input layer and an image input layer. In the radar input layer, the radar target point contains three-dimensional coordinate (x, y, z) information of each target. The radar input data is represented as a tensor with a shape of (N, 3), where N is the number of targets detected by the radar. The image input layer: the image target detection information is represented as M two-dimensional pixel points (u, v). In step S301, the feature extraction layer comprises a radar feature extraction layer and an image feature extraction layer, which are used to extract features of the radar and image target information input by the input layer. The feature extraction layer includes a plurality of fully connected layers and activation functions, the input is a (N, 3) tensor transmitted by the radar input layer, a ReLU activation function is connected after each fully connected layer, the data is expanded by the fully connected layer to enrich the network features, and the output is a (N, 128) tensor; The image feature extraction layer includes a plurality of fully connected layers and activation functions, the input is a (M, 2) tensor transmitted by the image input layer, a ReLU activation function is connected after each fully connected layer, and the output is a (M, 128) tensor.

8. The deep learning-based end-to-end radar-camera integrated machine automatic calibration method according to claim 7, characterized in that: In step S301, the feature fusion layer splices the data output by the feature extraction layer; the (N, 128) tensor output by the radar feature extraction layer and the (M, 128) tensor output by the image feature extraction layer are filled and spliced in the feature dimension, and output as a (max(N, M), 256) tensor; The output layer uses a plurality of fully connected layers to further process the fused features and reduce the dimension of the data, and finally outputs a (1, 12) tensor.

9. The deep learning-based end-to-end radar-camera integrated machine automatic calibration method according to claim 8, characterized in that: In step S303, the radar points are projected to the image to obtain radar projection points P proj (u proj , v proj ) are specifically: where K is an intrinsic camera matrix, R is a rotation matrix in the calibration matrix T, t is a translation vector in the calibration matrix T, and || denotes a normalization symbol. The radar point is projected to a projection point P of the image proj and the image target point P I The mean square error as the loss function L of the training network is specifically: where, u proj，i , v proj，i represent the projection point coordinates of the radar point projected to the image, N is the number of targets detected by the radar, u i , v i represent the pixel position of the i-th target on the image.

10. The deep learning-based end-to-end radar-camera integrated machine automatic calibration method according to claim 9, characterized in that: In step S304, the loss function L is used for back propagation of the network, and the back propagation specifically includes the following steps: In step S3041, the loss function L is calculated according to the output calibration matrix T of the network; In step S3042, the gradient of the loss function L to each parameter in the deep learning network is calculated, and the deep learning parameters include all parameters in each layer of the calibration network; In step S3043, according to the gradient, the Adam optimization algorithm is used to update each parameter in the deep learning network to obtain a new deep learning network; In step S3044, the training data is input into the updated network again, and steps S3041 to S3043 are repeated until the loss function L converges, and then the deep learning network at this time can be used for inference of the calibration matrix T.

Images