Target detection tracking method and device based on Leiyu fusion perception and medium

Abstract

The invention discloses a target detection tracking method and device based on radar visual fusion perception, and a medium, and the method employs visual detection as a leading part to establish a radar visual fusion tracking module, and solves a problem that the detection precision, tracking stability and environment robustness are difficult to give consideration to the existing single-mode perception in a complex traffic environment at the same time. Through unified multi-modal fusion of distance and speed information of visual detection, visual tracking and millimeter wave radar, stable, continuous and reliable target identification and track output of targets such as pedestrians and vehicles under a low-computing-power platform are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to a target detection and tracking method based on radar and vision fusion perception, a device and a medium, and belongs to the technical field of intelligent driving system fusion perception. BACKGROUND

[0002] With the rapid development of intelligent transportation, assisted driving and autonomous driving technology, the vehicle perception system puts forward higher requirements for the detection and continuous tracking of dynamic targets such as pedestrians and vehicles in front. In complex urban roads, expressways, and environments such as weak light, occlusion, and bad weather, it is often difficult to achieve high-precision target detection and stable trajectory output at the same time by relying on a single sensor. Therefore, multi-sensor fusion has gradually become the core direction of intelligent perception systems, among which the fusion application of vision sensors and millimeter wave radars is particularly widespread.

[0003] Most of the current mainstream target detection systems are based on cameras, which identify and locate vehicles and pedestrians in images through deep learning models (such as structures based on YOLO, CenterNet, etc.), and obtain the time sequence trajectory of the target by combining multi-target tracking algorithms (such as SORT, ByteTrack, etc.). This method performs well in terms of accuracy under conditions of sufficient light, no occlusion, and good image quality, but its detection stability decreases significantly in night, backlight, and rain and fog environments, and it is difficult to obtain reliable target distance and radial velocity information relying solely on vision. Compared with vision methods, millimeter wave radars have stronger environmental robustness in all-weather conditions and can directly output target motion parameters such as distance, azimuth angle, and radial velocity, with a speed measurement accuracy significantly better than vision. However, the angular resolution of millimeter wave radars is low, making it difficult to accurately distinguish different targets with close spatial distances, and they cannot directly provide information such as target appearance and category. Therefore, it is difficult to meet the target-level detection requirements in complex traffic environments using millimeter wave radars alone.

[0004] To address the limitations of single-modality sensors, some recent research attempts to use a joint perception framework for radar and vision. One approach matches the vision detection results with radar targets through geometric projection in the backend association, thereby supplementing the speed information for the vision trajectory. Another approach uses feature-level or decision-level fusion, taking radar point clouds or target information as network input to improve the robustness of the vision model. Although these methods improve the perception performance to some extent, many of them still rely on strong preconditions such as strict external parameter calibration, stable detection quality, or a large amount of synchronized labeled data, making it difficult to implement engineering deployment.

[0005] In summary, the existing technology mainly has the following disadvantages:

[0006] a. Visual detection and visual tracking are highly dependent on image quality and lack robustness. In the case of light changes, backlight, night, rain and fog, and occlusion, visual detection model is prone to false detection and missed detection, leading to unstable visual tracking (such as ID switching and track loss), and relying solely on vision can hardly ensure the continuity of target trajectory.

[0007] b. Monocular vision is difficult to provide reliable real distance and speed information. Relying solely on visual estimation of depth is usually not stable and has large noise, and the error is obvious in long-distance target, low-texture area and occlusion scene; vision is also difficult to provide accurate radial velocity, resulting in unreliable target motion state estimation.

[0008] c. Although millimeter wave radar has speed advantage, its angle resolution is low and cannot independently undertake vehicle-level target-level tracking task. The target angle and contour information output by millimeter wave radar is poor, which is difficult to distinguish categories and complete high-precision spatial positioning alone; traditional radar tracking usually relies on fixed threshold or simple association strategy, which cannot be stably corresponded with visual trajectory.

[0009] d. The existing front-end-to-end deep fusion method is highly dependent on data, has poor migration ability and high computing power requirement. Some current researches use feature-level or data-level end-to-end radar and vision fusion network to realize cross-modal information fusion through joint training. However, such method usually relies on a large number of labeled and well-aligned multi-sensor data sets; once the sensor installation position, resolution, radar model or scene changes, the model performance will decrease significantly, and the migration cost is high. At the same time, the end-to-end fusion network structure is complex and has large inference calculation, which is not suitable for deployment on vehicle-level or edge computing platform, and it is difficult to meet the engineering requirements of real-time and stability. SUMMARY

[0010] The present application aims to provide a target detection and tracking method, device and medium based on radar and vision fusion perception, which solves the problem that the existing single modal perception is difficult to simultaneously consider detection accuracy, tracking stability and environmental robustness in complex traffic environment, and realizes stable, continuous and reliable target identification and trajectory output for pedestrians, vehicles and other targets on a low-power platform.

[0011] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a target detection and tracking method based on radar and vision fusion perception, comprising:

[0012] synchronously triggering a visual sensor and a millimeter wave radar to collect time-synchronized visual sensor image data and millimeter wave radar raw signal data;

[0013] inputting the image data into a visual trajectory module for processing to output three-dimensional visual trajectory information in a visual coordinate system;

[0014] input the original signal data into a radar auxiliary correction module for processing, and output effective radar point cloud data in a radar coordinate system and vehicle motion parameters in a vehicle coordinate system;

[0015] convert the three-dimensional visual trajectory information and the effective radar point cloud data to a unified reference coordinate system based on calibration parameters between the millimeter wave radar and the visual sensor;

[0016] input the three-dimensional visual trajectory information and the effective radar point cloud data in the reference coordinate system into a radar-visual fusion tracking module for processing, and output a fusion track; wherein the radar-visual fusion tracking module comprises a track initiation module, a track prediction module, a track matching module, a track update module and a track extinction module;

[0017] the track initiation module is configured to generate an initial fusion track based on the three-dimensional visual trajectory information;

[0018] the track prediction module is configured to predict a historical track to obtain a current time prediction track, and an initial value of the historical track is the initial fusion track;

[0019] the track matching module is configured to match and judge the current time prediction track and the effective radar point cloud data;

[0020] the track update module is configured to update the fusion track according to a matching judgment result of the track matching module, output an updated fusion track, and continue track prediction;

[0021] the track extinction module is configured to judge loss of the three-dimensional visual trajectory information and the effective radar point cloud data, and perform extinction processing or track prediction on the fusion track according to a loss judgment result.

[0022] The application realizes deep fusion of millimeter wave radar measurement and visual space-time information through visual detection, overcomes the reliability degradation of a single mode under conditions such as occlusion, distance estimation drift and sparse measurement, and realizes a high-robustness real-time target detection and tracking system that can be deployed on an edge platform through a lightweight engineering fusion process without relying on massive end-to-end training data. The target detection and tracking method based on radar-visual fusion perception provided by the application can flexibly adjust the used sensors and installation positions, and greatly reduces the dependence on training data scenes and installation methods.

[0023] According to the embodiments of the application, the application can be further optimized, and the following is a technical solution formed after optimization:

[0024] In one preferred embodiment, the image data is input into a visual trajectory module for processing, and three-dimensional visual trajectory information in a visual coordinate system is output; specifically including:

[0025] The image data is input into a target detection module for processing, and target detection information in a visual coordinate system is output; wherein the target detection information includes target detection frame center position, target detection frame width, target detection frame height, target class and confidence;

[0026] The target detection information is input into a target tracking module for processing, and target tracking information in a visual coordinate system is output;

[0027] According to the target tracking information, the same target in a continuous time sequence is identified and associated to generate structured visual trajectory;

[0028] The target detection information is input into a monocular distance measurement module based on size prior, and target three-dimensional coordinate information in a visual coordinate system is output;

[0029] Based on the target three-dimensional coordinate information and the target tracking information, the same target in a continuous time image sequence is identified and associated, and three-dimensional visual trajectory information in a visual coordinate system is output ; wherein, is a tracking flag, is a target three-dimensional horizontal coordinate in a visual coordinate system, is a target three-dimensional vertical coordinate in a visual coordinate system, is a target three-dimensional vertical coordinate in a visual coordinate system, is a target class, is a tracking time;

[0030] Preferably, the target detection module uses a YOLO series detector; and the target tracking module uses a ByteTrack tracker.

[0031] In the above scheme, the target three-dimensional coordinate information provides a reference for a three-dimensional space required for cross-modal matching.

[0032] In one preferred embodiment, the process of obtaining the target three-dimensional coordinate information in a visual coordinate system specifically includes: in a continuous time sequence of the same target, a visual distance calculated from size prior and the target detection frame height is smoothed to obtain a target visual distance;

[0033] According to the target visual distance, target three-dimensional coordinate information in a visual coordinate system is calculated.

[0034] The monocular ranging module based on size prior provided by this invention avoids distance fluctuations caused by single-frame detection box jitter, enabling distance estimation to respond quickly to real-world changes. It also compensates for jump errors caused by small target pixel heights being close to noise at long distances. This ensures that the final obtained target visual distance is stable and continuous.

[0035] In one preferred embodiment, the visual distance calculated from the size prior and the target detection box height is smoothed to obtain the target visual distance. The specific calculation formula is as follows:

[0036] ;

[0037] ;

[0038] in, For the first Target visual distance in the frame For the current frame, For frame sliding window, For the first frame within the sliding window The monocular ranging value of the frame, The vertical focal length of the vision sensor. The height is the prior dimension. For the first Frame target detection bounding box height;

[0039] The target's three-dimensional coordinates in the visual coordinate system are calculated based on the target's visual distance. The specific calculation formula is as follows:

[0040] ;

[0041] ; ; ;

[0042] ;

[0043] ; ;

[0044] in, This refers to the target's three-dimensional coordinate information in the visual coordinate system. The target's three-dimensional horizontal coordinate in the visual coordinate system. The vertical coordinate of the target in the visual coordinate system. The target's three-dimensional vertical coordinates in the visual coordinate system. For the first Target visual distance in the frame The normalized imaging x-coordinate of the pixel center position of the target detection box. normalizing the pixel center position of the target detection frame into an imaging vertical coordinate, normalizing the pixel center position of the target detection frame into a horizontal coordinate, normalizing the principal point of the visual coordinate system into a horizontal coordinate, normalizing the pixel center position of the target detection frame into a vertical coordinate, normalizing the principal point of the visual coordinate system into a vertical coordinate, normalizing the horizontal focal length of the visual sensor, normalizing the vertical focal length of the visual sensor, normalizing the ray distance from the visual sensor to the pixel center position of the target detection frame.

[0045] In one preferred embodiment, the original signal data is input into a radar auxiliary correction module for processing, and effective radar point cloud data in a radar coordinate system and vehicle motion parameters in a vehicle coordinate system are output; specifically including:

[0046] The original signal data is sequentially subjected to distance dimension FFT, Doppler FFT, azimuth angle estimation, and elevation angle estimation to obtain polar coordinate information of the target, and the polar coordinate information is converted to a radar Cartesian coordinate system to obtain radar target original measurement and target radar point cloud data; wherein, is the distance of the target in the Cartesian coordinate system relative to the radar in the direction straight ahead of the radar, is the distance of the target in the Cartesian coordinate system relative to the radar in the direction to the right of the radar, is the distance of the target in the Cartesian coordinate system relative to the radar in the direction above the radar, is the movement speed of the target along the direction straight ahead of the radar, is the movement speed of the target along the direction to the right of the radar;

[0047] The radar point cloud data is converted to an image coordinate system, and projection point pixel coordinate data is output;

[0048] The radar point cloud data is input into a vehicle motion estimation module for processing, and vehicle motion parameters are output; wherein the vehicle motion estimation module uses a plane motion estimation method based on a random sample consensus algorithm to obtain the vehicle motion parameters ; wherein, is the longitudinal speed of the vehicle, is the yaw rate;

[0049] The projection point pixel coordinate data and the target visual distance are input into a visual cone screening module; wherein the visual cone screening module is used to construct a visual cone region and limit the millimeter wave radar search range using the target visual distance, and output effective radar point cloud data.

[0050] The application recovers spatial structure information and self-vehicle motion state from original signals and measurements of a vehicle-mounted millimeter wave radar, provides physically consistent motion parameters for cross-modal fusion, and the distance, speed and angle information provided by the millimeter wave radar is stable in time and has high accuracy in long-distance conditions, which is an effective supplement to visual detection.

[0051] In one preferred embodiment, a visual cone region is constructed and the millimeter wave radar search range is limited by the target visual distance, and effective radar point cloud data is output; specifically including:

[0052] Based on the position and height of the target on the image plane, the range of the target in the depth direction is calculated by monocular ranging, and an initial conical region in space is constructed;

[0053] The initial conical region is expanded outward by a preset angle along the horizontal and vertical directions to obtain the visual conical region, which is used for preliminary screening of radar point cloud data;

[0054] Based on the target visual distance, the radar search range is set to , and the matching screening of the visual target and the radar target is completed, and effective radar point cloud data after visual cone matching is obtained; wherein, is the target visual distance;

[0055] Preferably, .

[0056] The application sets the radar search range to , which can effectively reduce the risk of false matching caused by distant interference points, and through visual cone matching, the coarse screening of the visual target and the radar target is completed, effectively reducing the possibility of false matching.

[0057] The application constructs a three-dimensional space cone using the monocular ranging result of the visual detection target, and limits the maximum effective matching distance and angular range of the radar points. It is equivalent to establishing a "visual prior constraint" in the radar domain, effectively reducing the radar search range, and suppressing false association in cross-modal matching from the source. In particular, in the scenarios of multiple vehicles, parallel lanes, and long-distance weak reflection, the association accuracy is significantly improved.

[0058] In one preferred embodiment, the calibration parameters between the millimeter wave radar and the visual sensor include a radar extrinsic parameter matrix and a visual sensor extrinsic parameter matrix;

[0059] The radar extrinsic parameter matrix and the visual sensor extrinsic parameter matrix are corrected in real time by an extrinsic parameter calibration module during vehicle operation;

[0060] When the correction condition is triggered, the extrinsic parameter calibration module enters a dynamic calibration mode, specifically including:

[0061] a1, construct a common view target set, the common view target set being a set of targets detected by the visual sensor and the millimeter wave radar at the same time;

[0062] a2, convert the radar target to a camera coordinate system based on the radar extrinsic matrix and the visual sensor extrinsic matrix;

[0063] a3, project the radar target in the camera coordinate system to a pixel plane to obtain predicted pixel coordinates by using a camera intrinsic parameter, and construct a re-projection error, the re-projection error being an Euclidean distance between the predicted pixel coordinates and actual pixel coordinates detected by the visual sensor;

[0064] a4, filter out abnormal targets from the re-projection error by using a random sample consensus algorithm to obtain an effective common view target set;

[0065] a5, based on the effective common view target set, establish a nonlinear least squares optimization model with the objective of minimizing the overall re-projection error, the rotation vector and the translation vector of the extrinsic matrix being optimization parameters of the nonlinear least squares optimization model;

[0066] a6, calculate the nonlinear least squares optimization model by using a Levenberg-Marquardt optimization algorithm to obtain an optimal extrinsic increment;

[0067] a7, judge whether the overall re-projection error meets a preset convergence condition, if yes, update the radar extrinsic matrix and the visual sensor extrinsic matrix according to the optimal extrinsic increment, and if not, return to step a6 for continuous iteration; wherein the preset convergence condition is that a re-projection error change amount of a single iteration is less than a first preset threshold, or a norm of the extrinsic increment is less than a second preset threshold.

[0068] The extrinsic calibration module based on common view target statistical re-projection error provided by the application can recursively correct the extrinsic parameter in real time during the operation of the vehicle, ensure the stability of the cross-modal coordinate alignment, and realize the extrinsic dynamic correction mechanism without parking calibration and calibration board, so that the spatial consistency of radar and visual fusion remains stable in long-term operation.

[0069] In one preferred embodiment, an initial fusion track is generated based on the three-dimensional visual track information; specifically including:

[0070] An extended Kalman filter is used to recursively estimate the target state, and the target state is represented as: ; wherein, is the target position in the reference coordinate system, is a two-dimensional velocity component of the target;

[0071] When the radar and visual fusion tracking module tracks for a first set time After, the initial fusion track is acquired ; wherein, is a fusion track flag, and is set to -1 to indicate that the initial fusion track is temporary, is an initial position of the visual target in a reference coordinate system, is .

[0072] The application adopts a fusion extended Kalman filter taking position-velocity as state quantity to realize a dual-mode measurement selection strategy, so that the filter can dynamically select the most credible mode data and has self-recovery capability. Meanwhile, the Mahalanobis distance weighted robust fusion of multiple radar points of the same target is used to improve the estimation accuracy of distance and speed, and compared with single target single point update, it has higher robustness and noise resistance.

[0073] The application adopts an interpretable link of "detection-tracking-ranging-projection-matching-filtering"; each step can be independently debugged, replaced or extended, which not only reduces the overall computing power requirement, but also avoids the strong dependence of end-to-end methods on a large number of fusion data sets and complex training processes, so that the technical solution can run in real time on a low-computing-power platform and can adapt to changes in camera and radar positions without retraining data.

[0074] In one preferred embodiment, a current time prediction track is obtained by predicting a historical track, and an initial value of the historical track is the initial fusion track; specifically including:

[0075] If the change of the ego motion parameter is greater than a third preset threshold or the number of visual detected targets is greater than a fourth preset threshold, a uniform acceleration model is used for prediction, otherwise a uniform speed model is used for prediction;

[0076] The current time prediction track is obtained.

[0077] Visual tracking is prone to obvious forward and backward jumps in a vehicle-mounted scene due to target crossing the frame boundary, sudden size change or complex occlusion. The application adds physical constraints of radar measurement in the fusion track, such as continuous distance change rate, direction angle consistency, speed accessibility constraint, target RCS (reflective area) and SNR (signal-to-noise ratio), etc., which not only can correct unreasonable transition of the visual frame in time, but also can maintain the continuity of state estimation when the vision is temporarily invalid. Thus, the shaking problem during overtaking / being overtaken, lane changing and merging is significantly improved.

[0078] The application constructs a closed-loop time sequence fusion mechanism established by visual dominant navigation track, supplemented by radar physical information and cross-modal joint constraint, so that the track can obtain stability, spatial accuracy and tracking continuity better than single mode in a dynamic scene. The application constructs a closed-loop time sequence fusion mechanism established by visual dominant navigation track, supplemented by radar physical information and cross-modal joint constraint, so that the track can obtain stability, spatial accuracy and tracking continuity better than single mode in a dynamic scene.

[0079] In one preferred embodiment, the current time prediction track and the effective radar point cloud data are matched and judged; specifically, the matching judgment includes:

[0080] The four-degree-of-freedom Mahalanobis distance is used to determine whether the effective radar point cloud and the current time prediction track are matched; specifically, the Mahalanobis distance between the effective radar point cloud and the current time prediction track is calculated, if the Mahalanobis distance is greater than a fifth preset threshold, the matching is unsuccessful; if the Mahalanobis distance is not greater than the fifth preset threshold, there is a possibility of matching, when the number of the effective radar point clouds that may be matched is greater than 1, the effective radar point with the closest Mahalanobis distance is selected for fusion track updating;

[0081] According to the matching judgment result of the track matching module, the fusion track is updated, the updated fusion track is output, and the track prediction is continued; specifically, the updating includes:

[0082] When the effective radar point cloud and the prediction track continuously have a matching possibility for several frames, it is determined that the radar and visual matching is successful, and the data of the effective radar point with the closest Mahalanobis distance is used The fusion track is updated to obtain the updated fusion track ;

[0083] When the effective radar point cloud and the prediction track are not matched, the visual information The fusion track is updated in combination with the prediction track to obtain the updated fusion track ;

[0084] Wherein, According to the order of the generated prediction track, the track is updated from a transient state to a steady state; is the radar target position in the reference coordinate system; is the two-dimensional velocity component of the radar target; is the visual target position in the reference coordinate system.

[0085] The application associates the radar point cloud and the visual track through the Mahalanobis distance, can exclude most of the abnormal points while considering the measurement uncertainty, improve the matching robustness, and realize the continuous, stable and high-confidence track state updating.

[0086] In one preferred embodiment, loss determination is performed on the three-dimensional visual trajectory information and the effective radar point cloud data, and according to the loss determination result, the fusion track is processed or track prediction is performed; specifically comprising:

[0087] If the three-dimensional visual trajectory information and the effective radar point cloud data are continuously lost for a second set time , it is determined that the target leaves, and the fusion track is eliminated, otherwise track prediction is continued.

[0088] The disappearance determination of the track in the application is dominated by visual detection, and when the visual side fails to observe the target for a plurality of continuous frames, the track will be terminated. The above scheme avoids the false disappearance caused by radar noise points and prevents the "false alarm track" that may occur in the case of relying only on radar.

[0089] The application first establishes an initial track from the visual side, and keeps the track from disappearing under the condition that visual observation is continuously obtained, and then updates the track state through the matched millimeter wave radar points. This dual-mode "master-slave structure" avoids track breakage caused by short-term loss of a single mode, enables the tracking process to maintain track continuity in situations such as occlusion, rapid relative motion, out-of-frame / in-frame boundary, significantly reduces the ID switching rate, improves track stability, and improves tracking accuracy.

[0090] The application realizes continuous, stable, and high-confidence track state updating through the cross-modal fusion mechanism with causal coherence, which enables vision to maintain a dominant position in track establishment, track continuation, and track disappearance, and enables millimeter wave radar measurement to perform high-precision distance and speed compensation on the track state when available, and realizes robust matching through spatial cone constraint and Mahalanobis distance association.

[0091] Based on the same concept, the application also provides an electronic device comprising a memory, a processor, and a computer program or instructions stored on the memory, wherein the processor executes the computer program or instructions to implement the above-mentioned radar-visual fusion perception-based target detection and tracking method.

[0092] Compared with the prior art, the application has the following advantages:

[0093] The application provides a radar-visual fusion perception-based target detection and tracking method, device, and medium, which establishes a radar-visual fusion tracking module dominated by visual detection, solves the problem that existing single-modal perception cannot simultaneously consider detection accuracy, tracking stability, and environmental robustness in complex traffic environments, and realizes stable, continuous, and reliable target identification and track output for pedestrians, vehicles, and other targets on a low-power platform through unified multi-modal fusion of visual detection, visual tracking, and millimeter wave radar distance and speed information.

[0094] The present application makes the high resolution of visual detection and the stable distance and speed measurement of millimeter wave radar complementary to each other, reduces the influence of environmental changes on the detection result; the visual information and the motion parameters of the millimeter wave radar are updated under a unified state model, cross-modal noise compensation is realized, the trajectory continuity and the target motion estimation accuracy are improved; without relying on end-to-end huge training data, a real-time target detection and tracking system with high robustness can be deployed on an edge platform through a lightweight engineering fusion process, an integrated method can flexibly adjust the used sensors and their installation positions, and the dependence on training data scenes and installation methods is greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0095] Figure 1 is a target detection and tracking method flowchart based on radar and visual fusion perception of an embodiment of the present application; Figure 2 is a detailed flowchart of the target detection and tracking method of an embodiment of the present application; Figure 3 is a frustum matching schematic diagram combined with monocular ranging of an embodiment of the present application; Figure 4 is a radar and visual fusion tracking flowchart of an embodiment of the present application; Figure 5 is an actual radar and visual fusion application scene diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0096] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0097] Embodiment 1

[0098] As shown in Figure 1 , the target detection and tracking method based on radar and visual fusion perception provided by the embodiment of the present application includes the following steps:

[0099] Step 1, synchronously trigger the visual sensor and the millimeter wave radar to collect time-synchronized visual sensor image data and millimeter wave radar original signal data; input the image data to a visual trajectory module for processing, and output three-dimensional visual trajectory information in a visual coordinate system;

[0100] The vision side provides high-resolution appearance detection and accurate pixel spatial information for depth estimation and cross-modal matching. In this embodiment 1, the vision side uses a monocular camera as input image sequence. First, it extracts object detection information using a YOLO series detector, including the center position of the object detection box, the width of the object detection box, the height of the object detection box, the object category, and the confidence score. Pre-trained on a public large dataset and then fine-tuned using a proprietary dataset that matches the application scenario, the YOLO series detector possesses high object detection capabilities.

[0101] Due to issues such as target occlusion, cross-frame acceleration changes, and detection gaps in real-world scenarios, the detection results for a single frame are not temporally stable. Therefore, this embodiment 1 further introduces a tracking module to ensure the continuity of tracking flag IDs. Specifically, the ByteTrack tracker is used as the visual multi-target tracking module. ByteTrack maintains a large range of low-confidence targets and establishes primary matching using high-confidence targets during the association phase, thus maintaining the continuity of tracking flag IDs even in the presence of occlusion or short-term detection failures. The visual multi-target tracking module recursively estimates the target position and velocity using Kalman filtering, uses IoU (Intersection over Union) as the distance metric to obtain the corresponding cost matrix, and calculates the cost matrix using the Hungarian algorithm to achieve matching. Based on the frame-by-frame target tracking information output by the visual multi-target tracking module, a structured visual trajectory is generated based on the continuous image sequence, represented as: ;in, For visual navigation, For tracking purposes, The x-coordinate of the center position of the detection box pixels. The vertical coordinate of the center position of the detection box pixels. The width of the target detection box. The height of the target detection box, For the target category, To track the moment.

[0102] Since monocular vision itself lacks direct depth measurement capabilities, and distance is crucial data for correlating with radar information, this embodiment 1 constructs a monocular depth estimation model based on prior target size. For highly structured targets such as vehicles and pedestrians in traffic scenarios, their distance can be estimated through imaging scale relationships. Specifically, according to the pinhole camera imaging formula, we can obtain:

[0103] ;

[0104] in, For the first The initial monocular ranging value of the frame; The vertical focal length of the vision sensor; The size prior height is: The size prior height is: The frame target detection box height is:

[0105] In order to avoid the distance fluctuation caused by single frame detection box jitter, the embodiment 1 does not directly use the result calculated by the pinhole imaging formula, but introduces a "distance filtering" mechanism: in the time sequence of the same target, the distance calculated by the size prior is smoothed, and the smoothed distance is used as the final distance of the target in the frame. The target visual distance after frame sliding window smoothing is: The target visual distance after frame sliding window smoothing is:

[0106] ;

[0107] The target visual distance after frame sliding window smoothing is: The target visual distance after frame sliding window smoothing is: The target visual distance after frame sliding window smoothing is:

[0108] The monocular depth estimation model based on the target size prior can quickly respond to real changes, and can also offset the jump error caused by the small target pixel height close to noise at a long distance. The final visual distance is stable and continuous, and can be directly used for the construction of the view cone screening module and the matching with the radar information.

[0109] According to the target visual distance, the target three-dimensional coordinate information in the visual coordinate system is calculated, and the specific calculation formula is as follows:

[0110] ;

[0111] ; ; ;

[0112] ;

[0113] ; ;

[0114] The target three-dimensional coordinate information in the visual coordinate system is: The target three-dimensional coordinate information in the visual coordinate system is: The target three-dimensional horizontal coordinate in the visual coordinate system is: The target three-dimensional vertical coordinate in the visual coordinate system is: The target three-dimensional vertical coordinate in the visual coordinate system is: The intrinsic matrix of the visual sensor is: The horizontal principal point of the visual coordinate system is: The vertical principal point of the visual coordinate system is: The horizontal focal length of the visual sensor is: The vertical focal length of the visual sensor is: normalizing the pixel center position of the target detection frame into an imaging horizontal coordinate; normalizing the pixel center position of the target detection frame into an imaging vertical coordinate; a ray direction of the visual sensor to the pixel center position of the target detection frame; a corresponding ray distance.

[0115] The size prior height is also different for different target category labels, that is, the three-dimensional coordinate information of targets of different categories can be calculated. The three-dimensional coordinate information of the target, although not as accurate as the millimeter wave radar information, provides a three-dimensional spatial reference required for cross-modal matching. Based on the three-dimensional coordinate information of the target and the target tracking information, the same target is identified and associated in a continuous time image sequence, and three-dimensional visual trajectory information of the target in the visual coordinate system is output, represented as: .

[0116] Step 2, inputting the original signal data into a radar auxiliary correction module for processing, and outputting effective radar point cloud data in a radar coordinate system and ego motion parameters in a vehicle coordinate system;

[0117] Step 2 is used to recover spatial structure information and ego motion state from the original signal and measurement of the vehicle-mounted millimeter wave radar, to provide physically consistent motion parameters for subsequent cross-modal fusion.

[0118] The original received signal collected by the millimeter wave radar is first subjected to standard DSP-level processing, and distance dimension FFT, Doppler FFT, azimuth angle estimation, and elevation angle estimation are sequentially performed to screen out effective targets. Then, the polar coordinate information of the target is converted into radar Cartesian coordinates:

[0119] ;

[0120] Finally, the radar target original measurement and the corresponding target point cloud are obtained; wherein, is the distance of the target in the Cartesian coordinates relative to the radar in the direction directly in front of the radar, is the distance of the target in the Cartesian coordinates relative to the radar in the direction to the right of the radar, is the distance of the target in the Cartesian coordinates relative to the radar in the direction above the radar, is the motion speed of the target along the direction directly in front of the radar, is the motion speed of the target along the direction to the right of the radar, is the polar coordinate radial distance of the radar target relative to the phase center of the radar, is the polar coordinate azimuth angle of the radar target relative to the phase center of the radar, is the polar coordinate elevation angle of the radar target relative to the phase center of the radar, is the radial motion speed of the target relative to the radar.

[0121] The radial distance, radial motion speed and angle information provided by the millimeter wave radar are stable in time and have high accuracy in long distance conditions, and are a key supplement to visual information.

[0122] The embodiment 1 constructs a self-vehicle motion estimation model by using the measurement data of the radar on the static environmental target, and outputs self-motion parameters such as self-vehicle speed and yaw rate; based on the installation calibration parameters of the radar and the vehicle, the target measurement in the radar coordinate system is converted to the vehicle coordinate system, and then combined with the self-motion parameters, the target measurement in the vehicle coordinate system is converted to the world coordinate system, thereby providing a unified coordinate reference for subsequent radar-visual fusion.

[0123] Specifically, the radar point cloud set in the world coordinate system is subjected to inter-frame target point matching and coordinate transformation, the position offset of the same target point between adjacent frames is calculated, and a three-dimensional displacement vector and a motion speed vector corresponding to each frame of target are obtained.

[0124] Since the radar speed measurement mixes the self-vehicle motion component, the self-vehicle speed must be separated by a model. The embodiment 1 adopts a plane motion estimation method based on a random sample consensus algorithm RANSAC: assuming that the radial speed of the static object in the environment and the longitudinal speed and yaw rate of the self-vehicle satisfy a linear relationship; through the random sampling-model fitting-inlier counting iterative process of the RANSAC algorithm, the static target points are screened out from the radar point cloud; the longitudinal speed and yaw rate of the self-vehicle are robustly fitted by minimizing the square sum of the radial speed residuals of the static points as a constraint, and the self-vehicle motion parameters are obtained; wherein, is the longitudinal speed of the self-vehicle, is the yaw rate.

[0125] Based on the obtained self-vehicle motion parameters, the radial speed of the radar target is compensated for the self-vehicle motion component, and the absolute speed of the target is obtained; at the same time, the radar coordinate system coordinates of the radar target are converted to the unified world coordinate system in combination with the self-vehicle motion parameters, and the accurate denoised radar target trajectory is output, thereby providing a standardized input for subsequent multi-sensor trajectory fusion.

[0126] Step 3, based on the calibration parameters between the millimeter wave radar and the visual sensor, the three-dimensional visual trajectory information and the effective radar point cloud data are converted to a unified reference coordinate system;

[0127] In the radar-visual fusion system, the extrinsic matrix and The extrinsic parameters are the basis for cross-modal matching and spatial alignment. However, in actual deployment environment, the extrinsic parameters will drift over time, due to reasons including installation error of equipment, angle fine-tuning caused by vehicle vibration, structural deformation caused by temperature, bracket loosening after long-term operation, etc. If fixed extrinsic parameters are used, it will be difficult to ensure the consistency of the pixel coordinates of the visual detection output and the spatial coordinates measured by the millimeter wave radar, thereby introducing systematic errors in the fusion stage.

[0128] The extrinsic parameter dynamic calibration method based on statistical reprojection error of co-view target is adopted in this embodiment 1, which can recursively correct the extrinsic parameters in real time during the vehicle operation, and ensure the stability of cross-modal coordinate alignment. The intrinsic parameters of the camera and the extrinsic parameters of the radar are calibrated using the traditional method when the vehicle is shipped; after shipment, when a series of conditions such as clean environment, large vehicle interval, and large vehicle reflection intensity are met, the extrinsic parameter correction condition is triggered, and the extrinsic parameter calibration module enters the dynamic calibration mode, as follows:

[0129] Firstly, a co-view target set is constructed, which includes targets detected by the visual sensor and the millimeter wave radar at the same time, i.e. targets observed by both sensors in visual detection and radar measurement are selected.

[0130] Under the current extrinsic parameter matrix, the radar points are converted to the camera coordinate system:

[0131] ;

[0132] Then, the predicted pixel position is obtained by projecting it to the pixel plane using the intrinsic parameters of the visual sensor , and the reprojection error is constructed:

[0133] ;

[0134] Wherein, is the camera coordinate system; is the radar coordinate system, obtained by converting the polar coordinate measurement; is the target polar coordinate measurement obtained by radar measurement, is the radial distance, is the horizontal angle, is the azimuth angle; is the radar extrinsic parameter matrix, which is a rotation matrix; is the visual sensor extrinsic parameter matrix, which is a translation matrix; is the pixel center of the target obtained by visual detection.

[0135] To ensure robustness, embodiment 1 is based on the construction of a random sample consensus filter based on the reprojection error, and abnormal points such as high-speed changes, occlusions, and false detections are added to the outlier set for rejection. For valid points that meet the constraints, a nonlinear least squares optimization model is established to minimize the overall reprojection error, and the rotation vector and translation vector of the external parameter matrix are updated incrementally. The nonlinear least squares optimization model is calculated using the LM (Levenberg-Marquardt) optimization algorithm to obtain the optimal external parameter increment and , and the external parameter matrix is updated when the error convergence condition is met. Specifically, it is determined whether the overall reprojection error meets the error convergence condition. If so, the radar external parameter matrix and the visual sensor external parameter matrix are updated according to the optimal external parameter increment; if not, iteration is continued; wherein the error convergence condition is that the reprojection error change of a single iteration is less than a first preset threshold, or the norm of the external parameter increment is less than a second preset threshold. Through the above steps, embodiment 1 realizes an external parameter dynamic correction mechanism without the need for parking calibration and calibration board, so that the spatial consistency of radar-visual fusion remains stable in long-term operation.

[0136] Step 4, input the three-dimensional visual trajectory information in the reference coordinate system and the effective radar point cloud data into a radar-visual fusion tracking module for processing, and output a fusion track; wherein the radar-visual fusion tracking module includes a track initiation module, a track prediction module, a track matching module, a track update module, and a track extinction module.

[0137] After completing visual target detection, tracking, and monocular ranging, further combined with three-dimensional point cloud and radial velocity observation of millimeter wave radar, cross-modal joint estimation and track stable update are realized. As shown in Figure 4 , the cross-modal spatial fusion overall process follows the collaborative mechanism of “visual track establishment → radar assisted correction → no matching prediction maintenance” to improve the reliability of continuity and velocity direction estimation in occlusion scenarios.

[0138] In the trajectory establishment stage, first, the visual detection information is taken as the initial input, the target detection box obtained by the monocular camera is used for rough ranging, and a "visual cone region" is constructed as a three-dimensional space constraint window. Specifically, the visual cone region is based on the position and size of the target on the image plane, and the approximate range of the target in the depth direction is calculated by monocular ranging, thereby forming a conical region in space. In order to tolerate the uncertainty of visual ranging, while allowing a certain deviation in the millimeter wave radar point cloud, the visual cone in Embodiment 1 is expanded by a preset angle range in the horizontal and vertical directions to form an expanded visual cone. The expanded visual cone preliminarily constrains the subsequent radar point cloud screening, effectively reducing the possibility of false matching. Based on the forward distance estimated by vision , the radar search distance range in Embodiment 1 is set to , wherein By limiting the radar search distance range, the risk of false matching caused by distant interference points can be effectively reduced, and finally the visual target and radar target are coarsely screened through the visual cone matching.

[0139] The cross-modal spatial fusion RVST (Radar-Vision Spatial Tracking) process, specifically includes:

[0140] In the cross-modal fusion tracking framework of Embodiment 1, RVST is used as the core mechanism to realize the continuous association and complementary update between visual trajectory and millimeter wave radar measurement in complex traffic scenes. The entire process is based on the following fact assumptions: the visual detection in the vehicle scene can stably provide the pixel-level position of the target in most frames, so the visual tracking result can be used as the dominant source of the trajectory, and the high-stable distance measurement and radial velocity measurement of the radar are used for continuous correction, thereby significantly reducing the problems of drift error and unstable distance estimation that are prone to occur in pure visual tracking, and to some extent, the performance defects in low light environment are compensated.

[0141] In Embodiment 1, the visual detection is based on the data association of ByteTrack. Specifically, based on the position parameters and confidence parameters of the target detection box output by the visual detection model, the detection box is divided into high-score detection boxes and low-score detection boxes; the IOU matching algorithm is used to associate the high-score detection boxes with active trajectories in the first round; the low-score detection boxes are associated with the trajectories that are not matched in the first round in the second round; the no-match prediction keeping operation is performed on the trajectories that are not matched in both rounds, and new trajectories are created for the detection boxes that are not matched in both rounds, thereby completing the association between the detection boxes in the current frame and the target trajectories.

[0142] The embodiment 1 establishes the tracking trajectory based on the visual detection result, and uses an extended Kalman filter (EKF) to recursively estimate the target state. The filter state vector is designed as:

[0143] ;

[0144] wherein, is the target position, is the two-dimensional velocity component.

[0145] The tracking id and category information (cv_id, cls) of the visual detection output are reserved. When the stable output tracking id is obtained, the RVST tracker establishes the tracking trajectory after a stable tracking time of min_hits, and the initial position is provided by the visual detection, and since the visual detection cannot provide the initial velocity, i.e. at this time, st_id is set to -1, indicating that the tracking trajectory is transient. The tracking trajectory is represented as .

[0146] To provide a more stable tracking model, the embodiment 1 provides a “complementary acceleration model” (CV / CA dual model) to realize stable tracking in a regular urban road scene. The complementary acceleration model includes two motion models: a constant velocity model (CV) and a constant acceleration model (CA). The CV model is usually used in urban scenes to reduce the amount of calculation, and when the motion parameters of the ego vehicle are detected to change, for example drastically, significantly greater than 0, it means that the vehicle is frequently accelerating / decelerating / turning, or the number of surrounding targets detected by the visual detector is higher than , which means that the traffic flow is large and the lane is congested, then the CA model is switched to adapt to the acceleration change.

[0147] For the matching of the candidate radar points and the transient trajectory, the embodiment 1 uses a four-degree-of-freedom Mahalanobis distance, and the degrees of freedom correspond to the four key quantities in the trajectory state vector, i.e., the filter state vector.

[0148] Specifically, for each transient trajectory and its predicted state, the Mahalanobis distance between the candidate radar point and the predicted state of the transient trajectory is calculated. If the Mahalanobis distance is greater than a preset threshold (defined as a significance level of 0.01 in four degrees of freedom), the candidate radar point is considered to be matched with the transient trajectory. ​), it is considered that the candidate radar point does not match the predicted trajectory; otherwise, it is considered that there is a potential matching possibility, and if the number of candidate radar points with potential matching is greater than 1, the candidate radar point with the smallest Mahalanobis distance is selected. Through the method of matching by Mahalanobis distance, most of the abnormal points can be excluded while considering the measurement uncertainty, and the robustness of matching is improved.

[0149] To prevent false initialization caused by single-frame noise, embodiment 1 requires that the trajectory be stably matched for several consecutive frames before the transient trajectory is upgraded to a formal radar-visual fusion tracking (RVST) trajectory. At the same time, the tracking flag of the fusion tracking trajectory is enabled , indicating that the fusion tracking trajectory has completed the cross-modal initialization, that is, st_id is assigned, to reliably perform subsequent tracking and state updating.

[0150] When radar-visual information is successfully matched, the fusion filter uses radar information to update the RVST trajectory information , and outputs the steady-state trajectory.

[0151] If radar-visual information is not successfully matched, the visual information is used in combination with the prediction of the motion model to update the RVST trajectory information .

[0152] In the track continuation process, as long as visual detection exists or is not lost for a long time, the cross-modal fusion tracking method provided by embodiment 1 always keeps the track from being terminated, fundamentally reducing the problem of track breakage caused by sparse point tracks or intermittent measurements in traditional radar-dominated tracking. Only in the case where visual tracking is lost and radar point clouds are lost, and after max_hits frames, it is considered that the target has left the range or the occlusion is too severe, the tracking is stopped, and the track is terminated.

[0153] For the update strategy of the track, the cross-modal fusion tracking method supports a multi-point fusion-based manner, that is, when there are multiple radar points in the same visual cone, the data association between the radar points and the track is tracked through radar-visual fusion matching, which improves stability and resists occasional false alarm points. The disappearance of the track is completely dominated by visual detection, that is, when the visual fails to observe the target for a plurality of consecutive frames, the track will be terminated. This strategy avoids false disappearance caused by radar noise points, and prevents the possible "false alarm track" in the case of relying only on radar.

[0154] Through the complete RVST workflow described above, Embodiment 1 constructs a closed-loop time sequence fusion mechanism established by visual dominant navigation track, supplemented by radar physical information, and cross-modal joint constraint, so that the track can obtain stability, spatial accuracy and tracking continuity far superior to a single mode in a dynamic scene. The fusion mechanism reduces the mismatch and ID switching in visual tracking to a certain extent, significantly enhances the reliability of target distance and speed estimation, more significantly optimizes the false alarm and missed detection problems difficult to solve in radar tracking, and can be directly applied to real-time multi-target tracking systems in intelligent traffic monitoring and intelligent driving environments.

[0155] Embodiment 2

[0156] Embodiment 2 also provides an electronic device, which includes a memory, a processor, and a computer program or instructions stored on the memory, and the processor executes the computer program or instructions to implement the target detection and tracking method based on radar-visual fusion perception in Embodiment 1.

[0157] Although not shown, the electronic device includes a processor, which can perform various appropriate operations and processes according to programs and / or data stored in a read-only memory (ROM) or loaded from a storage section into a random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general-purpose main processor and one or more special-purpose coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP), and the like. In the RAM, various programs and data required for device operation are also stored. The processor, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0158] The above processor and memory are used together to execute programs / instructions stored in the memory, which can implement the methods, steps or functions described in the above embodiments when executed by a computer.

[0159] The content illustrated in the above embodiments should be understood as the embodiments being used only to more clearly illustrate the present application, and not for limiting the scope of the present application. After reading the present application, those skilled in the art can make various equivalent modifications to the embodiments, and all such modifications fall within the scope defined by the claims of the present application.

Claims

1. A target detection and tracking method based on radar and vision fusion perception, characterized in that, The method comprises the following steps: Synchronizing the visual sensor and the millimeter wave radar to collect time-synchronized visual sensor image data and millimeter wave radar raw signal data; Inputting the image data into a visual trajectory module for processing to output three-dimensional visual trajectory information in a visual coordinate system; Inputting the raw signal data into a radar auxiliary correction module for processing to output effective radar point cloud data in a radar coordinate system and vehicle motion parameters in a vehicle coordinate system; Converting the three-dimensional visual trajectory information and the effective radar point cloud data to a unified reference coordinate system based on calibration parameters between the millimeter wave radar and the visual sensor; Inputting the three-dimensional visual trajectory information and the effective radar point cloud data in the reference coordinate system into a radar-visual fusion tracking module for processing to output a fusion track; wherein the radar-visual fusion tracking module comprises a track initiation module, a track prediction module, a track matching module, a track update module, and a track elimination module; The track initiation module is configured to generate an initial fusion track based on the three-dimensional visual trajectory information; The track prediction module is configured to predict a historical track to obtain a current time prediction track, and an initial value of the historical track is the initial fusion track; The track matching module is configured to match and judge the current time prediction track and the effective radar point cloud data; The track update module is configured to update the fusion track according to the matching judgment result of the track matching module, output an updated fusion track, and continue track prediction; The track elimination module is configured to judge loss of the three-dimensional visual trajectory information and the effective radar point cloud data, and perform elimination processing or track prediction on the fusion track according to the loss judgment result. 2.The target detection and tracking method based on radar and visual fusion perception according to claim 1, characterized in that, Inputting the image data into a visual trajectory module for processing to output three-dimensional visual trajectory information in a visual coordinate system; specifically comprising: Inputting the image data into a target detection module for processing to output target detection information in a visual coordinate system; wherein the target detection information comprises a target detection frame center position, a target detection frame width, a target detection frame height, a target category, and a confidence; Inputting the target detection information into a target tracking module for processing to output target tracking information in a visual coordinate system; According to the target tracking information, identifying and associating a same target in a continuous time sequence to generate structured visual trajectory; Inputting the target detection information into a monocular distance measurement module based on size prior to output target three-dimensional coordinate information in a visual coordinate system; Based on the target three-dimensional coordinate information and the target tracking information, the same target in a continuous time image sequence is identified and associated, and three-dimensional visual trajectory information in a visual coordinate system is output . wherein, is a tracking mark, is a target three-dimensional horizontal coordinate in a visual coordinate system, is a target three-dimensional vertical coordinate in a visual coordinate system, is a target three-dimensional vertical coordinate in a visual coordinate system, is a target category, is a tracking time. 3.The target detection and tracking method based on the radar and visual fusion perception according to claim 2, characterized in that, The process of obtaining the target three-dimensional coordinate information in the visual coordinate system specifically comprises: In a continuous time sequence of a same target, smoothing a visual distance calculated from size prior and the target detection frame height to obtain target visual distance; According to the target visual distance, calculating target three-dimensional coordinate information in a visual coordinate system. 4.The target detection and tracking method based on the radar and visual fusion perception according to claim 3, characterized in that, The smoothing process of the visual distance calculated from size prior and the target detection frame height to obtain target visual distance is specifically calculated by the following formula: ； ； in, For the first Target visual distance in the frame For the current frame, For frame sliding window, For the first frame within the sliding window The monocular ranging value of the frame, The vertical focal length of the vision sensor. The height is the prior dimension. For the first Frame target detection bounding box height; According to the target visual distance, target three-dimensional coordinate information in a visual coordinate system is calculated, and a specific calculation formula is as follows: ； ； ； ； ； ； ； wherein, is the target three-dimensional coordinate information in the visual coordinate system, is the target three-dimensional horizontal coordinate in the visual coordinate system, is the target three-dimensional vertical coordinate in the visual coordinate system, is the target three-dimensional vertical coordinate in the visual coordinate system, is the target visual distance of the first frame, is the target detection frame pixel center position normalized imaging horizontal coordinate, is the target detection frame pixel center position normalized imaging vertical coordinate, is the target detection frame pixel center position horizontal coordinate, is the visual coordinate system principal point horizontal coordinate, is the target detection frame pixel center position vertical coordinate, is the visual coordinate system principal point vertical coordinate, is the visual sensor horizontal direction focal length, is the visual sensor vertical direction focal length, is the visual sensor to the target detection frame pixel center position ray distance. 5.The target detection and tracking method based on the lightning fusion perception awareness according to claim 3, characterized in that, The original signal data is input into a radar auxiliary correction module for processing, and effective radar point cloud data in a radar coordinate system and ego motion parameters in a vehicle coordinate system are output; specifically, the processing includes: The original signal data is sequentially subjected to distance dimension FFT, Doppler FFT, azimuth angle estimation, and elevation angle estimation to obtain polar coordinate information of the target, and the polar coordinate information is converted to a radar Cartesian coordinate system to obtain original radar target measurement data and target radar point cloud data; wherein, is a distance of the target relative to the radar in a direction directly in front of the radar in the Cartesian coordinate system, is a distance of the target relative to the radar in a direction to the right of the radar in the Cartesian coordinate system, is a distance of the target relative to the radar in a direction above the radar in the Cartesian coordinate system, is a movement speed of the target along a direction directly in front of the radar, is a movement speed of the target along a direction to the right of the radar. The radar point cloud data is converted to an image coordinate system, and projection point pixel coordinate data is output; The radar point cloud data is input to a self-motion estimation module for processing, and self-motion parameters are output; wherein the self-motion estimation module adopts a plane motion estimation method based on a random sample consensus algorithm to obtain the self-motion parameters ; wherein, is a longitudinal speed of the ego vehicle, is a yaw rate The projection point pixel coordinate data and the target visual distance are input into a visual cone screening module; the visual cone screening module is configured to construct a visual cone region and limit a millimeter wave radar search range by using the target visual distance, and output effective radar point cloud data. 6.The target detection and tracking method based on the lightning visual fusion perception according to claim 5, characterized in that, The visual cone region is constructed, and the millimeter wave radar search range is limited by using the target visual distance, and effective radar point cloud data is output; specifically, the processing includes: Based on the position and height and width of the target on the image plane, the range of the target in the depth direction is calculated by monocular ranging, and an initial conical region in space is constructed; The initial conical region is expanded outward by a preset angle along the horizontal and vertical directions to obtain the visual cone region, which is used for preliminary screening of the radar point cloud data; Based on the target visual distance, a radar search range is set as , the matching screening of the visual target and the radar target is completed, and the effective radar point cloud data after the visual cone matching is obtained; wherein, is the target visual distance; Preferably, . 7.The target detection and tracking method based on the lightning fusion perception awareness according to claim 1, characterized in that, The calibration parameters between the millimeter wave radar and the visual sensor include a radar extrinsic matrix and a visual sensor extrinsic matrix; The radar extrinsic matrix and the visual sensor extrinsic matrix are corrected in real time by an extrinsic calibration module during vehicle operation; When a correction condition is triggered, the extrinsic calibration module enters a dynamic calibration mode, and the processing includes: a1, a common view target set is constructed, the common view target set is a set of targets detected by the visual sensor and the millimeter wave radar at the same time; a2, based on the radar extrinsic matrix and the visual sensor extrinsic matrix, radar targets are converted to a camera coordinate system; a3, the radar targets in the camera coordinate system are projected to a pixel plane by using camera intrinsic parameters to obtain predicted pixel coordinates, a re-projection error is constructed, and the re-projection error is the Euclidean distance between the predicted pixel coordinates and actual pixel coordinates detected by the visual sensor; a4, a random sample consensus algorithm is used to filter abnormal targets based on the re-projection error, and an effective common view target set is obtained; a5, based on the effective common view target set, a nonlinear least squares optimization model is established, the nonlinear least squares optimization model aims to minimize the overall re-projection error, and the rotation vector and the translation vector of the extrinsic matrix are optimization parameters of the nonlinear least squares optimization model; a6, a Levenberg-Marquardt optimization algorithm is used to calculate the nonlinear least squares optimization model, and an optimal extrinsic increment is obtained; a7, it is judged whether the overall re-projection error meets a preset convergence condition, if yes, the radar extrinsic matrix and the visual sensor extrinsic matrix are updated according to the optimal extrinsic increment, if not, step a6 is returned to continue iteration; wherein, the preset convergence condition is that the re-projection error change of a single iteration is less than a first preset threshold, or the norm of the extrinsic increment is less than a second preset threshold. 8.The target detection and tracking method based on the visual fusion perception of thunder and lightning according to claim 1, characterized in that, Based on the three-dimensional visual trajectory information, an initial fusion track is generated; Specifically, the processing includes: The extended Kalman filter is used to recursively estimate the target state, which is represented as: ; wherein, is the target position in the reference coordinate system, is the target two-dimensional velocity component; When the radar and visual fusion tracking module tracks for a first set time Afterwards, the initial fusion track is acquired ; wherein, is a fusion track flag, and when set to -1, indicates that the initial fusion track is temporary, is an initial position of the visual target under a reference coordinate system, is ; The historical trajectory is predicted to obtain a current time prediction trajectory, and an initial value of the historical trajectory is the initial fusion trajectory; specifically comprising: determining whether the change of the ego motion parameter is greater than a third preset threshold or the number of the visual detected targets is greater than a fourth preset threshold, if yes, using a uniform acceleration model for prediction, and if no, using a uniform speed model for prediction; obtaining the current time prediction trajectory. 9.The target detection and tracking method based on the lightning fusion perception awareness according to claim 8, characterized in that, matching and judging the current time prediction trajectory and the effective radar point cloud data; specifically comprising: using a four-degree-of-freedom Mahalanobis distance to determine whether the effective radar point cloud and the current time prediction trajectory are matched; specifically, calculating the Mahalanobis distance between the effective radar point cloud and the current time prediction trajectory, if the Mahalanobis distance is greater than a fifth preset threshold, the matching is unsuccessful; if the Mahalanobis distance is not greater than the fifth preset threshold, there is a possibility of matching, when the number of the effective radar point clouds that may be matched is greater than 1, the effective radar point with the closest Mahalanobis distance is selected for fusion trajectory updating; updating the fusion trajectory according to the matching and judging result of the trajectory matching module, outputting the updated fusion trajectory, and continuing to perform trajectory prediction; specifically comprising: When the effective radar point cloud and the predicted track continuously exist matching possibility for several frames, it is determined that the radar-visual matching is successful, and the data of the nearest effective radar point is used The updated fusion track is updated to obtain an updated fusion track ; using visual information when the valid radar point cloud does not match the predicted track updating the fused track in combination with the predicted track to obtain an updated fused track ; wherein, The order of the predicted track generation is sequentially assigned, indicating that the track is updated from transient to steady state; is the radar target position in the reference coordinate system; is the radar target two-dimensional velocity component; is the visual target position in the reference coordinate system; performing loss determination on the three-dimensional visual trajectory information and the effective radar point cloud data, and performing extinction processing or trajectory prediction on the fusion trajectory according to the loss determination result; specifically comprising: If the three-dimensional visual track information and the valid radar point cloud data are continuously lost for a second set time then it is determined that the target has left, the fused track is eliminated, and otherwise the track prediction is continued.

10. An electronic device comprising a memory, a processor, and a computer program or instructions stored on the memory, wherein the computer program or instructions, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-9. the processor executes the computer program or instruction to realize the target detection and tracking method based on radar and visual fusion perception in any one of claims 1-9.

Images