Bimodal tracking sensing method and system for far small target based on point cloud inspiration

Abstract

The invention relates to the field of target detection and multi-sensor data fusion, in particular to a point cloud inspiration-based far and small target bimodal tracking sensing method and a point cloud inspiration-based far and small target bimodal tracking sensing system. The method comprises the steps that original radar point cloud and video data are acquired and preprocessed, and a point cloud cluster vector set and original image data are acquired; the point cloud cluster vector set, the original image data and the original radar point cloud pass through a bimodal fusion perception framework to obtain the position and identity characteristics of a far small target in an image domain; and performing bimodal joint tracking, completing sensing of a far small target, and outputting position information of the sensed target in the aerial view. According to the method, firstly, preprocessing is carried out to reduce noise points, then the image representation capability of the far-small target is inspired and enhanced through radar point cloud, then the detection accuracy of the far-small target is improved through bimodal feature fusion, and position information and identity features of the far-small target in an image domain are obtained. And finally, bimodal joint tracking is carried out to realize continuous sensing of a far small target.

Description

Technical Field

[0001] The present invention relates to the field of target detection and multi-sensor data fusion, and more specifically, to a point cloud-inspired dual-modal tracking perception method and system for distant small targets. Background Art

[0002] Millimeter-wave radar is the main sensor in the long-distance target perception technology of automobiles. Compared with other on-board sensors such as laser radar and ultrasonic radar, it has the advantages of long detection distance, strong all-weather working ability, high detection accuracy and good reliability. At present, the existing methods for perceiving distant and small targets based on the fusion of millimeter-wave radar and visual sensors can be divided into two categories. The first method fuses radar point cloud with image and extracts features for perception. This type of method usually requires additional training for distant and small targets. At the same time, for visual sensors, the size of distant and small targets in the image is often not enough to achieve target recognition. However, when millimeter-wave radar is detected at a long distance, it will be affected by environmental interference and multipath effects, resulting in the detected noise point cloud being more complex and frequent than the target point cloud, thus affecting the accuracy and continuity of the existing methods for perceiving distant and small targets. The second method relies on the detection results of the two sensors for decision-level fusion. The detection results of different sensors are weighted and then tracked to achieve the perception of distant and small targets. However, it is difficult for this type of method to improve the detection accuracy of distant and small targets in complex environments, which in turn affects the robustness of distant and small target detection.

[0003] The prior art discloses a driving environment perception method that combines image recognition and laser radar point cloud segmentation, including: (1) Collecting ground laser radar point cloud data and image data on a real road. (2) Using the collected image data as a reference, calibrating the laser radar point cloud data and image data, and marking the collected laser radar point cloud data. (3) Initializing the point cloud segmentation network, training the marked laser radar point cloud data, and updating the network parameters. (4) Transplanting the trained network to the unmanned vehicle industrial computer to obtain the category of the object to which the point cloud belongs. (5) Recognizing the image data. (6) Fusion of the segmented laser radar point cloud data and the image data after image recognition to obtain the exact location of the road and the object. However, this method will be affected by environmental interference and multipath effects when detecting at long distances, resulting in the detected noise point cloud being more complex and frequent than the target point cloud, thereby affecting the accuracy and continuity of the existing method in perceiving distant and small targets. Summary of the invention

[0004] The purpose of the present invention is to disclose a point cloud-inspired dual-modal tracking and perception method and system for distant small targets that has better perception effect on distant small targets.

[0005] In order to achieve the above object, the present invention provides a point cloud-inspired dual-modal tracking and perception method for distant small targets, comprising:

[0006] S1: Obtain the original radar point cloud and video data, pre-process the original radar point cloud and video data in the driving environment, and obtain the point cloud cluster vector set and original image data;

[0007] S2: The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of distant small targets in the image domain;

[0008] S3: Based on the position and identity features of the distant small target in the image domain and the point cloud cluster vector, dual-modal joint tracking is performed to complete the distant small target perception and output the position information of the perceived target in the bird's-eye view.

[0009] Furthermore, in step S1, it includes: using millimeter wave radar and camera to synchronously collect data, obtaining original radar point cloud and video data in the driving environment, preprocessing the original radar point cloud and video data in the driving environment, filtering out radar noise points, and obtaining a point cloud cluster vector set and original image data.

[0010] Furthermore, in step S1, the original radar point cloud and video data in the driving environment are preprocessed to filter out radar noise points, including:

[0011] Normalize raw radar point clouds and video data in driving environments;

[0012] Adjust the parameters of the filter or denoising algorithm; denoise the original radar point cloud and video data in the driving environment through the filter or denoising algorithm;

[0013] Verify the denoised data to ensure its accuracy and completeness and to ensure that the removed noise does not affect the accuracy of target detection and environmental understanding.

[0014] Furthermore, in step S1, it includes:

[0015] S1-1: Synchronously collect millimeter-wave radar and image data at time t to obtain the original radar point cloud data set PD t And the original image data VD t ,in H img and W img are the height and width of the original image respectively. The number of color channels of the original image is three

[0016] The original radar point cloud data set is represented as follows:

[0017]

[0018] in, Represents the nth point in the original radar point cloud data set at time t p radar points, n p =1,...,N p Represents the index of the radar point in the original radar point cloud data set, is the position information of the radar point in the radar space coordinate system, is the velocity information of the radar point in the radar space coordinate system;

[0019] S1-2: Get a static radar point cloud collection:

[0020]

[0021] S1-3: Get dynamic radar point cloud collection:

[0022]

[0023] S1-4: Yes Perform clustering to filter radar noise points and obtain radar point cloud cluster vector set

[0024]

[0025]

[0026] in, represents the mth radar point cloud cluster, m=1,2,...,M represents the radar point cloud cluster index, represents the nth point in the mth radar point cloud cluster at time t m radar points, n m =1,2,...,N m represents the index of the radar point in the mth cluster, express Position and velocity information in the radar space coordinate system;

[0027] S1-5: Statistics The radar point cloud information is as follows:

[0028]

[0029] Among them, cx t，m 、cy t，m 、cz t，m express The center coordinates in the radar space coordinate system, express The average velocity in all directions in the radar space coordinate system, σx t，m ,σy t，m ,σz t，m express The variance of each coordinate in the radar space coordinate system;

[0030] S1-6: Calculation The center coordinates on the pixel plane (cu t，m , cv t，m ), the formula is as follows:

[0031]

[0032] in, is the transformation matrix from the radar space coordinate system to the pixel plane;

[0033] S1-7: Get radar point cloud cluster vector set PR t :

[0034] PR t =[PR t，1 , ..., PR t，m ,...,PR t，M ]

[0035]

[0036] PR t，m is the mth radar point cloud cluster vector.

[0037] Further, in step S2, the dual-modal fusion perception framework includes: a candidate region generation subnetwork N21, an image generation module M21, an image enhancement subnetwork N22, a dual-modal fusion module M22, an image feature extraction subnetwork N23, a target detection subnetwork N24, and a target identity feature subnetwork N25;

[0038] The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of the distant small target in the image domain, including:

[0039] S2-1: PR t Input the candidate region generation subnetwork N21 to obtain the initial candidate region set RPN t :

[0040] RPN t =[RPN t，1 ,...,RPN t,m , ..., RPN t，M ]

[0041]

[0042] Among them, RPN t，m represents the mth initial candidate region, Represents RPNt，m At the center point on the pixel plane, Represents RPN t，m Width and height in pixel plane;

[0043] The candidate region generation subnetwork N21 input is PR t , the output is RPN t , which is used to obtain the position of the initial candidate area on the pixel plane by counting the radar point cloud cluster vector, wherein the subnetwork includes a first convolutional layer, a first activation layer, a first fully connected layer, a second activation layer, a first residual weighted summation point, and a second fully connected layer connected in sequence; wherein the input end of the first convolutional layer is connected to the input end of the first residual weighted summation point;

[0044] S2-2: RPN t and VD t Input into the slice image generation module M21 to obtain the normalized pre-processed slice image set

[0045] S2-3: Input the image enhancement subnetwork N22 to obtain the high-resolution slice image set HM t , where HM t =[HM t，1 , ..., H.M. t，m , ..., H.M. t，M ], H.M. t,m is the mth high-resolution slice image;

[0046] The input of the image enhancement subnetwork N22 is Output is HM t , the function of the sub-network is to improve the resolution of the normalized slice image set, enhance the appearance representation ability of the image set, and obtain high-resolution slice images, wherein the sub-network includes a second convolutional layer, several residual attention modules, an upsampling layer, a third convolutional layer, and a second residual weighted summing point connected in sequence; wherein the input end of the second convolutional layer is connected to the input end of the second residual weighted summing point after bilinear interpolation;

[0047] S2-4: HM to be obtained t 、RPN t , Input into the dual-modal fusion module M22 to obtain the dual-modal fusion slice image set RM t ;

[0048] S2-5: RM t Input into the fusion image feature extraction sub-network N23 to obtain the radar fusion image features

[0049] S2-6: Input into the target detection subnetwork N24 to obtain the image detection target Det t :

[0050]

[0051] Det t =[Det t,1 ,...,De t,n ,...,De t，N ]

[0052] Where N is the total number of detected targets, Det t，n is the nth target detected, They are the target category, the horizontal coordinate, vertical coordinate, width, and height of the target on the pixel plane;

[0053] S2-7: Det t and Input the target identity feature subnetwork N25 to obtain the target identity feature set Feat t :Feat t =[Feat t，1 , ..., Feat t，n , ..., Feat t，N ]

[0054] Among them, Feat t，n Represents the identity feature of the nth target

[0055] The target identity feature subnetwork N25 input is Det t and Output is Feat t ,The target identity feature sub-network N25 obtains the target appearance features according to the category of the detected target, the position information of the target on the pixel plane and the fusion features.

[0056] Further, in step S2-2, RPN t and VD t Input into the slice image generation module M21 to obtain the normalized pre-processed slice image set include:

[0057] S2-2-1: Calculate RPN t，m At the boundary of the pixel plane, the formula is as follows:

[0058]

[0059] in, RPN t，mThe right and left boundaries of RPN t，m The lower and upper boundaries of the candidate region, β is the scaling factor of the candidate region;

[0060] S2-2-2: Obtaining valid candidate regions The formula is as follows:

[0061]

[0062] in, is the mth valid candidate region, They are The right and left boundary values ​​of They are The lower and upper boundary values ​​of ;

[0063] S2-2-3: According to and VD t , get the initial slice image set The formula is as follows:

[0064]

[0065] S2-2-4: Obtain normalized slice image set The formula is as follows:

[0066]

[0067] Further, in step S2-4, the obtained HM t 、RPN t , Input into the dual-modal fusion module M22 to obtain the dual-modal fusion slice image set RM t include:

[0068] S2-4-1: Calculation Radial distance in bird's eye view and radial velocity The formula is as follows:

[0069]

[0070] S2-4-2: Calculation The coordinates on the pixel plane are given by the following formula:

[0071]

[0072] in, They are The horizontal and vertical coordinates on the pixel plane;

[0073] S2-4-3: Calculation The length of the projection line The formula is as follows:

[0074]

[0075] S2-4-4: Creation and HM t,m Initial pseudo-image WM of the same size but with two channels t，m , and set the initial value to 0;

[0076] S2-4-5: PR t，m Internal radar point cloud line projection to WM t，m , get the radar pseudo-image PR t，m :

[0077]

[0078] Where drawline means constructing the projection line, WM t，m represents a pseudo-graph used to construct the projection line, represents the horizontal coordinate of the center point of the projection line, represents the ordinate of the center point of the projection line, η represents the pixel width of the projection line, is expressed as the length of the projection line, Indicates that the projection line is in WM t，m The values ​​of the two channels on ;

[0079] S2-4-6: PR t，m and H.M. t，m Splice and obtain the dual-modal fusion slice image set RM t :

[0080]

[0081] RM t =[RM t，1 , ..., RM t，m , ..., RM t，M ]

[0082] Among them, RM t，m is the mth dual-modality fusion slice image, For splicing operation.

[0083] Furthermore, the target identity feature subnetwork N25 includes a fourth convolutional layer, a first batch of normalization layers, a third activation layer, a residual module, a fifth activation layer, a second batch of normalization layers and a third residual weighted summation point and a sixth activation layer connected in sequence, wherein the input end of the fourth convolutional layer is connected to the input end of the third residual weighted summation point.

[0084] Furthermore, in step S3, it includes:

[0085] S3-1: Get all perceived targets at time t:

[0086]

[0087] S3-2: Predict the existing tracker at time t-1 to obtain the predicted tracker:

[0088]

[0089] in, is the qth prediction tracker at time t, q=1,2,...Q represents the index of the tracker at time t-1, F is the state transfer matrix, is the qth existing tracker at time t-1, is the noise at time t-1, express The position of the plane in the bird's eye view, express The velocity in all directions, express identity characteristics, express The number of consecutive predictions;

[0090] S3-3:

[0091] S3-4: κ is the prediction number threshold;

[0092] S3-5: Remove tracker

[0093] S3-6: Yeah Det t Associate M and N targets, and return a set of successfully associated targets

[0094] S3-7: k = 1;

[0095] S3-8: Judgment If yes, execute S3-10-1, if no, execute S3-9;

[0096] S3-9: Judgment If yes, execute S3-10-2, if no, execute S3-20-3;

[0097] S3-10-1:

[0098] S3-10-2:

[0099] Among them, IPM means image inverse projection, Indicates the position coordinates of the image target in the bird's-eye view;

[0100] S3-11: The image perception objectives in the bird’s-eye view are:

[0101]

[0102] S3-12: Calculation and The distance D k，1：Q ;

[0103] S3-13: Determine MIN(D k，1：Q )≤μ, if yes, execute S3-14, if no, execute S3-20-1;

[0104] S3-14:

[0105] S3-15: Calculation and The cosine distance C k，q , the formula is as follows:

[0106]

[0107] S3-16: Judgment If yes, execute S3-17, if no, jump to S3-18;

[0108] S3-17:

[0109] S3-18: Determine q≤Q, if yes, execute S3-19, if no, execute S3-20-2;

[0110] S3-19: q=q+1, and jump to S3-15;

[0111] S3-20-1: Create a new tracker in,

[0112] S3-20-2: Update Tracker The formula is as follows:

[0113]

[0114] Where H is the observation matrix, G t is the filter gain;

[0115] S3-20-3: And jump to S3-22;

[0116] S3-21:

[0117] S3-22: k=k+1;

[0118] S3-23: Determine if k>K, if yes, jump to S3-8, if no, execute S3-24;

[0119] S3-24: Output all and

[0120] In addition, the present invention also provides a point cloud-inspired dual-mode tracking and perception system for distant small targets, comprising:

[0121] Acquisition module: acquires raw radar point cloud and video data, preprocesses the raw radar point cloud and video data in the driving environment, and obtains point cloud cluster vector sets and raw image data;

[0122] Position and identity feature module: The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of distant small targets in the image domain;

[0123] Position information output module: Based on the position and identity features of the distant small target in the image domain and the point cloud cluster vector, dual-modal joint tracking is performed to complete the perception of distant small targets and output the position information of the perceived target in the bird's-eye view.

[0124] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0125] The present invention first performs preprocessing to reduce noise points, then uses radar point clouds to inspire and enhance the image representation capabilities of distant small targets, and then improves the detection accuracy of distant small targets through dual-modal feature fusion, and obtains the position information and identity features of distant small targets in the image domain. Finally, dual-modal joint tracking is performed to achieve continuous perception of distant small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] Figure 1 This is a flow chart of a dual-modal tracking and perception method for distant small targets based on point cloud inspiration as described in Example 1;

[0127] Figure 2 A block diagram of a point cloud-inspired dual-modal tracking and perception system for distant small targets described in Example 2; DETAILED DESCRIPTION

[0128] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0129] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0130] Embodiment 1:

[0131] This embodiment provides Figure 1 A point cloud-inspired dual-modal tracking and perception method for distant small targets is shown, comprising:

[0132] S1: Obtain the original radar point cloud and video data, pre-process the original radar point cloud and video data in the driving environment, and obtain the point cloud cluster vector set and original image data;

[0133] S2: The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of distant small targets in the image domain;

[0134] S3: Based on the position and identity features of the distant small target in the image domain and the point cloud cluster vector, dual-modal joint tracking is performed to complete the distant small target perception and output the position information of the perceived target in the bird's-eye view.

[0135] This embodiment first performs preprocessing to reduce noise points, then uses radar point clouds to inspire and enhance the image representation capabilities of distant small targets, and then improves the detection accuracy of distant small targets through dual-modal feature fusion, and obtains the location information and identity features of distant small targets in the image domain. Finally, dual-modal joint tracking is performed to achieve continuous perception of distant small targets.

[0136] Embodiment 2:

[0137] This embodiment further discloses on the basis of the first embodiment:

[0138] In step S1, it includes: using millimeter wave radar and camera to synchronously collect data, obtaining original radar point cloud and video data in the driving environment, preprocessing the original radar point cloud and video data in the driving environment, filtering out radar noise points, and obtaining a point cloud cluster vector set and original image data.

[0139] In step S1, the original radar point cloud and video data in the driving environment are preprocessed to filter out radar noise points, including:

[0140] Normalize raw radar point clouds and video data in driving environments;

[0141] Adjust the parameters of the filter or denoising algorithm; denoise the original radar point cloud and video data in the driving environment through the filter or denoising algorithm;

[0142] Verify the denoised data to ensure its accuracy and completeness and to ensure that the removed noise does not affect the accuracy of target detection and environmental understanding.

[0143] In step S1, it includes:

[0144] S1-1: Synchronously collect millimeter-wave radar and image data at time t to obtain the original radar point cloud data set PD t And the original image data VD t ,in H img and W img are the height and width of the original image respectively. The number of color channels of the original image is three

[0145] The original radar point cloud data set is represented as follows:

[0146]

[0147] in, Represents the nth point in the original radar point cloud data set at time t p radar points, n p =1,...,N p Represents the index of the radar point in the original radar point cloud data set, is the position information of the radar point in the radar space coordinate system, is the velocity information of the radar point in the radar space coordinate system;

[0148] S1-2: Get a static radar point cloud collection:

[0149]

[0150] S1-3: Get dynamic radar point cloud collection:

[0151]

[0152] S1-4: Yes Perform clustering to filter radar noise points and obtain radar point cloud cluster vector set

[0153]

[0154]

[0155] in, represents the mth radar point cloud cluster, m=1,2,...,M represents the radar point cloud cluster index, represents the nth point in the mth radar point cloud cluster at time t m radar points, n m =1,2,...,N m represents the index of the radar point in the mth cluster, express Position and velocity information in the radar space coordinate system;

[0156] S1-5: Statistics The radar point cloud information is as follows:

[0157]

[0158] Among them, cx t，m 、cy t，m 、cz t，m express The center coordinates in the radar space coordinate system, express The average velocity in all directions in the radar space coordinate system, σx t，m ,σy t，m ,σz t，m express The variance of each coordinate in the radar space coordinate system;

[0159] S1-6: Calculation The center coordinates on the pixel plane (cu t，m , cv t，m ), the formula is as follows:

[0160]

[0161] in, is the transformation matrix from the radar space coordinate system to the pixel plane;

[0162] S1-7: Get radar point cloud cluster vector set PR t :

[0163] PR t =[PR t，1 , ..., PR t，m , ..., PR t,M ]

[0164]

[0165] PR t，m is the mth radar point cloud cluster vector.

[0166] In step S2, the dual-modal fusion perception framework includes: a candidate region generation subnetwork N21, an image generation module M21, an image enhancement subnetwork N22, a dual-modal fusion module M22, an image feature extraction subnetwork N23, a target detection subnetwork N24, and a target identity feature subnetwork N25;

[0167] The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of the distant small target in the image domain, including:

[0168] S2-1: PRt Input the candidate region generation subnetwork N21 to obtain the initial candidate region set RPN t :

[0169] RPN t =[RPN t,1 , ..., RPN t,m , ..., RPN t，M ]

[0170]

[0171] Among them, RPN t，m represents the mth initial candidate region, Represents RPN t，m At the center point on the pixel plane, Represents RPN t，m Width and height in pixel plane;

[0172] The candidate region generation subnetwork N21 input is PR t , the output is RPN t , which is used to obtain the position of the initial candidate area on the pixel plane by counting the radar point cloud cluster vector, wherein the subnetwork includes a first convolutional layer, a first activation layer, a first fully connected layer, a second activation layer, a first residual weighted summation point, and a second fully connected layer connected in sequence; wherein the input end of the first convolutional layer is connected to the input end of the first residual weighted summation point;

[0173] S2-2: RPN t and VD t Input into the slice image generation module M21 to obtain the normalized pre-processed slice image set

[0174] S2-3: Input the image enhancement subnetwork N22 to obtain the high-resolution slice image set HM t , where HM t =[HM t，1 , ..., H.M. t，m , ..., H.M. t，M ], H.M. t，m is the mth high-resolution slice image;

[0175] The input of the image enhancement subnetwork N22 is Output is HM t, the function of the sub-network is to improve the resolution of the normalized slice image set, enhance the appearance representation ability of the image set, and obtain high-resolution slice images, wherein the sub-network includes a second convolutional layer, several residual attention modules, an upsampling layer, a third convolutional layer, and a second residual weighted summing point connected in sequence; wherein the input end of the second convolutional layer is connected to the input end of the second residual weighted summing point after bilinear interpolation;

[0176] S2-4: HM to be obtained t 、RPN t , Input into the dual-modal fusion module M22 to obtain the dual-modal fusion slice image set RM t ;

[0177] S2-5: RM t Input into the fusion image feature extraction sub-network N23 to obtain the radar fusion image features

[0178] S2-6: Input into the target detection subnetwork N24 to obtain the image detection target Det t :

[0179]

[0180] Det t =[Det t,1 ,...,De t,n ,...,De t,N ]

[0181] Where N is the total number of detected targets, Det t，n is the nth target detected, They are the target category, the horizontal coordinate, vertical coordinate, width, and height of the target on the pixel plane;

[0182] S2-7: Det t and Input the target identity feature subnetwork N25 to obtain the target identity feature set Feat t :Feat t =[Feat t，1 , ..., Feat t，n , ..., Feat t，N ]

[0183] Among them, Feat t，n Represents the identity feature of the nth target

[0184] The target identity feature subnetwork N25 input is Det t and Output is Feat t ,The target identity feature sub-network N25 obtains the target appearance features according to the category of the detected target, the position information of the target on the pixel plane and the fusion features.

[0185] In step S2-2, RPN t and VD t Input into the slice image generation module M21 to obtain the normalized pre-processed slice image set include:

[0186] S2-2-1: Calculate RPN t，m At the boundary of the pixel plane, the formula is as follows:

[0187]

[0188]

[0189] in, RPN t，m The right and left boundaries of RPN t，m The lower and upper boundaries of , β is the scaling factor of the candidate region;

[0190] S2-2-2: Obtaining valid candidate regions The formula is as follows:

[0191]

[0192] in, is the mth valid candidate region, They are The right and left boundary values ​​of They are The lower and upper boundary values ​​of ;

[0193] S2-2-3: According to and VD t , get the initial slice image set The formula is as follows:

[0194]

[0195] S2-2-4: Obtain normalized slice image set The formula is as follows:

[0196]

[0197] Further, in step S2-4, the obtained HM t 、RPN t, Input into the dual-modal fusion module M22 to obtain the dual-modal fusion slice image set RM t include:

[0198] S2-4-1: Calculation Radial distance in bird's eye view and radial velocity The formula is as follows:

[0199]

[0200]

[0201] S2-4-2: Calculation The coordinates on the pixel plane are given by the following formula:

[0202]

[0203] in, They are The horizontal and vertical coordinates on the pixel plane;

[0204] S2-4-3: Calculation The length of the projection line The formula is as follows:

[0205]

[0206] S2-4-4: Creation and HM t，m Initial pseudo-image WM of the same size but with two channels t，m , and set the initial value to 0;

[0207] S2-4-5: PR t，m Internal radar point cloud line projection to WM t，m , get radar pseudo-image PM t，m :

[0208]

[0209] Where drawline means constructing the projection line, WM t，m represents a pseudo-graph used to construct the projection line, represents the horizontal coordinate of the center point of the projection line, represents the ordinate of the center point of the projection line, η represents the pixel width of the projection line, is expressed as the length of the projection line, Indicates that the projection line is in WM t，m The values ​​of the two channels on ;

[0210] S2-4-6: PM t，mand H.M. t，m Splice and obtain the dual-modal fusion slice image set RM t :

[0211]

[0212] RM t =[RM t，1 , ..., RM t，m , ..., RM t，M ]

[0213] Among them, RM t，m is the mth dual-modality fusion slice image, For splicing operation.

[0214] Among them, the target identity feature subnetwork N25 includes a fourth convolutional layer, a first batch of normalization layers, a third activation layer, a residual module, a fifth activation layer, a second batch of normalization layers and a third residual weighted summation point and a sixth activation layer connected in sequence, wherein the input end of the fourth convolutional layer is connected to the input end of the third residual weighted summation point.

[0215] In step S3, it includes:

[0216] S3-1: Get all perceived targets at time t:

[0217]

[0218] S3-2: Predict the existing tracker at time t-1 to obtain the predicted tracker:

[0219]

[0220] in, is the qth prediction tracker at time t, q=1,2,...Q represents the index of the tracker at time t-1, F is the state transfer matrix, is the qth existing tracker at time t-1, is the noise at time t-1, express The position of the plane in the bird's eye view, express The velocity in all directions, express identity characteristics, express The number of consecutive predictions;

[0221] S3-3:

[0222] S3-4: k is the prediction number threshold;

[0223] S3-5: Remove tracker

[0224] S3-6: Yeah Det t Associate M and N targets, and return a set of successfully associated targets

[0225] S3-7: k = 1;

[0226] S3-8: Judgment If yes, execute S3-10-1, if no, execute S3-9;

[0227] S3-9: Judgment If yes, execute S3-10-2, if no, execute S3-20-3;

[0228] S3-10-1:

[0229] S3-10-2:

[0230] Among them, IPM means image inverse projection, Indicates the position coordinates of the image target in the bird's-eye view;

[0231] S3-11: The image perception objectives in the bird’s-eye view are:

[0232]

[0233] S3-12: Calculation and The distance D k，1：Q ;

[0234] S3-13: Determine MIN(D k，1：Q )≤μ, if yes, execute S3-14, if no, execute S3-20-1;

[0235] S3-14:

[0236] S3-15: Calculation and The cosine distance C k，q , the formula is as follows:

[0237]

[0238] S3-16: Judgment If yes, execute S3-17, if no, jump to S3-18;

[0239] S3-17:

[0240] S3-18: Determine q≤Q, if yes, execute S3-19, if no, execute S3-20-2;

[0241] S3-19: q=q+1, and jump to S3-15;

[0242] S3-20-1: Create a new tracker in,

[0243] S3-20-2: Update Tracker The formula is as follows:

[0244]

[0245] Where H is the observation matrix, G t is the filter gain;

[0246] S3-20-3: And jump to S3-22;

[0247] S3-21:

[0248] S3-22: k=k+1;

[0249] S3-23: Determine if k>K, if yes, jump to S3-8, if no, execute S3-24;

[0250] S3-24: Output all and

[0251] This embodiment first performs preprocessing to reduce noise points, then uses radar point clouds to inspire and enhance the image representation capabilities of distant small targets, and then improves the detection accuracy of distant small targets through dual-modal feature fusion, and obtains the location information and identity features of distant small targets in the image domain. Finally, dual-modal joint tracking is performed to achieve continuous perception of distant small targets.

[0252] Embodiment three:

[0253] This embodiment provides Figure 2 A point cloud-inspired dual-modal tracking and perception system for distant small targets is shown, comprising:

[0254] Acquisition module: acquires raw radar point cloud and video data, preprocesses the raw radar point cloud and video data in the driving environment, and obtains point cloud cluster vector sets and raw image data;

[0255] Position and identity feature module: The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of distant small targets in the image domain;

[0256] Position information output module: Based on the position and identity features of the distant small target in the image domain and the point cloud cluster vector, dual-modal joint tracking is performed to complete the perception of distant small targets and output the position information of the perceived target in the bird's-eye view.

[0257] This embodiment first performs preprocessing to reduce noise points, then uses radar point clouds to inspire and enhance the image representation capabilities of distant small targets, and then improves the detection accuracy of distant small targets through dual-modal feature fusion, and obtains the location information and identity features of distant small targets in the image domain. Finally, dual-modal joint tracking is performed to achieve continuous perception of distant small targets.

[0258] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A point cloud-inspired dual-modal tracking and perception method for small and distant targets, characterized by: include: S1: Obtain the original radar point cloud and video data, pre-process the original radar point cloud and video data in the driving environment, and obtain the point cloud cluster vector set and original image data; S2: The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of distant small targets in the image domain; S3: Based on the position and identity features of the distant small target in the image domain and the point cloud cluster vector, dual-modal joint tracking is performed to complete the distant small target perception and output the position information of the perceived target in the bird's-eye view.

2. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 1 is characterized in that: In step S1, it includes: using millimeter wave radar and camera to synchronously collect data, obtaining original radar point cloud and video data in the driving environment, preprocessing the original radar point cloud and video data in the driving environment, filtering out radar noise points, and obtaining a point cloud cluster vector set and original image data.

3. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 1 is characterized in that: In step S1, the original radar point cloud and video data in the driving environment are preprocessed to filter out radar noise points, including: Normalize raw radar point clouds and video data in driving environments; Adjust the parameters of the filter or denoising algorithm; denoise the original radar point cloud and video data in the driving environment through the filter or denoising algorithm; Verify the denoised data to ensure its accuracy and completeness and to ensure that the removed noise does not affect the accuracy of target detection and environmental understanding.

4. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 2 is characterized in that: In step S1, it includes: S1-1: Synchronously collect millimeter-wave radar and image data at time t to obtain the original radar point cloud data set PD t And the original image data VD t ,in H img and W img are the height and width of the original image respectively. The number of color channels of the original image is three The original radar point cloud data set is represented as follows: in, Represents the nth point in the original radar point cloud data set at time t p radar points, n p =1,...,N p Represents the index of the radar point in the original radar point cloud data set, is the position information of the radar point in the radar space coordinate system, is the velocity information of the radar point in the radar space coordinate system; S1-2: Get a static radar point cloud collection: S1-3: Get dynamic radar point cloud collection: S1-4: Yes Perform clustering to filter radar noise points and obtain radar point cloud cluster vector set in, represents the mth radar point cloud cluster, m=1,2,...,M represents the radar point cloud cluster index, represents the nth point in the mth radar point cloud cluster at time t m radar points, n m =1,2,...,N m represents the index of the radar point in the mth cluster, express Position and velocity information in the radar space coordinate system; S1-5: Statistics The radar point cloud information is as follows: Among them, cx t，m 、cy t，m 、cz t，m express The center coordinates in the radar space coordinate system, express The average velocity in all directions in the radar space coordinate system, σx t，m ,σy t，m ,σz t，m express The variance of each coordinate in the radar space coordinate system; S1-6: Calculation The center coordinates on the pixel plane (cu t，m , cv t，m ), the formula is as follows: in, is the transformation matrix from the radar space coordinate system to the pixel plane; S1-7: Get radar point cloud cluster vector set PR t : PR t =[PR t，1 ,...,PR t，m ,...,PR t,M , PR t，m is the mth radar point cloud cluster vector.

5. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 1 is characterized in that: In step S2, the dual-modal fusion perception framework includes: a candidate region generation subnetwork N21, an image generation module M21, an image enhancement subnetwork N22, a dual-modal fusion module M22, an image feature extraction subnetwork N23, a target detection subnetwork N24, and a target identity feature subnetwork N25; The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of the distant small target in the image domain, including: S2-1: PR t Input the candidate region generation subnetwork N21 to obtain the initial candidate region set RPN t : RPN t =[RPN t，1 ,...,RPN t，m ,...,RPN t，M ] Among them, RPN t，m represents the mth initial candidate region, Represents RPN t，m At the center point on the pixel plane, Represents RPN t，m Width and height in pixel plane; The candidate region generation subnetwork N21 input is PR t , the output is RPN t , which is used to obtain the position of the initial candidate area on the pixel plane by counting the radar point cloud cluster vector, wherein the subnetwork includes a first convolutional layer, a first activation layer, a first fully connected layer, a second activation layer, a first residual weighted summation point, and a second fully connected layer connected in sequence; wherein the input end of the first convolutional layer is connected to the input end of the first residual weighted summation point; S2-2: RPN t and VD t Input into the slice image generation module M21 to obtain the normalized pre-processed slice image set S2-3: Input the image enhancement subnetwork N22 to obtain the high-resolution slice image set HM t , where HM t =[HM t，1 , ..., H.M. t，m , ..., H.M. t，M ], H.M. t，m is the mth high-resolution slice image; The input of the image enhancement subnetwork N22 is Output is HM t , the function of the sub-network is to improve the resolution of the normalized slice image set, enhance the appearance representation ability of the image set, and obtain high-resolution slice images, wherein the sub-network includes a second convolutional layer, several residual attention modules, an upsampling layer, a third convolutional layer, and a second residual weighted summing point connected in sequence; wherein the input end of the second convolutional layer is connected to the input end of the second residual weighted summing point after bilinear interpolation; S2-4: HM to be obtained t 、RPN t , Input into the dual-modal fusion module M22 to obtain the dual-modal fusion slice image set RM t ; S2-5: RM t Input into the fusion image feature extraction sub-network N23 to obtain the radar fusion image features S2-6: Input into the target detection subnetwork N24 to obtain the image detection target Det t : The t =[It t，1 ,...,The t，n ,...,The t，N ] Where N is the total number of detected targets, Det t，n is the nth target detected, They are the target category, the horizontal coordinate, vertical coordinate, width, and height of the target on the pixel plane; S2-7: Det t and Input the target identity feature subnetwork N25 to obtain the target identity feature set Feat t :Feat t =[Feat t，1 , ..., Feat t，n , ..., Feat t，N ] Among them, Feat t，n Represents the identity feature of the nth target The target identity feature subnetwork N25 input is Det t and Output is Feat t ,The target identity feature sub-network N25 obtains the target appearance features according to the category of the detected target, the position information of the target on the pixel plane and the fusion features.

6. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 5 is characterized in that: In step S2-2, RPN t and VD t Input into the slice image generation module M21 to obtain the normalized pre-processed slice image set include: S2-2-1: Calculate RPN t，m At the boundary of the pixel plane, the formula is as follows: in, RPN t，m The right and left boundaries of RPN t，m The lower and upper boundaries of the candidate region, β is the scaling factor of the candidate region; S2-2-2: Obtaining valid candidate regions The formula is as follows: in, is the mth valid candidate region, They are The right and left boundary values ​​of They are The lower and upper boundary values ​​of ; S2-2-3: According to and VD t , get the initial slice image set The formula is as follows: S2-2-4: Obtain normalized slice image set The formula is as follows:

7. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 5 is characterized in that: In step S2-4, the obtained HM t 、RPN t , Input into the dual-modal fusion module M22 to obtain the dual-modal fusion slice image set RM t include: S2-4-1: Calculation Radial distance in bird's eye view and radial velocity The formula is as follows: S2-4-2: Calculation The coordinates on the pixel plane are given by the following formula: in, They are The horizontal and vertical coordinates on the pixel plane; S2-4-3: Calculation The length of the projection line The formula is as follows: S2-4-4: Creation and HM t，m Initial pseudo-image WM of the same size but with two channels t，m , and set the initial value to 0; S2-4-5: PR t，m Internal radar point cloud line projection to WM t，m , get radar pseudo-image PM t，m : Where drawline means constructing the projection line, WM t，m represents a pseudo-graph used to construct the projection line, represents the horizontal coordinate of the center point of the projection line, represents the ordinate of the center point of the projection line, η represents the pixel width of the projection line, is expressed as the length of the projection line, Indicates that the projection line is in WM t，m The values ​​of the two channels on ; S2-4-6: PM t，m and H.M. t，m Splice and obtain the dual-modal fusion slice image set RM t : RM t =[RM t，1 ,...,RM t，m ,...,RM t，M ] Among them, RM t，m is the mth dual-modality fusion slice image, For splicing operation.

8. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 5 is characterized in that: in, The target identity feature subnetwork N25 includes a fourth convolutional layer, a first batch of normalization layers, a third activation layer, a residual module, a fifth activation layer, a second batch of normalization layers, a third residual weighted summing point and a sixth activation layer connected in sequence, wherein the input end of the fourth convolutional layer is connected to the input end of the third residual weighted summing point.

9. The point cloud-inspired dual-modal tracking and perception method for distant small targets according to claim 1, characterized in that: In step S3, it includes: S3-1: Get all perceived targets at time t: S3-2: Predict the existing tracker at time t-1 to obtain the predicted tracker: in, is the qth prediction tracker at time t, q=1,2,...Q represents the index of the tracker at time t-1, F is the state transfer matrix, is the qth existing tracker at time t-1, is the noise at time t-1, express Plane position in the bird's eye view, express The velocity in all directions, express identity characteristics, express The number of consecutive predictions; S3-3： S3-4: κ is the prediction number threshold; S3-5: Remove tracker S3-6: Yeah Det t Associate M and N targets, and return a set of successfully associated targets S3-7: k = 1; S3-8: Judgment If yes, execute S3-10-1, if no, execute S3-9; S3-9: Judgment If yes, execute S3-10-2, if no, execute S3-20-3; S3-10-1： S3-10-2： Among them, IPM means image inverse projection, Indicates the position coordinates of the image target in the bird's-eye view; S3-11: The image perception objectives in the bird’s-eye view are: S3-12: Calculation and The distance D k，1：Q ; S3-13: Determine MIN(D k，1：Q )≤μ, if yes, execute S3-14, if no, execute S3-20-1; S3-14： S3-15: Calculation and The cosine distance C k，q , the formula is as follows: S3-16: Judgment If yes, execute S3-17, if no, jump to S3-18; S3-17： S3-18: Determine q≤Q, if yes, execute S3-19, if no, execute S3-20-2; S3-19: q=q+1, and jump to S3-15; S3-20-1: Create a new tracker in, S3-20-2: Update Tracker The formula is as follows: Where H is the observation matrix, G t is the filter gain; S3-20-3: And jump to S3-22; S3-21： S3-22: k = k + 1; S3-23: Determine k>K, if yes, jump to S3-8, if no, execute S3-24; S3-24: Output all and 10. A point cloud-inspired dual-modal tracking and perception system for small and distant targets, characterized by: include: Acquisition module: acquires raw radar point cloud and video data, preprocesses the raw radar point cloud and video data in the driving environment, and obtains point cloud cluster vector sets and raw image data; Position and identity feature module: The point cloud cluster vector set, original image data and original radar point cloud are fused through a dual-modal perception framework to obtain the position and identity features of distant small targets in the image domain; Position information output module: Based on the position and identity features of the distant small target in the image domain and the point cloud cluster vector, dual-modal joint tracking is performed to complete the perception of distant small targets and output the position information of the perceived target in the bird's-eye view.

Images