Radar-based multi-stage fusion detection and tracking method, electronic device and storage device

Abstract

The present application discloses a multi-stage fusion detection and tracking method based on radar vision, an electronic device and a storage device, the method comprising: acquiring image data and radar data at the target moment acquired by an acquisition component; acquiring a fusion detection result based on the image data and the radar data; wherein the fusion detection result comprises a first detection target and a matching target category thereof within a first preset range; and, based on the radar data, acquiring a reference detection result; wherein the reference detection result comprises a second detection target within a second preset range, and the distance between any position within the second preset range and the acquisition component is greater than the distance between any position within the second preset range and the acquisition component; acquiring a target tracking result based on the fusion detection result and the reference detection result. By means of the above method, the present application can improve the accuracy of target tracking.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a detection and tracking method based on multi-stage fusion of radar vision, an electronic device and a storage device. Background Art

[0002] With the continuous development of sensing technology, single sensor systems have gradually shown limitations in complex application scenarios, especially in the field of target detection. In order to improve the accuracy of the detection system, the technology of combining radar and visual sensors has been applied in more and more fields. However, radar sensors and visual sensors often have corresponding disadvantages, which leads to low accuracy in target tracking after radar and visual fusion.

[0003] In view of this, how to improve the accuracy of target tracking after radar and vision fusion has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a detection and tracking method, electronic device and storage device based on multi-stage fusion of radar vision, which can improve the accuracy of target tracking.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a multi-stage fusion detection and tracking method based on radar vision, including: obtaining image data and radar data at the target moment collected by the acquisition component; obtaining a fusion detection result based on the image data and the radar data; wherein the fusion detection result includes a first detection target and a matching target category within a first preset range; and, based on the radar data, obtaining a reference detection result; wherein the reference detection result includes a second detection target within a second preset range, and the distance between any position within the second preset range and the acquisition component is greater than the distance between any position within the second preset range and the acquisition component; based on the fusion detection result and the reference detection result, obtaining a target tracking result.

[0006] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the radar-based multi-stage fusion detection and tracking method mentioned in the above-mentioned technical solution.

[0007] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a storage device storing program instructions that can be executed by a processor, and the program instructions are used to implement the radar-based multi-stage fusion detection and tracking method mentioned in the above technical solution.

[0008] The beneficial effects of the present application are as follows: Different from the prior art, the multi-stage fusion detection and tracking method based on radar vision proposed in the present application combines image data and radar data to obtain a fusion detection result within the first preset range close to the acquisition component, giving full play to the accuracy and stability of image data for target detection in a close range. In addition, the reference detection result within the second preset range far away from the acquisition component is determined using radar data, reducing the probability that the low clarity of long-distance image data will interfere with the detection result. Finally, the target tracking result is obtained based on the fusion detection result and the reference detection result to improve the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flowchart of an implementation method of the present application based on multi-stage fusion detection and tracking of radar vision;

[0011] Figure 2 yes Figure 1 Step S102 corresponds to a flow chart of another implementation method;

[0012] Figure 3 It is a structural schematic diagram of an implementation mode corresponding to the detection model of the present application;

[0013] Figure 4 yes Figure 2 Step S201 corresponds to a flow chart of another implementation method;

[0014] Figure 5 yes Figure 2 Step S202 corresponds to a flow chart of another implementation method;

[0015] Figure 6 yes Figure 1 Step S103 corresponds to a flow chart of another implementation method;

[0016] Figure 7 yes Figure 1 Step S104 corresponds to a flow chart of another implementation method;

[0017] Figure 8 It is a structural schematic diagram of an implementation method of the present application based on the multi-stage fusion detection and tracking system of radar vision;

[0018] Fig. 9It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0019] Fig.10 It is a structural diagram of an implementation method of a storage device of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] See also Figure 1 , Figure 1 This is a flow chart of an implementation method of a multi-stage fusion detection and tracking method based on radar vision in this application, and the method includes:

[0022] S101: Acquire image data and radar data at a target moment collected by a collection component.

[0023] In one embodiment, the target scene is collected by using the collection component to obtain image data and radar data at the target time, wherein the target scene is the range that the collection component can detect.

[0024] Specifically, the acquisition component includes an image collector and a radar collector. The image collector is used to collect RGB images of the target scene; the radar collector is used to transmit radar signals and collect radar echo signals. The image data and radar data at the target time are obtained by preprocessing the collected RGB images and radar echo signals respectively.

[0025] In one implementation scenario, the process of preprocessing the collected RGB image includes: converting the image in RGB format into a BEV (Bird's Eye View) image, and using the BEV image as image data. This preprocessing method projects the data in the three-dimensional space onto a two-dimensional plane to help align and combine the image data with the radar data. The method of preprocessing the radar echo signal includes removing noise, etc., and using the preprocessed radar echo signal as radar data. In addition, the above-mentioned image collector and radar collector are integrated into one body to serve as an acquisition component.

[0026] In another embodiment, after the image data and the radar data are obtained, the radar data and the image data are converted into coordinate representations in the same target coordinate system, so that the image data and the radar data are aligned according to the converted coordinates.

[0027] In a specific application scenario, taking the coordinate conversion in image data as an example, the specific calculation formula is as follows:

[0028]

[0029]

[0030] in, and Used to represent the current coordinates of each pixel in the image data. and Used to indicate the current coordinates of the acquisition component. and Used to represent the target coordinates of each pixel in the converted image data.

[0031] In another embodiment, the acquisition component may also include a split-type image collector and a radar collector, wherein the positional relationship between the image collector and the radar collector is determined so as to align and combine the image data with the radar data according to the positional relationship.

[0032] S102: Acquire a fusion detection result based on the image data and the radar data; wherein the fusion detection result includes a first detection target within a first preset range and a matching target category thereof.

[0033] In one embodiment, after acquiring the image data and radar data corresponding to the target moment, the image data and radar data are combined to detect a first detected target within a first preset range and a target category matched by the first detected target. The first detected target and its matched target category are used as fusion detection results. Among them, the target categories include motor vehicles and non-motor vehicles, etc.

[0034] In one implementation scenario, the first preset range is determined based on the distance between different positions in the target scene and the acquisition component. For example, in the target scene, the area whose distance to the acquisition component is less than or equal to the distance threshold is taken as the first preset range.

[0035] S103: Based on the radar data, obtain a reference detection result; wherein the reference detection result includes a second detection target within a second preset range, and the distance between any position within the second preset range and the acquisition component is greater than the distance between any position within the second preset range and the acquisition component.

[0036] In one embodiment, after preprocessing the radar data corresponding to the current moment, target detection is performed based on the preprocessed radar data to obtain a second detected target within a second preset range, and the second detected target is used as a reference detection result.

[0037] In one implementation scenario, the second preset range is also determined based on the distance between different positions in the target scene and the acquisition component. For example, in the target scene, the area whose distance to the acquisition component is greater than a distance threshold is used as the second preset range.

[0038] In another embodiment, in response to the acquisition component also including a split-type image collector and a radar collector, the acquisition component takes the image collector or the radar collector as the actual position, and determines the first preset range and the second preset range based on the actual position; or, the acquisition component takes the middle position between the image collector and the radar collector as the actual position, and determines the first preset range and the second preset range based on the actual position.

[0039] In addition, it should be noted that, in actual application, step S102 and step S103 are independent of each other, that is, step S102 can be executed first, and then step S103; or, step S103 can be executed first, and then step S102; or, step S102 and step S103 can be executed simultaneously.

[0040] S104: Obtain a target tracking result based on the fusion detection result and the reference detection result.

[0041] In one embodiment, in response to the fused detection result including a first detection target and its matching target category within a first preset range, and the reference detection result including a second detection target within a second preset range, the fused detection result and the reference detection result are fused to determine all detection targets included in the target scene at the target moment.

[0042] Furthermore, after acquiring all the detection targets contained in the target scene at multiple consecutive target moments, the detection targets in the target scene at different target moments are matched to determine the running trajectory of each detection target in the target scene, and the running trajectory is used as the target tracking result.

[0043] The multi-stage fusion detection and tracking method based on radar vision proposed in this application combines image data and radar data to obtain a fusion detection result within the first preset range close to the acquisition component, giving full play to the accuracy and stability of image data for target detection in a close range. In addition, radar data is used to determine the reference detection result within the second preset range far away from the acquisition component, reducing the probability that the low clarity of long-distance image data will interfere with the detection result. Finally, the target tracking result is obtained based on the fusion detection result and the reference detection result to improve the accuracy of target tracking.

[0044] See also Figure 2 , Figure 2 yes Figure 1Step S102 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S102 includes:

[0045] S201: Acquire target image features corresponding to image data and target radar features corresponding to radar data.

[0046] In one embodiment, image feature extraction is performed on the obtained image data to obtain corresponding target image features; and radar feature extraction is performed on the obtained radar data to obtain corresponding target radar features.

[0047] In one implementation scenario, see Figure 3 , Figure 3 It is a structural diagram of an embodiment of the detection model of the present application. In response to the fusion detection result being obtained using the trained detection model, the obtained image data is input into the first feature extraction network in the detection model to extract the corresponding target image features; and the radar data is input into the second feature extraction network in the detection model to extract the corresponding target radar features. Among them, the first feature extraction network and the second feature extraction network are commonly used convolutional neural networks, and the specific structure is not described in detail here.

[0048] In addition, the above detection model is obtained by training using multiple training samples, the training samples include training image data and its matching training radar data, and each training sample includes a corresponding training label, the training label includes a labeled training target and a matching target type. The specific training process can refer to the training process of the existing neural network model, and will not be elaborated in detail here.

[0049] S202: Acquire target fusion features based on target image features and target radar features.

[0050] In one embodiment, the obtained target image features and target radar features are fused to obtain target fusion features. The target fusion features contain information in both the image data and the radar data to help improve the accuracy of subsequent detection.

[0051] Specifically, the feature fusion method may be feature splicing.

[0052] S203: Obtain fusion detection results based on the target fusion features.

[0053] In one embodiment, target detection is performed on the target fusion feature to identify all first candidate targets in the target scene and the target category matched by each first candidate target. The first candidate target located within the first preset range is taken as the first detection target, and the first detection target and the target category matched by it are taken as the fusion detection result.

[0054] In an implementation scenario, please continue to refer to Figure 3 , input the target fusion feature into the detection network in the detection model to obtain a first detection target within a first preset range. And, input the target fusion feature into the classification network in the detection model to obtain a target category within the first preset range. Wherein, the distance between the first detection target and the acquisition component is less than or equal to the distance threshold.

[0055] Specifically, the detection network includes an encoder, a decoder and a detection layer connected in sequence, and the encoder and the decoder are used to further extract and analyze the target fusion features, and finally the detection layer outputs all the first candidate targets detected in the target scene. In addition, since the target fusion features are obtained based on the target radar features, the first candidate target output by the detection layer is also matched with a corresponding first reference distance, which is used to characterize the distance between the corresponding first candidate target and the acquisition component, and the first candidate target whose first reference distance is less than or equal to the distance threshold is used as the first detection target.

[0056] And, the classification network includes an encoder, a decoder and a classification layer connected in sequence, the encoder and the decoder are used to further extract and analyze the target fusion features, and finally the classification layer outputs all target categories in the target scene.

[0057] Furthermore, based on the first detection target and the target category within the first preset range, a fusion detection result is obtained.

[0058] Specifically, the detected first detection targets are matched with target categories to determine the target categories matched by each first detection target, and the first detection targets and the target categories matched by them are used as fusion detection results.

[0059] In another embodiment, after the target fusion feature is input into the detection network in the detection model, the detection network is used to detect the first candidate target within the first preset range, and all the first candidate targets detected are used as the first detection target. And, after the target fusion feature is input into the classification network in the detection model, the classification network is used to output the target category within the first preset range. Finally, according to the positional relationship, the first detection target is matched with the target category to obtain a fusion detection result.

[0060] The above scheme, since image data can clearly display information of closer areas, obtains fusion detection results within the first preset range through target fusion features, combines the advantages of using image data and radar data in target detection, and improves the accuracy of detection.

[0061] See also Figure 4 , Figure 4 yes Figure 2 Step S201 in the flowchart corresponds to another embodiment. Specifically, the implementation process of obtaining the target radar feature in step S201 may also include:

[0062] S301: Acquire initial radar features based on distance information, frequency information and angle information in radar data.

[0063] In one embodiment, in response to the radar data including frequency information of the pulse signal, distance information of the candidate target in the target scene, and angle information of the candidate target in the target scene, an initial radar feature is obtained based on the above frequency information, distance information, and angle information.

[0064] Specifically, in response to the second feature extraction network in the detection model including the first sub-network, the frequency information, the distance information and the angle information are input into the first sub-network, so that the first sub-network performs fast Fourier transform on the distance information and the angle information, and performs low-pass filtering on the frequency information, and then outputs a RAF (Range-Azimuth-Frequency) feature map corresponding to the radar data with richer information. The obtained RAF feature map is used as the initial radar feature.

[0065] S302: Perform convolution processing on the initial radar features to obtain target radar features in the target dimension.

[0066] In one implementation, a convolution process is performed on the initial radar features, and the features obtained after the convolution process are mapped to the target dimension to obtain the target radar features.

[0067] Specifically, in response to the second feature extraction network in the detection model further including a second sub-network, and the second sub-network including a convolution layer and a pooling layer, the obtained initial radar feature is input into the convolution layer, so that the convolution layer performs convolution processing on the initial radar feature using a preset convolution kernel to obtain a reference radar feature. The reference radar feature is input into the pooling layer, so that the pooling layer performs maximum pooling processing on the reference radar feature, and finally obtains the target radar feature under the target dimension.

[0068] In a specific implementation scenario, the initial radar feature is a feature map of dimension (N, H, W), where N represents the radar signal frequency, H represents the detection distance, and W represents the detection azimuth. After the initial radar feature is convolved with the convolution kernel (n, 1, 1), a feature map of dimension (C, H, W) is obtained, where C represents the number of convolution kernels. Then, the obtained feature map is subjected to maximum pooling to obtain the target radar feature of dimension (H, W).

[0069] It should be noted that, to facilitate subsequent detection, the resolution of the target radar feature is the same as the resolution of the target image feature.

[0070] The above scheme obtains the initial radar features based on the radar data first, and then obtains the target radar features based on the initial radar features, so that the target radar features finally obtained have richer information, which helps to improve the accuracy of subsequent detection.

[0071] See also Figure 5 , Figure 5 yes Figure 2 Step S202 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S202 includes:

[0072] S401: Based on the target image features, an activation image matching the image data is obtained; wherein the activation image includes a lane area and a background area different from the lane area.

[0073] In one embodiment, lane segmentation is performed on the target image features to obtain an activation image that matches the image data.

[0074] For details, please continue to see Figure 3 , the obtained target image features are input into the segmentation network in the detection model, so as to use the segmentation network to identify the lane area and non-lane area in the corresponding image data according to the target image features, and mark the lane area and non-lane area to obtain an activation image. By obtaining the activation image, the lane area can be focused in the subsequent detection process, thereby reducing the resource consumption of calculating the interference area.

[0075] In one implementation scenario, the segmentation network includes an encoder, a decoder, and a lane segmentation layer. The encoder further extracts features of the input target image and inputs the extracted features to the decoder. The decoder parses the input features and inputs the parsed features to the lane segmentation layer, which outputs an activation image that matches the image data. Different values ​​are set in the activation image, and different values ​​are used to distinguish lane areas or non-lane areas; for example, the lane area is marked with a first value, which is "1"; and the non-lane area is marked with a second value, which is "0", so that the non-lane area is shielded in subsequent processing.

[0076] S402: Based on the target image features, the target radar features and the activation image, a target fusion feature matching the lane area is obtained.

[0077] In one embodiment, the target image feature and the target radar feature are fused to obtain an initial fused feature. Based on the initial fused feature and the activated image, the target fused feature is obtained.

[0078] Specifically, the target image features and the target radar features are concatenated to obtain the initial fusion features, which are then multiplied by the activation image to obtain the target fusion features including the lane area information.

[0079] In another embodiment, the target image feature is multiplied with the activation image to obtain a first reference feature, and the target radar feature is multiplied with the activation image to obtain a second reference feature. The first reference feature and the second reference feature are feature fused to obtain a target fusion feature.

[0080] The above scheme obtains the activation image and obtains the target fusion feature according to the activation image, so that the lane area can be focused in the subsequent detection process, thereby reducing the resource consumption of calculating the interference area.

[0081] See also Figure 6 , Figure 6 yes Figure 1 Step S103 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S103 includes:

[0082] S501: Perform radar target detection on the initial radar features to obtain candidate targets within the detection range of the acquisition component.

[0083] In one embodiment, radar target detection is performed on the obtained initial radar features to obtain all second candidate targets located in the target scene. The process of obtaining the initial radar features can refer to step S301, which will not be elaborated in detail here.

[0084] In one implementation scenario, a constant false alarm rate (CFAR) detection technology may be used to perform radar target detection on the initial radar signature, and the specific implementation process will not be described in detail herein.

[0085] S502: Based on the distance between each candidate target and the acquisition component, a second detection target is screened from all candidate targets, and the second detection target is used as a reference detection result; wherein the distance between the second detection target and the acquisition component is greater than a distance threshold.

[0086] In one embodiment, in response to each second candidate target obtained through step S501 being matched with a second reference distance, the second reference distance is used to characterize the distance between the second candidate target and the acquisition component, and the second candidate target whose second reference distance is greater than the distance threshold is used as the second detection target, and all the second detection targets obtained by screening are used as reference detection results.

[0087] In another embodiment, the constant false alarm rate detection technology may be directly used to perform radar target detection on radar data to obtain a reference detection result; or, the constant false alarm rate detection technology may be directly used to perform radar target detection on target radar features to obtain a reference detection result.

[0088] See also Figure 7 , Figure 7 yes Figure 1 Step S104 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S104 includes:

[0089] S601: In response to obtaining fused detection results and reference detection results corresponding to multiple pairwise adjacent target moments, a first running trajectory of a first detection target is obtained based on the fused detection results corresponding to the multiple target moments.

[0090] In one embodiment, in response to obtaining fusion detection results corresponding to multiple consecutive target moments, the fusion detection results corresponding to any two adjacent target moments are analyzed to determine the first matching relationship between the first detection targets respectively included in the adjacent fusion detection results. After traversing multiple target moments, the first running trajectory of each first detection target in the video stream corresponding to the multiple target moments is determined according to the first matching relationship corresponding to all adjacent target moments.

[0091] In a specific implementation scenario, for adjacent target moments X1 and X2, the fusion detection result corresponding to target moment X1 includes the first detection target a and the first detection target b, and the fusion detection result corresponding to target moment X2 includes the first detection target a' and the first detection target b'. Then, according to the positions in the target scene, the first detection target a and the first detection target a' that are close in distance are determined to be the same target, and the first detection target b and the first detection target b' that are close in distance are determined to be the same target. Thus, according to the positions corresponding to the first detection target a and the first detection target a' that match each other, their corresponding first running trajectories are determined; and according to the positions corresponding to the first detection target b and the first detection target b' that match each other, their corresponding first running trajectories are determined.

[0092] S602: Based on the reference detection results corresponding to the multiple target moments, obtain a second running trajectory of the second detection target.

[0093] In one embodiment, in response to obtaining reference detection results corresponding to a plurality of consecutive target moments, the reference detection results corresponding to any two adjacent target moments are analyzed to determine a second matching relationship between the second detection targets respectively included in the adjacent reference detection results. After traversing a plurality of target moments, the second running trajectories of different second detection targets are determined according to the second matching relationships corresponding to all adjacent target moments. The specific implementation process can refer to the above-mentioned corresponding implementation method, which will not be elaborated in detail here.

[0094] S603: Obtain a target tracking result based on the first running trajectory and the second running trajectory.

[0095] In one embodiment, the first running trajectory and the second running trajectory are fused to obtain a target tracking result, so as to intuitively display the running trajectories of all targets in the target scene.

[0096] See also Figure 8 , Figure 8 1 is a schematic diagram of a structure of a target tracking system based on multi-stage fusion of radar vision corresponding to an embodiment of the present application. Specifically, the target tracking system includes an acquisition module 10, a first processing module 20, a second processing module 30 and a tracking module 40 coupled to each other.

[0097] Specifically, the acquisition module 10 is used to acquire the image data and radar data at the target moment acquired by the acquisition component.

[0098] The first processing module 20 is used to obtain a fusion detection result based on the image data and the radar data; wherein the fusion detection result includes a first detection target within a first preset range and a matching target category thereof.

[0099] The second processing module 30 is used to obtain a reference detection result based on the radar data; wherein the reference detection result includes a second detection target within a second preset range, and the distance between any position within the second preset range and the acquisition component is greater than the distance between any position within the second preset range and the acquisition component.

[0100] The tracking module 40 is used to obtain the target tracking result based on the fusion detection result and the reference detection result.

[0101] See also Fig. 9 , Fig. 9: is a structural diagram of an embodiment of an electronic device of the present application. The electronic device includes: a memory 50 and a processor 60 coupled to each other. The memory 50 stores program instructions, and the processor 60 is used to execute the program instructions to implement the multi-stage fusion detection and tracking method based on radar vision mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 60 can also be called a CPU (Center Processing Unit). The processor 60 may be an integrated circuit chip with signal processing capabilities. The processor 60 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 60 can be implemented by an integrated circuit chip.

[0102] See also Fig.10 , Fig.10 The storage device 70 stores program instructions 80 that can be executed by a processor, and when the program instructions 80 are executed by the processor, the detection and tracking method based on multi-stage fusion of radar vision mentioned in any of the above embodiments is implemented.

[0103] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0107] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A detection and tracking method based on multi-stage fusion of radar vision, characterized in that: include: Obtain image data and radar data at the target moment collected by the acquisition component; Based on the image data and the radar data, a fusion detection result is obtained; wherein the fusion detection result includes a first detection target within a first preset range and a matching target category thereof; and, Based on the radar data, a reference detection result is obtained; wherein the reference detection result includes a second detection target within a second preset range, and the distance between any position within the second preset range and the acquisition component is greater than the distance between any position within the first preset range and the acquisition component; Based on the fusion detection result and the reference detection result, obtaining a target tracking result; Wherein, obtaining the fusion detection result based on the image data and the radar data includes: obtaining the target image features corresponding to the image data and the target radar features corresponding to the radar data; obtaining the target fusion features based on the target image features and the target radar features; obtaining the fusion detection result based on the target fusion features; Among them, obtaining the target radar feature corresponding to the radar data includes: obtaining an initial radar feature based on distance information, frequency information and angle information in the radar data; performing convolution processing on the initial radar feature to obtain the target radar feature in the target dimension; specifically including: performing convolution processing on the initial radar feature to obtain a reference radar feature, and performing maximum pooling processing on the reference radar feature to obtain the target radar feature.

2. The method according to claim 1, characterized in that The acquiring target fusion features based on the target image features and the target radar features includes: Based on the target image features, acquiring an activation image matching the image data; wherein the activation image includes a lane area and a background area different from the lane area; The target fusion feature matching the lane area is acquired based on the target image feature, the target radar feature and the activation image.

3. The method according to claim 2, characterized in that The acquiring the target fusion feature matching the lane area based on the target image feature, the target radar feature and the activation image includes: Fusing the target image feature and the target radar feature to obtain an initial fused feature; The target fusion feature is obtained based on the initial fusion feature and the activation image.

4. The method according to claim 1, characterized in that: In response to the fusion detection result being obtained using the trained detection model, the obtaining of the fusion detection result based on the target fusion feature includes: Inputting the target fusion feature into the detection network in the detection model to obtain the first detection target within the first preset range; wherein the distance between the first detection target and the acquisition component is less than or equal to a distance threshold; and, Inputting the target fusion features into the classification network in the detection model to obtain the target category within the first preset range; The fusion detection result is obtained based on the first detection target and the target category within the first preset range.

5. The method according to claim 1, characterized in that: The obtaining of a reference detection result based on the radar data includes: Performing radar target detection on the initial radar features to obtain candidate targets within the detection range of the acquisition component; Based on the distance between each of the candidate targets and the acquisition component, a second detection target is screened from all the candidate targets, and the second detection target is used as the reference detection result; wherein the distance between the second detection target and the acquisition component is greater than a distance threshold.

6. The method according to claim 1, characterized in that The obtaining of the target tracking result based on the fusion detection result and the reference detection result includes: In response to obtaining a plurality of fused detection results and reference detection results respectively corresponding to two or more adjacent target moments, obtaining a first running trajectory of a first detection target based on the fused detection results corresponding to the plurality of target moments; and Based on the reference detection results corresponding to the multiple target moments, obtaining a second running trajectory of the second detection target; The target tracking result is obtained based on the first running trajectory and the second running trajectory.

7. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the radar-based multi-stage fusion detection and tracking method according to any one of claims 1-6.

8. A storage device, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the radar-based multi-stage fusion detection and tracking method described in any one of claims 1 to 6.

Images