Target detection method, target detection device and computer readable storage medium

Abstract

The present application provides a target detection method, a target detection device and a computer-readable storage medium. In the target detection method of the present application, a first image feature and a second image feature are fused to obtain a third image feature; for a first sub-region at the same position, according to a preset rule, one of the first image feature in the first sub-region, the second image feature in the first sub-region and the third image feature in the first sub-region is used as a target sub-feature of the first sub-region; a target image feature is obtained according to the target sub-feature corresponding to the first sub-region; target detection is performed according to the target image feature to obtain a target detection result of the target to be detected. The target detection method of the present application can effectively improve the accuracy of target detection.

Description

Technical Field

[0001] The present application belongs to the field of target detection, and specifically relates to a target detection method, a target detection device and a computer-readable storage medium. Background Art

[0002] Currently, in the field of intelligent transportation, a single visible light camera is usually used as a sensor. However, with the development of technology and the increase in demand, a single visible light camera can no longer solve the complex and changeable situations in intelligent transportation. Radar cameras are an effective supplement to visible light cameras. How to comprehensively utilize radar cameras and visible light cameras, that is, how to perform radar and visual fusion, has become a new trend and problem in the field of intelligent transportation. Summary of the invention

[0003] The present application provides a target detection method, a target detection device and a computer-readable storage medium to solve the problem of being unable to accurately detect targets.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: to provide a target detection method, the method comprising: acquiring a first image and a second image, wherein the first image is obtained based on data collected by a radar on a target scene, and the second image is obtained based on data collected by a camera on the target scene; performing feature extraction on the first image to obtain first image features, and performing feature extraction on the second image to obtain second image features; performing fusion processing on the first image features and the second image features to obtain third image features; using the same division method, respectively dividing the first image features, the second image features and the third image features into at least one first sub-region; for the first sub-region at the same position, according to a preset rule, one of the first image feature in the first sub-region, the second image feature in the first sub-region, and the third image feature in the first sub-region is used as a target sub-feature of the first sub-region; according to the target sub-feature corresponding to the first sub-region, a target image feature is obtained; according to the target image feature, target detection is performed to obtain a target detection result of the target to be detected.

[0005] Another technical solution adopted in the present application is to provide a target detection device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the method described in the above embodiment.

[0006] Another technical solution adopted by the present application is: providing a computer-readable storage medium, the computer-readable storage medium storing a computer program, and the computer program can be executed by a processor to implement any of the steps in the above method embodiments.

[0007] The beneficial effect of the present application is that in the target detection method of the present application, for the first sub-region at each position, one of the first image feature, the second image feature and the third image feature is used as the target sub-feature of the first sub-region, and the target image feature is obtained according to the target sub-feature corresponding to the first sub-region at each position, and finally the target detection is performed according to the target image feature, which can effectively improve the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0009] Figure 1 It is a flowchart of an embodiment of a target detection method in this application;

[0010] Figure 2 It is a structural diagram of an embodiment of a target detection model in this application;

[0011] Figure 3 It is a structural schematic diagram of an implementation method of a target detection device in the present application;

[0012] Figure 4 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] See also Figure 1 , the present application provides a target detection method, the method comprising:

[0015] S10: Acquire a first image and a second image, wherein the first image is obtained based on data collected by a radar on a target scene, and the second image is obtained based on data collected by a camera on the target scene.

[0016] Specifically, the first image is obtained by collecting data of the target scene through the radar, and the second image is obtained by collecting data of the target scene through the camera. In one embodiment, the first image and the second image have the same size.

[0017] The execution subject of this embodiment is a target detection device, which can receive data collected by the radar and the data collected by the camera, and process the data collected by the radar to obtain a first image, and process the data collected by the camera to obtain a second image. In one embodiment, the target detection device is independent of the radar and the camera; in another embodiment, the target detection device and at least one of the radar and the camera can be integrated on the same device at the same time.

[0018] S20: performing feature extraction on the first image to obtain first image features, and performing feature extraction on the second image to obtain second image features.

[0019] Specifically, by performing feature extraction on both the acquired first image and the second image, first image features in the first image and second image features in the second image are obtained. The same extraction method can be used to extract features from the first image and the second image, or different extraction methods can be used to extract features from the first image and the second image.

[0020] S30: Fusing the first image feature and the second image feature to obtain a third image feature.

[0021] Specifically, by fusing the acquired first image feature and the second image feature, a third image feature is obtained, and the third image feature combines the advantages of the first image feature and the second image feature.

[0022] S40: Using the same division method, the first image feature, the second image feature and the third image feature are respectively divided into at least one first sub-region.

[0023] Specifically, the first image feature, the second image feature, and the third image feature all include the same photographing area, and the same division method can be used to divide the first image feature, the second image feature, and the third image feature into at least one first sub-area. In one embodiment, the first image feature, the second image feature, and the third image feature can be divided into one first sub-area; in another embodiment, the first image feature, the second image feature, and the third image feature can be divided into more than two first sub-areas. In one implementation, the first image feature, the second image feature, and the third image feature have the same size.

[0024] The same division method means that the first image feature, the second image feature and the third image feature are respectively divided into the first sub-regions of equal number, and the multiple first sub-regions corresponding to the first image feature, the multiple first sub-regions corresponding to the second image feature and the multiple first sub-regions corresponding to the third image feature correspond to each other one by one, and the corresponding three first sub-regions have the same position in their respective corresponding image features. For ease of understanding, an example is provided here for explanation:

[0025] In one example, the first image feature, the second image feature, and the third image feature are all divided into N×M first sub-regions, where N is the number of rows and M is the number of columns.

[0026] S50: For the first sub-region at the same position, according to a preset rule, one of the first image feature in the first sub-region, the second image feature in the first sub-region, and the third image feature in the first sub-region is used as the target sub-feature of the first sub-region.

[0027] Specifically, for the first sub-region at the same position, one of the corresponding first image feature, second image feature and third image feature can be selected as the target sub-feature according to the preset rule. In one embodiment, the preset rule is random selection, that is, for the first sub-region at the same position, one of the first image feature in the first sub-region, the second image feature in the first sub-region and the third image feature in the first sub-region is randomly selected as the target sub-feature of the first sub-region. In another embodiment, the first sub-region at the same position is set in advance to select the first image feature in the first sub-region, the second image feature in the first sub-region or the third image feature in the first sub-region as the target sub-feature of the first sub-region at the position, and then for the first sub-region at each position, the corresponding target sub-feature is selected according to the preset regulations. For ease of understanding, an example is given here for illustration:

[0028] Assume that in this example, the first image feature, the second image feature and the third image feature are all divided into 3×3 first sub-regions, and for the first sub-region in the first row and the first column, the first image feature in the first sub-region is pre-set as the target sub-feature corresponding to the first sub-region, for the first sub-region in the first row and the second column, the third image feature in the first sub-region is pre-set as the target sub-feature corresponding to the first sub-region, and for the first sub-region in the first row and the third column, the second image feature in the first sub-region is pre-set as the target sub-feature corresponding to the first sub-region, and so on. Corresponding target sub-features are also set for the first sub-regions at the other 6 positions. When target detection is performed subsequently, the corresponding target sub-feature is selected for the first sub-region at each position according to the pre-set rules.

[0029] S60: Obtain target image features according to the target sub-features corresponding to the first sub-region.

[0030] Specifically, the target sub-features corresponding to the first sub-regions at all positions may be spliced ​​to obtain the target image feature. For example, in one embodiment, if the first image feature, the second image feature, and the third image feature are all divided into 3×3 first sub-regions, the target sub-features corresponding to the 9 first sub-regions are finally spliced ​​to obtain the target image feature.

[0031] S70: Perform target detection according to the target image features to obtain the target detection result of the target to be detected.

[0032] Specifically, target detection is performed on the target image features to obtain a detection result, wherein the target detection result of the target to be detected may include information such as the position of the target to be detected and the confidence level of the target to be detected.

[0033] Among them, the advantage of radar lies in the longitudinal distance, and the advantage of camera lies in the horizontal field of view. There are overlaps and differences in the advantageous areas of the two. In the prior art, usually after the first image feature and the second image feature are fused, target detection is directly performed based on the third image feature, but this method will cause the application effect of some areas to be inferior to that of a single sensor. However, it can be seen from the above content that in the process of generating target image features, for each first sub-area, the present application selects one of the corresponding first image features, second image features, and third image features according to a preset rule as the target sub-feature corresponding to the first sub-area, that is, the target image feature combines the first image feature, the second image feature, and the third image feature at the same time, so that the target image feature can fuse the advantages of the first image feature, the second image feature, and the third image feature, so that the target image feature can finally detect the target based on the target image feature to ensure the accuracy of detection.

[0034] In one embodiment, before step S10, the method further includes:

[0035] S11: Acquire radar data collected by the radar on the target scene.

[0036] S12: Convert the radar data from the polar coordinate system to the image coordinate system to obtain a third image.

[0037] S13: Perform grayscale processing on the third image to obtain the first image.

[0038] Specifically, since the radar cannot directly collect image data, the coordinate system of the radar data collected by the radar is first converted, that is, from the polar coordinate system to the image coordinate system to obtain the third image, and then the third image is gray-processed to obtain the first image, that is, the first image is a grayscale image, wherein the image grayscale value in the first image represents the reflection intensity.

[0039] In one embodiment, step S10: the step of acquiring the first image and the second image includes:

[0040] S101: Acquire radar data collected by a radar on a target scene at a first moment, and obtain a first image based on the radar data.

[0041] Specifically, the first image is obtained based on radar data collected by the radar for the target scene at the first moment.

[0042] S102: Acquire a second image of the target scene captured by the camera at the first moment, wherein the acquisition frequency of the radar is lower than the acquisition frequency of the camera.

[0043] Specifically, the second image captured by the camera at the first moment of the target scene is selected, and the acquisition time corresponding to the second image and the first image is the same. Since the acquisition frequency of the radar is lower than that of the camera, that is, in the same time period, the number of radar acquisitions is lower than the number of camera acquisitions, in order to ensure that the acquisition time of the first image and the second image is the same, the first moment of the radar acquisition is taken as the basis, and the image captured by the camera at the first moment is taken as the second image.

[0044] In one embodiment, step S50 includes:

[0045] S51: In response to satisfying the first condition, the first image feature in the first sub-region is used as the target sub-feature corresponding to the first sub-region; in response to satisfying the second condition, the second image feature in the first sub-region is used as the target sub-feature corresponding to the first sub-region; otherwise, the third image feature in the first sub-region is used as the target sub-feature corresponding to the first sub-region.

[0046] The first condition includes: a first target confidence corresponding to the first sub-region is greater than or equal to a first threshold, and a second target confidence corresponding to the first sub-region is less than a second threshold, wherein the first target confidence represents the confidence that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first image, and the second target confidence represents the confidence that the target to be detected exists in the third sub-region corresponding to the first sub-region in the second image;

[0047] The second condition includes: the first target confidence corresponding to the first sub-region is less than the first threshold, and the second target confidence corresponding to the first sub-region is greater than or equal to the second threshold.

[0048] Specifically, the first image and the second image are also divided in the manner of dividing the first image feature, the second image feature, and the third image feature, so that the first image is divided into a plurality of second sub-regions, and the second image is divided into a plurality of third sub-regions. The second sub-region corresponding to the first sub-region in the first image refers to the second sub-region in the first image that has the same position as the first sub-region in the first image feature / the second image feature / the third image feature. The third sub-region corresponding to the first sub-region in the second image refers to the third sub-region in the second image that has the same position as the first sub-region in the first image feature / the second image feature / the third image feature.

[0049] For ease of understanding, an example is given here: the first image feature, the second image feature, and the third image feature are all divided into N×M first sub-regions, and correspondingly, the first image is divided into N×M second sub-regions, and the second image is divided into N×M third sub-regions, where N is the number of rows and M is the number of columns. Then the first sub-region in the first row and the first column corresponds to the second sub-region in the first row and the first column, and the third sub-region in the first row and the first column, the first sub-region in the first row and the second column corresponds to the second sub-region in the first row and the second column, and the third sub-region in the first row and the second column, and so on.

[0050] The first target confidence represents the confidence that the target to be detected exists in the second subregion corresponding to the first subregion in the first image. The higher the first target confidence, the greater the probability that the target to be detected exists in the second subregion corresponding to the first subregion in the first image. The first image feature in the first subregion includes more features of the target to be detected. When the target to be detected does not exist in the second subregion corresponding to the first subregion in the first image, the first target confidence is 0. The second target confidence represents the confidence that the target to be detected exists in the third subregion corresponding to the first subregion in the second image. The higher the second target confidence, the greater the probability that the target to be detected exists in the third subregion corresponding to the first subregion in the second image. The second image feature in the first subregion includes more features of the target to be detected. When the target to be detected does not exist in the third subregion corresponding to the first subregion in the second image, the second target confidence is 0.

[0051] Therefore, when the first target confidence corresponding to the first sub-region is greater than or equal to the first threshold, and the second target confidence corresponding to the first sub-region is less than the second threshold (that is, when the first condition is met), it means that the probability that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first image is relatively high, and the probability that the target to be detected exists in the third sub-region corresponding to the first sub-region in the second image is relatively low, so that the first image feature in the first sub-region includes more features of the target to be detected, and the second image feature in the first sub-region includes fewer features of the target to be detected, and therefore the first image feature in the first sub-region is used as the target sub-feature corresponding to the first sub-region.

[0052] When the first target confidence corresponding to the first sub-region is less than the first threshold, and the second target confidence corresponding to the first sub-region is greater than or equal to the second threshold (that is, when the second condition is met), it means that the probability that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first image is small, and the probability that the target to be detected exists in the third sub-region corresponding to the first sub-region in the second image is large, so that the first image feature in the first sub-region includes fewer features of the target to be detected, and the second image feature in the first sub-region includes more features of the target to be detected, and therefore the second image feature in the first sub-region is used as the target sub-feature corresponding to the first sub-region.

[0053] When neither the first condition nor the second condition is met, it means that the probability that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first image is small, and the probability that the target to be detected exists in the third sub-region corresponding to the first sub-region in the second image is also small; or, the probability that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first image is large, and the probability that the target to be detected exists in the third sub-region corresponding to the first sub-region in the second image is also large. Therefore, the third image feature in the first sub-region is used as the target sub-feature corresponding to the first sub-region.

[0054] The first threshold and the second threshold may be equal or unequal in magnitude, and the magnitudes of the first threshold and the second threshold may be set according to actual conditions.

[0055] In one embodiment, the first target confidence level corresponding to the first sub-region and the second target confidence level corresponding to the first sub-region are determined in the following manner:

[0056] S21: Perform target detection according to the first image feature to obtain a first detection result of the target to be detected, wherein the first detection result includes a first detection frame of the target to be detected and a first confidence level corresponding to the first detection frame.

[0057] Specifically, target detection is performed on the first image feature to obtain a first detection frame of the target to be detected and first confidence information corresponding to the first detection frame, thereby obtaining a first detection result. It is understandable that the first detection result may include a first detection frame of a target to be detected and a first confidence corresponding to the first detection frame, or may include a first detection frame of each of multiple targets to be detected and a first confidence corresponding to each of the first detection frames, depending on the number of targets to be detected that exist in the second sub-region corresponding to the first sub-region in the first image. Among them, the first confidence information corresponding to the first detection frame represents the probability that the target to be detected exists in the first detection frame.

[0058] S22: Perform target detection according to the second image feature to obtain a second detection result of the target to be detected, wherein the second detection result includes a second detection frame of the target to be detected and a second confidence level corresponding to the second detection frame.

[0059] Specifically, target detection is performed on the second image feature to obtain a second detection frame of the target to be detected and second confidence information corresponding to the second detection frame, thereby obtaining a second detection result. It is understandable that the second detection result may include a second detection frame of a target to be detected and a second confidence corresponding to the second detection frame, or may include a second detection frame of each of a plurality of targets to be detected and a second confidence corresponding to each of the second detection frames, depending on the number of targets to be detected that exist in the third sub-region corresponding to the first sub-region in the second image. Among them, the second confidence information corresponding to the second detection frame represents the probability that the target to be detected exists in the second detection frame.

[0060] S23: Determine a first target confidence corresponding to the first sub-region based on a first confidence corresponding to a first detection box existing in the first sub-region, and determine a second target confidence corresponding to the first sub-region based on a second confidence corresponding to a second detection box existing in the first sub-region.

[0061] In one embodiment, in step S23: the step of determining the first target confidence corresponding to the first sub-region according to the first confidence corresponding to the first detection box existing in the first sub-region includes:

[0062] S231: Calculate the average value of the first confidence corresponding to the first detection box in the first sub-region to obtain the first target confidence corresponding to the first sub-region.

[0063] Specifically, the average value of the first confidences corresponding to the first detection boxes in the first sub-region can be used as the first target confidence corresponding to the first sub-region, that is, the first confidences corresponding to all the first detection boxes in the first sub-region are first summed to obtain a sum value, and then the sum value is divided by the number of first detection boxes in the first sub-region to obtain the first target confidence corresponding to the first sub-region.

[0064] Of course, in other embodiments, the median, mode, maximum value or minimum value of the first confidence corresponding to the first detection box in the first sub-region may also be used as the first target confidence corresponding to the first sub-region.

[0065] And, in step S23: the step of determining the second target confidence corresponding to the first sub-region according to the second confidence corresponding to the second detection frame existing in the first sub-region includes:

[0066] S232: Calculate the average value of the second confidence corresponding to the second detection box in the first sub-region to obtain the second target confidence corresponding to the first sub-region.

[0067] Specifically, the average value of the second confidence corresponding to the second detection box in the first sub-region can be used as the second target confidence corresponding to the first sub-region, that is, the second confidence corresponding to all the second detection boxes in the first sub-region are first summed to obtain a sum value, and then the sum value is divided by the number of second detection boxes in the first sub-region to obtain the second target confidence corresponding to the first sub-region.

[0068] Of course, in other embodiments, the median, mode, maximum value or minimum value of the second confidence corresponding to the second detection box in the first sub-region may also be used as the second target confidence corresponding to the first sub-region.

[0069] No matter whether the positions of the camera and the radar are fixed or not, the above steps S21-S23 can be used to determine the first target confidence corresponding to the first sub-area and the second target confidence corresponding to the first sub-area.

[0070] In another embodiment, the first target confidence level corresponding to the first sub-region and the second target confidence level corresponding to the first sub-region are determined in the following manner:

[0071] S24: Determine the confidence of the first target corresponding to the first sub-area based on the confidence that there is a target to be detected in a second sub-area corresponding to the first sub-area in at least one first historical image, wherein the first historical image is obtained based on data collected by the radar.

[0072] Specifically, the first historical image is an image obtained based on historical data collected by the radar. Therefore, the confidence that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first historical image can also reflect the confidence that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first image. Therefore, the confidence of the first target corresponding to the first sub-region can be determined based on the confidence that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first historical image.

[0073] S25: Determine the confidence of the second target corresponding to the first sub-area based on the confidence that there is a target to be detected in a third sub-area corresponding to the first sub-area in at least one second historical image, wherein the second historical image is obtained based on data collected by the camera.

[0074] Specifically, the first historical image is an image obtained based on historical data collected by the camera. Therefore, the confidence that the target to be detected exists in the third sub-area corresponding to the first sub-area in the second historical image can also reflect the confidence that the target to be detected exists in the third sub-area corresponding to the first sub-area in the second image. Therefore, the confidence of the second target corresponding to the first sub-area can be determined based on the confidence that the target to be detected exists in the third sub-area corresponding to the first sub-area in the second historical image.

[0075] In one embodiment, in step S24: according to the confidence that the target to be detected exists in the second sub-region corresponding to the first sub-region in at least one first historical image, the step of determining the confidence of the first target corresponding to the first sub-region includes:

[0076] S241: Calculate the average value of the confidence that the target to be detected exists in the second sub-region in the first historical image, and obtain the first target confidence corresponding to the first sub-region.

[0077] Specifically, the average value of the confidence that the target to be detected exists in the second sub-region in the first historical image can be used as the first target confidence corresponding to the first sub-region. The mode, median, maximum value or minimum value of the confidence that the target to be detected exists in the second sub-region in the first historical image can also be used as the first target confidence corresponding to the first sub-region. In one embodiment, when there is no target to be detected in the second sub-region in the first historical image, the corresponding confidence is 0, and the first target confidence corresponding to the first sub-region is also 0.

[0078] And, in step S25: according to the confidence that the target to be detected exists in the third sub-region corresponding to the first sub-region in at least one second historical image, the step of determining the confidence of the second target corresponding to the first sub-region includes:

[0079] S251: Calculate the average value of the confidence that the target to be detected exists in the third sub-region in the second historical image, and obtain the second target confidence corresponding to the first sub-region.

[0080] Specifically, the average value of the confidence that the target to be detected exists in the third sub-region in the second historical image can be used as the second target confidence corresponding to the first sub-region. In other implementations, the mode, median, maximum value or minimum value of the confidence that the target to be detected exists in the third sub-region in the second historical image can also be used as the second target confidence corresponding to the first sub-region. In one embodiment, when there is no target to be detected in the third sub-region in the second historical image, the corresponding confidence is 0, and the second target confidence corresponding to the third sub-region is also 0.

[0081] In one embodiment, in response to the positions of the radar and the camera being unchanged, steps S24-S25 are used to determine the first target confidence corresponding to the first sub-area and the second target confidence corresponding to the first sub-area.

[0082] Specifically, when the positions of the radar and the camera remain unchanged, steps S24-S25 are used to determine the first target confidence corresponding to the first sub-area and the second target confidence corresponding to the first sub-area. When the positions of the radar and the camera change, steps S21-S23 are used to determine the first target confidence corresponding to the first sub-area and the second target confidence corresponding to the first sub-area.

[0083] In one embodiment, the process of obtaining the third image feature in step S30 includes:

[0084] S31: performing splicing processing on the first image feature and the second image feature.

[0085] In one implementation, the first image feature and the second image feature may be concatenated in a channel dimension.

[0086] S32: Process the spliced ​​image features based on the attention mechanism to obtain a third image feature.

[0087] Specifically, the concatenated feature results can be sent to attention mechanism modules such as Squeeze-and-Excitation attention (SE) and convolutional block attention module (CBAM) to obtain the third image feature.

[0088] It should be noted that the present application does not limit the specific process of obtaining the third image feature.

[0089] See also Figure 2In one embodiment, in order to improve the image processing speed, the target detection method is participated in by using the target detection model. In this implementation, step S20 includes:

[0090] S26: Input the first image 10 into the first detection network 30 in the target detection model to obtain a first image feature.

[0091] Specifically, the target detection model includes a first detection network 30, and the first image feature can be obtained by inputting the first image 10 into the first detection network 30. In one embodiment, the first detection network 30 includes a first feature extraction subnetwork 31 and a first detection subnetwork 32. After the first image 10 is input into the first detection network 30, the first feature extraction subnetwork 31 extracts features from the first image 10 to obtain first image features, and then after receiving the first image features output by the first feature extraction subnetwork 31, the first detection subnetwork 32 performs target detection on the first image features to obtain a first detection frame of the target to be detected and a first confidence level corresponding to the first detection frame.

[0092] S27: Input the second image 20 into the second detection network 40 in the target detection model to obtain second image features.

[0093] Specifically, the target detection model also includes a second detection network 40. The second image 20 is input into the second detection network 40 to obtain the second image features. In one embodiment, the second detection network 40 includes a second feature extraction subnetwork 41 and a second detection subnetwork 42. After the second image 20 is input into the second detection network 40, the second feature extraction subnetwork 41 extracts features from the second image 20 to obtain second image features. After receiving the second image features output by the second feature extraction subnetwork 41, the second detection subnetwork 42 performs target detection on the second image features to obtain a second detection frame of the target to be detected and a second confidence level corresponding to the second detection frame.

[0094] And, step S30 includes:

[0095] S31: Input the first image feature and the second image feature into the fusion module 50 in the target detection model to obtain the third image feature.

[0096] Specifically, the target detection model also includes a fusion module 50. After receiving the first image feature output by the first feature extraction subnetwork 31 and the second image feature output by the second feature extraction subnetwork 41, the fusion module 50 fuses the first image feature and the second image feature to obtain a third image feature.

[0097] And, step S70 includes:

[0098] S71: Input the target image features into the detection module 70 in the target detection model to obtain the target detection result.

[0099] Specifically, the target detection model also includes an adaptation module 60. After receiving the first image feature, the second image feature and the third image feature, the adaptation module 60 obtains the target image feature according to the above steps. Finally, the target image feature is sent to the detection module 70, and then the detection module 70 processes the target image feature to obtain the target detection result.

[0100] In one embodiment, the target detection method further includes:

[0101] S81: Obtain a sample set, where the sample set includes multiple first sample images and multiple second sample images, and the multiple first sample images and the multiple second sample images correspond to each other one by one. The first sample images are obtained based on data collected by the radar, and the second sample images are obtained based on data collected by the camera. The corresponding first sample images and second sample images correspond to the same collection scene.

[0102] Specifically, for the same acquisition scene, the radar is used to acquire the corresponding first sample image, and the camera is used to acquire the corresponding second sample image. The first sample image and the second sample image correspond one to one to ensure the accuracy of data acquisition. Multiple first sample images and multiple second sample images constitute a sample set.

[0103] Before training the sample set, the first sample image and the second sample image are also labeled, and the labeled labels are the detection boxes and categories of the targets to be detected in the first sample image and the second sample image.

[0104] During the training process, the sample set is further divided into a training set and a validation set. The first sample image in the training set corresponds to the second sample image one-to-one, and the first sample image in the validation set corresponds to the second sample image one-to-one.

[0105] S82: Use the sample set to train the first detection network 30 and the second detection network 40, and freeze the parameters of the trained first detection network 30 and the second detection network 40.

[0106] Specifically, the first detection network 30 and the second detection network 40 are first trained using a sample set, and after the training of the first detection network 30 and the second detection network 40 is completed, the first detection network 30 and the second detection network 40 are frozen, that is, in the subsequent training process, the first detection network 30 and the second detection network 40 will no longer be trained.

[0107] S83: Use the sample set to train the fusion module 50 and the detection module 70.

[0108] The sample set is further used to train the fusion module 50 and the detection module 70 to further improve the operating accuracy of the target detection model.

[0109] It should be noted that this application does not limit the training process of the target detection model.

[0110] See also Figure 3 The present application also provides a target detection device 200, which includes a memory 210 and a processor 220 connected to each other, wherein the memory 210 is used to store a computer program, and when the computer program is executed by the processor 220, it is used to implement the method in any of the above-mentioned embodiments.

[0111] See also Figure 4 The present application also provides a computer-readable storage medium 400, wherein the computer-readable storage medium 400 stores a computer program 410, and the computer program 410 can be executed by a processor to implement the steps as described in any one of the above method embodiments.

[0112] The computer-readable storage medium 400 may specifically be a device that can store the computer program 410, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer program 410. The server may send the stored computer program 410 to other devices for execution, or may also execute the stored computer program 410 by itself.

[0113] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire a first image and a second image, wherein the first image is obtained based on data collected by a radar on a target scene, and the second image is obtained based on data collected by a camera on the target scene; Performing feature extraction on the first image to obtain first image features, and performing feature extraction on the second image to obtain second image features; fusing the first image feature and the second image feature to obtain a third image feature; Using the same division method, the first image feature, the second image feature, and the third image feature are respectively divided into at least one first sub-region; For the first sub-region at the same position, according to a preset rule, one of the first image feature in the first sub-region, the second image feature in the first sub-region, and the third image feature in the first sub-region is used as a target sub-feature of the first sub-region; Obtaining a target image feature according to the target sub-feature corresponding to the first sub-region; Performing target detection according to the target image features to obtain a target detection result of the target to be detected; The step of taking, according to a preset rule, one of the first image feature in the first sub-region, the second image feature in the first sub-region, and the third image feature in the first sub-region as the target sub-feature of the first sub-region includes: In response to satisfying a first condition, taking a first image feature in the first sub-region as a target sub-feature corresponding to the first sub-region; in response to satisfying a second condition, taking a second image feature in the first sub-region as a target sub-feature corresponding to the first sub-region; otherwise, taking a third image feature in the first sub-region as a target sub-feature corresponding to the first sub-region; Wherein, the first condition includes: a first target confidence corresponding to the first sub-region is greater than or equal to a first threshold, and a second target confidence corresponding to the first sub-region is less than a second threshold, wherein the first target confidence represents the confidence that the target to be detected exists in the second sub-region corresponding to the first sub-region in the first image, and the second target confidence represents the confidence that the target to be detected exists in the third sub-region corresponding to the first sub-region in the second image; The second condition includes: the first target confidence corresponding to the first sub-area is less than the first threshold, and the second target confidence corresponding to the first sub-area is greater than or equal to the second threshold; The first image and the second image are divided in a manner of dividing the first image features, the second image features and the third image features, the first image is divided into a plurality of second sub-regions, and the second image is divided into a plurality of third sub-regions.

2. The target detection method according to claim 1, characterized in that: The method further comprises: Performing target detection according to the first image feature to obtain a first detection result of the target to be detected, wherein the first detection result includes a first detection frame of the target to be detected and a first confidence level corresponding to the first detection frame; Performing target detection according to the second image feature to obtain a second detection result of the target to be detected, wherein the second detection result includes a second detection frame of the target to be detected and a second confidence level corresponding to the second detection frame; The first target confidence corresponding to the first sub-region is determined based on the first confidence corresponding to the first detection box existing in the first sub-region, and the second target confidence corresponding to the first sub-region is determined based on the second confidence corresponding to the second detection box existing in the first sub-region.

3. The target detection method according to claim 1, characterized in that: The method further comprises: Determining the confidence of the first target corresponding to the first sub-region according to the confidence that the target to be detected exists in a second sub-region corresponding to the first sub-region in at least one first historical image, wherein the first historical image is obtained based on data collected by the radar; The confidence of the second target corresponding to the first sub-area is determined based on the confidence that the target to be detected exists in a third sub-area corresponding to the first sub-area in at least one second historical image, wherein the second historical image is obtained based on data collected by a camera.

4. The target detection method according to claim 3, characterized in that: The method further comprises: In response to the unchanged positions of the radar and the camera, the step of determining the confidence of the first target corresponding to the first sub-area based on the confidence that the target to be detected exists in a second sub-area corresponding to the first sub-area in at least one first historical image, and the step of determining the confidence of the second target corresponding to the first sub-area based on the confidence that the target to be detected exists in a third sub-area corresponding to the first sub-area in at least one second historical image are performed.

5. The target detection method according to claim 1, characterized in that: The steps of extracting features from the first image to obtain first image features, and extracting features from the second image to obtain second image features, include: Inputting the first image into a first detection network in an object detection model to obtain the first image feature; Inputting the second image into a second detection network in the target detection model to obtain the second image features; Furthermore, the step of fusing the first image feature and the second image feature to obtain a third image feature includes: Inputting the first image feature and the second image feature into a fusion module in the object detection model to obtain the third image feature; And, the step of performing target detection according to the target image features to obtain a target detection result of the target to be detected includes: The target image features are input into the detection module in the target detection model to obtain the target detection result.

6. The target detection method according to claim 5, characterized in that: The method further comprises: Acquire a sample set, the sample set comprising a plurality of first sample images and a plurality of second sample images, the plurality of first sample images and the plurality of second sample images being in one-to-one correspondence, The first sample image is obtained based on data collected by the radar, and the second sample image is obtained based on data collected by the camera, and the corresponding first sample image and second sample image correspond to the same collection scene; Using the sample set to train the first detection network and the second detection network, and freezing the parameters of the trained first detection network and the second detection network; The fusion module and the detection module are trained using the sample set.

7. The target detection method according to claim 1, characterized in that: The step of acquiring the first image and the second image comprises: Acquire radar data collected by the radar on the target scene at a first moment, and obtain the first image according to the radar data; The second image of the target scene captured by the camera at the first moment is acquired, wherein an acquisition frequency of the radar is lower than an acquisition frequency of the camera.

8. A target detection device, characterized in that: The method comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the method according to any one of claims 1 to 7.

Images