A target determination method and a target determination apparatus

Abstract

The application discloses a target determination method, belongs to the field of perception fusion, can be applied to intelligent automobiles and intelligent networked automobiles, and comprises the following steps: acquiring an image to be processed and a plurality of millimeter wave detection points, the image to be processed and the plurality of millimeter wave detection points being data acquired synchronously for the same detection target, and each millimeter wave detection point comprising depth information. The plurality of millimeter wave detection points are mapped to the image to be processed. A plurality of candidate boxes of the detection target on the image to be processed are determined according to first information, and the first information comprises the depth information and position information of each millimeter wave detection point. Non-maximum suppression (NMS) processing is performed on the plurality of candidate boxes according to the depth information, so as to output a target box and a target millimeter wave detection point, and the target box is determined according to the depth information and position information of the target millimeter wave detection point. The scheme provided in the application can improve the accuracy of the correlation of the detection results of a visual sensor and a millimeter wave radar.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a target determination method and a target determination apparatus. BACKGROUND

[0002] Artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0003] Target detection and recognition refers to finding a target from a scene (for example, an image), which can include detection and recognition. Among them, detection specifically refers to judging whether a target exists, and if a target exists, determining the position of the target. Recognition specifically refers to determining the category of the target. Target detection and recognition have a wide range of applications in many fields of life, such as the field of autonomous driving, the field of driving assistance warning, etc. In the process of target detection and recognition, multi-sensor fusion is usually required, for example, the data collected by a laser radar, a millimeter wave radar, a vision sensor, an infrared sensor, etc. are fused to obtain vehicle surrounding environment information, that is, to detect and recognize target objects in the vehicle surrounding environment.

[0004] However, to accurately fuse the data of multiple sensors, one-to-one association of targets in different sensors is required, that is, multi-sensor target matching. After completing multi-sensor target matching, accurate information of the target can be obtained through fusion. Due to the different characteristics of different sensors, it is difficult to associate the detection results of heterogeneous sensors. Among them, the association of the detection results of the vision sensor and the millimeter wave radar is particularly difficult. SUMMARY

[0005] The embodiments of the present application provide a target determination method, which can improve the accuracy of the association of the detection results of the vision sensor and the millimeter wave radar.

[0006] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0007] The first aspect of the application provides a target determination method. The method provided by the application can be applied in the field of automatic driving or monitoring. The method can include: obtaining a to-be-processed image and a plurality of millimeter wave detection points. The to-be-processed image and the plurality of millimeter wave detection points are data obtained synchronously for the same detection target. The working principle of a millimeter wave radar is to use a high-frequency circuit to generate electromagnetic waves of a specific modulation frequency, and to send and receive electromagnetic waves from a target through an antenna. The parameters of the transmitted and received electromagnetic waves are used to calculate the parameters of the target. The millimeter wave radar can simultaneously measure the distance, speed, and direction of a plurality of targets. The speed is measured based on the Doppler effect, and the direction (including the horizontal angle and the vertical angle) is measured through an array of antennas. It can be understood that each millimeter wave detection point includes the parameters of the target. Specifically, the millimeter wave detection point in the present application includes depth information, i.e., a parameter obtained by distance measurement. Of course, the millimeter wave detection point can also include other parameters, such as a millimeter wave detection point that can include the depth information of the target, the speed information (a parameter obtained by speed measurement) of the target, and the direction information (a parameter obtained by direction measurement) of the target. The synchronously obtained data can be understood as data collected simultaneously by the millimeter wave radar and the image sensor, or as data collected by the millimeter wave radar and the image sensor with a frame rate deviation within a preset range. For example, for the same detection target, the millimeter wave radar collects millimeter wave detection points at a first frame rate, and the image sensor collects a to-be-processed image at a second frame rate. If the deviation between the first frame rate and the second frame rate is less than a preset threshold, it can be considered that the millimeter wave radar and the to-be-processed image are synchronously obtained data. The to-be-processed image can be obtained by a visual sensor, and the plurality of millimeter wave detection points can be obtained by a millimeter wave radar. When the method provided by the application is applied in the field of automatic driving, the to-be-processed image can be an image obtained by a vehicle through a visual sensor. Specifically, the to-be-processed image can be an image captured by a camera installed on the vehicle. When the method provided by the application is applied in a monitoring scenario, the to-be-processed image can be an image obtained by a visual sensor installed on the roadside. Specifically, the to-be-processed image can be an image captured by a camera installed on the roadside. Each millimeter wave detection point can include depth information, which is used to represent the distance between the detection target and the millimeter wave radar. The millimeter wave radar is used to obtain a plurality of millimeter wave detection points. The detection target can be a vehicle, a person, a tree, or any target. The plurality of millimeter wave detection points are mapped to the to-be-processed image. A plurality of candidate boxes of the detection target on the to-be-processed image are determined according to first information. The first information can include the depth information and the position information of each millimeter wave detection point. The position information is used to represent the position of each millimeter wave detection point mapped on the to-be-processed image. According to the depth information and the position information of each millimeter wave detection point, a group of candidate boxes can be determined. The group of candidate boxes includes a plurality of candidate boxes.The multiple candidate boxes are subjected to non-maximum suppression (NMS) processing according to the depth information, to output a target box and a target millimeter wave detection point, the target box being determined according to the depth information and the position information of the target millimeter wave detection point. According to the first aspect, by mapping the millimeter wave detection point to the image to be processed, determining the multiple candidate boxes of the image to be processed according to the position information and the depth information of the millimeter wave detection point, and subjecting the multiple candidate boxes to NMS processing according to the depth information, when the final candidate box is determined, the millimeter wave detection point associated with the candidate box can be output, and the accuracy of target matching is improved.

[0008] Optionally, in combination with the first aspect, in a first possible implementation, the NMS processing of the multiple candidate boxes according to the depth information can include: subjecting the multiple candidate boxes to NMS processing according to a first score and a second score, the first score representing a probability that a detection target in each candidate box belongs to each of N categories determined according to the classifier, the N categories being pre-set categories, and N being a positive integer, and the second score representing a probability that the detection target in each candidate box belongs to each of the N categories determined according to a first probability distribution between the depth information and each category. According to the first possible implementation of the first aspect, a specific manner of NMS processing of the multiple candidate boxes according to the depth information is given. The first possible implementation of the first aspect subjects the multiple candidate boxes to NMS processing according to the first score and the second score, improves the accuracy of data association, and improves the accuracy of one-to-one matching of the same target in different sensors.

[0009] Optionally, in combination with the first possible implementation of the first aspect, in a second possible implementation, the method can further include: performing statistics on the data in the first set to determine a probability distribution of a first size of a statistical target corresponding to each category, the first set can include multiple statistical targets corresponding to each category and size information of each statistical target, and the first probability distribution is determined according to the probability distribution of the first size and a first relationship between the size of the statistical target and the depth information of the millimeter wave detection point corresponding to the statistical target. According to the second possible implementation of the first aspect, a specific manner of determining the first probability distribution is given, and the diversity of the scheme is increased.

[0010] Optionally, in a third possible implementation form of the first aspect as above, the method further comprises: performing statistics on the data in the second set, determining a probability distribution of the second size of the statistical target corresponding to each category, the second set comprising a plurality of statistical targets corresponding to each category, and the size information of each statistical target; and determining the second probability distribution according to the second size distribution and a second relationship, the second probability distribution being used to update the first probability distribution, the second relationship being a relationship between the size of the statistical target and the depth information of the millimeter wave detection point corresponding to the statistical target. According to the third possible implementation form of the first aspect, the data can be updated. For example, in the scenario of autonomous driving, the probability distribution between the depth information and each category can be determined by using the updated data.

[0011] Optionally, in a fourth possible implementation form of the second or third possible implementation form of the first aspect as above, the size information is height information of the statistical target. According to the fourth possible implementation form of the first aspect, a specific category of size information is given, and the diversity of the scheme is increased.

[0012] Optionally, in a fifth possible implementation form of the first aspect or any one of the first to fourth possible implementation forms of the first aspect as above, the position information is used to determine the position of the candidate box in the to-be-processed image in combination with the distribution characteristics of the millimeter wave detection point on the vehicle. According to the fifth possible implementation form of the first aspect, when determining the position of the candidate box in the to-be-processed image, the distribution characteristics of the millimeter wave detection point on the vehicle are taken into account, and the position of the detection target in the to-be-processed image can be better determined by using the position of the millimeter wave detection point. For example, if it is determined according to the distribution characteristics of the millimeter wave detection point on the vehicle that the relationship between the vehicle and the millimeter wave detection point is that the millimeter wave detection point is generally located at the lower left corner of the target vehicle, then the plurality of prior boxes can be determined with the millimeter wave detection point at the lower left corner of the prior box. If the distribution characteristics of the millimeter wave detection point are not taken into account, and the position of the prior box is determined at random according to the millimeter wave detection point, for example, the plurality of prior boxes are determined with the millimeter wave detection point at the upper right corner of the prior box, then the probability that the detection target is included in the prior box will be reduced.

[0013] Optionally, in a sixth possible implementation form of the first aspect or any one of the first to fifth possible implementation forms of the first aspect as above, the depth information is used to determine the size of the candidate box, and the size of the candidate box is negatively correlated with the depth information.

[0014] Optionally, in combination with the first aspect or the first to the seventh possible implementation manners of the first aspect, in a seventh possible implementation manner, the method further can comprise: processing the to-be-processed image by a faster region with convolution neural network (Faster-RCNN) to obtain a first feature map of the to-be-processed image; extracting a second feature map corresponding to the plurality of candidate boxes from the first feature map; processing the second feature map by a regression network and a classifier to obtain a first result, the first result being used for non-maximum suppression (NMS) processing.

[0015] Optionally, in combination with the first aspect or the first to the seventh possible implementation manners of the first aspect, in an eighth possible implementation manner, the to-be-processed image is acquired by a visual sensor, a sampling frequency of the visual sensor is a first frequency, a sampling frequency of the millimeter wave radar is a second frequency, and a difference between the first frequency and the second frequency is not greater than a preset threshold.

[0016] The second aspect of the present application provides a target determination apparatus, which can comprise: an acquisition module configured to acquire a to-be-processed image and a plurality of millimeter wave detection points, the to-be-processed image and the plurality of millimeter wave detection points being data acquired synchronously for a same target, each millimeter wave detection point can comprise depth information, the depth information being used to indicate a distance between a detected target and a millimeter wave radar, and the millimeter wave radar being used to acquire the plurality of millimeter wave detection points; a mapping module configured to map the plurality of millimeter wave detection points acquired by the acquisition module to the to-be-processed image acquired by the acquisition module; and a processing module configured to determine, according to first information, a plurality of candidate boxes of the detected target on the to-be-processed image, the first information comprising depth information and position information of each millimeter wave detection point, the position information being used to indicate a position of each millimeter wave detection point mapped on the to-be-processed image, and the processing module being further configured to perform non-maximum suppression (NMS) processing on the plurality of candidate boxes according to the depth information to output a target box and a target millimeter wave detection point, the target box being determined according to the depth information and the position information of the target millimeter wave detection point.

[0017] Optionally, in combination with the second aspect, in a first possible implementation manner, the processing module is specifically configured to perform non-maximum suppression (NMS) processing on the plurality of candidate boxes according to a first score and a second score, the first score indicating a probability of a detected target in each candidate box belonging to each of N categories determined according to the classifier, the N categories being pre-set categories, and N being a positive integer, and the second score indicating a probability of the detected target in each candidate box belonging to each of the N categories determined according to a first probability distribution between the depth information and each category.

[0018] Optionally, in combination with the first possible implementation manner of the second aspect, in a second possible implementation manner, the target determination apparatus further includes a statistics module, configured to: perform statistics on the data in the first set, and determine a probability distribution of the first size of the statistical target corresponding to each category, the first set including a plurality of statistical targets corresponding to each category and size information of each statistical target; and determine the first probability distribution according to the probability distribution of the first size and a first relationship, the first relationship being a relationship between the size of the statistical target and the depth information of the millimeter wave detection point corresponding to the statistical target.

[0019] Optionally, in combination with the second possible implementation manner of the second aspect, in a third possible implementation manner, the statistics module is further configured to: perform statistics on the data in the second set, and determine a probability distribution of the second size of the statistical target corresponding to each category, the second set including a plurality of statistical targets corresponding to each category and size information of each statistical target; and determine the second probability distribution according to the second size distribution and a second relationship, the second probability distribution being used to update the first probability distribution, the second relationship being a relationship between the size of the statistical target and the depth information of the millimeter wave detection point corresponding to the statistical target.

[0020] Optionally, in combination with the second or third possible implementation manner of the second aspect, in a fourth possible implementation manner, the size information is height information of the statistical target.

[0021] Optionally, in combination with the second aspect or the first to fourth possible implementation manners of the second aspect, in a fifth possible implementation manner, the position information is used to determine the position of the candidate box in the to-be-processed image in combination with the distribution characteristics of the millimeter wave detection point on the vehicle.

[0022] Optionally, in combination with the second aspect or the first to fifth possible implementation manners of the second aspect, in a sixth possible implementation manner, the depth information is used to determine the size of the candidate box, and the size of the candidate box is negatively correlated with the depth information.

[0023] Optionally, in combination with the second aspect or the first to sixth possible implementation manners of the second aspect, in a seventh possible implementation manner, the processing module is further configured to: perform processing on the to-be-processed image by using a Faster-RCNN to obtain a first feature map of the to-be-processed image; extract a second feature map corresponding to the plurality of candidate boxes from the first feature map; and perform processing on the second feature map by using a regression network and a classifier to obtain a first result, the first result being used for non-maximum suppression (NMS) processing.

[0024] Optionally, in combination with the second aspect or the first to seventh possible implementation manners of the second aspect, in an eighth possible implementation manner, the to-be-processed image is acquired by a visual sensor, a sampling frequency of the visual sensor is a first frequency, a sampling frequency of the millimeter wave radar is a second frequency, and a difference between the first frequency and the second frequency is not greater than a preset threshold.

[0025] The third aspect of the present application provides an intelligent automobile, which can include a processor, the processor and a memory are coupled, the memory stores program instructions, when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any one of the possible implementation manners of the first aspect is executed.

[0026] The fourth aspect of the present application provides a monitoring device, which can include a processor, the processor and a memory are coupled, the memory stores program instructions, when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any one of the possible implementation manners of the first aspect is executed.

[0027] The fifth aspect of the present application provides a computer readable storage medium, which can include a program, when the program is executed on a computer, the computer executes the method described in the first aspect or any one of the possible implementation manners of the first aspect.

[0028] The sixth aspect of the present application provides a target determination system, which can include an end-side device and a cloud-side device, the end-side device is configured to acquire a to-be-processed image and a plurality of millimeter wave detection points, the to-be-processed image and the plurality of millimeter wave detection points are data acquired synchronously for the same detection target, each millimeter wave detection point can include depth information, the depth information is used to represent the distance between the detection target and the millimeter wave radar, and the millimeter wave radar is used to acquire the plurality of millimeter wave detection points. The cloud-side device is configured to receive the to-be-processed image and the plurality of millimeter wave detection points sent by the end-side device. The cloud-side device is further configured to map the plurality of millimeter wave detection points onto the to-be-processed image. The cloud-side device is further configured to determine a plurality of candidate boxes of the detection target on the to-be-processed image according to first information, the first information can include the depth information and position information of each millimeter wave detection point, and the position information is used to represent the position of each millimeter wave detection point mapped on the to-be-processed image. The cloud-side device is further configured to perform non-maximum suppression (NMS) processing on the plurality of candidate boxes according to the depth information, to output a target box and a target millimeter wave detection point, and the target box is determined according to the depth information and the position information of the target millimeter wave detection point.

[0029] Optionally, in combination with the sixth aspect, in a first possible implementation, the cloud-side device is specifically configured to perform non-maximum suppression (NMS) processing on the plurality of candidate boxes according to a first score and a second score, the first score representing a probability of a detection target in each candidate box belonging to each of N categories determined according to the classifier, the N categories being pre-set categories, and N being a positive integer, and the second score representing a probability of the detection target in each candidate box belonging to each of the N categories determined according to the first probability distribution between the depth information and each category.

[0030] Optionally, in combination with the first possible implementation of the sixth aspect, in a second possible implementation, the cloud-side device is further configured to perform statistics on data in the first set to determine a probability distribution of the first size of the statistical target corresponding to each category, the first set can include a plurality of statistical targets corresponding to each category and size information of each statistical target, and determine the first probability distribution according to the probability distribution of the first size and the first relationship, the first relationship being a relationship between the size of the statistical target and the depth information of the millimeter wave detection point corresponding to the statistical target.

[0031] Optionally, in combination with the second possible implementation of the sixth aspect, in a third possible implementation, the cloud-side device is further configured to perform statistics on data in the second set to determine a probability distribution of the second size of the statistical target corresponding to each category, the second set can include a plurality of statistical targets corresponding to each category and size information of each statistical target, and determine the second probability distribution according to the second size distribution and the second relationship, the second probability distribution being used to update the first probability distribution, and the second relationship being a relationship between the size of the statistical target and the depth information of the millimeter wave detection point corresponding to the statistical target.

[0032] Optionally, in combination with the second possible implementation or the third possible implementation of the sixth aspect, in a fourth possible implementation, the size information is height information of the statistical target.

[0033] Optionally, in combination with the sixth aspect or the first to fourth possible implementations of the sixth aspect, in a fifth possible implementation, the position information is used to determine the position of the candidate box in the image to be processed in combination with the distribution characteristics of the millimeter wave detection point on the vehicle.

[0034] Optionally, in combination with the sixth aspect or the first to fifth possible implementations of the sixth aspect, in a sixth possible implementation, the depth information is used to determine the size of the candidate box, and the size of the candidate box is negatively correlated with the depth information.

[0035] Optionally, in combination with the sixth aspect or the first possible implementation to the sixth possible implementation of the sixth aspect, in a seventh possible implementation, the cloud-side device is further configured to process the to-be-processed image by the Faster-RCNN to obtain a first feature map of the to-be-processed image, extract a second feature map corresponding to the plurality of candidate boxes from the first feature map, and process the second feature map by a regression network and a classifier to obtain a first result, wherein the first result is used for non-maximum suppression (NMS) processing.

[0036] Optionally, in combination with the sixth aspect or the first possible implementation to the seventh possible implementation of the sixth aspect, in an eighth possible implementation, the end-side device acquires the to-be-processed image by a visual sensor, a sampling frequency of the visual sensor is a first frequency, a sampling frequency of the millimeter wave radar is a second frequency, and a difference between the first frequency and the second frequency is not greater than a preset threshold.

[0037] The seventh aspect of the present application provides a model training method, which can include: acquiring a training image and a plurality of millimeter wave detection points, the training image and the plurality of millimeter wave detection points being data acquired synchronously for the same detection target, each millimeter wave detection point can include depth information, the depth information being used to represent the distance between the detection target and the millimeter wave radar, and the millimeter wave radar being used to acquire the plurality of millimeter wave detection points. Mapping the plurality of millimeter wave detection points to the training image. Determining a plurality of candidate boxes of the detection target on the training image according to first information, the first information can include the depth information and position information of each millimeter wave detection point, and the position information is used to represent the position of each millimeter wave detection point mapped on the training image. Training the model according to a feature map corresponding to the plurality of candidate boxes.

[0038] Optionally, in combination with the seventh aspect, in a first possible implementation, the position information is used to determine the position of the candidate box in the training image in combination with the distribution characteristics of the millimeter wave detection point on the vehicle.

[0039] Optionally, in combination with the seventh aspect or the first possible implementation of the seventh aspect, in a second possible implementation, the depth information is used to determine the size of the candidate box, and the size of the candidate box is negatively correlated with the depth information.

[0040] Optionally, in combination with the seventh aspect or the first or second possible implementation of the seventh aspect, in a third possible implementation, the method can further include performing convolution processing on the training image to obtain a first feature map of the training image, extracting a second feature map corresponding to the plurality of candidate boxes from the first feature map, and training the model according to the second feature map.

[0041] Optionally, in combination with the seventh aspect or the first possible implementation to the third possible implementation of the seventh aspect, in a fourth possible implementation, the training image is acquired by the visual sensor, a sampling frequency of the visual sensor is a first frequency, a sampling frequency of the millimeter wave radar is a second frequency, and a difference between the first frequency and the second frequency is not greater than a preset threshold. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1a A flowchart of a target level detection result fusion process;

[0043] Figure 1b A flowchart of a feature level fusion process;

[0044] Figure 2 A schematic diagram of detection performance of heterogeneous sensors in different dimensions;

[0045] Figure 3 A structure schematic diagram of a convolutional neural network provided by an embodiment of the present application;

[0046] Figure 4 A structure schematic diagram of another convolutional neural network provided by an embodiment of the present application;

[0047] Figure 5 A schematic diagram of an efficient region convolutional neural network;

[0048] Figure 6 A flowchart of a target determination method provided by an embodiment of the present application;

[0049] Figure 7a A schematic diagram of an application scenario of a target determination method provided by the present application;

[0050] Figure 7b A schematic diagram of an application scenario of another target determination method provided by the present application;

[0051] Figure 7c A schematic diagram of an application scenario of another target determination method provided by the present application;

[0052] Figure 7d A schematic diagram of an application scenario of another target determination method provided by the present application;

[0053] Figure 7e A schematic diagram of an application scenario of another target determination method provided by the present application;

[0054] Figure 8 A flowchart of another target determination method provided by an embodiment of the present application;

[0055] Figure 9aAnother schematic diagram of an application scenario of a target determination method provided in the application is shown in FIG. 6.

[0056] Figure 9b Another schematic diagram of an application scenario of a target determination method provided in the application is shown in FIG. 6.

[0057] Figure 10 A schematic diagram of a probability distribution of a first size provided in the application is shown in FIG. 7.

[0058] Figure 11 A schematic diagram of a flow of another target determination method provided in an embodiment of the application is shown in FIG. 8.

[0059] Figure 12 A schematic diagram of an application scenario of a target determination method provided in the application is shown in FIG. 6.

[0060] Figure 13 A comparison diagram of effects of a scheme provided in an embodiment of the application and other schemes is shown in FIG. 10.

[0061] Figure 14 A schematic diagram of a flow of a model training method provided in an embodiment of the application is shown in FIG. 11.

[0062] Figure 15 A structural schematic diagram of a target determination apparatus provided in the application is shown in FIG. 12.

[0063] Figure 16 A structural schematic diagram of a model training apparatus provided in the application is shown in FIG. 13.

[0064] Figure 17 A structural schematic diagram of another target determination apparatus provided in the application is shown in FIG. 14.

[0065] Figure 18 A structural schematic diagram of a chip provided in an embodiment of the application is shown in FIG. 15. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Those skilled in the art can know that, with the development of technology and the appearance of new scenes, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0067] In order to better understand the fields and scenes to which the schemes provided in the present application can be applied, before the technical solutions provided in the present application are specifically introduced, the related knowledge of multi-sensor information fusion is introduced.

[0068] Multi-sensor information fusion (MSIF) is an information processing process that uses computer technology to automatically analyze and synthesize information or data from multiple sensors or multiple sources under certain criteria to complete the required decision and estimation. The definition of sensor data fusion can be summarized as follows: the local data resources provided by multiple sensors of the same type or different types distributed in different locations are synthesized, computer technology is used to analyze them, the possible redundancies and contradictions among the multi-sensor information are eliminated, they are complemented, the uncertainty is reduced, a consistent interpretation and description of the measured target are obtained, and thus the rapidity and correctness of system decision, planning and reaction are improved, and the system obtains more sufficient information. In this application, the same type of sensors are also referred to as homogeneous sensors, and the different types of sensors are also referred to as heterogeneous sensors. When the distinction between the two is not emphasized, both mean the same. In addition, it should be noted that in this application, multi-sensor information fusion is also referred to as multi-sensor data fusion or multi-sensor fusion, and when the distinction between them is not emphasized, they mean the same.

[0069] The information fusion of the sensors can fuse information at different levels, such as high-level fusion of detection results of targets and feature-level fusion. The high-level fusion refers to fusing the detection results of targets at the target level from multiple homogeneous or heterogeneous sensors after obtaining the detection results of targets at the target level from the data of a single sensor. The feature-level fusion refers to fusing the extracted features from multiple homogeneous or heterogeneous sensors before forming the detection results of targets at the target level after feature extraction from the measurement data of a single sensor. The following will be described in combination with Figure 1a and Figure 1b The flowchart of the high-level fusion of the detection results of targets is shown in Figure 1a The flowchart of the feature-level fusion is shown in Figure 1b Figure 1a ​As shown, it is assumed that there are multiple sensors, which are respectively a first sensor, a second sensor and a third sensor. The first sensor obtains data through a first perception algorithm to output a first target level detection result of the target. The second sensor obtains data through a second perception algorithm to output a second target level detection result of the target. The third sensor obtains data through a third perception algorithm to output a third target level detection result of the target. The first target level detection result, the second target level detection result and the third target level detection result are further fused. For the fusion of the target level detection result, each sensor independently processes to generate target data. Each sensor has its own independent perception, such as laser radar perception, camera perception and millimeter wave radar perception. After all the sensors complete target data generation, the main processor performs data fusion. Figure 1b As shown, it is assumed that there are multiple sensors, which are respectively a first sensor, a second sensor and a third sensor. In the feature level fusion scenario, there is only one perception algorithm for perceiving the fused multi-dimensional comprehensive data. Since there is only one perception algorithm, the data obtained by each sensor needs to be synchronized in time and space. The time synchronization is to ensure that the data collected by different sensors is synchronized in time, and the space synchronization is to convert the measurement values in the respective coordinate systems of different sensors to the same coordinate system, that is, the unification of the coordinate systems.

[0070] Although multi-sensor data fusion has not formed a complete theoretical system and effective fusion algorithm, many mature and effective fusion methods have been proposed in many application fields according to their specific application backgrounds. The commonly used methods of multi-sensor data fusion can be basically summarized into two categories of random and artificial intelligence. The random method includes weighted average method, Kalman filter method, multi-Bayes estimation method, evidence reasoning and production rule; and the artificial intelligence method includes fuzzy logic theory, neural network, rough set theory and expert system.

[0071] It should be noted that to accurately fuse multi-sensor information, one-to-one matching of the target in different sensors is required, that is, multi-sensor target matching. After completing the multi-sensor target matching, the accurate information of the target can be obtained through fusion. The target matching in the present application is also referred to as the association of sensor output data. When the difference between the two is not emphasized, the two mean the same. The scheme provided by the present application focuses on how to ensure the correctness or accuracy of the association between the target level detection results of heterogeneous sensors to obtain better data fusion results, that is, to better ensure the robustness of the subsequent output detection results. It is generally believed that if the detection performance of heterogeneous sensors in a certain dimension (or function) is good, the accuracy of the association between the two is generally high. The following will be combined with the description of the embodiments to further illustrate the association between the target level detection results of heterogeneous sensors. Figure 2Please provide an explanation. For example... Figure 2 The diagram shown illustrates the detection performance of heterogeneous sensors in different dimensions. Figure 2 The document showcases three sensors: a camera, millimeter-wave radar, and lidar, along with their detection performance across seven different dimensions: object detection, object recognition, distance measurement, object edge detection, lane tracking, performance in adverse weather conditions, and performance in low light or severe exposure. Figure 2 As can be seen, both millimeter-wave radar and lidar have excellent target detection performance, meaning that the accuracy of data association between millimeter-wave radar and lidar is relatively high for target detection. However, for object edge detection, the accuracy of data association between millimeter-wave radar and lidar is lower. Furthermore, from... Figure 2 As can be seen, neither cameras nor millimeter-wave radar can perform well in all seven functions, resulting in generally low accuracy in data correlation between them. However, the complementary advantages of their measurement characteristics are also significant. Therefore, finding a way to correlate the detection results from cameras and millimeter-wave radar is of great importance.

[0072] The solution provided in this application requires correlating the output data of heterogeneous sensors using a neural network. The following section will involve extensive knowledge related to neural networks. To better understand the technical solution provided in this application, relevant knowledge about neural networks will be introduced below. It should be noted that the solution provided in this application does not limit the type of neural network; any neural network suitable for target detection can be used in the embodiments of this application.

[0073] Convolutional neural networks (CNNs) are deep neural networks with convolutional structures. They are a type of deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNNs are feed-forward artificial neural networks where each neuron responds to overlapping regions in the input image.

[0074] like Figure 3 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0075] Convolutional / pooling layers 120:

[0076] Convolutional layers:

[0077] like Figure 3 The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0078] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0079] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0080] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0081] Pooling layer:

[0082] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 3 Layers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0083] Neural network layer 130:

[0084] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 3 As shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0085] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 3 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 3 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0086] It should be noted that, as Figure 3 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 4 The multiple convolutional / pooling layers shown are used in parallel, and the extracted features are all input into the neural network layer 130 for processing.

[0087] In a preferred embodiment, the neural network of this application can employ a faster regions with convolutional neural network (Faster-RCNN). Faster RCNN is a typical object detection algorithm. In this algorithm, for an input image, multiple convolutional layers are first used to extract the basic feature map of the image. Based on the basic feature map, the region proposal network (RPN) in the Faster RCNN algorithm generates a large number of candidate boxes. These candidate boxes are then filtered, and only a fixed number are selected for input into the next stage module. Subsequently, a deeper classification analysis is performed on these fixed number of candidate boxes to finally obtain the final candidate boxes containing the target. It should be noted that the solution provided in this application does not generate a large number of candidate boxes through the RPN network, which will be explained later. The following section combines... Figure 5 This section introduces Faster R-CNN. Figure 5 This is a schematic diagram of an efficient regional convolutional neural network.

[0088] like Figure 5As shown, Faster R-CNN can be divided into four parts: convolutional layers, an RPN network, a region pooling (ROI) pooling layer, a classification layer, and a regression network. These are explained below. The convolutional layers, introduced above, are mainly used to extract image features. The input is the entire image, and the output is the extracted features, generally called feature maps. The RPN network is used to recommend candidate regions. The input is the image, and the output is multiple candidate regions. It should be noted that the scheme provided in this application does not output candidate regions through the RPN network, which will be explained later. Furthermore, it should be noted that this application sometimes refers to candidate regions as candidate boxes; unless a distinction is emphasized, the two terms are used interchangeably. The ROI pooling process can be understood as the process of pooling candidate regions. When extracting features from the original image, the corresponding first feature map is extracted. The corresponding candidate regions will then be mapped onto the first feature map; this mapping process is part of ROI pooling. Typically, a max-pooling process is also performed to obtain a second feature map, which is then fed into subsequent calculations. This second feature map corresponds to the feature map of the candidate region. The classification layer and regression network further process the second feature map, outputting the class to which the candidate region belongs and the location of the candidate region in the image.

[0089] The solution provided in this application may include two parts: an "inference" process and a "training" process. These will be described separately below.

[0090] I. Reasoning Process – Goal Determination Method

[0091] Figure 6 This is a flowchart illustrating a target determination method provided in an embodiment of this application.

[0092] like Figure 6 As shown, a target determination method provided in this application embodiment may include the following steps:

[0093] 601. Obtain the image to be processed.

[0094] The solution provided in this application can be applied to a variety of scenarios, specifically, Figure 6 The method shown can be applied in scenarios such as autonomous driving and monitoring.

[0095] when Figure 6 When the method shown is applied to the field of autonomous driving, the image to be processed in step 601 can be an image obtained by the vehicle through a vision sensor. Specifically, the image to be processed can be an image captured by a camera installed on the vehicle.

[0096] when Figure 4 When the method shown is applied to a monitoring scenario, the image to be processed in step 601 can be an image acquired by a visual sensor installed on the roadside. Specifically, the image to be processed can be an image captured by a camera installed on the roadside.

[0097] The solution provided in this application can acquire images to be processed through a visual sensor. It should be noted that the visual sensor is sometimes referred to as a camera in this application. Unless the difference between the two is emphasized, they are used to refer to the same thing.

[0098] In one possible implementation, the vision sensor may include a lens and an image sensor. An optical image of the scene generated by the lens is projected onto the image sensor, which converts it into an electrical signal. After processing such as analog-to-digital conversion (A / D), the image to be processed is obtained. This vision sensor can be any of the following specific forms: a camera, camcorder, scanner, or other device with photographic capabilities (e.g., mobile phone, tablet computer, etc.).

[0099] 602. Acquire multiple millimeter-wave detection points.

[0100] Multiple millimeter-wave detection points and the image to be processed are acquired synchronously. Each millimeter-wave detection point includes depth information, which indicates the distance between the detected target and the millimeter-wave radar. The detected target can be any target such as a vehicle, person, or tree. The synchronous acquisition of data can be understood as the millimeter-wave radar and image sensor acquiring data simultaneously, or as the frame rate deviation between the data acquisition by the millimeter-wave radar and image sensor being within a preset range. For example, for the same detected target, the millimeter-wave radar acquires millimeter-wave detection points at a first frame rate, and the image sensor acquires the image to be processed at a second frame rate. If the deviation between the first and second frame rates is less than a preset threshold, the millimeter-wave radar and the image to be processed can be considered synchronously acquired data. The millimeter-wave radar emits high-frequency millimeter waves, which are reflected by the target and collected by the receiving system. The distance to the target is determined by frequency measurement, thus forming multiple millimeter-wave detection points.

[0101] when Figure 6 When the method shown is applied to autonomous driving scenarios, the millimeter-wave detection point in step 602 can be data acquired by a millimeter-wave radar installed on the vehicle.

[0102] when Figure 6 When the method shown is applied to a monitoring scenario, the millimeter-wave detection point in step 602 can be data acquired by millimeter-wave radar on monitoring equipment installed on the road.

[0103] In this application, depth information is sometimes referred to as distance information. Without emphasizing the difference between the two, both refer to the distance between the target acquired by the millimeter-wave radar and the millimeter-wave radar.

[0104] 603. Map multiple millimeter-wave detection points onto the image to be processed.

[0105] This application can map multiple millimeter-wave detection points onto an image to be processed in various ways. For example, one method for mapping multiple millimeter-wave detection points onto an image to be processed is given below. It should be noted that all methods for mapping multiple millimeter-wave detection points onto an image to be processed in related technologies can be used in the embodiments of this application.

[0106] Mapping multiple millimeter-wave detection points onto the image to be processed involves spatial fusion of data from millimeter-wave radar and visual sensors. Specifically, this can be achieved by unifying the coordinate systems, mapping the millimeter-wave detection points acquired by the millimeter-wave radar onto the image to be processed acquired by the visual sensor. The millimeter-wave detection points determined by the millimeter-wave radar and the target determined by the visual sensor must be in the same coordinate system for better correlation and matching.

[0107] Assume the visual sensor coordinate system is (Xc, Yc, Zc), the millimeter-wave radar coordinate system is (Xr, Yr, Zr), and the three-dimensional world coordinate system is (Xw, Yw, Zw).

[0108] The coordinate system of the millimeter-wave radar can be used as a reference, and the coordinate system of the millimeter-wave radar can be set to coincide with the world coordinate system. This can be expressed by the following formula:

[0109]

[0110] Mapping image data from the visual sensor coordinate system to the world coordinate system yields the coordinates of the image data in the visual sensor coordinate system in the world coordinate system, which can be expressed by the following formula:

[0111]

[0112] f represents the focal length of the vision sensor, (u0, v0) represents the principal point of the vision sensor, dx and dy represent the pixel unit size of the vision sensor in the x and y directions, respectively, and [-a, -b, 0]. T Let θ represent the translation vector between the installation positions of the vision sensor and the millimeter-wave radar, and let θ represent the rotation angle between the millimeter-wave radar and the vision sensor. Using Equations 1-1 and 1-2 above, the coordinates of the millimeter-wave radar can be converted to the coordinates of the vision sensor, thus mapping the millimeter-wave radar detection points onto the image to be processed.

[0113] 604. Determine multiple candidate bounding boxes for the image to be processed based on the depth and location information of each millimeter-wave detection point.

[0114] Location information is used to represent the location of each millimeter-wave detection point mapped to the image to be processed.

[0115] A set of candidate bounding boxes can be determined based on the depth and location information of each millimeter-wave detection point. This set of candidate bounding boxes includes multiple candidate bounding boxes. The following sections explain how to determine candidate bounding boxes based on depth and location information.

[0116] In the scheme provided in this application, the size of the candidate bounding box is determined based on the depth information of the millimeter-wave detection point. This scheme utilizes the principle of pinhole imaging, where the closer the object distance, the larger the image; the farther the object distance, the smaller the image. According to the principle of pinhole imaging, the greater the depth information of the millimeter-wave detection point, the smaller the size of the candidate bounding box; conversely, the smaller the depth information of the millimeter-wave detection point, the larger the size of the candidate bounding box. Furthermore, the scheme provided in this application allows for the setting of prior bounding boxes. Multiple prior bounding boxes can be set; specifically, multiple regions with different sizes or aspect ratios can be set as prior bounding boxes. The candidate bounding boxes are based on these prior bounding boxes, reducing training difficulty to some extent. The size of the prior bounding boxes can be determined based on the sizes of preset categories. For example, the scheme provided in this application can identify three categories: trucks, cars, and buses. The average size of trucks, cars, and buses can be obtained through a large amount of statistical data. Therefore, for each millimeter-wave detection point, at least three sizes of prior bounding boxes can be determined. When determining the size of the candidate bounding box based on the depth information, the size of each of the three sizes of the prior bounding box can be adjusted according to the depth information of the millimeter-wave detection point.

[0117] The solution provided in this application determines the position of the candidate box based on the location of the millimeter-wave detection point mapped onto the image to be processed. In other words, the position of the candidate box is determined based on the position of each millimeter-wave detection point on the image to be processed. The solution provided in this application determines the position of the candidate box based on the distribution characteristics of the millimeter-wave detection points and their positions on the image to be processed. The distribution characteristics of the millimeter-wave detection points may vary in different scenarios. In one possible implementation, the distribution characteristics of the millimeter-wave detection points in a particular application scenario can be obtained through extensive experimental statistics for each possible application scenario. An example is given below. Assuming the solution provided in this application is applied to the field of autonomous driving, it is necessary to obtain the distribution characteristics of the millimeter-wave detection points on the vehicle. For example, the vehicle can be placed in a clean background environment (a clean background environment can be understood as minimizing other targets in the scene besides the vehicle). The millimeter-wave radar emits high-frequency millimeter waves, which are reflected by the vehicle and collected by the receiving system to obtain statistical data. By repeatedly emitting high-frequency millimeter waves with the millimeter-wave radar and collecting data multiple times for the same vehicle, or by changing the vehicle or adding different numbers of vehicles, multiple statistical analyses can be performed to obtain the distribution characteristics of the millimeter-wave detection points on the vehicle. For example, it can also obtain the distribution characteristics of millimeter-wave detection points on people, animals, or goods (such as shipping containers) according to the application needs of different scenarios.

[0118] To better understand how the solution provided in this application determines candidate bounding boxes based on depth and location information, an example of an autonomous driving scenario will be used below for illustration.

[0119] Figures 7a to 7c This is a schematic diagram illustrating an application scenario of the target determination method provided in this application. For example... Figure 7a The diagram shows the millimeter-wave detection points obtained when the target is a vehicle. Each millimeter-wave detection point contains the distance between the target and the millimeter-wave radar. Some of these points are noise points caused by multipath reflection or ray tracing, but they also contain distance information. Figure 7b As shown, with Figure 7a Taking a millimeter-wave detection point as an example, this section explains how to determine candidate boxes based on location information. It should be noted that the principle behind determining candidate boxes based on the location information of each millimeter-wave detection point is the same, and will not be repeated here. For example... Figure 7bAs shown, assuming the relationship between the vehicle and the millimeter-wave detection points is determined based on their distribution characteristics, and the millimeter-wave detection points are generally located at the lower left corner of the target vehicle, multiple prior boxes can be determined based on the lower left corner of the millimeter-wave detection points. The number of prior boxes is determined according to the predefined target category. Figure 7b As shown, assuming there are three predefined categories: Category 1, Category 2, and Category 3, the average size of Category 1 is determined through extensive statistical data; the average size of Category 2 is determined; and the average size of Category 3 is determined; then, for... Figure 7b The millimeter-wave detection points shown can acquire three different sizes of prior boxes. For example, as shown in 7c, assuming the relationship between the vehicle and the millimeter-wave detection points is determined based on their distribution characteristics on the vehicle, and the millimeter-wave detection points are generally located below the target vehicle, multiple prior boxes can be determined with the millimeter-wave detection points located below the prior boxes. In one possible implementation, multiple prior boxes can be determined at the center position of the lower edge of the prior box where the millimeter-wave detection points are located. The number and size of the prior boxes can be determined according to... Figure 7a The description in the text is for reference only and will not be repeated here. In one possible implementation, assuming the relationship between the vehicle and the millimeter-wave detection points is determined based on the distribution characteristics of the millimeter-wave detection points on the vehicle, and the millimeter-wave detection points are generally located below the target vehicle, then... Figure 7d As shown, multiple prior boxes can be determined by placing the millimeter-wave detection point at the middle position of the lower edge of the prior box.

[0120] Multiple prior boxes are determined by placing millimeter-wave detection points at the lower left corner of the prior box. From Figures 7c to 7d As can be seen, the solution provided in this application can determine multiple prior boxes based on the distribution characteristics of millimeter-wave detection points on a target, such as the distribution characteristics on a vehicle. It should be noted that... Figures 7a to 7d The methods described in this application, such as determining multiple prior boxes by placing millimeter-wave detection points at the lower left edge of the prior box and at the lower left corner of the prior box, are merely preferred embodiments of the solution provided. In some possible implementations, other methods for determining prior boxes can be selected based on the distribution characteristics of millimeter-wave detection points on the target. For example, multiple candidate boxes can be determined based on the center position of the millimeter-wave detection points within the prior box, or multiple candidate boxes can be determined based on any position to the left of the prior box. This application determines the positions of candidate boxes based on the distribution characteristics of millimeter-wave detection points on the target, which allows for better target selection; in other words, it allows for a better association between the locations of millimeter-wave detection points and the target.

[0121] In one possible implementation, when the millimeter-wave radar is installed on the front bumper of an autonomous vehicle, this application has obtained through extensive experiments that the millimeter-wave detection points are mostly distributed on the bottom and sides of the vehicle.

[0122] like Figure 7e The diagram shown is a schematic representation of an application scenario provided by an embodiment of this application. Figure 7e As shown, taking two millimeter-wave detection points as an example, this paper explains how to determine candidate bounding boxes based on the depth information of the millimeter-wave detection points. Figure 7e As shown, assuming the depth information of millimeter-wave detection point A is less than that of millimeter-wave detection point B, then the size of the candidate box determined based on millimeter-wave detection point A should be smaller than the size of the candidate box determined based on millimeter-wave detection point B. The negative correlation between depth information and candidate box size can be understood by referring to the principle of pinhole imaging, and will not be elaborated further in this application.

[0123] 605. Perform non-maximum suppression (NMS) on multiple candidate boxes based on depth information to output the target box and the target millimeter-wave detection point corresponding to the detected target in the target box.

[0124] In other words, NMS processing is performed on multiple candidate bounding boxes based on depth information to output the target bounding box and the target millimeter-wave detection point. The target bounding box is determined based on the depth and position information of the target millimeter-wave detection point. This application uses a single target as an example for illustration; however, it should be noted that the solution provided in this application is equally applicable when there are multiple targets.

[0125] Non-maximum suppression (NMS) suppresses elements that are not maxima. This method primarily aims to reduce the number of candidate boxes. In step 604, this scheme determines a large number of candidate boxes based on the depth and location information of the millimeter-wave detection point. Each candidate box, after being processed by a classifier, has a probability value belonging to a certain category, and each candidate box also corresponds to a depth value of the millimeter-wave detection point. Redundant candidate boxes can be removed using the NMS method to determine the final candidate boxes. It should be noted that this application sometimes refers to the final candidate boxes as target boxes. Unless otherwise emphasized, both refer to the boxes output after processing by the NMS method, which represent the location of the target.

[0126] The input of NMS is N candidate bounding boxes that have been sorted in descending order of scores. When there are multiple targets, multiple candidate bounding boxes will be output. For example, M candidate bounding boxes with the highest scores and not suppressed will be output, where N is a positive integer greater than M, and the scores of the candidate bounding boxes are determined according to depth information. For example, assume that there are 3 categories of targets, namely the first category, the second category, and the third category. Assume that through a large number of statistics or neural network learning, etc., the probability distribution (assumed to be the A probability distribution) between the size and depth information of the targets corresponding to the first category, the probability distribution (assumed to be the B probability distribution) between the size and depth information of the targets corresponding to the second category, and the probability distribution (assumed to be the C probability distribution) between the size and depth information of the targets corresponding to the third category are determined. Then, according to the relationship between the depth information of the targets in the candidate bounding boxes and the A probability distribution, B probability distribution, and C probability distribution, the probability that the targets in the candidate bounding boxes belong to a certain category can be determined.

[0127] To better understand the solution provided by this application, the NMS processing of multiple candidate bounding boxes according to depth information will be illustrated by an example below. First, assume that there are 6 candidate bounding boxes. For each category, the probabilities of each candidate bounding box belonging to that category are sorted according to depth information. Assume that for a certain category, the probabilities of belonging to that category from smallest to largest are A < B < C < D < E < F. Then, starting from the candidate bounding box F with the highest probability, it is judged whether the intersection over union (IOU) of the candidate bounding boxes A, B, C, D, E and F is greater than a set threshold. The IOU can be used to represent the overlap degree of two candidate bounding boxes. Assume that the overlap degrees of the candidate bounding boxes B and D with F exceed the threshold, then the candidate bounding boxes B and D are discarded, and the first candidate bounding box F is marked as the one to be retained. From the remaining candidate bounding boxes A, C, E, the candidate bounding box E with the highest probability is selected, and then the overlap degrees of the candidate bounding boxes A and C with E are judged. If the overlap degree is greater than a certain threshold, then they are discarded; and the candidate bounding box E is marked as the second candidate bounding box to be retained by us. Repeat this process to find all the candidate bounding boxes to be retained, which are the final candidate bounding boxes. And the millimeter-wave detection points corresponding to the detected targets in the final candidate bounding boxes are output.

[0128] From Figure 6 the corresponding embodiments, it can be seen that multiple candidate bounding boxes of the image to be processed are determined through the position information and depth information of the millimeter-wave detection points, and the NMS processing is performed on the multiple candidate bounding boxes according to the depth information. When the final candidate bounding boxes are determined, the millimeter-wave detection points associated with the candidate bounding boxes can be output, improving the accuracy of target matching.

[0129] From Figure 6As can be seen from the corresponding embodiments, NMS processing can be performed on multiple candidate boxes based on depth information. In some possible implementations, NMS processing can also be performed on multiple candidate boxes based on depth information combined with other information. Furthermore, there are multiple ways to determine the probability distribution between depth information and a certain category, which will be discussed below. Figure 6 Based on the corresponding embodiments, Figure 6 The corresponding embodiments can be further refined or expanded.

[0130] Figure 8 This is a flowchart illustrating another target determination method provided in an embodiment of this application.

[0131] like Figure 8 As shown, another target determination method provided in this application embodiment may include the following steps:

[0132] 801. Obtain the image to be processed.

[0133] 802. Acquire multiple millimeter-wave detection points.

[0134] 803. Map multiple millimeter-wave detection points onto the image to be processed.

[0135] 804. Determine multiple candidate bounding boxes for the image to be processed based on the depth and location information of each millimeter-wave detection point.

[0136] Steps 801 to 804 can be referred to Figure 6 Steps 601 to 604 in the corresponding embodiments are understood and will not be repeated here.

[0137] 805. Perform NMS processing on multiple candidate boxes based on the first score and the second score.

[0138] The first score represents the probability, determined by the classifier, that the detected object in each candidate box belongs to each of the N categories, where N is a predefined positive integer. The second score represents the probability, determined by the first probability distribution between depth information and each category, that the detected object in each candidate box belongs to each of the N categories.

[0139] In one possible implementation, NMS takes N candidate boxes as input, which are sorted in descending order of score, and outputs M candidate boxes with the highest score that are not suppressed, where N is a positive integer greater than M, and the score of the candidate box is determined by the product of the first score and the second score.

[0140] This application does not limit the type of classifier. The classifier scores each input candidate box; the higher the score, the greater the probability that the candidate box contains a target of the corresponding category. Related technologies that determine the score of each candidate box based on a classifier can all be used in this application. However, for candidate boxes processed by a regression network, if NMS is applied to multiple candidate boxes based only on the first score, issues may arise such as… Figure 9a The results shown exhibit significant duplication and interference. This method introduces depth information from millimeter-wave detectors, adding a dimension to determine the millimeter-wave detectors corresponding to candidate regions, thus improving the accuracy of correlation and target detection. For example... Figure 9a As shown in the diagram, assuming the candidate boxes are sorted from highest to lowest based on the first score, after the first NMS processing, there might be three millimeter-wave detector points that could be associated with the final output candidate box. By comparing the depth information of these three millimeter-wave detector points, assuming that the depth information of millimeter-wave detector point A has the highest probability of corresponding to the category of that candidate box, then after NMS processing, as shown... Figure 9b As shown, the output includes the final candidate boxes and B millimeter-wave detection points. This description is for ease of understanding, as introducing depth information can improve the accuracy of correlation. In one possible implementation, the score of each candidate box can be determined according to the following formula, i.e., the score of each candidate box can be determined based on a first score and a second score. The input to NMS is N candidate boxes sorted from highest to lowest score according to the first and second scores, and the output is M final candidate boxes with the highest scores that are not suppressed. The score of each candidate box can be expressed by the following formula:

[0141] score = p(depth) = ∑ classesp (depth, classes) = ∑ classesp (depth|classes)p(class)

[0142] p(class) = softmax(classes)

[0143] p(depth|class)~N(mean height(class) std height(class) )

[0144] Here, `score` represents the score determined based on the first and second scores, `depth` represents the depth information of each millimeter-wave detection point, and `classes` represents the target types. The number of types is preset, as explained earlier, and will not be repeated here. `p(A,B)` represents the probability that A and B occur simultaneously, i.e., the probability distribution between depth information and class. `p(A|B)` represents the probability that A occurs given the probability of B occurring, i.e., the probability distribution of depth information corresponding to a certain class. `mean` represents calculating the average, and `std` represents calculating the standard deviation. `N` represents a Gaussian distribution.

[0145] Depend on Figure 8 As can be seen from the corresponding embodiments, the solution provided in this application performs NMS processing on multiple candidate boxes through a first score and a second score, thereby improving the accuracy of data association, that is, improving the accuracy of matching the same target one by one with different sensors.

[0146] Below Figure 6 and Figure 8 Based on the corresponding implementation examples, this paper explains how to determine the probability distribution between depth information and a certain category.

[0147] In one possible implementation, the data in the first set is statistically analyzed to determine the probability distribution of the first size of the statistical target corresponding to each category. The first set includes multiple statistical targets corresponding to each category, and the size information of each statistical target. A first probability distribution is determined based on the probability distribution of the first size and a first relationship, which is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target. For example, suppose the first set includes three categories: trucks, cars, and buses, where there are 1000 samples corresponding to trucks, 1000 statistical targets corresponding to cars, and 1000 statistical targets corresponding to buses. For each statistical target, size information is included. For example, suppose statistical target A is one of the 1000 statistical targets corresponding to trucks; then A includes size information, such as at least one of the following: A's physical dimensions, or A's length, or A's width, or A's height. Based on the category of each statistical target, the size information of each statistical target can yield the probability distribution of the first size, i.e., the probability distribution of the size of the statistical target under each category. Figure 10The diagram illustrates the probability distribution of a first size when the size information is the target's height information. Furthermore, the relationship between depth information and target size can be determined using the pinhole imaging principle. In one possible implementation, the relationship between depth information and target size can be obtained by repeatedly adjusting the target's distance from the millimeter-wave radar. Once the relationship between depth information and target size, as well as the probability distribution of target size under each category, are obtained, the probability distribution between depth information and each category can be determined.

[0148] In one possible implementation, the first set can be updated, and the probability distribution between depth information and each probability category can be determined using the updated set. For example, the data in the second set can be statistically analyzed to determine a second size distribution of the statistical targets corresponding to each category. The second size distribution is used to update the first size distribution. The second set includes multiple statistical targets corresponding to each category, as well as the size information of each statistical target. A first probability distribution is determined based on the second size distribution and a second relationship, which is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target.

[0149] It should be noted that the solution provided in this application may include other steps besides those described in detail above, and the embodiments of this application do not limit these steps. A specific embodiment will be described below.

[0150] Figure 11 This is a flowchart illustrating another target determination method provided in an embodiment of this application.

[0151] like Figure 11 As shown, images are acquired using a visual sensor, and millimeter-wave detection points are obtained using a millimeter-wave radar. The millimeter-wave detection points and image frames in the video are then time-aligned. In one possible implementation, the sampling frequency of the visual sensor is a first frequency, and the sampling frequency of the millimeter-wave radar is a second frequency, with the difference between the first and second frequencies not exceeding a preset threshold. The millimeter-wave detection points are mapped onto the images acquired by the visual sensor, and then the image mapped with the millimeter-wave detection points is input into a convolutional neural network. The convolutional neural network can obtain a first feature map of the image. Based on the position and depth information of the millimeter-wave detection points in the image, multiple candidate boxes are generated, and second feature maps corresponding to the multiple candidate boxes are extracted from the first feature map. The second feature maps are processed through classification and regression layers, and then the second feature maps processed by the classification and regression layers are subjected to NMS processing based on the depth information to output the detection results of the image and the millimeter-wave detection points associated with the detection results.

[0152] Figure 12This is a schematic diagram illustrating an application scenario for the target determination method provided in this application. For example... Figure 12 The solution provided in this application can be applied to the field of autonomous driving. When applied to autonomous driving, it can detect and identify objects on the road, such as vehicles. It can detect the position of vehicles within the image range acquired by the visual sensor, the type of vehicle, and the distance between the vehicle and the driver vehicle. Here, "driver vehicle" refers to a vehicle equipped with the visual sensor. Figure 12 As shown, for each target, the detection results of the visual sensor and the detection results of the millimeter-wave detection point are matched and correlated one-to-one. It should be noted that the solution provided in this application can be applied to any scenario requiring correlation between the target-level detection results of the visual sensor and the millimeter-wave radar. For example, the solution provided in this application can be applied in surveillance scenarios. When applied to surveillance scenarios, the solution can detect and identify targets within the monitored area, such as vehicles or people. For each target, the detection results of the visual sensor and the detection results of the millimeter-wave detection point are matched and correlated one-to-one.

[0153] See Figure 13 This includes a comparison between the correlation results determined by the target determination method provided in this application and the results of a first approach, which is a method that does not perform NMS processing based on depth information. This application can be tested using a self-built database, which may include multiple images with millimeter-wave detection points and manually categorized annotations. 50 This indicates that the overlap between the final candidate bounding box and the target object in the image is 50%. It can be understood that the method provided in this application improves the accuracy of object detection by performing NMS processing based on the first score determined by the classifier and the second score determined by depth information. Therefore, by Figure 13 As can be seen, the target determination method provided in this application uses a convolutional neural network to recognize the AP output of the image. 50 This is significantly better than the first approach. Therefore, the target determination method provided in this application can further improve the accuracy of target detection.

[0154] II. Training Process – A Model Training Method

[0155] Figure 14 This is a flowchart illustrating a model training method provided in an embodiment of this application.

[0156] like Figure 14 As shown, a model training method provided in this application embodiment may include the following steps:

[0157] 1401. Obtain training data.

[0158] The training data includes multiple training images mapped with millimeter-wave detection points. The training images and millimeter-wave detection points are data acquired synchronously for the same target. In this application, the target is sometimes referred to as the target object; unless a distinction is emphasized, the two terms are used interchangeably.

[0159] Training images carry the label information of the target objects, which can be obtained through manual annotation. The training images are the original images used to train the object detection model, and the label information of the target objects can be understood as the ground truth (GT) used to train the object detection model.

[0160] For details on how to map millimeter-wave detection points onto training images, please refer to [reference needed]. Figure 6 In the corresponding embodiment, step 1402 maps multiple millimeter-wave detection points onto the image to be processed for understanding, which will not be repeated here.

[0161] 1402. Determine multiple candidate bounding boxes for the image to be processed based on the depth and location information of each millimeter-wave detection point.

[0162] Based on the depth and location information of each millimeter-wave detection point, a set of candidate bounding boxes can be determined. This set of candidate bounding boxes includes multiple candidate bounding boxes. (See reference...) Figure 6 The steps 604 in the corresponding embodiments will be understood and will not be repeated here.

[0163] 1403. Train the model based on the features corresponding to multiple candidate boxes to obtain a trained model.

[0164] Training data can be input into the model, such as Fast R-CNN, Faster R-CNN, Mask R-CNN, etc. The model can obtain the first feature map of the training data. Based on the position and depth information of the millimeter-wave detector points in the image, multiple candidate boxes are generated. Second feature maps corresponding to these candidate boxes are extracted from the first feature map. The model can then be trained based on the second feature maps until the model's loss function converges, indicating that training is complete.

[0165] The foregoing has provided a detailed description of the target determination method and model training method provided in this application. Based on the aforementioned target determination method and model training method, the following describes the target determination device and model training device provided in this application. This target determination device is used to execute the aforementioned... Figures 6-12The corresponding method steps, the model training device is used to execute the aforementioned steps. Figure 14 The steps of the corresponding method.

[0166] See Figure 15 This application provides a schematic diagram of a target determination device. The target determination device includes:

[0167] The acquisition module 1501 is used to acquire an image to be processed and multiple millimeter-wave detection points. The image to be processed and the multiple millimeter-wave detection points are data acquired synchronously for the same target. The image to be processed can be acquired through a visual sensor, and the millimeter-wave detection points can be acquired through millimeter-wave radar. Each millimeter-wave detection point can include depth information, which represents the distance between the detected target and the millimeter-wave radar. The millimeter-wave radar is used to acquire multiple millimeter-wave detection points. The mapping module is used to map the multiple millimeter-wave detection points acquired by the acquisition module 1501 onto the image to be processed acquired by the acquisition module 1501. The processing module 1502 is used to determine multiple candidate bounding boxes of the detected target on the image to be processed based on first information. The first information can include the depth information and position information of each millimeter-wave detection point, where the position information represents the position of each millimeter-wave detection point mapped onto the image to be processed. The processing module 1502 is also used to perform non-maximum suppression (NMS) processing on the multiple candidate bounding boxes based on the depth information to output the target bounding box and the target millimeter-wave detection point (i.e., output the target-level detection results of different sensors for the same detected target), i.e., output the association result. The target bounding box is determined based on the depth and position information of the target millimeter-wave detection point.

[0168] In one possible implementation, the processing module 1502 is specifically configured to: perform non-maximum suppression (NMS) processing on a plurality of candidate boxes according to a first score and a second score, wherein the first score represents the probability, determined by the classifier, that the detected target in each candidate box belongs to each of the N categories, where the N categories are pre-defined categories and N is a positive integer, and the second score represents the probability, determined by the first probability distribution between depth information and each category, that the detected target in each candidate box belongs to each of the N categories.

[0169] In one possible implementation, the target determination device may further include a statistics module 1503, which is used to perform statistics on the data in the first set to determine the probability distribution of the first size of the statistical target corresponding to each category. The first set may include multiple statistical targets corresponding to each category, and the size information of each statistical target. A first probability distribution is determined based on the probability distribution of the first size and a first relationship, whereby the first relationship is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target.

[0170] In one possible implementation, the statistics module 1503 is further configured to: perform statistics on the data in the second set to determine the probability distribution of the second size of the statistical target corresponding to each category. The second set may include multiple statistical targets corresponding to each category, and the size information of each statistical target. A second probability distribution is determined based on the second size distribution and a second relationship, wherein the second probability distribution is used to update the first probability distribution, and the second relationship is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target.

[0171] In one possible implementation, the size information is the height information of the statistical target.

[0172] In one possible implementation, location information is used to determine the location of candidate boxes in the image to be processed, taking into account the distribution characteristics of millimeter-wave detection points on the vehicle.

[0173] In one possible implementation, depth information is used to determine the size of the candidate box, and the size of the candidate box is negatively correlated with the depth information.

[0174] In one possible implementation, the processing module 1502 is further configured to: perform convolution processing on the image to be processed to obtain a first feature map of the image to be processed; extract second feature maps corresponding to multiple candidate boxes from the first feature map; and process the second feature maps through a regression network and a classifier to obtain a first result, the first result being used for non-maximum suppression (NMS) processing.

[0175] Optionally, in conjunction with the second aspect or the first to seventh possible implementations of the second aspect, in the eighth possible implementation, the image to be processed is acquired by a visual sensor, the sampling frequency of the visual sensor is a first frequency, the sampling frequency of the millimeter-wave radar is a second frequency, and the difference between the first frequency and the second frequency is not greater than a preset threshold.

[0176] See Figure 16 This application provides a schematic diagram of the structure of a model training device. The model training device includes:

[0177] Get module 1601, used for execution Figure 14 Step 1401 in the corresponding embodiment.

[0178] Training module 1602 is used for execution Figure 14 Steps 1402 and 1403 in the corresponding embodiments.

[0179] Please see Figure 17 The following is a schematic diagram of another target determination device provided in this application.

[0180] The target determination device may include a processor 1701 and a memory 1702. The processor 1701 and the memory 1702 are interconnected via a circuit. The memory 1702 stores program instructions and data.

[0181] The aforementioned are stored in memory 1702 Figure 6 or Figure 8 The steps in the code include the corresponding program instructions and data.

[0182] Processor 1701 is used to perform the aforementioned Figure 6 or Figure 8 The method steps performed by the target determination apparatus shown in any of the embodiments.

[0183] This application also provides a computer-readable storage medium storing a program for generating vehicle speed, which, when used on a computer, causes the computer to execute the aforementioned... Figure 6 or Figure 8 The steps in the method described in the illustrated embodiment.

[0184] Alternatively, the aforementioned Figure 17 The target determination device shown is a chip.

[0185] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the processor 1701 described above, or the functions of processor 1701, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the target determination device in the foregoing embodiments based on the program code stored in the external memory.

[0186] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 6 or Figure 8 The steps performed by the target determination device in the method described in the illustrated embodiment.

[0187] The target determination device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the server to perform the aforementioned operations. Figure 6 or Figure 8The target determination method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM), etc.

[0188] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0189] For details, please refer to Figure 18 , Figure 18 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 180. The NPU 180 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1803, which is controlled by the controller 1804 to extract matrix data from the memory and perform multiplication operations.

[0190] In some implementations, the arithmetic circuit 1803 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1803 is a two-dimensional pulsating array. The arithmetic circuit 1803 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1803 is a general-purpose matrix processor.

[0191] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1802 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1801 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1808.

[0192] Unified memory 1806 is used to store input and output data. Weight data is directly transferred to weight memory 1802 via direct memory access controller (DMAC) 1805. Input data is also transferred to unified memory 1806 via DMAC.

[0193] The bus interface unit (BIU) 1810 is used for interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1809.

[0194] The bus interface unit 1810 (BIU) is used by the instruction fetch memory 1809 to fetch instructions from external memory, and also by the memory access controller 1805 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0195] The DMAC is mainly used to move input data from external memory DDR to unified memory 1806, or to weight data to weight memory 1802, or to input data to input memory 1801.

[0196] The vector computation unit 1807 includes multiple arithmetic processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computations in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0197] In some implementations, the vector computation unit 1807 can store the processed output vector in the unified memory 1806. For example, the vector computation unit 1807 can apply linear and / or nonlinear functions to the output of the computation circuit 1803, such as performing linear interpolation on feature planes extracted by convolutional layers, or accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1807 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1803, for example, for use in subsequent layers of the neural network.

[0198] The instruction fetch buffer 1809 connected to the controller 1804 is used to store the instructions used by the controller 1804;

[0199] Unified memory 1806, input memory 1801, weighted memory 1802, and instruction fetch memory 1809 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0200] The operations of each layer in the recurrent neural network can be performed by the arithmetic circuit 1803 or the vector calculation unit 1807.

[0201] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors used to control the above. Figure 6 or Figure 8 The method of program execution of integrated circuits.

[0202] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0203] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0204] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0205] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0206] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A target determination method, characterized in that, include: The system acquires an image to be processed and multiple millimeter-wave detection points, wherein the image to be processed and the multiple millimeter-wave detection points are data acquired synchronously for the same detection target. Each millimeter-wave detection point includes depth information, which is used to represent the distance between the detection target and the millimeter-wave radar. The millimeter-wave radar is used to acquire the multiple millimeter-wave detection points. Map the multiple millimeter-wave detection points onto the image to be processed; Based on the first information, multiple candidate bounding boxes of the detection target are determined on the image to be processed. The first information includes the depth information and position information of each millimeter-wave detection point. The position information is used to indicate the position of each millimeter-wave detection point mapped on the image to be processed. The candidate boxes are processed by non-maximum suppression (NMS) based on the depth information to output target boxes and target millimeter-wave detection points. The target boxes are determined based on the depth information and the position information of the target millimeter-wave detection points. The step of performing Non-Maximum Suppression (NMS) processing on the plurality of candidate boxes based on the depth information includes: Non-maximum suppression (NMS) is applied to the plurality of candidate boxes based on a first score and a second score. The first score represents the probability that the detected target in each candidate box belongs to each of the N categories, as determined by the classifier. The N categories are pre-defined categories, and N is a positive integer. The second score represents the probability that the detected target in each candidate box belongs to each of the N categories, as determined by a first probability distribution between the depth information and each category.

2. The method according to claim 1, characterized in that, The method further includes: The data in the first set are statistically analyzed to determine the probability distribution of the first size of the statistical target corresponding to each category. The first set includes multiple statistical targets corresponding to each category, as well as the size information of each statistical target. The first probability distribution is determined based on the probability distribution of the first size and the first relationship, wherein the first relationship is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target.

3. The method according to claim 2, characterized in that, The method further includes: The data in the second set are statistically analyzed to determine the probability distribution of the second size of the statistical target corresponding to each category. The second set includes multiple statistical targets corresponding to each category, as well as the size information of each statistical target. A second probability distribution is determined based on the second size distribution and the second relationship. The second probability distribution is used to update the first probability distribution. The second relationship is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target.

4. The method according to claim 2 or 3, characterized in that, The size information refers to the height information of the statistical target.

5. The method according to any one of claims 1 to 3, characterized in that, The location information is used to determine the position of the candidate box in the image to be processed, based on the distribution characteristics of the millimeter-wave detection points on the vehicle.

6. The method according to any one of claims 1 to 3, characterized in that, The depth information is used to determine the size of the candidate box, and the size of the candidate box is negatively correlated with the depth information.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The image to be processed is processed by a high-efficiency region convolutional neural network Faster-RCNN to obtain the first feature map of the image to be processed; Extract the second feature map corresponding to the plurality of candidate boxes from the first feature map; The second feature map is processed by a regression network and a classifier to obtain a first result, which is then used for non-maximum suppression (NMS) processing.

8. The method according to any one of claims 1 to 3, characterized in that, The image to be processed is acquired by a visual sensor, the sampling frequency of the visual sensor is a first frequency, the sampling frequency of the millimeter-wave radar is a second frequency, and the difference between the first frequency and the second frequency is not greater than a preset threshold.

9. A target determination device, characterized in that, include: An acquisition module is used to acquire an image to be processed and multiple millimeter-wave detection points. The image to be processed and the multiple millimeter-wave detection points are acquired synchronously. Each millimeter-wave detection point includes depth information, which is used to represent the distance between the detected target and the millimeter-wave radar. The millimeter-wave radar is used to acquire the multiple millimeter-wave detection points. A mapping module is used to map the plurality of millimeter-wave detection points acquired by the acquisition module onto the image to be processed acquired by the acquisition module; The processing module is configured to determine multiple candidate bounding boxes of the detection target on the image to be processed based on first information, wherein the first information includes the depth information and position information of each millimeter-wave detection point, and the position information is used to indicate the position of each millimeter-wave detection point mapped on the image to be processed; The processing module is further configured to perform non-maximum suppression (NMS) processing on the plurality of candidate boxes based on the depth information to output a target box and a target millimeter-wave detection point, wherein the target box is determined based on the depth information and the position information of the target millimeter-wave detection point; The processing module is specifically used for: Non-maximum suppression (NMS) is applied to the plurality of candidate boxes based on a first score and a second score. The first score represents the probability, determined by the classifier, that the detected target in each candidate box belongs to each of the N categories, where the N categories are pre-defined categories and N is a positive integer. The second score represents the probability, determined by the first probability distribution between the depth information and each category, that the detected target in each candidate box belongs to each of the N categories.

10. The target determination device according to claim 9, characterized in that, The target determination device also includes a statistics module. The statistics module is used to perform statistics on the data in the first set and determine the probability distribution of the first size of the statistical target corresponding to each category. The first set includes multiple statistical targets corresponding to each category and size information of each statistical target. The first probability distribution is determined based on the probability distribution of the first size and the first relationship, wherein the first relationship is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target.

11. The target determination device according to claim 10, characterized in that, The statistics module is also used for: The data in the second set are statistically analyzed to determine the probability distribution of the second size of the statistical target corresponding to each category. The second set includes multiple statistical targets corresponding to each category, as well as the size information of each statistical target. A second probability distribution is determined based on the second size distribution and the second relationship. The second probability distribution is used to update the first probability distribution. The second relationship is the relationship between the size of the statistical target and the depth information of the millimeter-wave detection point corresponding to the statistical target.

12. The target determination device according to claim 10 or 11, characterized in that, The size information refers to the height information of the statistical target.

13. The target determination apparatus according to any one of claims 9 to 11, characterized in that, The location information is used to determine the position of the candidate box in the image to be processed, based on the distribution characteristics of the millimeter-wave detection points on the vehicle.

14. The target determination apparatus according to any one of claims 9 to 11, characterized in that, The depth information is used to determine the size of the candidate box, and the size of the candidate box is negatively correlated with the depth information.

15. The target determination apparatus according to any one of claims 9 to 11, characterized in that, The processing module is further configured to: The image to be processed is subjected to convolution processing to obtain the first feature map of the image to be processed; Extract the second feature map corresponding to the plurality of candidate boxes from the first feature map; The second feature map is processed by a regression network and a classifier to obtain a first result, which is then used for non-maximum suppression (NMS) processing.

16. The target determination apparatus according to any one of claims 9 to 11, characterized in that, The image to be processed is acquired by a visual sensor, the sampling frequency of the visual sensor is a first frequency, the sampling frequency of the millimeter-wave radar is a second frequency, and the difference between the first frequency and the second frequency is not greater than a preset threshold.

17. An intelligent vehicle, characterized in that, The intelligent vehicle includes a processor coupled to a memory, the memory storing program instructions that, when executed by the processor, implement the method of any one of claims 1 to 8.

18. A monitoring device, characterized in that, The monitoring device processor is coupled to a memory, which stores program instructions. When the program instructions stored in the memory are executed by the processor, the method of any one of claims 1 to 8 is implemented.

19. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to perform the method as claimed in any one of claims 1 to 8.

20. A target determination system, characterized in that, The target determination system includes edge-side devices and cloud-side devices. The end-side device is used to acquire an image to be processed and multiple millimeter-wave detection points. The image to be processed and the multiple millimeter-wave detection points are acquired synchronously. Each millimeter-wave detection point includes depth information, which is used to represent the distance between the detected target and the millimeter-wave radar. The millimeter-wave radar is used to acquire the multiple millimeter-wave detection points. The cloud-side device is used to receive the image to be processed and multiple millimeter-wave detection points sent by the end-side device; The cloud-side device is also used to map the plurality of millimeter-wave detection points onto the image to be processed; The cloud-side device is further configured to determine multiple candidate boxes of the detection target on the image to be processed based on first information, wherein the first information includes the depth information and position information of each millimeter-wave detection point, and the position information is used to indicate the position of each millimeter-wave detection point mapped on the image to be processed; The cloud-side device is further configured to perform non-maximum suppression (NMS) processing on the plurality of candidate boxes based on the depth information to output a target box and a target millimeter-wave detection point, wherein the target box is determined based on the depth information and the position information of the target millimeter-wave detection point; The cloud-side device is specifically used to perform non-maximum suppression (NMS) processing on the plurality of candidate boxes based on a first score and a second score. The first score represents the probability that the detected target in each candidate box belongs to each of the N categories, as determined by the classifier. The N categories are pre-defined categories, and N is a positive integer. The second score represents the probability that the detected target in each candidate box belongs to each of the N categories, as determined by the first probability distribution between the depth information and each category.

Images