Target Classification Method

Abstract

The present invention provides a target classification method, including: calibrating a radar coordinate system and an image coordinate system; filtering the radar signal for noise to obtain a point cloud signal, and defogging the image data to obtain a refined defogging image; clustering and denoising the point cloud signal to obtain multiple groups of point trace sets, and using a deep learning model to train the refined defogging image to obtain a visual classification result; obtaining parameter information of the point trace set; classifying the point trace set to obtain a radar classification result; fusing the radar classification result with the visual classification result to obtain a final classification result. The classification method provided in the present application has multiple data sources, denoises and classifies the radar signal to obtain the radar classification result, and defogs and trains the image data to obtain the visual classification result, and finally fuses the radar classification result with the visual classification result, thereby avoiding misclassification, improving the accuracy of target recognition, and improving the reliability and accuracy of target classification.

Description

Technical Field

[0001] The present application relates to the technical field of target detection and classification, and in particular to a target classification method. Background Art

[0002] Traditional millimeter-wave radar data only contains distance, azimuth and speed information, which can be used to assist computer vision in target detection tasks, but the data dimension is still insufficient for target classification tasks. The emergence of 4D millimeter-wave radar makes it possible to use radar data for target classification. This is because the detection information of 4D millimeter-wave radar includes distance, azimuth, speed and pitch angle. For the detection of targets, the spatial expansion from plane space to three-dimensional space can effectively determine the shape and spatial characteristics of the target. In addition, 4D millimeter-wave radar provides denser point cloud and point trace information, which can be used to classify targets.

[0003] Although 4D millimeter-wave radar has made significant progress compared to previous radars, it still has problems such as sparse point cloud data and low small target recognition accuracy. For target classification tasks, some experts in this field have proposed using multi-frame point traces of 4D millimeter-wave radar and target length, width and height information for probability calculation and classification. However, using only 4D millimeter-wave radar data for classification, the data obtained has certain limitations, and the processing ability for similar targets is poor. In addition, it relies on experience to adjust the threshold, which will lead to additional instability factors. In addition, there is still a risk of misclassification when relying solely on probability calculation for accuracy.

[0004] Therefore, there is an urgent need to propose a new target classification method to solve problems such as single data source, low target recognition accuracy, and target misclassification, so as to provide new ideas for the industry, which is of great significance and very necessary. Summary of the invention

[0005] The present application provides a target classification method, which can solve at least one of the problems of single data source for target classification, low target recognition accuracy, target misclassification, etc.

[0006] On the one hand, an embodiment of the present application provides a target classification method, including:

[0007] The millimeter wave radar is used to obtain the radar signal of the target to be detected in the detection area, and at the same time, the camera is used to obtain the image data of the target to be detected in the detection area;

[0008] Calibrate the radar coordinate system and the image coordinate system to unify the coordinate systems;

[0009] Performing noise filtering on the radar signal to obtain a point cloud signal, and processing the image data to obtain a refined defogging image;

[0010] Clustering and denoising the point cloud signal to obtain multiple groups of point trace sets, and training the refined dehazed image using a deep learning model to obtain a visual classification result, wherein the visual classification result includes: a visual detection box and a visual classification confidence;

[0011] Acquire parameter information of the point trace set, wherein the parameter information includes: length, width, height, Doppler velocity and radar reflection intensity of the target to be measured;

[0012] According to the parameter information of the point trace set and the preset parameter threshold interval, the point trace set is classified to obtain a radar classification result, and the radar classification confidence of the target to be measured is calculated, wherein the radar classification result includes: a radar detection frame;

[0013] The radar classification result is fused with the visual classification result to obtain a final classification result.

[0014] Optionally, in the target classification method, the step of filtering the radar signal sent back by the millimeter wave radar to obtain a point cloud signal, and processing the image data sent back by the camera to obtain a refined defogging image includes:

[0015] Performing a Fourier transform on the radar signal to convert the radar signal from a time domain to a frequency domain;

[0016] Adaptively removing noise from the radar signal using an OS-CFAR algorithm;

[0017] Mapping the radar signal into a three-dimensional coordinate system to obtain a point cloud signal;

[0018] Dehazing processing is performed on the image data.

[0019] Optionally, in the target classification method, the step of performing defogging processing on the image data includes:

[0020] Performing atmospheric light value estimation on the image data to obtain an estimated value of atmospheric light;

[0021] Performing a rough transmittance estimation on the image data to obtain a rough dehazed image;

[0022] The transmittance of the rough dehazed image is precisely estimated by using a guided filtering method to obtain the refined dehazed image.

[0023] Optionally, in the target classification method, the step of estimating the atmospheric light value of the image data to obtain an estimated value of the atmospheric light includes:

[0024] The first step: divide the image data into four rectangular areas; wherein the image data satisfies the following atmospheric scattering model:

[0025] I(p)=t(p)J(p)+(1-t(p))A

[0026] Among them, I(p) is the foggy image, J(p) is the fog-free image, t(p) is the transmittance, and A is the atmospheric light;

[0027] Step 2: according to the average value and standard deviation of the pixel values ​​of each rectangular area, obtain the difference between the average value of the pixel values ​​of each rectangular area and the standard deviation of the pixel values ​​of each rectangular area, and select the rectangular area with the largest difference;

[0028] The third step: take the rectangular area with the largest difference as the new image data, and repeat the first step and the second step on the new image data until the area of ​​the rectangular area with the largest current difference is smaller than the preset area threshold. At this time, the pixel value of the brightest point in the rectangular area whose area is smaller than the preset area threshold is used as the estimated value of the atmospheric light.

[0029] Optionally, in the target classification method, the step of clustering and denoising the point cloud signal to obtain multiple groups of point trace sets, and training the refined dehazed image using a deep learning model to obtain a visual classification result includes:

[0030] Clustering the point cloud signal using a DBSCAN algorithm to obtain a plurality of point trace sets;

[0031] Eliminate a portion of the point trace set at the boundary position of the detection area to filter out obstacles, wherein the portion of the point trace set to be eliminated includes at least 20 signal points;

[0032] Grouping the point trace sets to assign corresponding IDs to each of the point trace sets;

[0033] A deep learning model is used to train the refined dehazed image to detect and classify the target to be detected to obtain the visual classification result, wherein the visual classification result includes: a visual target detection box and a visual classification confidence; the deep learning model is a YOLOX-S model.

[0034] Optionally, in the target classification method, the step of obtaining parameter information of the point trace set, wherein the parameter information includes: length, width, height, speed and radar reflection intensity, comprises:

[0035] Mapping the point trace set from the radar coordinate system to the plane coordinate system;

[0036] In the current frame, the cluster center of each point trace set is set as a virtual measurement point, and the virtual measurement point in the current frame is used as a measurement base point;

[0037] respectively detecting whether each of the virtual measurement points in the next two frames is offset; if any of the virtual measurement points in the next two frames is offset, translating the displaced virtual measurement points until they coincide with the position of the measurement base point in the current frame;

[0038] Get the maximum and minimum values ​​of each point set on the x-axis and y-axis respectively, where the maximum value on the x-axis is recorded as x i _max, the minimum value on the x-axis is recorded as x i _min, the maximum value on the y-axis is recorded as y i _max, the minimum value on the y-axis is recorded as y i _min, where i is an integer greater than or equal to 1;

[0039] According to x i _max and x i _min, get the width of each point set, and according to y i _max and y i _min obtains the length of each point trace set;

[0040] Map each of the point trace sets from the plane coordinate system to the three-dimensional coordinate system, and obtain the maximum value of each of the point trace sets on the z axis as the height of each of the point trace sets, wherein the maximum value on the z axis is recorded as z i _max;

[0041] The Doppler velocity and radar reflection intensity of each point track set are obtained.

[0042] Optionally, in the target classification method, the step of classifying the target to be measured according to the parameter information of the point trace set and the preset parameter threshold interval to obtain a radar classification result, and calculating the radar classification confidence of the target to be measured includes:

[0043] According to the length, width and height of each of the point trace sets, the aspect ratio and the width-to-height ratio of each of the point trace sets are obtained, and the parameter threshold intervals include: a first threshold interval, a second threshold interval, a third threshold interval, a fourth threshold interval, a fifth threshold interval, a sixth threshold interval and a seventh threshold interval;

[0044] Comparing the aspect ratio of each of the point trace sets with the first threshold interval, comparing the width-to-height ratio of each of the point trace sets with the second threshold interval, and comparing the radar reflection intensity of each of the point trace sets with the third threshold interval, so as to distinguish whether the point trace set is a vehicle target or a pedestrian target;

[0045] Using a Kalman filter to predict all point trace sets to keep tracking the vehicle target and / or the pedestrian target;

[0046] According to the current frame image and the previous two frames of images, the radar classification confidence of the target to be measured is obtained.

[0047] Optionally, in the target classification method, the step of obtaining the radar classification confidence of the target to be measured based on the current frame image and the previous two frames of images includes:

[0048] The evaluation index of the current frame image and the evaluation index of the previous two frames of images are obtained respectively, and the standard deviation is obtained according to the evaluation index of the current frame image and the evaluation index of the previous two frames of images, wherein the standard deviation satisfies the following formula:

[0049]

[0050] Where S is the standard deviation; n is the number of samples, and n=3; X j is the sample value of the evaluation index, and X j ∈(0,1), j=1,2,3; is the sample mean of the evaluation index;

[0051] According to the standard deviation, the standard error is obtained, and the standard error satisfies the following formula:

[0052]

[0053] Among them, SEM is standard error;

[0054] The standard score is obtained based on the sample mean, standard error, and the preset lower limit threshold of the confidence interval, where the standard score satisfies the following formula:

[0055]

[0056] Wherein, z is the standard score; SEM is the standard error; a is the lower limit threshold of the preset confidence interval, and a=0.5;

[0057] According to the standard score, the normal distribution table is queried to obtain the radar classification confidence of the target to be measured.

[0058] Optionally, in the target classification method, after comparing the aspect ratio of each of the point track sets with the first threshold interval, comparing the width-to-height ratio of each of the point track sets with the second threshold interval, and comparing the radar reflection intensity of each of the point track sets with the third threshold interval to distinguish whether the point track sets are vehicle targets or pedestrian targets, and before using the Kalman filter to predict all the point track sets to keep tracking the vehicle targets and / or the pedestrian targets, the following steps are also included:

[0059] If the point trace set is a vehicle target, it is determined whether the length of the point trace set is within the fourth threshold interval; if the length of the point trace set is within the fourth threshold interval, it is determined that the point trace set is a small four-wheeled vehicle; if the length of the point trace set is lower than the lower limit of the fourth threshold interval, it is determined that the point trace set is a two-wheeled vehicle; if the length of the point trace set exceeds the upper limit of the fourth threshold interval, it is determined that the point trace set is a large four-wheeled vehicle;

[0060] If the target to be detected is a pedestrian target, determine whether the aspect ratio of the point track set is within the fifth threshold interval, whether the aspect ratio of the point track set is within the sixth threshold interval, and whether the Doppler velocity of the point track set is within the seventh threshold interval; if the aspect ratio of the point track set is within the fifth threshold interval, and the aspect ratio of the point track set is within the sixth threshold interval, and the Doppler velocity of the point track set is within the seventh threshold interval, then determine that the point track set is a moving pedestrian.

[0061] Optionally, in the target classification method, the step of fusing the radar classification result with the visual classification result to obtain a final classification result includes:

[0062] According to the visual detection frame and the radar detection frame, obtaining an overlapping area of ​​the visual detection frame and the radar detection frame, and according to the overlapping area, obtaining an overlapping ratio of the overlapping area to the sum of the areas of the visual detection frame and the radar detection frame;

[0063] If the overlap ratio is greater than 0.7, the corresponding visual detection frame and radar detection frame are retained, and the visual classification confidence corresponding to the retained visual detection frame and the radar classification confidence corresponding to the radar detection frame are compared; if the classification categories of the targets to be measured are the same, and the average of the visual classification confidence and the radar classification confidence is greater than 0.7, the corresponding radar classification result and the corresponding visual classification result are retained as the final classification result;

[0064] If the overlap ratio is greater than 0.3 and less than 0.7, the corresponding visual classification confidence and radar classification confidence are compared; if the visual classification confidence is greater than the radar classification confidence, the visual detection box is retained as the final classification result;

[0065] If the overlap ratio is less than 0.3, it is determined whether the visual detection box or the radar detection box exists in the detection area; if the visual detection box exists, the visual classification result corresponding to the visual detection box is used as the final classification result; if the radar detection box exists, the radar classification result corresponding to the radar detection box is used as the final classification result.

[0066] In summary, the present invention provides a target classification method, including: calibrating the radar coordinate system and the image coordinate system; filtering the radar signal for noise to obtain a point cloud signal, and processing the image data to obtain a refined dehazed image; clustering and denoising the point cloud signal to obtain multiple groups of point trace sets, and using a deep learning model to train the refined dehazed image to obtain a visual classification result; obtaining parameter information of the point trace set; classifying the point trace set to obtain a radar classification result; fusing the radar classification result with the visual classification result to obtain a final classification result. The target classification method proposed in this application has multiple data sources, denoises and classifies the radar signal to obtain the radar classification result, and demists and trains the image data to obtain the visual classification result, and finally fuses the radar classification result with the visual classification result, thereby avoiding the misclassification of different targets, improving the target recognition accuracy, and improving the reliability and accuracy of target classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 is a flow chart of a target classification method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0071] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0072] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0073] This application embodiment provides a target classification method, please refer to Figure 1 , Figure 1 is a flow chart of a target classification method according to an embodiment of the present invention, wherein the target classification method comprises:

[0074] S10: using the millimeter wave radar to obtain radar signals of the target to be detected in the detection area, and at the same time, using the camera to obtain image data of the target to be detected in the detection area.

[0075] S20: According to the radar signal (data) returned by the millimeter-wave radar, the radar coordinate system and the image coordinate system are calibrated, and the coordinate systems are unified so that the two are on the same horizontal plane for subsequent fusion.

[0076] S30: Filter the radar signal for noise removal to obtain a point cloud signal, and process the image data to obtain a refined defogging image. Specifically, the steps of filtering the radar signal for noise removal and defogging the image data may include:

[0077] Step S30.1: Performing Fourier transform on the radar signal to convert the radar signal from the time domain to the frequency domain;

[0078] Step S30.2: Adaptively remove noise from the radar signal using the OS-CFAR algorithm. In this embodiment, an adaptive threshold value may be used to remove false alarms.

[0079] Step S30.3: Mapping the radar signal into a three-dimensional coordinate system to obtain a point cloud signal;

[0080] Step S30.4: Defogging the image data, wherein the image data collected by the camera is an image frame of a fixed size, which is sensitive to visibility (for example, foggy days, strong light, night, etc.). Defogging the image after the image data is transmitted back can improve the reliability and accuracy of target recognition. In this embodiment, the camera can use a binocular camera.

[0081] Furthermore, the step of performing defogging processing on the image data may include:

[0082] Step S30.41: Estimating the atmospheric light value of the image data to obtain an estimated value of the atmospheric light. Preferably, the step of estimating the atmospheric light value of the image data to obtain an estimated value of the atmospheric light may include:

[0083] Step S30.411: Divide the image data into four rectangular areas; wherein the image data satisfies the following atmospheric scattering model:

[0084] I(p)=t(p)J(p)+(1-t(p))A

[0085] Among them, I(p) is the foggy image, J(p) is the fog-free image, t(p) is the transmittance, and A is the atmospheric light;

[0086] Step S30.412: according to the average value and standard deviation of the pixel values ​​of each rectangular area, obtain the difference between the average value of the pixel values ​​of each rectangular area and the standard deviation of the pixel values ​​of each rectangular area, and select the rectangular area with the largest difference;

[0087] Step S30.413: Take the rectangular area with the largest difference as the new image data, and repeat steps S30.411 and S30.412 on the new image data until the area of ​​the rectangular area with the largest current difference is smaller than the preset area threshold. At this time, the pixel value of the brightest point in the rectangular area whose area is smaller than the preset area threshold is taken as the estimated value of the atmospheric light.

[0088] Step S30.42: roughly estimate the transmittance of the image data to obtain a roughly defogging image. Specifically, roughly estimate the transmittance of the image data, give an initial transmittance value t(p)0, bring it into the atmospheric scattering model, and obtain the initial defogging result J(p)0, that is:

[0089]

[0090] After obtaining the results, iterate repeatedly and use a loss function to evaluate the contrast error and information loss. The formula is as follows:

[0091]

[0092]

[0093] E=E contrast +λE loss ;

[0094] Among them, E is the total loss; E contrast is the contrast error; E loss is the image information loss; c∈{r,g,b} is a pixel channel; and For J c (p) and I c (p) the mean value in image B; N b is the number of pixels in image B; h c (i) is the histogram value of pixel i in channel c of the input image; α c and β c is the pixel value cutoff boundary; the total loss E is the weighted sum of contrast error and image information loss; λ is the weight.

[0095] In the formula, t is calculated by the following formula:

[0096]

[0097] Step S30.43: Use the guided filtering method to perform a precise estimate of the transmittance of the coarse dehazed image to obtain a final transmittance value, and bring it into the atmospheric scattering model to obtain the final refined dehazed image.

[0098] In this embodiment, the main purpose of step S30 is to remove noise interference from radar signals and image data respectively, and to make foreground targets independent and easy to detect.

[0099] S40: Clustering and denoising the point cloud signal to obtain multiple sets of point traces, and training the refined defogging image using a deep learning model to obtain a visual classification result, wherein the visual classification result includes: a visual detection box and a visual classification confidence. Specifically, the step of clustering and denoising the point cloud signal to obtain multiple sets of point traces, and training the refined defogging image using a deep learning model to obtain a visual classification result may include:

[0100] S40.1: clustering the point cloud signal using a DBSCAN algorithm to obtain a plurality of point trace sets;

[0101] S40.2: Eliminate a portion of the point trace set at the boundary position of the detection area to further filter out echoes of unnecessary targets and obstacles such as buildings and vegetation, wherein the boundary position may be within 20% of the boundary of the detection area; the portion of the point trace set to be eliminated includes at least 20 signal points;

[0102] S40.3: Grouping the point trace sets to assign corresponding IDs to each of the point trace sets;

[0103] S40.4: Use a deep learning model to train the refined dehazed image to detect and classify the target to be detected to obtain the visual classification result, wherein the visual classification result includes: a visual target detection box and a visual classification confidence; the deep learning model is a YOLOX-S model. Before implementing step S40.4, the BDD100K autonomous driving data set can be trained using the YOLOX-S model to perform real-time road target detection and classification, and then the trained YOLOX-S model is used in this step to train the refined dehazed image.

[0104] S50: Obtain parameter information of the point trace set, wherein the parameter information includes: length, width, height, Doppler velocity and radar reflection intensity of the target to be measured. Specifically, the step of obtaining the parameter information of the point trace set may include:

[0105] S50.1: Mapping the point trace set from the radar coordinate system to the plane coordinate system;

[0106] S50.2: In the current frame, the cluster center of each point trace set is set as a virtual measurement point, and the virtual measurement point in the current frame is used as a measurement base point;

[0107] S50.3: Detecting whether each virtual measurement point in the next two frames is offset; if any virtual measurement point in the next two frames is offset, translating the displaced virtual measurement point until it coincides with the position of the measurement base point in the current frame;

[0108] S50.4: Obtain the maximum and minimum values ​​of each point trace set on the x-axis and y-axis respectively, wherein the maximum value on the x-axis is recorded as x i _max, the minimum value on the x-axis is recorded as x i _min, the maximum value on the y-axis is recorded as y i _max, the minimum value on the y-axis is recorded as y i_min, where i is an integer greater than or equal to 1;

[0109] S50.5: Calculate x i _max and x i _min, and use this difference as the width of each point set, and calculate y i _max and y i _min, and use this difference as the length of each point trace set;

[0110] S50.6: Map each of the point trace sets from the plane coordinate system to the three-dimensional coordinate system, and obtain the maximum value of each of the point trace sets on the z axis as the height of each of the point trace sets, wherein the maximum value on the z axis is recorded as z i _max;

[0111] S50.7: Obtain the Doppler velocity and radar reflection intensity of each of the point trace sets.

[0112] S60: According to the parameter information of the point trace set and the preset parameter threshold interval, the point trace set is classified to obtain a radar classification result, and the radar classification confidence of the target to be measured is calculated, wherein the radar classification result includes: a radar detection frame; in this embodiment, the parameter threshold interval includes: a first threshold interval, a second threshold interval, a third threshold interval, a fourth threshold interval, a fifth threshold interval, a sixth threshold interval and a seventh threshold interval. Specifically, according to the parameter information of the point trace set and the preset parameter threshold interval, the step of classifying the target to be measured to obtain a radar classification result and calculating the radar classification confidence of the target to be measured includes:

[0113] S60.1: Obtaining the length-to-width ratio and the aspect-to-height ratio of each of the point trace sets according to the length, width and height of each of the point trace sets;

[0114] S60.2: Compare the aspect ratio of each of the point trace sets with the first threshold interval, compare the width-to-height ratio of each of the point trace sets with the second threshold interval, and compare the radar reflection intensity of each of the point trace sets with the third threshold interval, so as to distinguish whether the point trace set is a vehicle target or a pedestrian target. For example, the first threshold interval can be set to 0.8-2.0, the second threshold interval can be set to 0.5-1.5, and the third threshold interval can be set to 3.5dBsm-6.8dBsm. If the aspect ratio of each of the point trace sets exceeds the upper limit of the first threshold interval, and the width-to-height ratio of each of the point trace sets is within the second threshold interval, and the radar reflection intensity of each of the point trace sets (target to be measured) is within the third threshold interval, then the point trace set is determined to be a vehicle target (target to be measured); if the point trace set is not a vehicle target, it is determined to be a pedestrian target.

[0115] Furthermore, after step S60.2 and before step S60.3, the vehicle target and the pedestrian target may be further determined, which may mainly include the following steps:

[0116] If the point trace set is a vehicle target, it is determined whether the length of the point trace set is within the fourth threshold interval (for example, 2m to 6m); if the length of the point trace set is within the fourth threshold interval, it is determined that the point trace set is a small four-wheeled vehicle; if the length of the point trace set exceeds the upper limit of the fourth threshold interval, it is determined that the point trace set is a large four-wheeled vehicle; if the length of the point trace set is lower than the lower limit of the fourth threshold interval, it is determined that the point trace set is a two-wheeled vehicle. Furthermore, it can be further determined whether the two-wheeled vehicle is a motorcycle, an electric vehicle, or a bicycle. Specifically, a speed threshold interval is preset (for example, 10.0km / s to 35.0km / s). If the Doppler speed of the point track set is within the speed threshold interval of 10.0km / s to 35.0km / s, the two-wheeled vehicle is determined to be an electric vehicle; if the Doppler speed of the point track set is lower than 10.0km / s, the two-wheeled vehicle is determined to be a bicycle; if the Doppler speed of the point track set is higher than 35.0km / s, the two-wheeled vehicle is determined to be a motorcycle.

[0117] If the target to be detected is a pedestrian target, then continue to judge whether the aspect ratio of the point track set is within the fifth threshold interval (for example, 0.8 to 1.2), whether the aspect ratio of the point track set is within the sixth threshold interval (for example, 0.2 to 0.45), and whether the Doppler velocity of the point track set is within the seventh threshold interval (0.5m / s to 5m / s); if the aspect ratio of the point track set is within the fifth threshold interval, and the aspect ratio of the point track set is within the sixth threshold interval, and the Doppler velocity of the point track set is within the seventh threshold interval, and the length and width of the point track set are both less than or equal to 1m, and the height of the point track set is between 1.0m and 2.5m, then the point track set is determined to be a moving pedestrian, otherwise the point track set is determined to be a non-moving pedestrian / standing pedestrian (the Doppler velocity of the non-moving pedestrian / standing pedestrian (the point track set) is less than 0.5m / s).

[0118] In this embodiment, if a point set does not meet all the above conditions, it is determined to be an unclassified target.

[0119] S60.3: Use a Kalman filter to predict all point trace sets to keep tracking the vehicle target and / or the pedestrian target;

[0120] S60.4: Obtain the radar classification confidence of the target to be measured based on the current frame image and the previous two frames of images.

[0121] Preferably, the step of obtaining the radar classification confidence of the target to be measured according to the current frame image and the previous two frames of images includes:

[0122] S60.41: Obtain the evaluation index of the current frame image and the evaluation index of the previous two frames of images respectively, and obtain the standard deviation according to the evaluation index of the current frame image and the evaluation index of the previous two frames of images, wherein the standard deviation satisfies the following formula:

[0123]

[0124] Where S is the standard deviation; n is the number of samples, and n=3; X j is the sample value of the evaluation index, and X j ∈(0,1), j=1,2,3; is the sample mean of the evaluation index; in this embodiment, X j (Evaluation index) is obtained by judging whether the object in the current detection frame (current frame image) is consistent with the label; if the object in the current detection frame (current frame image) is consistent with the label, then X j (Evaluation index) is marked as 1; if the object in the current detection frame (current frame image) is inconsistent with the label, then X j(Evaluation index) is marked as 0.

[0125] S60.42: Obtain a standard error based on the standard deviation, where the standard error satisfies the following formula:

[0126]

[0127] Among them, SEM is standard error;

[0128] S60.43: Obtain the standard score based on the sample mean, standard error, and the preset lower limit threshold of the confidence interval, where the standard score satisfies the following formula:

[0129]

[0130] Wherein, z is the standard score; SEM is the standard error; a is the lower limit threshold of the preset confidence interval, and a=0.5;

[0131] S60.44: Based on the standard score, query the normal distribution table to obtain the radar classification confidence of the target to be measured.

[0132] S70: Fusing the radar classification result with the visual classification result to obtain a final classification result. Specifically, the step of fusing the radar classification result with the visual classification result to obtain a final classification result may include:

[0133] S70.1: Obtain, according to the visual detection frame and the radar detection frame, an overlapping area of ​​the visual detection frame and the radar detection frame, and, according to the overlapping area, obtain an overlapping ratio of the overlapping area to the sum of the areas of the visual detection frame and the radar detection frame;

[0134] S70.2: (1) If the overlap ratio is greater than 0.7, retain the corresponding visual detection frame and radar detection frame, and compare the visual classification confidence corresponding to the retained visual detection frame and the radar classification confidence corresponding to the radar detection frame; if the classification categories of the targets to be measured are the same, and the average of the visual classification confidence and the radar classification confidence is greater than 0.7, retain the corresponding radar classification result and the corresponding visual classification result as the final classification result; (2) If the overlap ratio is greater than 0.3 and less than 0.7, compare the corresponding visual classification confidence and radar classification confidence; if the visual classification confidence is greater than the radar classification confidence, retaining the visual detection box as the final classification result; (3) if the overlap ratio is less than 0.3, determining whether the visual detection box or the radar detection box exists in the detection area; if the visual detection box exists, determining it as a radar miss detection, and taking the visual classification result corresponding to the visual detection box as the final classification result; if the radar detection box exists, determining it as a visual miss detection, and taking the radar classification result corresponding to the radar detection box as the final classification result.

[0135] In summary, an embodiment of the present application provides a target classification method, including: using a millimeter-wave radar to obtain a radar signal of a target to be detected in a detection area; calibrating a radar coordinate system and an image coordinate system; filtering the radar signal to obtain a point cloud signal, and processing the image data to obtain a refined dehazed image; clustering and denoising the point cloud signal to obtain multiple groups of point trace sets, and using a deep learning model to train the refined dehazed image to obtain a visual classification result; obtaining parameter information of the point trace set; classifying the point trace set to obtain a radar classification result; and fusing the radar classification result with the visual classification result to obtain a final classification result. The target classification method proposed in this application combines computer vision and 4D millimeter-wave radar for multi-sensor fusion classification. It has multiple data sources, denoises and classifies radar signals to obtain radar classification results, and defogs and trains image data to obtain visual classification results. Finally, the radar classification results and the visual classification results are fused to perform real-time target classification around the clock and at all times, avoiding misclassification of different targets. Multiple data sources complement each other, improve target recognition accuracy, improve the reliability and accuracy of target classification, and also improve the robustness and stability of target detection and classification.

[0136] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A target classification method, characterized in that: include: The millimeter wave radar is used to obtain the radar signal of the target to be detected in the detection area, and at the same time, the camera is used to obtain the image data of the target to be detected in the detection area; Calibrate the radar coordinate system and the image coordinate system to unify the coordinate systems; Performing noise filtering on the radar signal to obtain a point cloud signal, and processing the image data to obtain a refined defogging image; Clustering and denoising the point cloud signal to obtain multiple groups of point trace sets, and training the refined dehazed image using a deep learning model to obtain a visual classification result, wherein the visual classification result includes: a visual detection box and a visual classification confidence; Acquire parameter information of the point trace set, wherein the parameter information includes: length, width, height, Doppler velocity and radar reflection intensity of the target to be measured; According to the parameter information of the point trace set and the preset parameter threshold interval, the point trace set is classified to obtain a radar classification result, and the radar classification confidence of the target to be measured is calculated, wherein the radar classification result includes: a radar detection frame; Fusion of the radar classification result with the visual classification result to obtain a final classification result; The step of obtaining parameter information of the point trace set, wherein the parameter information includes length, width, height, speed and radar reflection intensity, comprises: Mapping the point trace set from the radar coordinate system to the plane coordinate system; In the current frame, the cluster center of each point trace set is set as a virtual measurement point, and the virtual measurement point in the current frame is used as a measurement base point; respectively detecting whether each of the virtual measurement points in the next two frames is offset; if any of the virtual measurement points in the next two frames is offset, translating the displaced virtual measurement points until they coincide with the position of the measurement base point in the current frame; Get the maximum and minimum values ​​of each point set on the x-axis and y-axis respectively, where the maximum value on the x-axis is recorded as x i _max, the minimum value on the x-axis is recorded as x i _min, the maximum value on the y-axis is recorded as y i _max, the minimum value on the y-axis is recorded as y i _min, where i is an integer greater than or equal to 1; According to x i _max and x i _min, get the width of each point set, and according to y i _max and y i _min obtains the length of each point trace set; Map each of the point trace sets from the plane coordinate system to the three-dimensional coordinate system, and obtain the maximum value of each of the point trace sets on the z axis as the height of each of the point trace sets, wherein the maximum value on the z axis is recorded as z i _max; The Doppler velocity and radar reflection intensity of each point track set are obtained.

2. The target classification method according to claim 1, characterized in that: The step of filtering out noise on the radar signal sent back by the millimeter wave radar to obtain a point cloud signal, and processing the image data sent back by the camera to obtain a refined defogging image comprises: Performing a Fourier transform on the radar signal to convert the radar signal from a time domain to a frequency domain; Adaptively removing noise from the radar signal using an OS-CFAR algorithm; Mapping the radar signal into a three-dimensional coordinate system to obtain a point cloud signal; Dehazing processing is performed on the image data.

3. The target classification method according to claim 2, characterized in that: The step of performing defogging processing on the image data comprises: Performing atmospheric light value estimation on the image data to obtain an estimated value of atmospheric light; Performing a rough transmittance estimation on the image data to obtain a rough dehazed image; The transmittance of the rough dehazed image is precisely estimated by using a guided filtering method to obtain the refined dehazed image.

4. The target classification method according to claim 3, characterized in that: The step of estimating the atmospheric light value of the image data to obtain an estimated value of the atmospheric light comprises: The first step: divide the image data into four rectangular areas; wherein the image data satisfies the following atmospheric scattering model: Among them, I(p) is the foggy image, J(p) is the fog-free image, t(p) is the transmittance, and A is the atmospheric light; Step 2: according to the average value and standard deviation of the pixel values ​​of each rectangular area, obtain the difference between the average value of the pixel values ​​of each rectangular area and the standard deviation of the pixel values ​​of each rectangular area, and select the rectangular area with the largest difference; The third step: take the rectangular area with the largest difference as the new image data, and repeat the first step and the second step on the new image data until the area of ​​the rectangular area with the largest current difference is smaller than the preset area threshold. At this time, the pixel value of the brightest point in the rectangular area whose area is smaller than the preset area threshold is used as the estimated value of the atmospheric light.

5. The target classification method according to claim 1, characterized in that: The step of clustering and denoising the point cloud signal to obtain multiple groups of point trace sets, and training the refined dehazed image using a deep learning model to obtain a visual classification result comprises: Clustering the point cloud signal using a DBSCAN algorithm to obtain a plurality of point trace sets; Eliminate a portion of the point trace set at the boundary position of the detection area to filter out obstacles, wherein the portion of the point trace set to be eliminated includes at least 20 signal points; Grouping the point trace sets to assign corresponding IDs to each of the point trace sets; A deep learning model is used to train the refined dehazed image to detect and classify the target to be detected to obtain the visual classification result, wherein the visual classification result includes: a visual target detection box and a visual classification confidence; the deep learning model is a YOLOX-S model.

6. The target classification method according to claim 1, characterized in that: The step of classifying the target to be measured to obtain a radar classification result according to the parameter information of the point trace set and the preset parameter threshold interval, and calculating the radar classification confidence of the target to be measured includes: According to the length, width and height of each of the point trace sets, the aspect ratio and the width-to-height ratio of each of the point trace sets are obtained, and the parameter threshold intervals include: a first threshold interval, a second threshold interval, a third threshold interval, a fourth threshold interval, a fifth threshold interval, a sixth threshold interval and a seventh threshold interval; Comparing the aspect ratio of each of the point trace sets with the first threshold interval, comparing the width-to-height ratio of each of the point trace sets with the second threshold interval, and comparing the radar reflection intensity of each of the point trace sets with the third threshold interval, so as to distinguish whether the point trace set is a vehicle target or a pedestrian target; Using a Kalman filter to predict all point trace sets to keep tracking the vehicle target and / or the pedestrian target; According to the current frame image and the previous two frames of images, the radar classification confidence of the target to be measured is obtained.

7. The target classification method according to claim 6, characterized in that: The step of obtaining the radar classification confidence of the target to be measured according to the current frame image and the previous two frames of images includes: The evaluation index of the current frame image and the evaluation index of the previous two frames of images are obtained respectively, and the standard deviation is obtained according to the evaluation index of the current frame image and the evaluation index of the previous two frames of images, wherein the standard deviation satisfies the following formula: Where S is the standard deviation; n is the number of samples, and n=3; is the sample value of the evaluation index, and ϵ(0,1), j=1,2,3; is the sample mean of the evaluation index; According to the standard deviation, the standard error is obtained, and the standard error satisfies the following formula: Among them, SEM is standard error; The standard score is obtained based on the sample mean, standard error, and the preset lower limit threshold of the confidence interval, where the standard score satisfies the following formula: Wherein, z is the standard score; SEM is the standard error; is the preset lower limit threshold of the confidence interval, and a=0.5; According to the standard score, the normal distribution table is queried to obtain the radar classification confidence of the target to be measured.

8. The target classification method according to claim 6, characterized in that: After comparing the aspect ratio of each of the point track sets with the first threshold interval, comparing the width-to-height ratio of each of the point track sets with the second threshold interval, and comparing the radar reflection intensity of each of the point track sets with the third threshold interval to distinguish whether the point track sets are vehicle targets or pedestrian targets, and before using the Kalman filter to predict all the point track sets to keep tracking the vehicle targets and / or the pedestrian targets, the method further includes the following steps: If the point trace set is a vehicle target, then determine whether the length of the point trace set is within the fourth threshold interval; if the length of the point trace set is within the fourth threshold interval, then determine that the point trace set is a small four-wheeled vehicle; If the length of the point trace set is lower than the lower limit of the fourth threshold interval, the point trace set is determined to be a two-wheeled vehicle; if the length of the point trace set exceeds the upper limit of the fourth threshold interval, the point trace set is determined to be a large four-wheeled vehicle; If the target to be detected is a pedestrian target, determining whether the aspect ratio of the point set is within the fifth threshold interval, whether the aspect ratio of the point set is within the sixth threshold interval, and whether the Doppler velocity of the point set is within the seventh threshold interval; If the aspect ratio of the point set is within the fifth threshold interval, the width-to-height ratio of the point set is within the sixth threshold interval, and the Doppler velocity of the point set is within the seventh threshold interval, then the point set is determined to be a moving pedestrian.

9. The target classification method according to claim 1, characterized in that: The step of fusing the radar classification result with the visual classification result to obtain a final classification result includes: According to the visual detection frame and the radar detection frame, obtaining an overlapping area of ​​the visual detection frame and the radar detection frame, and according to the overlapping area, obtaining an overlapping ratio of the overlapping area to the sum of the areas of the visual detection frame and the radar detection frame; If the overlap ratio is greater than 0.7, the corresponding visual detection frame and radar detection frame are retained, and the visual classification confidence corresponding to the retained visual detection frame and the radar classification confidence corresponding to the radar detection frame are compared; if the classification categories of the targets to be measured are the same, and the average of the visual classification confidence and the radar classification confidence is greater than 0.7, the corresponding radar classification result and the corresponding visual classification result are retained as the final classification result; If the overlap ratio is greater than 0.3 and less than 0.7, the corresponding visual classification confidence and radar classification confidence are compared; if the visual classification confidence is greater than the radar classification confidence, the visual detection box is retained as the final classification result; If the overlap ratio is less than 0.3, it is determined whether the visual detection box or the radar detection box exists in the detection area; if the visual detection box exists, the visual classification result corresponding to the visual detection box is used as the final classification result; if the radar detection box exists, the radar classification result corresponding to the radar detection box is used as the final classification result.

Images