Data processing method and related device

Abstract

A data processing method is applied to the field of radars and comprises the following steps: acquiring point cloud data of a first object; obtaining first prediction information of the category of the first object according to the point cloud data of the first object; wherein the point cloud data of the first object comprises continuous multi-frame point cloud data, and the first prediction information indicates that the category corresponding to the point cloud data of the first object is an unknown category. According to the method and the device, when the category identification of the unknown category object is carried out, the continuous multi-frame unknown category judgment result can be output, and equivalently, a more accurate and stable category judgment result can be output.

Description

Technical Field

[0001] This application relates to the field of radar, and in particular, to a data processing method and related devices. Background Art

[0002] Traffic situation awareness requires real-time and accurate perception of data, which is the basis for subsequent realization of intelligent management and control such as traffic monitoring, control, decision-making, dispatching, and dredging. However, traffic management and control require all-time and all-scene perception. However, traditional sensors such as cameras have a sharp deterioration in performance in environments such as rainy and foggy weather and strong light irradiation, and cannot provide reliable perception results.

[0003] Millimeter-wave radar has strong anti-interference ability and has the perception ability of all-day and all-weather and large detection range. By identifying targets of interest such as motor vehicles (large, medium, and small cars), non-motor vehicles, and pedestrians (alighting passengers, pedestrians crossing the road, construction workers, etc.) through millimeter-wave radar, high-accuracy and stable identification can be achieved in all-day and all-weather, complex open, and long-distance scenarios, realizing high-precision situation awareness on highways, urban roads, and intersections, and providing an effective monitoring means and decision-making basis for traffic management. It can also be applied to scenarios such as intrusion detection of vehicles, ships, and people at the border and coastal defense; and perimeter intrusion of foreign objects such as railways and oil and gas pipelines.

[0004] In the prior art, a classifier is trained with training samples so that the classifier can have the ability to identify the categories included in the training samples. However, when encountering a target of an unknown class (that is, a class not included in the training samples), the target will be classified as a known class, and the target cannot be rejected, resulting in an incorrect classification result. Summary of the Invention

[0005] This application provides a data processing method, which can output a more accurate and stable category determination result when identifying the category of an object of an unknown class.

[0006] In a first aspect, this application provides a data processing method, which includes: obtaining point cloud data of a first object; obtaining first prediction information of the category of the first object according to the point cloud data of the first object; wherein, the point cloud data of the first object includes point cloud data of multiple consecutive frames, and the category corresponding to the point cloud data of the first object is an unknown class.

[0007] For example, the first prediction information may indicate that the category corresponding to each frame of point cloud data in the multiple consecutive frames is an unknown class.

[0008] When this application identifies the category of an object of an unknown class, it can output judgment results of an unknown class for multiple consecutive frames, which is equivalent to being able to output a more accurate and stable category determination result.

[0009] In a possible implementation, the number of consecutive multiple frames is greater than 5, for example, it can be 6, 7, 8, 10, 15, 20, etc.

[0010] In a possible implementation, the category of the first object is a category that is not predefined. For example, according to the point cloud data of the first object, through a classification model, first prediction information about the category of the first object can be obtained; training samples including the category of the first object are not used when training the classification model. That is to say, for the classification model, it does not have the ability to recognize the category of the first object.

[0011] In a possible implementation, the classification model includes multiple different classifiers; the consecutive multiple frames include a first target frame; obtaining the first prediction information about the category of the first object through the classification model according to the point cloud data of the first object includes: obtaining multiple first prediction information about the category of the first object through multiple different classifiers according to the point cloud data of the first target frame;

[0012] In the existing target recognition method based on radar point cloud, when encountering an unknown class target in a complex open scene, the target will be classified into a known category and it is impossible to reject the judgment of the target.

[0013] To solve the above problems, the idea of this application is to train multiple different classifiers. When multiple different classifiers process the same unknown class target, there will be a large difference in the processing results between them. Therefore, it is possible to determine whether an unknown class target is encountered through the difference between the processing results of multiple classifiers. For example, when the difference between the processing results of multiple classifiers is large, it can be considered that the target is an unknown class target, and then the category of the target can be output as unknown. When the difference between the processing results of multiple classifiers is small, it can be considered that the target is a known class target, and then the specific category of the target can be output.

[0014] In a possible implementation, the multiple different classifiers include a first classifier and a second classifier. The model structures of the first classifier and the second classifier are different, or the numerical values of the model parameters of the first classifier and the second classifier are different, and the category recognition accuracies of the first classifier and the second classifier are both greater than a threshold.

[0015] In a possible implementation, the classifier is a random forest constructed by sampling multiple decision trees, and the decision trees included in different classifiers are different.

[0016] In a possible implementation, the difference between multiple first prediction information (or multiple second prediction information introduced later) can be represented by, but not limited to, the entropy of the multiple prediction information. That is, based on multiple prediction information (such as class prediction probabilities), the entropy value can be calculated using these prediction information, and this is used as the basis for estimating the uncertainty of the class. The greater the uncertainty of the sample, the less reliable the sample. Therefore, when an unknown class sample appears, the reliability of the sample is estimated using the above calculation method. If the sample uncertainty exceeds a set threshold, the sample is rejected as an unknown class, thereby achieving the ability to reject unknown class samples in complex open scenarios.

[0017] In a possible implementation, the method further includes: obtaining point cloud data of a second object; based on the point cloud data of the second object, through multiple different classifiers, obtaining multiple second prediction information of the class of the second object; when the difference between the multiple second prediction information meets a preset condition, determining the class of the second object according to the multiple second prediction information.

[0018] In a possible implementation, the point cloud data of the second object includes point cloud data of multiple consecutive frames; the multiple second prediction information includes multiple second prediction information corresponding to each frame of point cloud data; the determining the class of the second object according to the multiple second prediction information includes: when the difference between the multiple second prediction information corresponding to the point cloud data of more than a preset proportion of frames in each frame or multiple frames meets the preset condition, determining the class of the second object according to the fusion result of the multiple second prediction information of the frames whose difference meets the preset condition.

[0019] In a possible implementation, the method further includes: using the class of the second object as the class of the second object in the frame after the multiple consecutive frames.

[0020] In a possible implementation, the point cloud data of the second object includes point cloud data of multiple consecutive frames; the multiple second prediction information includes multiple second prediction information corresponding to each frame of point cloud data; the method further includes: obtaining point cloud data of the second object in the frame after the multiple consecutive frames; when the difference between the multiple second prediction information corresponding to the point cloud data of more than a preset proportion of frames in the multiple frames does not meet the preset condition, determining the class of the second object in the frame after the multiple consecutive frames according to the point cloud data of the frame after the multiple consecutive frames.

[0021] In the above manner, for each sample, the decision results of multiple consecutive frames of highly reliable samples are fused as the prediction result of the entire trajectory, and subsequent trajectories are no longer predicted, thereby saving the device resource overhead, improving the stability and accuracy of the final output result, and avoiding the output result from jumping when a single-frame sample prediction is incorrect during the prediction process.

[0022] In one possible implementation, the method further includes:

[0023] Obtain the point cloud data of a third object and the point cloud data of a fourth object; wherein, the point cloud data of the third object and the point cloud data of the fourth object are consistent in a first point cloud feature; the point cloud data of the third object and the point cloud data of the fourth object are inconsistent in a second point cloud feature; the first point cloud feature is a necessary feature used for object classification; according to the point cloud data of the third object and the point cloud data of the fourth object, obtain third prediction information about the category of the third object and fourth prediction information about the category of the fourth object; the third prediction information and the fourth prediction information indicate that the categories of the third object and the fourth object are different.

[0024] In one possible implementation, the point cloud data of the first object is the point cloud data within one frame; the obtaining, according to the point cloud data of the first object, of first prediction information about the category of the first object through a classification model includes: determining, according to the point cloud data of the first object, a plurality of point cloud features and the correlations between different point cloud features; and obtaining, according to the plurality of point cloud features and the correlations, the first prediction information of the first object through the classification model.

[0025] In one possible implementation, the correlations between some basic characteristics of the target can be extracted from a single-frame point cloud, and the correlations are used to enhance the separability of various complex targets, thereby improving the accuracy of subsequent category confirmation. Specifically, the point cloud data is the point cloud data within one frame, and a plurality of point cloud features and the correlations between different point cloud features can be determined according to the point cloud data.

[0026] In one possible implementation, the point cloud data includes point cloud data of multiple consecutive frames; the obtaining, according to the point cloud data of the first object, of first prediction information about the category of the first object through a classification model includes: determining, according to the point cloud data of the first object, a plurality of point cloud features of each frame and the changes between the point cloud features of different frames; and obtaining, according to the plurality of point cloud features and the changes, the first prediction information of the first object through the classification model.

[0027] In a possible implementation, it is also possible to extract the changes between the point cloud features of different frames from the multi-frame point cloud (this change can also be referred to as spatio-temporal features). For example, spatio-temporal features such as the change features of the target short-time scattering center, density change features, and velocity change features are used to enhance the richness and stability of the features, thereby improving the accuracy of subsequent category confirmation.

[0028] In a possible implementation, the method further includes: obtaining the point cloud data of a third object; the point cloud data of the first object is independent scattering points, and the point cloud data of the third object is group scattering points; according to the point cloud data of the third object, through the classification model, the third prediction information of the category of the third object is obtained.

[0029] In a complex open scene, there are problems such as different target sizes and large differences in the relative positions of the targets with respect to the radar. Therefore, there are also significant differences in the number of point clouds of the targets. Samples of different categories have different numbers of point clouds, which can also be used as classification information and subsequently input into the classifier. For independent scattering points and group scattering points, in the embodiments of the present application, in the case where there are independent scattering points and group scattering points for the target, the same classifier is used to uniformly identify the independent scattering points and group scattering points. The additional information carried by the independent scattering points and group scattering points is fully utilized to improve the accuracy of the final identification. The reason for the improvement in the accuracy of target recognition is that there are significant differences in the number of point clouds of different categories of targets themselves. Most targets with independent scattering points belong to categories with small volumes and weak reflection coefficients, such as two-wheeled vehicles and pedestrians. Most motor vehicles are group scattering point targets at close range. Therefore, the independent scattering points and group scattering points themselves carry a certain amount of information and have a certain degree of distinguishability. Processing the two types of target point clouds jointly in the same classifier can fully utilize this part of the information, making the classifier more distinguishable for different categories of targets. Therefore, in the embodiments of the present application, the independent scattering points and group scattering points are uniformly identified.

[0030] In a possible implementation, the method is applied to an edge device.

[0031] In a possible implementation, the classification model is implemented based on a machine learning algorithm that is not based on deep learning.

[0032] In a second aspect, the present application provides a data processing device, which includes:

[0033] An acquisition module, configured to acquire the point cloud data of a first object;

[0034] A processing module, configured to obtain first prediction information about the category of the first object according to the point cloud data of the first object; wherein, the point cloud data of the first object includes point cloud data of multiple consecutive frames, and the category corresponding to the point cloud data of the first object is an unknown category.

[0035] In a possible implementation, the number of the multiple consecutive frames is greater than 5.

[0036] In a possible implementation, the category of the first object is a category that is not predefined in advance.

[0037] In a possible implementation, the processing module is specifically configured to:

[0038] According to the point cloud data of the first object, obtain first prediction information about the category of the first object through a classification model; when training the classification model, training samples including the category of the first object are not used.

[0039] In a possible implementation, the classification model includes multiple different classifiers; the multiple consecutive frames include a first target frame;

[0040] The processing module is specifically configured to:

[0041] According to the point cloud data of the first target frame, obtain multiple first prediction information about the category of the first object through multiple different classifiers;

[0042] When the difference between the multiple first prediction information does not meet a preset condition, determine that the category corresponding to the point cloud data of the first target frame is an unknown category.

[0043] In a possible implementation, the multiple different classifiers include a first classifier and a second classifier, the model structures of the first classifier and the second classifier are different, or, the numerical values of the model parameters of the first classifier and the second classifier are different, and the category recognition accuracies of the first classifier and the second classifier are both greater than a threshold.

[0044] In a possible implementation, the classifier is constructed by sampling multiple base classifiers, and the base classifiers included in different classifiers are not completely the same.

[0045] In a possible implementation, the base classifier is a decision tree, and the classifier is a random forest.

[0046] In a possible implementation, the difference is represented by the entropy of the multiple prediction information.

[0047] In a possible implementation, the processing module is further configured to:

[0048] Obtain the point cloud data of the second object;

[0049] Based on the point cloud data of the second object, obtain multiple second prediction information on the category of the second object through multiple different classifiers;

[0050] When the difference between the multiple second prediction information meets a preset condition, determine the category of the second object according to the multiple second prediction information.

[0051] In a possible implementation, the processing module is specifically configured to:

[0052] Based on the point cloud data of multiple consecutive frames of the second object, obtain multiple second prediction information on the category of the second object in each frame through multiple different classifiers;

[0053] When the difference between the multiple second prediction information meets a preset condition, determine the category of the second object in the second target frame according to the multiple second prediction information.

[0054] In a possible implementation, the processing module is specifically configured to:

[0055] When the difference between the multiple second prediction information corresponding to the point cloud data of each frame or multiple frames that exceed a preset proportion of frames all meets the preset condition, determine the category of the second object according to the fusion result of the multiple second prediction information of the frames whose differences all meet the preset condition.

[0056] In a possible implementation, the obtaining module is further configured to:

[0057] Obtain the point cloud data of the third object and the point cloud data of the fourth object; wherein, the point cloud data of the third object and the point cloud data of the fourth object are consistent in the first point cloud feature; the point cloud data of the third object and the point cloud data of the fourth object are inconsistent in the second point cloud feature; the first point cloud feature is a necessary feature used for object classification;

[0058] The processing module is further configured to obtain third prediction information on the category of the third object and fourth prediction information on the category of the fourth object according to the point cloud data of the third object and the fourth object; the third prediction information and the fourth prediction information indicate that the categories of the third object and the fourth object are different.

[0059] In a possible implementation, the point cloud data of the first object is the point cloud data within one frame; the processing module is specifically configured to:

[0060] Based on the point cloud data of the first object, determine multiple point cloud features and the correlation between different point cloud features;

[0061] Based on the multiple point cloud features and the correlation, first prediction information of the first object is obtained through a classification model.

[0062] In a possible implementation, the point cloud data includes point cloud data of multiple consecutive frames; the processing module is specifically configured to:

[0063] Determine multiple point cloud features of each frame and the change situation between the point cloud features of different frames according to the point cloud data of the first object;

[0064] Based on the multiple point cloud features and the change situation, first prediction information of the first object is obtained through a classification model.

[0065] In a possible implementation, the obtaining module is further configured to:

[0066] Obtain point cloud data of a third object; the point cloud data of the first object is independent scattering points, and the point cloud data of the third object is group scattering points;

[0067] The processing module is further configured to:

[0068] Based on the point cloud data of the third object, third prediction information of the category of the third object is obtained through the classification model.

[0069] In a possible implementation, the method is applied to an edge device.

[0070] In a possible implementation, the classification model is implemented based on a machine learning algorithm that does not rely on deep learning.

[0071] In a third aspect, the present application provides a data processing device, which includes a memory and a processor; the memory stores code, and the processor is configured to obtain the code and execute the method according to any one of the first aspect.

[0072] In a possible implementation, the device further includes a radar system for:

[0073] Providing a radar field;

[0074] Sensing reflections from objects in the radar field to obtain the point cloud data.

[0075] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and when it runs on a computer, it causes the computer to execute the method according to the first aspect and any of its optional methods.

[0076] Fifth aspect, an embodiment of the present application provides a computer program product including instructions, which when running on a computer, causes the computer to execute the method according to the first aspect and any optional method thereof as described above.

[0077] Sixth aspect, the present application provides a chip system, which includes a processor for supporting a device to implement some or all of the functions involved in the above aspects. For example, sending or processing the data or information involved in the above method. In a possible design, the chip system further includes a memory for storing necessary program instructions and data for an execution device or a training device. The chip system may be composed of chips or may include chips and other discrete devices. Description of the Drawings

[0078] Figure 1 Schematic diagram of the architecture provided by the embodiment of the present application;

[0079] Figure 2 Schematic diagram of the radar system provided by the embodiment of the present application;

[0080] Figure 3 Schematic diagram of a data processing method provided by the embodiment of the present application;

[0081] Figure 4 Schematic diagram of point clouds of different categories of targets provided by the embodiment of the present application;

[0082] Figure 5 Schematic diagram of an architecture provided by the embodiment of the present application;

[0083] Figure 6 Schematic diagram of the entropy value spatial distribution provided by the embodiment of the present application;

[0084] Figure 7 Schematic diagram of a data processing device provided by the embodiment of the present application;

[0085] Figure 8 Schematic diagram of a data processing device provided by the embodiment of the present application;

[0086] Figure 9 Schematic structural diagram of a chip provided by the embodiment of the present application. Detailed Embodiments

[0087] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the embodiments of the present application are only for explaining the specific embodiments of the present application and are not intended to limit the present application.

[0088] In the description, claims and the above-mentioned drawings of this application, terms such as "first" and "second" are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing embodiments of this application. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0089] The embodiments of this application will be described below with reference to the drawings. As can be known to those of ordinary skill in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. First, the application scenarios of the embodiments of this application will be introduced:

[0090] The data processing method provided by the embodiments of the present invention can be applied to scenarios such as railway monitoring and highway monitoring that require real-time monitoring of targets appearing in the defense area. Schematically, the scenarios to which the data processing method provided by the embodiments of this application can be applied include but are not limited to:

[0091] Scenario 1: Railway monitoring scenario.

[0092] The perimeter of a railway line is an important part of railway line safety protection. In recent years, when the perimeter of a railway line has been invaded by a target, due to the lack of timely warning, many accidents have occurred, seriously affecting the normal operation of trains and causing huge losses. The situation of railway lines is complex and the environment is relatively harsh, which is a scenario that requires key attention and protection. To ensure the operation safety of railway lines, cameras are usually installed along the railway lines to monitor the railway defense area, track and photograph targets such as animals or non-motor vehicles that break into the railway defense area, and realize perimeter intrusion warning of the line. Schematically, referring to Figure 1 , Figure 1 is a schematic diagram of an application scenario of a data processing method provided by the embodiments of the present invention. As shown in Figure 1 , installation poles are erected at certain intervals along the railway line, and radars and cameras are installed on the installation poles. By means of the linkage between the radar and the camera (when the camera is a dome camera, this linkage method is called radar-dome camera linkage), the railway defense area is monitored. When the radar detects that a target invades the perimeter of the line and enters the railway defense area, the position of the target is calculated, and the camera is controlled to track and photograph the target and report an alarm, so that railway staff can timely learn about the railway operation status and ensure the operation safety of the railway.

[0093] Scenario 2: Highway monitoring scenario.

[0094] To ensure the safety of vehicle driving on highways, cameras are often installed on both sides or above the highways to monitor the defense areas of the highways, track and photograph targets such as animals or non-motor vehicles that break into the defense areas, and realize perimeter intrusion alarm for highways. In this scenario, for example, mounting poles are erected at certain intervals along both sides of the highway, and radars and cameras are installed on the mounting poles. By means of the linkage between the radar and the camera (when the camera is a PTZ camera, this linkage method is called radar-PTZ linkage), the defense area of the highway is monitored. When the radar detects that a target invades the highway perimeter and enters the defense area, the position of the target is calculated, and the camera is controlled to track and photograph the target and report an alarm, so that highway staff can timely learn the real-time road conditions of the highway and ensure the safety of vehicle driving.

[0095] Scenario Three: Traffic Situation Awareness.

[0096] In this application, the radar can be deployed along the road to detect and distinguish multiple targets such as people, vehicles, and foreign objects, and realize traffic situation awareness on highways, urban roads, and intersections.

[0097] In addition, it can also be applied to intrusion detection of vehicles, ships, people, etc. at the border and coastal defense; perimeter intrusion of foreign objects such as railways and oil and gas pipelines, etc. It can also be used for vehicle-mounted millimeter-wave radars.

[0098] Regarding the radar system:

[0099] According to different specific implementations of the radar system, the radar signal can have multiple carriers. For example: when the radar system is a microwave radar, the radar signal is a microwave signal; when the radar system is an ultrasonic radar, the radar signal is an ultrasonic signal; when the radar system is a lidar, the radar signal is a laser signal. It should be noted that when the radar system integrates multiple different radars, the radar signal can be a collection of multiple radar signals, which is not limited here.

[0100] The radar system can generate a radar signal and transmit the radar signal into the area being monitored by the radar system. Referring to Figure 2 , the generation and transmission of the signal can be realized by a radio frequency (RF) signal generator 12, a radar transmission circuit 14, and a transmitting antenna 32. The radar transmission circuit 14 generally includes any circuit required to generate the signal transmitted via the transmitting antenna 32, such as a pulse shaping circuit, a transmission trigger circuit, an RF switch circuit, or other appropriate transmission circuits. The RF signal generator 12 and the radar transmission circuit 14 can be controlled via a processor 20, and the processor issues commands and control signals via a control line 34, so that a desired RF signal with a desired configuration and signal parameters is transmitted at the transmitting antenna 32.

[0101] The radar system can also receive the returned radar signal at the analog processing circuit 16 via the receiving antenna 30. This returned radar signal can be referred to as "echo", "radar data", "echo signal", "echo data", or "reflected signal". The analog processing circuit 16 generally includes any circuits required to process the signals received via the receiving antenna 30 (such as signal separation, mixing, heterodyne and / or homodyne conversion, amplification, filtering, received signal triggering, signal switching and routing, and / or other appropriate radar signal receiving functions). Thus, the analog processing circuit 16 generates one or more analog signals, such as in-phase (I) analog signals and quadrature (Q) analog signals. The resulting analog signals are transmitted to the analog-to-digital converter circuit (ADC) 18 and digitized by this circuit. Then, the digitized signals are forwarded to the processor 20 for reflected signal processing.

[0102] Regarding the processor:

[0103] The processor 20 can be one of various types of processors that can process the digitized received signals and control the RF signal generator 12 and the radar transmitting circuit 14 to provide the radar operations and functions of the terminal device 100. Thus, the processor 20 can be a digital signal processor (DSP), a microprocessor, a microcontroller, or other such devices.

[0104] In some implementations, the processor 20 can include hardware circuits (such as application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor, or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor 20 can be a hardware system with the function of executing instructions, such as a CPU, DSP, etc., or a hardware system without the function of executing instructions, such as an ASIC, FPGA, etc., or a combination of the above hardware systems without the function of executing instructions and hardware systems with the function of executing instructions.

[0105] To perform the radar operations and functions of the radar system, the processor 20 interfaces via the system bus 22 with one or more other required circuits (such as one or more memory devices 24 composed of one or more types of memories, any required peripheral circuits 26 identified, and any required input / output circuits 28).

[0106] As described above, the processor 20 may interface with the RF signal generator 12 and the radar transmitting circuit 14 via the control line 34. In an alternative embodiment, the RF signal generator 12 and / or the radar transmitting circuit 14 may be connected to the bus 22 such that they may communicate with one or more of the processor 20, the memory device 24, the peripheral circuit 26, and the input / output circuit 28 via the bus 22.

[0107] Wherein, the target object may be located within the monitoring area of the radar system, so the radar system may receive the reflected signal after the target object reflects the radar signal. And based on the reflection information, an identification result of the target within the monitoring area may be obtained. The target may be a vehicle, a pedestrian, a tree, etc.

[0108] In the embodiment of the present application, the processor 20 may obtain the code stored in the memory device 24 (or a memory device separately deployed from the processor 20) to implement the data processing method in the embodiment of the present application.

[0109] Specifically, the processor 20 may be a hardware system having an instruction execution function. The data processing method provided in the embodiment of the present application may be software code stored in the memory. The processor 20 may obtain the software code from the memory and execute the obtained software code to implement the data processing method provided in the embodiment of the present application.

[0110] It should be understood that the processor 20 may also be a combination of a hardware system without an instruction execution function and a hardware system with an instruction execution function. Some steps in the data processing method provided in the embodiment of the present application may also be implemented by the hardware system without an instruction execution function in the processor 20, which is not limited herein.

[0111] In some possible implementations, the above step of determining the identity of the target object may also be implemented based on the interaction with the cloud server 300.

[0112] The radar system may be configured with a wireless communication module. The radar system may establish a communication connection with the cloud server 300 through the wireless communication module.

[0113] Regarding the wireless communication module:

[0114] The wireless communication module can provide one or more of wireless communication methods such as wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), near field communication (NFC), and infrared technology (IR) applied to radar systems. In some embodiments, the smart home device 200 may also be configured with a mobile communication module, and the mobile communication module can provide solutions including wireless communication technologies such as 2G / 3G / 4G / 5G applied to the electronic device 100.

[0115] In an alternative implementation, after receiving the radar data, the processor 20 can transfer the radar data to the cloud server 300, and then the server can process the radar data to obtain a data processing result.

[0116] In an alternative implementation, the processor 20 can obtain data processing results sent from the cloud server, etc.

[0117] Traffic situation awareness requires real-time and accurate perception of data, which is the basis for subsequent realization of intelligent management and control such as traffic monitoring, control, decision-making, scheduling, and evacuation. However, traffic management and control require all-time and all-scenario perception. However, traditional sensors such as cameras deteriorate sharply in environments such as rainy and foggy weather and strong light irradiation, and cannot provide reliable perception results.

[0118] Millimeter-wave radar has strong anti-interference ability and has the perception ability of all-day and all-weather and large detection range. By identifying targets of interest such as motor vehicles (large, medium, and small cars), non-motor vehicles, and pedestrians (getting off vehicles, pedestrians crossing, construction workers, etc.) with millimeter-wave radar, high-accuracy and stable identification in all-day and all-weather, complex open, and long-distance scenarios can be achieved, and high-precision situation awareness of highways, urban roads, and intersections can be realized, providing an effective monitoring means and decision-making basis for traffic management. It can also be applied to scenarios such as intrusion detection of vehicles, ships, and people in border and coastal defense; and perimeter intrusion of foreign objects such as railways and oil and gas pipelines.

[0119] In the prior art, a classifier is trained with training samples so that the classifier can have the ability to identify the categories included in the training samples. However, when encountering a target of an unknown category (that is, a category not included in the training samples), the target will be classified as a known category, and the target cannot be rejected, resulting in incorrect classification results.

[0120] To solve the above problems, the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0121] Referring to Figure 3 , Figure 3 FIG. is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application. The data processing method provided by the embodiment of the present application can be applied to a processor in an electronic device, a server, or a radar system (for example, deployed on a radar chip). Among them, the electronic device can be a product such as a vehicle-mounted device. As Figure 3 shown, the data processing method provided by the embodiment of the present application can include:

[0122] 301. Obtain the point cloud data of the first object.

[0123] In a possible implementation, the object to be detected (for example, including the first object) may be located in the monitoring area of the radar system. The radar system can emit radar signals to the monitored area and receive the reflected signals of the object to be detected on the radar signals. For example, the generation and emission of the signals can be realized by the RF signal generator 12, the radar transmitting circuit 14, and the transmitting antenna 32 in the above embodiments.

[0124] Among them, in the scenario of road condition monitoring, the object to be detected can be a motor vehicle (large, medium, and small cars), a non-motor vehicle, a pedestrian (getting off the vehicle, pedestrian crossing, construction worker, etc.).

[0125] Among them, the radar system can generate radar signals, and the types of radar signals can include but are not limited to continuous wave (CW) signals and chirp signals (or called chirp).

[0126] Taking the chirp signal as an example, a chirp signal is an electromagnetic signal whose frequency varies with time. Generally, the frequency of a rising chirp signal increases over time, while the frequency of a falling chirp signal decreases over time. The frequency variation of a chirp signal can take many different forms. For example, the frequency of a linear frequency modulated (LFM) signal varies linearly. Other forms of frequency variation in a chirp signal include exponential variation. In addition to the type of chirp signal in which the frequency varies continuously according to some predetermined function (i.e., a linear function or an exponential function), a chirp signal in the form of a stepped chirp signal can also be generated, where the frequency changes in steps. That is, a typical stepped chirp signal includes multiple frequency steps, where the frequency is constant for a certain predetermined duration at each step. The stepped chirp signal can also be pulsed on and off, where the pulse is on during a certain predetermined time period during each step of the chirp scan.

[0127] In one possible implementation, a radar system can transmit a chirp signal, and the mathematical expression of the chirp signal can be exemplarily:

[0128]

[0129] Where, B is the bandwidth, is the fixed initial phase, t c is the chirp signal period, A is the amplitude, f 0 is the starting frequency.

[0130] In one possible implementation, a radar system can transmit a radar signal and receive a reflected signal from a first object.

[0131] In one possible implementation, point cloud data of the first object can be obtained based on the reflected signal. The point cloud data can include multiple point clouds, and each point cloud can include, but is not limited to, position information, color information, reflection intensity information, and echo count information.

[0132] In one possible implementation, point cloud features can be constructed based on the point cloud data of the first object, and the point cloud features can be used to determine the category of the target. Existing feature extraction methods do not make full use of the target motion information and are not comprehensive enough in extracting the target point cloud features. Only simple basic features such as point cloud distance, speed, angle, and RCS measurement values are used, which cannot meet the requirements of recognition accuracy and generalization in complex open scenarios.

[0133] In a possible implementation, the correlation features between some basic features of the target can be extracted from a single-frame point cloud, and the separability of various complex targets can be enhanced by using the correlation features, thereby improving the accuracy of subsequent category confirmation. Specifically, the point cloud data is the point cloud data within one frame, and multiple point cloud features and the correlations between different point cloud features can be determined according to the point cloud data.

[0134] For example, the point cloud features can be instantaneous position, shape, velocity, electromagnetic scattering characteristics, and the correlations between different point cloud features can be but are not limited to: the correlation features between instantaneous position and shape, the correlation features between instantaneous position and velocity, the correlation features between instantaneous position and electromagnetic scattering, the correlation features between shape and velocity, the correlation features between shape and electromagnetic scattering, the correlation features between velocity and electromagnetic scattering, etc.

[0135] More specifically, the correlations between different point cloud features can be but are not limited to: the correlation features between various basic features such as Y-RCS feature, X-RCS feature, R-V, A-RCS, V-A, etc. Compared with the numerical features directly output by the radar, the correlation features have better inter-class distinguishability, and at the same time are ranked higher than most other features in the feature importance ranking.

[0136] In a possible implementation, the change conditions (which can also be referred to as spatio-temporal features) between the point cloud features of different frames can also be extracted from multiple-frame point clouds. For example, spatio-temporal features such as the short-time scattering center change feature, density change feature, velocity change feature of the target, etc. are used to enhance the richness and stability of the features, thereby improving the accuracy of subsequent category confirmation.

[0137] In a possible implementation, the point cloud data includes consecutive multiple-frame point cloud data; multiple point cloud features of each frame and the change conditions between the point cloud features of different frames can be determined according to the point cloud data.

[0138] Exemplarily, the change conditions between the point cloud features of different frames can be but are not limited to the change amounts of the means of R, A, X, Y, V, RCS within two or three frames, the change amounts of the circumscribed rectangle density, area, convex hull area, etc. within two frames. The temporal features can fuse the features of consecutive multiple frames and can reflect the motion state of the object. Compared with single-frame features, the introduction of multi-frame features enhances the richness and stability of the features, and at the same time, the target motion state information is more fully utilized, ensuring the generalization ability of subsequent target recognition.

[0139] In addition, in complex open scenarios, there are problems such as various target sizes and large differences in the relative positions of targets with respect to the radar. Therefore, there are also significant differences in the number of point clouds of targets. Samples of different categories have different numbers of point clouds, which can also be used as classification information and subsequently input into a classifier. For independent scattering points and group scattering points, there are significant differences in the feature extraction methods. Therefore, in the embodiments of the present application, for the case where target samples have independent scattering points and group scattering points, multiple features with different dimensions can be constructed according to the characteristics of independent scattering points and group scattering points.

[0140] Regarding the feature extraction of independent scattering points: The target of independent scattering points mainly includes the physical information output by the radar itself and the variation information of multi-frame features. Since the information carried by independent scattering points themselves is less, the number of feature dimensions that can be extracted is less, and most of them are composed of the variation information of basic features.

[0141] Regarding the feature extraction of group scattering points: Group scattering points can extract multi-dimensional features such as the statistics, shape, point cloud density, correlation features of each feature, and temporal variation features of multiple frames of each target point. Compared with the features of independent scattering point targets, the features of group scattering points have higher dimensions and richer information.

[0142] 302. Obtain first prediction information about the category of the first object according to the point cloud data of the first object; wherein, the point cloud data of the first object includes point cloud data of multiple consecutive frames, and the category corresponding to the point cloud data of the first object is an unknown category.

[0143] For example, the first prediction information may indicate that the category corresponding to the point cloud data of the first object is an unknown category.

[0144] For example, the number of multiple consecutive frames is the number of frames (or most of the frames) when the first target appears within the radar detection range.

[0145] In a possible implementation, the number of multiple consecutive frames is greater than 5, and for example, it can be 6, 7, 8, 10, 15, 20, etc.

[0146] In a possible implementation, the category of the first object is a category that is not predefined in advance. For example, according to the point cloud data of the first object, through a classification model, first prediction information about the category of the first object can be obtained; training samples including the category of the first object are not used when training the classification model. That is to say, for the classification model, it does not have the ability to recognize the category of the first object.

[0147] When the present application performs category recognition on an object of an unknown category, it can output the judgment results of multiple consecutive frames of the unknown category, which is equivalent to being able to output more accurate and stable category determination results.

[0148] In a possible implementation, the classification model includes a plurality of different classifiers; the continuous multi-frame includes a first target frame; obtaining first prediction information about the category of the first object based on the point cloud data of the first object through the classification model includes: obtaining a plurality of first prediction information about the category of the first object through a plurality of different classifiers based on the point cloud data of the first target frame;

[0149] In the existing target recognition method based on radar point cloud, when encountering an unknown class target in a complex open scene, the target will be classified into a known category and it is impossible to reject the judgment of the target.

[0150] To solve the above problems, the idea of this application is to train a plurality of different classifiers. When the plurality of different classifiers process the same unknown class target, there will be great differences in the processing results among them. Therefore, it is possible to determine whether an unknown class target is encountered based on the differences among the processing results of the plurality of classifiers. For example, when the differences among the processing results of the plurality of classifiers are large, it can be considered that the target is an unknown class target, and then the category of the target can be output as unknown. When the differences among the processing results of the plurality of classifiers are small, it can be considered that the target is a known class target, and then the specific category of the target can be output.

[0151] In a possible implementation, the plurality of different classifiers include a first classifier and a second classifier, and the model structures of the first classifier and the second classifier are different, or the numerical values of the model parameters of the first classifier and the second classifier are different.

[0152] For example, the first classifier and the second classifier can be completely different model types (for example, one is a decision tree and the other is implemented based on a neural network), or the same model type but different model parameters (for example, obtained by training with different training samples), or the same type of models but different structures (for example, both are neural networks, and the number of network layers or the types of network layers of the two classifiers are not completely the same).

[0153] In a possible implementation, the category recognition accuracies of the first classifier and the second classifier are both greater than a threshold. Among them, the recognition accuracy of each classifier for at least known category targets is relatively high.

[0154] Next, a schematic implementation of the classifier will be introduced taking a random forest as an example:

[0155] In a possible implementation, the classifier is a random forest constructed by sampling a plurality of decision trees, and the decision trees included in different classifiers are different.

[0156] That is to say, a random forest classifier composed of multiple different decision trees can be trained, and then, by means of sampling with replacement, multiple random forest classifiers for the current prediction can be extracted.

[0157] In one possible implementation, each classifier can obtain the probabilities of the first object for each known category.

[0158] In one possible implementation, when the difference between the multiple pieces of prediction information does not meet the preset condition, it can be determined that the category of the first object is unknown.

[0159] In one possible implementation, the difference between multiple pieces of prediction information can be represented by, but not limited to, the entropy of the multiple pieces of prediction information. That is, based on multiple pieces of prediction information (such as category prediction probabilities), the entropy value can then be calculated using these prediction information, and this is used as the basis for estimating the uncertainty of the category. The greater the uncertainty of the sample, the less reliable the sample. Therefore, when an unknown category sample appears, the reliability of the sample is estimated using the above calculation method. If the sample uncertainty exceeds the set threshold, the sample is rejected as an unknown category, thereby achieving the ability to reject unknown category samples in complex open scenarios.

[0160] It should be understood that the mainstream technology of the existing technology is to first use a neural network or other classifier to output a prediction probability vector, and then directly use the prediction probability as the confidence level of the sample. However, the existing technology mainly has the following two disadvantages: First, using the prediction probability for judgment, the recognition result is unreliable: The prediction probability output by a single classifier only represents the degree of conformity of the current judgment of the sample with the features of each category, and cannot provide the reliability information of the prediction probability, which is not the same concept as the confidence level and reliability. Second, because the probability values vary greatly, the thresholds for different scenarios and different categories are different, and the generalization ability for scenarios is poor: The prediction probability is a vector containing multiple probability values, and there is no unified measurement value. Moreover, the prediction probability distributions of different categories vary greatly, and it is difficult to determine the threshold for some categories.

[0161] Taking the classifier as a random forest as an example, an ensemble method is used to estimate the uncertainty of sample judgment. By performing multiple samplings with replacement on the trained decision trees, multiple different random forest models are obtained. The advantage of sampling with replacement is that it can ensure that the classification effect of each random forest is relatively good. Then, multiple prediction probability vectors are output using multiple different initialized random forest models, and the average value of the multiple prediction probability vectors is calculated as the sample entropy value, which is used as the basis for estimating the sample uncertainty, thereby reflecting the reliability of the recognition result.

[0162] In the case of K classification, the interval of the entropy value is

[0163] Such as Figure 6As shown, the left figure is the distribution of known and unknown classes in the entropy value space, and the right figure is the distribution of known and unknown classes in the probability space. It can be seen that in the entropy value space, most of the unknown classes and known classes do not overlap, while in the probability space, most of the unknown classes and known classes have obvious overlap, and it is impossible to distinguish unknown class samples from known class samples. Thus, the advantage of rejecting unknown classes in the entropy value space is demonstrated.

[0164] In a possible implementation, the method further includes: obtaining point cloud data of a second object; according to the point cloud data of the second object, through a plurality of different classifiers, obtaining a plurality of second prediction information on the class of the second object; when the difference between the plurality of second prediction information satisfies a preset condition, determining the class of the second object according to the plurality of second prediction information.

[0165] For example, when the difference between the plurality of second prediction information satisfies a preset condition, determining the class of the second object according to the plurality of second prediction information. When determining the class of the second object according to the plurality of second prediction information, a mean fusion algorithm can be used, that is, after fusing the plurality of second prediction information (for example, taking the average), then using the fusion result (for example, if each second prediction information is the probability of each class, the fusion result is the fusion result of the probability distribution, which is still the probability of each class) as the class of the second object.

[0166] In a possible implementation, the method further includes: obtaining point cloud data of a third object; the point cloud data of the first object is independent scattering points, and the point cloud data of the third object is group scattering points; according to the point cloud data of the third object, through the classification model, obtaining third prediction information on the class of the third object.

[0167] Due to problems such as different target sizes and large differences in the relative positions of targets to the radar in complex open scenarios, there are also significant differences in the number of point clouds of targets. Samples of different classes have different numbers of point clouds, which can also be used as classification information and subsequently input into the classifier. For independent scattering points and group scattering points, in the embodiments of the present application, for the situation where the target has independent scattering points and group scattering points, the same classifier is used to uniformly identify independent scattering points and group scattering points. The additional information carried by independent scattering points and group scattering points is fully utilized to improve the accuracy of the final identification. The reason for improving the accuracy of target identification is that there are significant differences in the number of point clouds of different classes of targets themselves. Most of the targets with independent scattering points belong to categories with small volumes and weak reflection coefficients, such as two-wheeled vehicles and pedestrians (exemplarily, reference can be made to Figure 4As shown). Most motor vehicles are group scattering point targets at close range. Therefore, the independent scattering points and group scattering points themselves carry a certain amount of information and have a certain degree of distinguishability. Combining the point clouds of the two types of targets in the same classifier can make full use of this part of the information, making the classifier better able to distinguish different types of targets. Therefore, in the embodiments of the present application, the independent scattering points and group scattering points are uniformly recognized.

[0168] An exemplary process is schematically shown as follows:

[0169] First, unify the feature dimensions of the independent scattering points and group scattering points, and then use the same classifier for training and recognition to form an end-to-end recognition system, enabling the classifier to make more full use of the existing information. This method can use ensemble methods such as random forest as the classifier. Random forest achieves good generalization performance through the randomness of sample selection and the randomness of decision tree feature extraction. Moreover, random forest is less sensitive to partial missing features compared to other mainstream classifiers and can better handle the problem of inconsistent feature dimensions of independent scattering points and group scattering points. It also has better applicability to occluded targets and can better solve various application problems in actual scenarios.

[0170] In addition, due to volume and cost limitations, the on-board radar is not equipped with a high-computing-power processor. Deep networks have high computing power requirements and cannot meet the computing power limitations of the on-board radar. Moreover, the radar has a high frame rate and a large amount of echo data, with high requirements for algorithm real-time performance. Many deep learning algorithms cannot meet the real-time requirements. In the embodiments of the present application, using the above-mentioned uncertainty estimation calculation method, an estimation result of sample uncertainty can be output for each frame in the sample trajectory. Then, during the sample prediction process, the uncertainty values of each frame in each sample trajectory are detected. When the difference degrees of the recognition results of multiple consecutive frames all meet the preset conditions (for example, when there are N consecutive frames of uncertainty values, each value is less than a pre-set threshold M), it is determined that the current road section belongs to a high-reliability target road section. Therefore, the prediction results of these consecutive N frames are selected, and a fusion calculation is performed on the prediction results of these N frames. The mean fusion algorithm is used in this method, and then the fusion result is used as the final decision result of the current sample trajectory, and the result is not updated in the subsequent trajectory of the sample.

[0171] In a possible implementation, the method further includes: obtaining the point cloud data of a second object; according to the point cloud data of the second object, obtaining multiple second prediction information about the category of the second object through multiple different classifiers; and when the difference between the multiple second prediction information meets the preset conditions, determining the category of the second object according to the multiple second prediction information.

[0172] In a possible implementation, the point cloud data of the second object includes point cloud data of multiple consecutive frames; the multiple second prediction information includes multiple second prediction information corresponding to the point cloud data of each frame; determining the category of the second object according to the multiple second prediction information includes: when the differences between the multiple second prediction information corresponding to the point cloud data of each frame or more than a preset proportion of frames among multiple frames all meet a preset condition, determining the category of the second object according to the fusion result of the multiple second prediction information of the frames whose differences all meet the preset condition.

[0173] In a possible implementation, the method further includes: using the category of the second object as the category of the second object in the frame after the multiple consecutive frames.

[0174] In a possible implementation, the point cloud data of the second object includes point cloud data of multiple consecutive frames; the multiple second prediction information includes multiple second prediction information corresponding to the point cloud data of each frame; the method further includes: obtaining the point cloud data of the second object in the frame after the multiple consecutive frames; when the differences between the multiple second prediction information corresponding to the point cloud data of more than a preset proportion of frames among the multiple frames do not meet a preset condition, determining the category of the second object in the frame after the multiple consecutive frames according to the point cloud data of the frame after the multiple consecutive frames.

[0175] In the above manner, for each sample, the decision results of multiple consecutive frames of highly reliable samples are fused as the prediction result of the entire trajectory, and subsequent trajectories are no longer predicted, thereby saving the device resource overhead, improving the stability and accuracy of the final output result, and avoiding the output result from jumping when a single-frame sample is predicted incorrectly during the prediction process.

[0176] Exemplarily, referring to Figure 5 , Figure 5 is a schematic framework of an embodiment of the present application.

[0177] Referring to Figure 7 , Figure 7 is a structural schematic of a data processing device provided by an embodiment of the present application. As Figure 7 shown, the device includes:

[0178] An acquisition module 701, configured to acquire the point cloud data of the first object;

[0179] Among them, the specific description of the acquisition module 701 can refer to the introduction of step 301 in the above embodiment, and the similarities will not be elaborated here.

[0180] A processing module 702, configured to obtain first prediction information about the category of the first object according to the point cloud data of the first object; wherein, the point cloud data of the first object includes point cloud data of multiple consecutive frames, and the first prediction information indicates that the category corresponding to the point cloud data of the first object is an unknown category.

[0181] Wherein, the specific description of the processing module 702 can refer to the introduction of step 302 in the above embodiment, and the similarities will not be elaborated here.

[0182] In a possible implementation, the number of the multiple consecutive frames is greater than 5.

[0183] In a possible implementation, the category of the first object is a category that is not predefined in advance.

[0184] In a possible implementation, the processing module 702 is specifically configured to:

[0185] According to the point cloud data of the first object, obtain first prediction information about the category of the first object through a classification model; no training samples including the category of the first object are used when training the classification model.

[0186] In a possible implementation, the classification model includes multiple different classifiers; the multiple consecutive frames include a first target frame;

[0187] The processing module 702 is specifically configured to:

[0188] According to the point cloud data of the first target frame, obtain multiple first prediction information about the category of the first object through multiple different classifiers;

[0189] When the difference between the multiple first prediction information does not meet a preset condition, determine that the category corresponding to the point cloud data of the first target frame is an unknown category.

[0190] In a possible implementation, the multiple different classifiers include a first classifier and a second classifier, the model structures of the first classifier and the second classifier are different, or, the numerical values of the model parameters of the first classifier and the second classifier are different, and the category recognition accuracies of the first classifier and the second classifier are both greater than a threshold.

[0191] In a possible implementation, the classifier is constructed by sampling multiple base classifiers, and the base classifiers included in different classifiers are not exactly the same.

[0192] In a possible implementation, the base classifier is a decision tree and the classifier is a random forest.

[0193] In a possible implementation, the difference is represented by the entropy of the multiple pieces of prediction information.

[0194] In a possible implementation, the processing module 702 is further configured to:

[0195] Obtain the point cloud data of a second object;

[0196] Based on the point cloud data of the second object, obtain multiple pieces of second prediction information about the category of the second object through multiple different classifiers;

[0197] When the difference between the multiple pieces of second prediction information meets a preset condition, determine the category of the second object according to the multiple pieces of second prediction information.

[0198] In a possible implementation, the processing module 702 is specifically configured to:

[0199] Based on the point cloud data of multiple consecutive frames of the second object, obtain multiple pieces of second prediction information about the category of the second object in each frame through multiple different classifiers;

[0200] When the difference between the multiple pieces of second prediction information meets a preset condition, determine the category of the second object in the second target frame according to the multiple pieces of second prediction information.

[0201] In a possible implementation, the processing module 702 is specifically configured to:

[0202] When the difference between the multiple pieces of second prediction information corresponding to the point cloud data of more than a preset proportion of frames in each frame or multiple frames all meets the preset condition, determine the category of the second object according to the fusion result of the multiple pieces of second prediction information of the frames whose differences all meet the preset condition.

[0203] In a possible implementation, the acquisition module 701 is further configured to:

[0204] Obtain the point cloud data of a third object and the point cloud data of a fourth object; wherein, the point cloud data of the third object and the point cloud data of the fourth object are consistent in a first point cloud feature; the point cloud data of the third object and the point cloud data of the fourth object are inconsistent in a second point cloud feature; the first point cloud feature is a necessary feature used for object classification;

[0205] The processing module 702 is further configured to obtain a third prediction information about the category of the third object and a fourth prediction information about the category of the fourth object according to the point cloud data of the third object and the point cloud data of the fourth object; the third prediction information and the fourth prediction information indicate that the categories of the third object and the fourth object are different.

[0206] In a possible implementation, the point cloud data of the first object is the point cloud data within one frame; the processing module 702 is specifically configured to:

[0207] Determine a plurality of point cloud features and the correlation between different point cloud features according to the point cloud data of the first object;

[0208] Obtain first prediction information of the first object through a classification model according to the plurality of point cloud features and the correlation.

[0209] In a possible implementation, the point cloud data includes point cloud data of multiple consecutive frames; the processing module 702 is specifically configured to:

[0210] Determine a plurality of point cloud features for each frame and the change situation between the point cloud features of different frames according to the point cloud data of the first object;

[0211] Obtain first prediction information of the first object through a classification model according to the plurality of point cloud features and the change situation.

[0212] In a possible implementation, the acquisition module 701 is further configured to:

[0213] Acquire the point cloud data of a third object; the point cloud data of the first object is independent scattering points, and the point cloud data of the third object is group scattering points;

[0214] The processing module 702 is further configured to:

[0215] Obtain third prediction information of the category of the third object through the classification model according to the point cloud data of the third object.

[0216] In a possible implementation, the method is applied to a terminal device.

[0217] In a possible implementation, the classification model is implemented based on a machine learning algorithm that is not based on deep learning.

[0218] Next, a data processing device provided in an embodiment of the present application will be introduced. Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of a data processing device provided in an embodiment of the present application. Specifically, the data processing device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (where the number of processors 803 in the data processing device 800 can be one or more, Figure 8Take a processor as an example. Among them, the processor 803 may include an application processor 8031 and a communication processor 8032. In some embodiments of the present application, the receiver 801, the transmitter 802, the processor 803, and the memory 804 may be connected through a bus or other means.

[0219] The memory 804 may include a read-only memory and a random access memory, and provide instructions and data to the processor 803. A part of the memory 804 may also include a non-volatile random access memory (NVRAM). The memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, where the operation instructions may include various operation instructions for implementing various operations.

[0220] The processor 803 controls the operation of the radar system (including the antenna, the receiver 801, and the transmitter 802). In a specific application, the various components of the radar system are coupled together through a bus system, where the bus system may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, all kinds of buses are referred to as the bus system in the figure.

[0221] The data processing method disclosed in the above embodiments of the present application ( Figure 3As shown, it can be applied to or implemented by the processor 803. The processor 803 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 803 or the instructions in the form of software. The above-mentioned processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 803 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 804, and the processor 803 reads the information in the memory 804 and combines its hardware to complete the steps of the data processing method provided in the above embodiments.

[0222] The receiver 801 can be used to receive input digital or character information, and generate signal inputs related to the relevant settings and function controls of the radar system. The transmitter 802 can be used to output digital or character information through the first interface; the transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group.

[0223] In a possible implementation, the device further includes a radar system for:

[0224] Providing a radar field;

[0225] Sensing reflections from objects in the radar field to obtain the point cloud data.

[0226] The embodiments of the present application also provide a computer program product, which when running on a computer, causes the computer to execute the data processing method described in the above embodiments.

[0227] In an embodiment of the present application, a computer-readable storage medium is further provided. A program for signal processing is stored in the computer-readable storage medium. When it runs on a computer, the computer is caused to execute the data processing method described in the above embodiment.

[0228] The data processing device provided in the embodiment of the present application may specifically be a chip. The chip includes: a processing unit and a communication unit. The processing unit may be a processor, for example, and the communication unit may be an input / output interface, a pin, a circuit, etc. The processing unit may execute the computer-executable instructions stored in the storage unit to cause the chip in the execution device to execute the image enhancement method described in the above embodiment, or to cause the chip in the training device to execute the image enhancement method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit outside the chip in the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0229] Specifically, please refer to Figure 9 , Figure 9 which is a schematic structural diagram of the chip provided in the embodiment of the present application. The chip may be embodied as a neural network processor NPU90. The NPU 90 is mounted on the main CPU (Host CPU) as a coprocessor, and tasks are assigned by the HostCPU. The core part of the NPU is the arithmetic circuit 903. The arithmetic circuit 903 is controlled by the controller 904 to extract matrix data from the memory and perform multiplication operations.

[0230] In some implementations, the arithmetic circuit 903 includes multiple processing units (Process Engine, PE) inside. In some implementations, the arithmetic circuit 903 is a two-dimensional systolic array. The arithmetic circuit 903 may also be a one-dimensional systolic array or other electronic circuits that can perform mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 903 is a general matrix processor.

[0231] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 902 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 901 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 908.

[0232] The unified memory 906 is used to store input data and output data. The weight data is directly transferred through the direct memory access controller (DMAC) 905 and is moved to the weight memory 902. The input data is also moved to the unified memory 906 through the DMAC.

[0233] The BIU is the Bus Interface Unit, i.e., the bus interface unit 99, which is used for the interaction between the AXI bus, the DMAC, and the Instruction Fetch Buffer (IFB) 909.

[0234] The bus interface unit 99 (Bus Interface Unit, abbreviated as BIU) is used for the instruction fetch buffer 909 to obtain instructions from the external memory, and is also used for the storage unit access controller 905 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0235] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 906, or transfer the weight data to the weight memory 902, or transfer the input data to the input memory 901.

[0236] The vector calculation unit 907 includes multiple arithmetic processing units, which, if necessary, further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, etc. It is mainly used for neural network non-convolution / full connection layer network calculations, such as Batch Normalization, pixel-level summation, upsampling of the feature plane, etc.

[0237] In some implementations, the vector calculation unit 907 can store the processed output vector in the unified memory 906. For example, the vector calculation unit 907 can apply a linear function and / or a non-linear function to the output of the arithmetic circuit 903, such as performing linear interpolation on the feature plane extracted by the convolutional layer, or, for another example, a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit 907 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as the activation input to the arithmetic circuit 903, such as for use in subsequent layers in the neural network.

[0238] The instruction fetch buffer 909 connected to the controller 904 is used to store the instructions used by the controller 904;

[0239] The unified memory 906, the input memory 901, the weight memory 902, and the fetch memory 909 are all On-Chip memories. The external memory is private to the NPU hardware architecture.

[0240] Wherein, the processor mentioned anywhere above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program steps related to the data processing method described in the above embodiments.

[0241] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0242] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware. Of course, it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits. However, for this application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disc of a computer, and includes several instructions for causing a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods of the various embodiments of this application.

[0243] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0244] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, they implement all or part of the processes or functions according to the embodiments of the present application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (such as coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. A data processing method, characterized in that, the method includes: obtaining point cloud data of a first object; obtaining first prediction information of the category of the first object according to the point cloud data of the first object; wherein, the point cloud data of the first object includes point cloud data of multiple consecutive frames, and the first prediction information indicates that the category corresponding to the point cloud data of the first object is an unknown category.

2. The method according to claim 1, characterized in that, the number of the multiple consecutive frames is greater than 5.

3. The method according to claim 1 or 2, characterized in that, the category of the first object is a category that is not predefined in advance.

4. The method according to any one of claims 1 to 3, characterized in that, the obtaining first prediction information of the category of the first object according to the point cloud data of the first object includes: obtaining first prediction information of the category of the first object through a classification model according to the point cloud data of the first object; no training samples including the category of the first object are used when training the classification model.

5. The method according to claim 4, characterized in that, the classification model includes multiple different classifiers; the multiple consecutive frames include a first target frame; the obtaining first prediction information of the category of the first object through a classification model according to the point cloud data of the first object includes: obtaining multiple first prediction information of the category of the first object through multiple different classifiers according to the point cloud data of the first target frame; when the difference between the multiple first prediction information does not meet a preset condition, determining that the category corresponding to the point cloud data of the first target frame is an unknown category.

6. The method according to claim 5, characterized in that, the multiple different classifiers include a first classifier and a second classifier, the model structures of the first classifier and the second classifier are different, or, the numerical values of the model parameters of the first classifier and the second classifier are different, and the category recognition accuracies of the first classifier and the second classifier are both greater than a threshold.

7. The method according to claim 5 or 6, characterized in that, the classifier is constructed by sampling multiple base classifiers, and the base classifiers included in different classifiers are not completely the same.

8. The method according to claim 7, characterized in that, the base classifier is a decision tree, and the classifier is a random forest.

9. The method according to any one of claims 5 to 8, characterized in that, the difference is represented by the entropy of the multiple prediction information.

10. The method according to any one of claims 1 to 9, characterized in that, the method further includes: obtaining point cloud data of multiple consecutive frames of a second object; determining the category of the second object according to the point cloud data of multiple consecutive frames of the second object, and using the category of the second object as the category of the frame of the second object after the multiple consecutive frames.

11. The method according to claim 10, characterized in that, the determining the category of the second object according to the point cloud data of multiple consecutive frames of the second object includes: Based on the point cloud data of consecutive multiple frames of the second object, multiple second prediction information of the category of the second object in each frame is obtained through multiple different classifiers; When the difference between the multiple second prediction information satisfies a preset condition, the category of the second object in the second target frame is determined according to the multiple second prediction information.

12. The method according to claim 11, wherein, the determining the category of the second object according to the multiple second prediction information includes: When the difference between the multiple second prediction information corresponding to the point cloud data of more than a preset proportion of frames in each frame or multiple frames all satisfies the preset condition, the category of the second object is determined according to the fusion result of the multiple second prediction information of the frames whose differences all satisfy the preset condition.

13. The method according to any one of claims 1 to 12, wherein, the method further includes: Obtaining the point cloud data of the third object and the point cloud data of the fourth object; wherein, the point cloud data of the third object and the point cloud data of the fourth object are consistent in the first point cloud feature; the point cloud data of the third object and the point cloud data of the fourth object are inconsistent in the second point cloud feature; the first point cloud feature is a necessary feature used for object classification; According to the point cloud data of the third object and the fourth object, third prediction information of the category of the third object and fourth prediction information of the category of the fourth object are obtained; the third prediction information and the fourth prediction information indicate that the categories of the third object and the fourth object are different.

14. The method according to any one of claims 1 to 13, wherein, the point cloud data of the first object is the point cloud data within one frame; the obtaining the first prediction information of the category of the first object according to the point cloud data of the first object includes: Determining multiple point cloud features and the correlation between different point cloud features according to the point cloud data of the first object; Obtaining the first prediction information of the first object according to the multiple point cloud features and the correlation.

15. The method according to any one of claims 1 to 14, wherein, the point cloud data includes consecutive multiple frames of point cloud data; the obtaining the first prediction information of the category of the first object according to the point cloud data of the first object includes: Determining multiple point cloud features in each frame and the change situation between the point cloud features of different frames according to the point cloud data of the first object; Obtaining the first prediction information of the first object according to the multiple point cloud features and the change situation.

16. The method according to any one of claims 1 to 15, wherein, the method further includes: Obtaining the point cloud data of the third object; the point cloud data of the first object is independent scattered points, and the point cloud data of the third object is group scattered points; Obtaining the third prediction information of the category of the third object according to the point cloud data of the third object.

17. The method according to any one of claims 1 to 16, wherein, the method is applied to an edge device.

18. The method according to any one of claims 1 to 17, It is characterized in that the classification model is implemented based on a machine learning algorithm that is not based on deep learning.

19. A data processing device It is characterized in that the device includes: an acquisition module for acquiring point cloud data of a first object; a processing module for obtaining first prediction information of the category of the first object according to the point cloud data of the first object; wherein, the point cloud data of the first object includes point cloud data of multiple consecutive frames, and the first prediction information indicates that the category corresponding to the point cloud data of the first object is an unknown category.

20. The device according to claim 19 It is characterized in that the number of the multiple consecutive frames is greater than 5.

21. The device according to claim 19 or 20 It is characterized in that the category of the first object is a category that is not predefined in advance.

22. The device according to any one of claims 19 to 21 It is characterized in that the processing module is specifically configured to: obtain first prediction information of the category of the first object through a classification model according to the point cloud data of the first object; no training samples including the category of the first object are used when training the classification model.

23. The device according to claim 22 It is characterized in that the classification model includes a plurality of different classifiers; the multiple consecutive frames include a first target frame; the processing module is specifically configured to: obtain multiple first prediction information of the category of the first object through a plurality of different classifiers according to the point cloud data of the first target frame; when the difference between the multiple first prediction information does not meet a preset condition, determine that the category corresponding to the point cloud data of the first target frame is an unknown category.

24. The device according to claim 23 It is characterized in that the plurality of different classifiers include a first classifier and a second classifier, the model structures of the first classifier and the second classifier are different, or, the numerical values of the model parameters of the first classifier and the second classifier are different, and, the category recognition accuracies of the first classifier and the second classifier are both greater than a threshold.

25. The device according to claim 23 or 24 It is characterized in that the classifier is constructed by sampling a plurality of base classifiers, and the base classifiers included in different classifiers are not completely the same.

26. The device according to claim 25 It is characterized in that the base classifier is a decision tree, and the classifier is a random forest.

27. The device according to any one of claims 23 to 26 It is characterized in that the difference is represented by the entropy of the multiple prediction information.

28. The device according to any one of claims 19 to 27 It is characterized in that the processing module is further configured to: acquire point cloud data of multiple consecutive frames of a second object; determine the category of the second object according to the point cloud data of the multiple consecutive frames of the second object, and use the category of the second object as the category of the frame of the second object after the multiple consecutive frames.

29. The device according to claim 28 It is characterized in that the processing module is specifically configured to: Based on the point cloud data of consecutive multiple frames of the second object, multiple second prediction information of the category of the second object in each frame is obtained through multiple different classifiers; When the difference between the multiple second prediction information meets a preset condition, the category of the second object in the second target frame is determined according to the multiple second prediction information.

30. The device according to claim 29, wherein, the processing module is specifically configured to: When the difference between the multiple second prediction information corresponding to the point cloud data of each frame or multiple frames exceeding a preset proportion of frames all meets the preset condition, the category of the second object is determined according to the fusion result of the multiple second prediction information of the frames whose differences all meet the preset condition.

31. The device according to any one of claims 19 to 30, wherein, the obtaining module is further configured to: Obtain the point cloud data of a third object and the point cloud data of a fourth object; wherein, the point cloud data of the third object and the point cloud data of the fourth object are consistent in the first point cloud feature; the point cloud data of the third object and the point cloud data of the fourth object are inconsistent in the second point cloud feature; the first point cloud feature is a necessary feature used for object classification; the processing module is further configured to obtain third prediction information of the category of the third object and fourth prediction information of the category of the fourth object according to the point cloud data of the third object and the fourth object; the third prediction information and the fourth prediction information indicate that the categories of the third object and the fourth object are different.

32. The device according to any one of claims 19 to 31, wherein, the point cloud data of the first object is the point cloud data within one frame; the processing module is specifically configured to: Determine multiple point cloud features and the correlation between different point cloud features according to the point cloud data of the first object; Obtain the first prediction information of the first object according to the multiple point cloud features and the correlation.

33. The device according to any one of claims 19 to 32, wherein, the point cloud data includes consecutive multiple frames of point cloud data; the processing module is specifically configured to: Determine multiple point cloud features in each frame and the change situation between the point cloud features of different frames according to the point cloud data of the first object; Obtain the first prediction information of the first object according to the multiple point cloud features and the change situation.

34. The device according to any one of claims 19 to 33, wherein, the obtaining module is further configured to: Obtain the point cloud data of a third object; the point cloud data of the first object is independent scattering points, and the point cloud data of the third object is group scattering points; the processing module is further configured to: Obtain third prediction information of the category of the third object according to the point cloud data of the third object.

35. The device according to any one of claims 19 to 34, wherein, the device is applied to an edge device.

36. The device according to any one of claims 19 to 35, wherein, the classification model is implemented based on a machine learning algorithm that is not based on deep learning.

37. A data processing device, characterized in that, the device includes a memory and a processor; the memory stores code, and the processor is configured to obtain the code and execute the method according to any one of claims 1 to 18.

38. The device according to claim 37, characterized in that, the device further includes a radar system for: providing a radar field; sensing reflections from objects in the radar field to obtain the point cloud data.

39. A computer-readable storage medium, characterized in that, it includes computer-readable instructions that, when run on a computer device, cause the computer device to execute the method according to any one of claims 1 to 18.

40. A computer program product, characterized in that, it includes computer-readable instructions that, when run on a computer device, cause the computer device to execute the method according to any one of claims 1 to 18.

Images