Method and apparatus for detecting object, electronic device, and storage medium

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for detecting objects, belonging to the field of intelligent monitoring technology. It includes: acquiring target detection data from a lidar sensor; determining information about a first object detected by the lidar sensor based on the target detection data using a first target detection model; the first target detection model undergoing at least one update training, whereby the update training updates the parameters of the target detection model with detection function based on point cloud data labeled with second object information; the point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data, and the edge point cloud data includes point cloud data from the first detection data that failed to match the object in the millimeter-wave radar data; the first detection data is obtained by the lidar sensor and is time-matched with the millimeter-wave radar data. This application can accurately detect objects at a distance, thereby improving the effectiveness of object detection.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring technology, and in particular to a method, apparatus, electronic device and storage medium for detecting objects. Background Technology

[0002] The application of lidar sensors in the field of intelligent surveillance technology is becoming increasingly widespread. Currently, a detection model can be pre-trained based on a large number of training samples to obtain a target detection model with detection capabilities. In application, the lidar sensor acquires detection data, and then, based on this data, the target detection model can determine the object detected by the lidar sensor.

[0003] However, the detection range of lidar sensors is limited by their detection distance and characteristics, resulting in sparse or scarce detection data at distant points within the sensor's detection range. Consequently, when processing the detection data using a target detection model, it becomes impossible to identify distant objects, thus affecting the effectiveness of object detection. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for detecting objects, which can solve the problem in related technologies where distant objects cannot be determined when processing detection data through target detection models, thus affecting the effectiveness of object detection. The technical solution is as follows:

[0005] Firstly, a method for detecting an object is provided, the method comprising:

[0006] Obtain the current detection data from the lidar sensor to obtain target detection data;

[0007] Based on the target detection data, the first object information of the object currently detected by the lidar sensor is determined by the first target detection model;

[0008] The first target detection model undergoes at least one update training. The update training refers to updating the parameters of the target detection model with detection function based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data in the first detection data that fails to match the object in the millimeter-wave radar data. The first detection data is detected by the lidar sensor and is time-matched with the millimeter-wave radar data.

[0009] In this way, the edge point cloud data in the first detection data is labeled using the second object information in the millimeter-wave radar data used to describe the distant object. The target detection model is then updated based on the point cloud data labeled with the second object information, so that the obtained first target detection model can accurately detect the distant object, thereby improving the effectiveness of object detection.

[0010] As an example of this application, the method for obtaining the point cloud data labeled with the second object information includes:

[0011] During the operation of the millimeter-wave radar sensor and the lidar sensor, two frames of detection data with time matching are acquired. The two frames of detection data include the first detection data and the second detection data, and the second detection data is detected by the millimeter-wave radar sensor.

[0012] The second target detection model processes the second detection data to obtain the millimeter-wave radar data. The second target detection model is used to determine the object detected by the millimeter-wave radar sensor based on the second detection data.

[0013] Based on the first detection data and the millimeter-wave radar data, the point cloud data labeled with the second object information is determined.

[0014] In this way, by acquiring two frames of detection data that match the time, and labeling the data based on the acquired two frames of detection data, the effectiveness of the subsequent model update training based on the labeled point cloud data can be guaranteed, thereby enabling the updated and trained model to effectively detect distant objects.

[0015] As an example of this application, determining the point cloud data labeled with the second object information based on the first detection data and the millimeter-wave radar data includes:

[0016] The target detection model processes the first detection data and outputs information about at least one first object in the point cloud coordinate system.

[0017] The millimeter-wave radar data is mapped to the point cloud coordinate system to obtain information about at least one second object in the point cloud coordinate system.

[0018] Object matching is performed based on the information of the at least one first object in the point cloud coordinate system and the information of the at least one second object in the point cloud coordinate system.

[0019] Based on the object matching results, the point cloud data labeled with the second object information is determined.

[0020] In this way, by mapping the detection data of both lidar and millimeter-wave radar sensors to the same coordinate system, and performing object matching based on the mapped object information to identify mismatched objects, distant objects can be filtered out, thereby effectively annotating the point cloud data of distant objects.

[0021] As an example of this application, determining the point cloud data labeled with the second object information based on the object matching result includes:

[0022] Based on the object matching result, the object's 3D bounding box is determined in the point cloud coordinate system. The object's 3D bounding box refers to the 3D bounding box of the second object that does not match the at least one first object.

[0023] If the number of point clouds included in the three-dimensional bounding box of the object in the first detection data is greater than the number of points, then the point cloud data included in the three-dimensional bounding box of the object is labeled based on the data corresponding to the three-dimensional bounding box of the object in the millimeter-wave radar data, so as to obtain the point cloud data labeled with the second object information.

[0024] In this way, the 3D bounding box of the distant object is determined, and if the number of point clouds included in the 3D bounding box is greater than the number of points, the point cloud data in the 3D bounding box is labeled, so as to use millimeter-wave radar data to label objects detected by lidar sensors that include a small amount of point cloud data.

[0025] As an example of this application, the acquisition of two frames of time-matched detection data includes:

[0026] The timestamp of the detection data from the millimeter-wave radar sensor is obtained to obtain a first timestamp, and the timestamp of the detection data from the lidar sensor is obtained to obtain a second timestamp;

[0027] If the time difference between the first timestamp and the second timestamp is less than the duration threshold, then the detection data of the millimeter-wave radar sensor and the detection data of the lidar sensor are determined as the two frames of time-matched detection data.

[0028] As an example of this application, the method further includes:

[0029] If the time difference between the first timestamp and the second timestamp is greater than or equal to the duration threshold, then the timestamp of the next frame of detection data from the millimeter-wave radar sensor is obtained to obtain the third timestamp.

[0030] Compare the second timestamp with the third timestamp;

[0031] If the time difference between the second timestamp and the third timestamp is less than the duration threshold, then the detection data corresponding to the third timestamp and the detection data corresponding to the second timestamp are determined as the two frames of time-matched detection data.

[0032] As an example of this application, the method further includes:

[0033] If the time difference between the second timestamp and the third timestamp is greater than or equal to the duration threshold, then the next frame of detection data from the lidar sensor is obtained.

[0034] The detection data corresponding to the first timestamp and the next frame detection data of the lidar sensor are determined as the two frames of detection data that are time-matched.

[0035] Secondly, an apparatus for detecting an object is provided, the apparatus comprising:

[0036] The acquisition module is used to acquire the current detection data of the lidar sensor to obtain target detection data;

[0037] The determination module is used to determine the first object information of the object currently detected by the lidar sensor based on the target detection data and through the first target detection model;

[0038] The first target detection model undergoes at least one update training. The update training refers to updating the parameters of the target detection model with detection function based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data in the first detection data that fails to match the object in the millimeter-wave radar data. The first detection data is detected by the lidar sensor and is time-matched with the millimeter-wave radar data.

[0039] As an example of this application, the acquisition module is also used for:

[0040] During the operation of the millimeter-wave radar sensor and the lidar sensor, two frames of detection data with time matching are acquired. The two frames of detection data include the first detection data and the second detection data, and the second detection data is detected by the millimeter-wave radar sensor.

[0041] The second target detection model processes the second detection data to obtain the millimeter-wave radar data. The second target detection model is used to determine the object detected by the millimeter-wave radar sensor based on the second detection data.

[0042] Based on the first detection data and the millimeter-wave radar data, the point cloud data labeled with the second object information is determined.

[0043] As an example of this application, the acquisition module is used for:

[0044] The target detection model processes the first detection data and outputs information about at least one first object in the point cloud coordinate system.

[0045] The millimeter-wave radar data is mapped to the point cloud coordinate system to obtain information about at least one second object in the point cloud coordinate system.

[0046] Object matching is performed based on the information of the at least one first object in the point cloud coordinate system and the information of the at least one second object in the point cloud coordinate system.

[0047] Based on the object matching results, the point cloud data labeled with the second object information is determined.

[0048] As an example of this application, the acquisition module is used for:

[0049] Based on the object matching result, the object's 3D bounding box is determined in the point cloud coordinate system. The object's 3D bounding box refers to the 3D bounding box of the second object that does not match the at least one first object.

[0050] If the number of point clouds included in the three-dimensional bounding box of the object in the first detection data is greater than the number of points, then the point cloud data included in the three-dimensional bounding box of the object is labeled based on the data corresponding to the three-dimensional bounding box of the object in the millimeter-wave radar data, so as to obtain the point cloud data labeled with the second object information.

[0051] As an example of this application, the acquisition module is used for:

[0052] The timestamp of the detection data from the millimeter-wave radar sensor is obtained to obtain a first timestamp, and the timestamp of the detection data from the lidar sensor is obtained to obtain a second timestamp;

[0053] If the time difference between the first timestamp and the second timestamp is less than the duration threshold, then the detection data of the millimeter-wave radar sensor and the detection data of the lidar sensor are determined as the two frames of time-matched detection data.

[0054] As an example of this application, the acquisition module is used for:

[0055] If the time difference between the first timestamp and the second timestamp is greater than or equal to the duration threshold, then the timestamp of the next frame of detection data from the millimeter-wave radar sensor is obtained to obtain the third timestamp.

[0056] Compare the second timestamp with the third timestamp;

[0057] If the time difference between the second timestamp and the third timestamp is less than the duration threshold, then the detection data corresponding to the third timestamp and the detection data corresponding to the second timestamp are determined as the two frames of time-matched detection data.

[0058] As an example of this application, the acquisition module is used for:

[0059] If the time difference between the second timestamp and the third timestamp is greater than or equal to the duration threshold, then the next frame of detection data from the lidar sensor is obtained.

[0060] The detection data corresponding to the first timestamp and the next frame detection data of the lidar sensor are determined as the two frames of detection data that are time-matched.

[0061] Thirdly, a computer-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, implement the method described in any one of the first aspects above.

[0062] Fourthly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any one of the first aspects above.

[0063] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0064] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0065] The beneficial effects of the technical solutions provided in this application are:

[0066] By acquiring target detection data from a lidar sensor, and then using this data, a first target detection model is used to determine the first object information of the object currently detected by the lidar sensor. The first target detection model undergoes at least one update training, which involves updating the parameters of the target detection model with detection capabilities based on point cloud data labeled with second object information. This labeled point cloud data is obtained by labeling edge point cloud data from millimeter-wave radar data. The edge point cloud data includes point cloud data from the first detection data that failed to match the object in the millimeter-wave radar data. The first detection data is obtained by the lidar sensor and is time-matched with the millimeter-wave radar data. Thus, by using the second object information from the millimeter-wave radar data, which describes distant objects, to label the edge point cloud data in the first detection data, and updating the target detection model based on this labeled point cloud data, the resulting first target detection model can accurately detect distant objects, thereby improving the effectiveness of object detection. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating a method for detecting an object according to an exemplary embodiment;

[0069] Figure 2 This is a flowchart illustrating a method for acquiring point cloud data labeled with information about a second object, according to an exemplary embodiment.

[0070] Figure 3 This is a schematic diagram of the structure of a device for detecting an object according to an exemplary embodiment;

[0071] Figure 4 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0073] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0074] Before providing a detailed description of the detection method for the objects provided in the embodiments of this application, a brief introduction to the technical terms involved in the embodiments of this application will be given first.

[0075] Point cloud data: The raw data output by a lidar sensor is a continuous three-dimensional data stream composed of disordered three-dimensional points.

[0076] Target mapping matrix: Used to unify the detection data of millimeter-wave radar sensors and lidar sensors into the same coordinate system.

[0077] The execution subject involved in the embodiments of this application will be introduced next.

[0078] By way of example and not limitation, the method for detecting an object provided in this application embodiment can be executed by an electronic device configured with or connected to a lidar sensor and a millimeter-wave radar sensor. Exemplarily, the lidar sensor can be, but is not limited to, any one of 8-line lidar, 16-line lidar, 24-line lidar, 32-line lidar, 64-line lidar, and 128-line lidar. The millimeter-wave radar sensor can be, but is not limited to, any one of 77GHz millimeter-wave radar and 24GHz millimeter-wave radar.

[0079] In practice, lidar sensors and millimeter-wave radar sensors can be installed according to actual needs. For example, they can be fixed to roadside markers (horizontal or vertical poles) to detect the objects to be detected using lidar sensors and millimeter-wave radar sensors respectively. As an example, the objects to be detected can include, but are not limited to, vehicles, pedestrians, non-motorized vehicles, and trees.

[0080] As an example of this application, before the lidar sensor and the millimeter-wave radar sensor are operating normally, their sampling frequencies can be pre-adjusted to ensure that their detection data correspond. That is, the lidar sensor and the millimeter-wave radar sensor operate as synchronously as possible. Alternatively, this can be understood as ensuring that the object detected by the lidar sensor is the same as the object detected by the millimeter-wave radar sensor at the same or similar times. Exemplarily, the sampling frequencies of the lidar sensor and the millimeter-wave radar sensor are adjusted to be the same, or their sampling frequencies are adjusted to a multiple relationship, etc. This application does not limit the scope of this embodiment.

[0081] In some embodiments, the electronic device may include, but is not limited to, wearable devices, terminal devices, in-vehicle systems, camera devices, and roadside base stations. For example, wearable devices may include, but are not limited to, smartwatches, smart bracelets, and smart earmuffs. Additionally, terminal devices may include, but are not limited to, mobile phones, tablets, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs).

[0082] Roadside base stations are crucial infrastructure for intelligent transportation vehicle-road cooperation, serving as service stations integrating sensing, computing, and communication capabilities. In one embodiment, a roadside base station can also be referred to as a smart base station or a roadside fusion sensing system.

[0083] After introducing the technical terms and execution entities involved in the embodiments of this application, the method for detecting objects provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0084] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for detecting an object according to an exemplary embodiment. This method can be applied to the aforementioned electronic device. As an example and not a limitation, the method for detecting an object may include the following steps:

[0085] Step 101: Obtain the current detection data of the lidar sensor to obtain target detection data.

[0086] In one embodiment, the lidar sensor performs detection operations according to a first sampling frequency. This first sampling frequency can be set according to actual needs, and this application embodiment does not limit it. For example, in monitoring scenarios with high object mobility, such as stations and roads, the first sampling frequency can be set to a higher value. Conversely, in monitoring scenarios with low object mobility, the first sampling frequency can be set to a lower value.

[0087] In addition, the millimeter-wave radar sensor performs detection operations according to a second sampling frequency. This second sampling frequency can be set according to actual needs. For example, the second sampling frequency can be the same as the first sampling frequency, or it can be a multiple of the first sampling frequency.

[0088] Step 102: Based on the target detection data, determine the first object information of the object currently detected by the lidar sensor through the first target detection model.

[0089] In practice, step 102 may include: calling the first target detection model, inputting target detection data into the first target detection model, and outputting the first object information of the object currently detected by the lidar sensor.

[0090] In one embodiment, the first object information may include, but is not limited to, at least one of the following: object location, object category, object color, object size, and object heading angle detected by the lidar sensor.

[0091] The first target detection model undergoes at least one update training. The update training refers to updating the parameters of the target detection model with detection function based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data in the first detection data that failed to match the object in the millimeter-wave radar data. The first detection data is detected by a lidar sensor and is time-matched with the millimeter-wave radar data.

[0092] In one embodiment, the target detection model is obtained by training a network model based on a large number of LiDAR data samples. Each LiDAR data sample can be obtained by pre-annotating the detection data of the LiDAR sensor. For example, the user can determine at least one of the following information: object size, object position, object type, object color, etc., of the object corresponding to the detection data of the LiDAR sensor. Then, the detection data of the LiDAR sensor is annotated to obtain the LiDAR data sample.

[0093] As an example, the network model to be trained can be a deep learning model, etc.

[0094] Subsequently, a large number of LiDAR data samples can be input into the network model to be trained. When the training meets the first training termination condition, the trained network model is determined as the target detection model. The first training termination condition can be set according to actual needs.

[0095] Since the target detection model may not be able to accurately determine the distant object when processing the detection data at the far end of the detection range of the lidar sensor, in this embodiment of the application, the target detection model is updated and trained at least once to obtain a first target detection model, so that the first target detection model can determine the distant object at the far end of the detection range of the lidar sensor as accurately as possible.

[0096] For this purpose, millimeter-wave radar data can be acquired. Millimeter-wave radar data includes object information describing objects detected by the millimeter-wave radar sensor. This object information may include, but is not limited to, at least one of the following: object location, object size, object category, object color, and object heading angle. Thus, the edge point cloud data in the first detection data is annotated with the second object information using the second object information from the millimeter-wave radar data, resulting in point cloud data annotated with the second object information. Then, the target detection model is updated and trained based on the point cloud data annotated with the second object information, enabling the resulting first target detection model to accurately detect distant objects and improving the detection range of the lidar sensor.

[0097] As an example of this application, millimeter-wave radar data is obtained by processing second detection data using a second target detection model, the second detection data being detected by a millimeter-wave radar sensor. The second target detection model is used to determine the object detected by the millimeter-wave radar sensor based on the second detection data.

[0098] The second target detection model can be obtained by training a large number of millimeter-wave radar data samples on the data processing model to be trained. Each millimeter-wave radar data sample can be the detection data of a labeled millimeter-wave radar sensor. For example, the user can determine the object information corresponding to the detection data of the millimeter-wave radar sensor and label the detection data of the millimeter-wave radar sensor.

[0099] Subsequently, a large number of millimeter-wave radar data samples are input into the data processing model to be trained. When the second training termination condition is met, the trained data processing model is determined as the second target detection model. The second training termination condition can be set according to actual needs.

[0100] As an example, the data processing model to be trained can be a DBF (Digital Beam-Forming) model.

[0101] It should be noted that after determining the first object information of the object currently detected by the lidar sensor based on the target detection data using the first target detection model, the current first target detection model is defined as a target detection model with detection capabilities. That is, the current first target detection model is subsequently updated and trained based on the target detection data and the detection data from the millimeter-wave radar sensor that matches the target detection data. In other words, after each time an object is detected based on the first target detection model, the first target detection model is continuously updated and trained to make its detection accuracy increasingly precise.

[0102] In this embodiment, target detection data from a lidar sensor is acquired. Then, based on this data, a first object information of the object currently detected by the lidar sensor is determined using a first target detection model. The first target detection model undergoes at least one update training. This update training refers to updating the parameters of the target detection model with detection capabilities based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data from the first detection data that failed to match the object in the millimeter-wave radar data. The first detection data is obtained by the lidar sensor and is time-matched with the millimeter-wave radar data. Thus, by using the second object information from the millimeter-wave radar data, which describes distant objects, to label the edge point cloud data in the first detection data, and updating the target detection model based on this labeled point cloud data, the resulting first target detection model can accurately detect distant objects, thereby improving the effectiveness of object detection.

[0103] The following describes the method for obtaining point cloud data labeled with second object information involved in the embodiments of this application. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a method for acquiring point cloud data labeled with second object information according to an exemplary embodiment. This method can be executed by the aforementioned electronic device. As an example and not a limitation, the method may include the following steps:

[0104] Step 201: During the operation of the millimeter-wave radar sensor and the lidar sensor, acquire two frames of detection data with time matching. The two frames of detection data include the first detection data and the second detection data. The second detection data is obtained by the millimeter-wave radar sensor.

[0105] As mentioned earlier, since the sampling frequencies of millimeter-wave radar sensors and lidar sensors are adjusted, in order to ensure the effectiveness of subsequent updates and training of the target detection model with detection capabilities, two frames of time-matched detection data are acquired during the data labeling process to obtain the first detection data and the second detection data.

[0106] In one embodiment, the specific implementation of acquiring two time-matched detection data frames may include: acquiring the timestamp of the detection data from the millimeter-wave radar sensor to obtain a first timestamp, and acquiring the timestamp of the detection data from the lidar sensor to obtain a second timestamp. If the time difference between the first timestamp and the second timestamp is less than a duration threshold, then the detection data from the millimeter-wave radar sensor and the detection data from the lidar sensor are determined as two time-matched detection data frames.

[0107] The duration threshold can be set by the user according to actual needs, or it can be set by default by the electronic device. This application embodiment does not limit this.

[0108] If the time difference between the first timestamp and the second timestamp is less than the duration threshold, it can be determined that the detection data corresponding to the first timestamp and the detection data corresponding to the second timestamp were collected at the same time or close to the same time. In this case, it can be determined that the detection data corresponding to the first timestamp and the detection data corresponding to the second timestamp are two frames of detection data with time matching. Therefore, the detection data corresponding to the first timestamp is determined as the second detection data, and the detection data corresponding to the second timestamp is determined as the first detection data.

[0109] In one embodiment, if the time difference between the first and second timestamps is greater than or equal to a duration threshold, the timestamp of the next frame of detection data from the millimeter-wave radar sensor is obtained to obtain a third timestamp. The second and third timestamps are then compared. If the time difference between the second and third timestamps is less than the duration threshold, the detection data corresponding to the third timestamp and the detection data corresponding to the second timestamp are determined as two frames of detection data with time matching.

[0110] If the time difference between the first and second timestamps is greater than or equal to the duration threshold, it can be determined that the detection data corresponding to the first and second timestamps were not collected at the same time, or at close to each other. In this case, the timestamp of the next frame of detection data from the millimeter-wave radar sensor can be obtained to obtain the third timestamp. If the time difference between the second and third timestamps is less than the duration threshold, it indicates that the detection data corresponding to the second and third timestamps were collected at the same or close to each other. Therefore, the detection data corresponding to the second timestamp can be used to determine the first detection data, and the detection data corresponding to the third timestamp can be used to determine the second detection data.

[0111] In one embodiment, if the time difference between the second and third timestamps is greater than or equal to a duration threshold, the next frame of detection data from the lidar sensor is acquired. The detection data corresponding to the first timestamp and the next frame of detection data from the lidar sensor are determined as two time-matched frames of detection data.

[0112] If the time difference between the second and third timestamps is greater than or equal to the duration threshold, it indicates that the detection data corresponding to the second and third timestamps were not collected at the same time, or at close to each other. In this case, since the millimeter-wave radar sensor and the lidar sensor operate synchronously, the next frame of detection data from the lidar sensor should be collected at the same or close to the detection data corresponding to the first timestamp. Therefore, the electronic device can directly identify the next frame of detection data from the lidar sensor as the first detection data and the detection data corresponding to the first timestamp as the second detection data. That is, during the operation of the millimeter-wave radar sensor and the lidar sensor, the two time-matched frames of detection data are usually at most one frame apart.

[0113] It should be noted that the above description is based on the example of comparing the third timestamp of the next frame of detection data from the millimeter-wave radar sensor when the timestamp between the first and second timestamps is greater than or equal to the duration threshold. In another embodiment, when the timestamp between the first and second timestamps is greater than or equal to the duration threshold, the timestamp of the next frame of detection data from the lidar sensor can also be obtained first to obtain a fourth timestamp. Then, the fourth timestamp is compared with the first timestamp. If the difference between the fourth timestamp and the first timestamp is greater than or equal to the duration threshold, the next frame of detection data from the millimeter-wave radar sensor is obtained, and this next frame of detection data is determined as the second detection data, while the detection data corresponding to the second timestamp is determined as the first detection data. That is, when the timestamp between the first and second timestamps is greater than or equal to the duration threshold, this embodiment does not limit which radar sensor's detection data timestamp is obtained first for comparison.

[0114] Step 202: Process the second detection data using the second target detection model to obtain millimeter-wave radar data. The second target detection model is used to determine the objects detected by the millimeter-wave radar sensor based on the second detection data.

[0115] As mentioned earlier, millimeter-wave radar data includes object information of objects detected by millimeter-wave radar sensors. It is easy to understand that millimeter-wave radar data includes second object information.

[0116] Step 203: Based on the first detection data and millimeter-wave radar data, determine the point cloud data labeled with the second object information.

[0117] As an example of this application, the specific implementation of determining point cloud data labeled with second object information based on the first detection data and millimeter-wave radar data may include the following 2031 to 2034:

[0118] 2031: Process the first detection data using the target detection model and output information about at least one first object in the point cloud coordinate system.

[0119] In practice, the specific implementation of 2031 may include: invoking a target detection model, inputting the first detection data into the target detection model for processing, and having the target detection model output information about at least one first object in the point cloud coordinate system. At least one first object refers to an object detected by a lidar sensor.

[0120] 2032: Map millimeter-wave radar data to a point cloud coordinate system to obtain information about at least one second object in the point cloud coordinate system.

[0121] In one embodiment, millimeter-wave radar data can be mapped to a point cloud coordinate system using a target mapping matrix to obtain information about at least one second object in the point cloud coordinate system. The target mapping matrix can be obtained through a calibration method. For example, a coordinate transformation matrix containing multiple degrees of freedom, such as translation and rotation, can be obtained through a calibration method, and this coordinate transformation matrix is ​​then determined as the target mapping matrix.

[0122] At least one second object refers to an object detected by a millimeter-wave radar sensor. It is easy to understand that at least one second object may include some or all of the first object, that is, some objects in at least one second object and at least one first object are the same object.

[0123] 2033: Perform object matching based on information of at least one first object in the point cloud coordinate system and information of at least one second object in the point cloud coordinate system.

[0124] As an example of this application, IOU (Intersection over Union) calculation can be performed based on the information of at least one first object in the point cloud coordinate system and the information of at least one second object in the point cloud coordinate system to determine the IOU value between each first object in at least one first object and each second object in at least one second object, thereby obtaining the object matching result.

[0125] 2034: Based on the object matching results, determine the point cloud data labeled with the second object information.

[0126] As an example of this application, the specific implementation of determining point cloud data labeled with second object information based on the object matching result may include: determining the object's three-dimensional bounding box in the point cloud coordinate system based on the object matching result, wherein the object's three-dimensional bounding box refers to the three-dimensional bounding box of at least one second object that does not match at least one first object. If the number of point clouds included in the object's three-dimensional bounding box in the first detection data is greater than a point count threshold, then the point cloud data included in the object's three-dimensional bounding box is labeled based on the data corresponding to the object's three-dimensional bounding box in the millimeter-wave radar data to obtain point cloud data labeled with the second object information.

[0127] The point threshold can be set by the user according to actual needs, or it can be set by default by the electronic device. This application embodiment does not limit this.

[0128] For any one of the at least one first object, if the Interchange of Union (IOU) value between that first object and some of the at least one second object is greater than or equal to a matching threshold, then that first object is considered a match for that second object. Otherwise, if the IOU value between that first object and that second object is less than the matching threshold, then that first object is considered a mismatch for that second object. This process can identify at least one second object that does not match at least one first object.

[0129] The matching threshold can be set by the user according to actual needs, or it can be set by default by the electronic device. This application embodiment does not limit this.

[0130] Next, the 3D bounding box corresponding to the unmatched second object is determined in the point cloud coordinate system. Because millimeter-wave radar sensors have a wide detection range, the millimeter-wave radar data includes a significant amount of information about the second object corresponding to the 3D bounding box. However, because lidar sensors have a limited detection range, the first detection data may only include a small amount of point cloud data corresponding to the 3D bounding box. If the number of points in the determined 3D bounding box is greater than a point count threshold, it indicates that some features of the second object corresponding to the 3D bounding box have been detected, but the detected point cloud data is sparse. In this case, the point cloud data included within the 3D bounding box can be labeled using the data from the millimeter-wave radar data, resulting in point cloud data labeled with information about the second object.

[0131] In one embodiment, if the number of point clouds included in the determined object's 3D bounding box is less than or equal to a point count threshold, it indicates that the object's 3D bounding box contains only a very small amount of point cloud data, such as only a single-digit number of point cloud data. In this case, the second object corresponding to the object's 3D bounding box can be ignored. In this situation, the point cloud data within the object's 3D bounding box does not need to be labeled.

[0132] It should be noted that the point cloud data labeled with the second object information used to update and train the object detection model can be one or more sets. In the case of multiple sets, each set of point cloud data labeled with the second object information can be obtained through the above method.

[0133] It should also be noted that after acquiring a set of point cloud data labeled with the second object, the target detection model with detection function can be updated and trained immediately, or multiple sets of point cloud data labeled with the second object can be acquired before updating and training the target detection model with detection function. The embodiments of this application do not limit the timing of updating and training.

[0134] As an example of this application, the first object detection model can be further de-emphasized and retrained if certain preconditions are met. These preconditions can be set according to actual needs; for example, the preconditions could be that the detection performance of the first object detection model reaches a certain pre-defined standard, or that the number of times the first object detection model has been retrained has reached a pre-defined number.

[0135] In this embodiment, during the operation of the lidar sensor and the millimeter-wave radar sensor, two frames of time-matched detection data are acquired to obtain first detection data and second detection data. Based on the first and second detection data, mismatched objects detected by the lidar sensor and the millimeter-wave radar sensor are identified. Then, based on the second object information of the mismatched objects in the millimeter-wave radar data, the edge point cloud data in the first detection data is labeled to obtain point cloud data labeled with the second object information. The target detection model is updated and trained based on the point cloud data labeled with the second object information to obtain a first target detection model. Subsequently, target detection data from the lidar sensor is acquired, and based on the target detection data, the first object information of the objects currently detected by the lidar sensor is determined through the first target detection model. The first target detection model can accurately detect objects at a distance, thereby improving the effectiveness of object detection.

[0136] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0137] Figure 3 This is a schematic diagram illustrating the structure of a device for detecting an object according to an exemplary embodiment. The device may be part or all of an electronic device, comprised of software, hardware, or a combination of both. The device for detecting the object may include:

[0138] The acquisition module 310 is used to acquire the current detection data of the lidar sensor to obtain target detection data;

[0139] The determination module 320 is used to determine the first object information of the object currently detected by the lidar sensor based on the target detection data and through the first target detection model;

[0140] The first target detection model undergoes at least one update training. The update training refers to updating the parameters of the target detection model with detection function based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data in the first detection data that fails to match the object in the millimeter-wave radar data. The first detection data is detected by the lidar sensor and is time-matched with the millimeter-wave radar data.

[0141] As an example of this application, the acquisition module 310 is further configured to:

[0142] During the operation of the millimeter-wave radar sensor and the lidar sensor, two frames of detection data with time matching are acquired. The two frames of detection data include the first detection data and the second detection data, and the second detection data is detected by the millimeter-wave radar sensor.

[0143] The second target detection model processes the second detection data to obtain the millimeter-wave radar data. The second target detection model is used to determine the object detected by the millimeter-wave radar sensor based on the second detection data.

[0144] Based on the first detection data and the millimeter-wave radar data, the point cloud data labeled with the second object information is determined.

[0145] As an example of this application, the acquisition module 310 is used for:

[0146] The target detection model processes the first detection data and outputs information about at least one first object in the point cloud coordinate system.

[0147] The millimeter-wave radar data is mapped to the point cloud coordinate system to obtain information about at least one second object in the point cloud coordinate system.

[0148] Object matching is performed based on the information of the at least one first object in the point cloud coordinate system and the information of the at least one second object in the point cloud coordinate system.

[0149] Based on the object matching results, the point cloud data labeled with the second object information is determined.

[0150] As an example of this application, the acquisition module 310 is used for:

[0151] Based on the object matching result, the object's 3D bounding box is determined in the point cloud coordinate system. The object's 3D bounding box refers to the 3D bounding box of the second object that does not match the at least one first object.

[0152] If the number of point clouds included in the three-dimensional bounding box of the object in the first detection data is greater than the number of points, then the point cloud data included in the three-dimensional bounding box of the object is labeled based on the data corresponding to the three-dimensional bounding box of the object in the millimeter-wave radar data, so as to obtain the point cloud data labeled with the second object information.

[0153] As an example of this application, the acquisition module 310 is used for:

[0154] The timestamp of the detection data from the millimeter-wave radar sensor is obtained to obtain a first timestamp, and the timestamp of the detection data from the lidar sensor is obtained to obtain a second timestamp;

[0155] If the time difference between the first timestamp and the second timestamp is less than the duration threshold, then the detection data of the millimeter-wave radar sensor and the detection data of the lidar sensor are determined as the two frames of time-matched detection data.

[0156] As an example of this application, the acquisition module 310 is used for:

[0157] If the time difference between the first timestamp and the second timestamp is greater than or equal to the duration threshold, then the timestamp of the next frame of detection data from the millimeter-wave radar sensor is obtained to obtain the third timestamp.

[0158] Compare the second timestamp with the third timestamp;

[0159] If the time difference between the second timestamp and the third timestamp is less than the duration threshold, then the detection data corresponding to the third timestamp and the detection data corresponding to the second timestamp are determined as the two frames of time-matched detection data.

[0160] As an example of this application, the acquisition module 310 is used for:

[0161] If the time difference between the second timestamp and the third timestamp is greater than or equal to the duration threshold, then the next frame of detection data from the lidar sensor is obtained.

[0162] The detection data corresponding to the first timestamp and the next frame detection data of the lidar sensor are determined as the two frames of detection data that are time-matched.

[0163] In this embodiment, target detection data from a lidar sensor is acquired. Then, based on this data, a first object information of the object currently detected by the lidar sensor is determined using a first target detection model. The first target detection model undergoes at least one update training. This update training refers to updating the parameters of the target detection model with detection capabilities based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data from the first detection data that failed to match the object in the millimeter-wave radar data. The first detection data is obtained by the lidar sensor and is time-matched with the millimeter-wave radar data. Thus, by using the second object information from the millimeter-wave radar data, which describes distant objects, to label the edge point cloud data in the first detection data, and updating the target detection model based on this labeled point cloud data, the resulting first target detection model can accurately detect distant objects, thereby improving the effectiveness of object detection.

[0164] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown in the diagram), memory 41, and computer program 42 stored in said memory 41 and executable on said at least one processor 40, which, when executed, implements the steps in any of the above method embodiments.

[0165] The electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0166] The processor 40 may be a CPU (Central Processing Unit), or it may be other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0167] In some embodiments, the memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. In other embodiments, the memory 41 may be an external storage device of the electronic device 4, such as a plug-in hard disk, SMC (Smart Media Card), SD (Secure Digital) card, flash card, etc., equipped on the electronic device 4. Furthermore, the memory 41 may include both internal and external storage units of the electronic device 4. The memory 41 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0168] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0170] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of detecting an object, characterized by, The method includes: Obtain the current detection data from the lidar sensor to obtain target detection data; Based on the target detection data, the first object information of the object currently detected by the lidar sensor is determined by the first target detection model; The first target detection model undergoes at least one update training, which refers to updating the parameters of the target detection model with detection function based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data in the first detection data that fails to match the object in the millimeter-wave radar data. The first detection data is detected by the lidar sensor and is time-matched with the millimeter-wave radar data. The methods for obtaining the point cloud data labeled with the second object information include: During the operation of the millimeter-wave radar sensor and the lidar sensor, two frames of detection data with time matching are acquired. The two frames of detection data include the first detection data and the second detection data. The second detection data is detected by the millimeter-wave radar sensor. The sampling frequency of the millimeter-wave radar sensor is the same as or a multiple of the sampling frequency of the lidar sensor. The second target detection model processes the second detection data to obtain the millimeter-wave radar data. The second target detection model is used to determine the object detected by the millimeter-wave radar sensor based on the second detection data. Based on the first detection data and the millimeter-wave radar data, the point cloud data labeled with the second object information is determined.

2. The method as described in claim 1, characterized in that, The step of determining the point cloud data labeled with the second object information based on the first detection data and the millimeter-wave radar data includes: The target detection model processes the first detection data and outputs information about at least one first object in the point cloud coordinate system. The millimeter-wave radar data is mapped to the point cloud coordinate system to obtain information about at least one second object in the point cloud coordinate system. Object matching is performed based on the information of the at least one first object in the point cloud coordinate system and the information of the at least one second object in the point cloud coordinate system. Based on the object matching results, the point cloud data labeled with the second object information is determined.

3. The method as described in claim 2, characterized in that, The step of determining the point cloud data labeled with the second object information based on the object matching result includes: Based on the object matching result, the object's 3D bounding box is determined in the point cloud coordinate system. The object's 3D bounding box refers to the 3D bounding box of the second object that does not match the at least one first object. If the number of point clouds included in the three-dimensional bounding box of the object in the first detection data is greater than the point number threshold, then the point cloud data included in the three-dimensional bounding box of the object is labeled based on the data corresponding to the three-dimensional bounding box of the object in the millimeter-wave radar data, to obtain the point cloud data labeled with the second object information.

4. The method as described in claim 1, characterized in that, The acquisition of two frames of time-matched detection data includes: The timestamp of the detection data from the millimeter-wave radar sensor is obtained to obtain a first timestamp, and the timestamp of the detection data from the lidar sensor is obtained to obtain a second timestamp; If the time difference between the first timestamp and the second timestamp is less than the duration threshold, then the detection data of the millimeter-wave radar sensor and the detection data of the lidar sensor are determined as the two frames of time-matched detection data.

5. The method as described in claim 4, characterized in that, The method further includes: If the time difference between the first timestamp and the second timestamp is greater than or equal to the duration threshold, then the timestamp of the next frame of detection data from the millimeter-wave radar sensor is obtained to obtain the third timestamp. Compare the second timestamp with the third timestamp; If the time difference between the second timestamp and the third timestamp is less than the duration threshold, then the detection data corresponding to the third timestamp and the detection data corresponding to the second timestamp are determined as the two frames of time-matched detection data.

6. The method as described in claim 5, characterized in that, The method further includes: If the time difference between the second timestamp and the third timestamp is greater than or equal to the duration threshold, then the next frame of detection data from the lidar sensor is obtained. The detection data corresponding to the first timestamp and the next frame detection data of the lidar sensor are determined as the two frames of detection data that are time-matched.

7. A device for detecting an object, characterized in that, The device includes: The acquisition module is used to acquire the current detection data of the lidar sensor to obtain target detection data; The determination module is used to determine the first object information of the object currently detected by the lidar sensor based on the target detection data and through the first target detection model; The first target detection model undergoes at least one update training, which refers to updating the parameters of the target detection model with detection function based on point cloud data labeled with second object information. The point cloud data labeled with second object information is obtained by labeling edge point cloud data based on millimeter-wave radar data. The edge point cloud data includes point cloud data in the first detection data that fails to match the object in the millimeter-wave radar data. The first detection data is detected by the lidar sensor and is time-matched with the millimeter-wave radar data. The acquisition module is further configured to: acquire two frames of time-matched detection data during the operation of the millimeter-wave radar sensor and the lidar sensor, the two frames of detection data including the first detection data and the second detection data, the second detection data being detected by the millimeter-wave radar sensor, the sampling frequency of the millimeter-wave radar sensor being the same as or a multiple of the sampling frequency of the lidar sensor; process the second detection data using a second target detection model to obtain the millimeter-wave radar data, the second target detection model being used to determine the object detected by the millimeter-wave radar sensor based on the second detection data; and determine the point cloud data labeled with the second object information based on the first detection data and the millimeter-wave radar data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the method of any one of claims 1 to 6.

Images