Target detection method, device and unmanned vehicle

Abstract

The disclosure provides a target detection method, device and unmanned vehicle, and relates to the technical field of computer vision. The target detection method comprises the following steps: obtaining sensor data to be processed, wherein the sensor data to be processed comprises point cloud data; determining a detection model to be enabled according to whether the sensor data to be processed comprises image data corresponding to the point cloud data, wherein the detection model comprises a first detection model and a second detection model, the first detection model is obtained based on point cloud sample data, and the second detection model is obtained based on the point cloud sample data and image sample data; and processing the sensor data to be processed based on the detection model to be enabled to obtain a detection result of a target to be recognized. Through the above steps, the accuracy and detection efficiency of the target detection result can be improved, and the safety of automatic driving can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and particularly relates to a target detection method, device and unmanned vehicle. BACKGROUND

[0002] Target detection is an important task in automatic driving. For example, when a vehicle is driving on a road, it needs to detect obstacles according to data collected by sensors, and perform autonomous control and path planning of the vehicle according to the detection result. Because of the limitation of the computing power at the vehicle end, it is necessary to reasonably design the entire detection framework to achieve the highest accuracy rate under limited computing power.

[0003] In related technologies, after a detection model trained based on sample data is actually deployed to the vehicle end of automatic driving, the data type of input detection data may not be consistent with that of sample data due to the delay of bandwidth or problems of the sensor itself, which seriously affects the performance of the detection model, reduces the accuracy and efficiency of target detection, and further seriously affects the safety of unmanned driving. SUMMARY

[0004] One of the technical problems to be solved by the present disclosure is to provide a solution that can improve the accuracy and efficiency of target detection and improve the safety of unmanned driving.

[0005] According to a first aspect of the present disclosure, a target detection method is provided, comprising: acquiring to-be-processed sensor data, wherein the to-be-processed sensor data comprises point cloud data; determining a to-be-enabled detection model according to whether the to-be-processed sensor data further comprises image data corresponding to the point cloud data, wherein the detection model comprises a first detection model and a second detection model, the first detection model is trained based on point cloud sample data, and the second detection model is trained based on point cloud sample data and image sample data; and processing the to-be-processed sensor data based on the to-be-enabled detection model to obtain a detection result of a to-be-identified target.

[0006] In some embodiments, the determining the to-be-enabled detection model according to whether the to-be-processed sensor data further comprises image data corresponding to the point cloud data comprises: in a case where the to-be-processed sensor data does not comprise image data corresponding to the point cloud data, taking the first detection model as the to-be-enabled detection model; and in a case where the to-be-processed sensor data comprises image data corresponding to the point cloud data, taking the second detection model as the to-be-enabled detection model.

[0007] In some embodiments, in a case where the to-be-processed sensor data does not include image data corresponding to the point cloud data, the first detection model is taken as the to-be-activated detection model in a case where the to-be-processed sensor data does not include image data, or the time stamps of the image data and the point cloud data included in the to-be-processed sensor data are inconsistent.

[0008] In some embodiments, in a case where the to-be-processed sensor data includes image data corresponding to the point cloud data, the second detection model is taken as the to-be-activated detection model in a case where the time stamps of the image data and the point cloud data included in the to-be-processed sensor data are consistent.

[0009] In some embodiments, the to-be-activated model is the first detection model, and the processing of the to-be-processed sensor data based on the to-be-activated detection model to obtain the detection result of the to-be-identified target includes: performing feature encoding on the point cloud data to obtain a first feature map; and inputting the first feature map into the first detection model to obtain the detection result of the to-be-identified target.

[0010] In some embodiments, the feature encoding on the point cloud data to obtain a point cloud feature map includes: voxelization encoding on the point cloud data to obtain a voxel feature map; generating a bird's eye view feature map according to the voxel feature map; and inputting the bird's eye view feature map into a point cloud feature extraction network model to obtain the point cloud feature map.

[0011] In some embodiments, the to-be-activated model is the second detection model, and the processing of the to-be-processed sensor data based on the to-be-activated detection model to obtain the detection result of the to-be-identified target includes: performing feature encoding on the point cloud data to obtain a first feature map; performing feature encoding on the image data to obtain a second feature map; fusing the first feature map and the second feature map to obtain a fused feature map; and inputting the fused feature map into the second detection model to obtain the detection result of the to-be-identified target.

[0012] In some embodiments, the feature encoding on the image data to obtain a second feature map includes: performing semantic segmentation on the image data to obtain semantic information of each pixel point in the image data; determining semantic information of a point cloud point corresponding to the pixel point according to the semantic information of the pixel point in the image data and a coordinate system conversion relationship; and performing feature encoding on the semantic information of the point cloud point to obtain the second feature map.

[0013] In some embodiments, the feature encoding the semantic information of the point cloud point to obtain the second feature map comprises: voxelizing the semantic information of the point cloud point to obtain a voxel feature map; generating a bird's eye view feature map according to the voxel feature map; and downsampling the bird's eye view feature map to obtain the second feature map, wherein the second feature map is consistent in size with the first feature map.

[0014] According to a second aspect of the present disclosure, a target detection apparatus is provided, comprising: an acquisition module configured to acquire to-be-processed sensor data, wherein the to-be-processed sensor data comprises point cloud data; a determination module configured to determine a to-be-activated detection model according to whether the to-be-processed sensor data comprises image data corresponding to the point cloud data, wherein the detection model comprises a first detection model and a second detection model, the first detection model is trained based on point cloud sample data, and the second detection model is trained based on point cloud sample data and image sample data; and a detection module configured to process the to-be-processed sensor data based on the to-be-activated detection model to obtain a detection result of a to-be-identified target.

[0015] In some embodiments, the determination module is configured to: in a case where the to-be-processed sensor data does not comprise image data corresponding to point cloud data, take the first detection model as the to-be-activated detection model; and in a case where the to-be-processed sensor data comprises image data corresponding to point cloud data, take the second detection model as the to-be-activated detection model.

[0016] In some embodiments, the determination module is configured to: in a case where the to-be-processed sensor data does not comprise image data, or the image data comprised by the to-be-processed sensor data and the point cloud data are not consistent in time stamp, take the first detection model as the to-be-activated detection model.

[0017] According to a third aspect of the present disclosure, a target detection apparatus is also provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a target detection method as described above based on instructions stored in the memory.

[0018] According to a fourth aspect of the present disclosure, a computer readable storage medium is also provided, having computer program instructions stored thereon, the instructions being executed by a processor to implement a target detection method as described above.

[0019] According to a fifth aspect of the present disclosure, an unmanned vehicle is also provided, comprising a target detection apparatus as described above.

[0020] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] The present disclosure can be more clearly understood and appreciated from the following detailed description, taken in conjunction with the following drawings of which:

[0023] Figure 1 A flowchart of a target detection method according to some embodiments of the present disclosure.

[0024] Figure 2 A flowchart of determining a detection model to be enabled according to some embodiments of the present disclosure.

[0025] Figure 3 A flowchart of target detection based on a first detection model according to some embodiments of the present disclosure.

[0026] Figure 4 A flowchart of target detection based on a second detection model according to some embodiments of the present disclosure.

[0027] Figure 5 A structural diagram of a target detection apparatus according to some embodiments of the present disclosure.

[0028] Figure 6 A structural diagram of a target detection apparatus according to some other embodiments of the present disclosure.

[0029] Figure 7 A structural diagram of a computer system according to some embodiments of the present disclosure.

[0030] Figure 8 A structural diagram of an unmanned vehicle according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0031] Various exemplary embodiments of the present disclosure will now be described in detail by referring to the drawings. It should be noted that the relative arrangement, numerical expressions, and numerical values of the components and steps set forth in these embodiments are not intended to limit the scope of the present disclosure unless otherwise specifically stated.

[0032] It should be understood, however, that the sizes of the components shown in the drawings are illustrative and not meant to be limiting.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the scope of the present disclosure and its applications or uses.

[0034] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art(s) can not be discussed in detail herein. However, where appropriate, the techniques, methods, and apparatus should be considered as being part of the disclosure.

[0035] In all of the compositions and methods shown and discussed herein, any specific values should be interpreted as merely exemplary, and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0036] It should be noted that like reference numerals and letters refer to like items throughout the attached drawings, and thus once an item is defined in one drawing, it is not necessary that it be further discussed in subsequent drawings.

[0037] In order to make the purposes, technical solutions, and advantages of the present disclosure clearer, the present disclosure is further described in detail below with reference to the specific embodiments and in conjunction with the accompanying drawings.

[0038] Figure 1 A flowchart of a target detection method according to some embodiments of the present disclosure. As shown in FIG. 1, the target detection method according to some embodiments of the present disclosure includes the following steps. Figure 1

[0039] Step S110: Obtain sensor data to be processed.

[0040] In some embodiments, the target detection method is performed by a target detection device. For example, in an autonomous driving scenario, the target detection device can be disposed in a vehicle-mounted electronic device or a server that controls the driving of a vehicle.

[0041] In some embodiments, the target detection device obtains the sensor data to be processed in a timely manner. For example, the target detection device pulls the sensor data to be processed from an external module in a timely manner.

[0042] In other embodiments, the target detection device obtains the sensor data to be processed in response to a request from an external module. For example, the target detection device receives a detection request sent by an external module and obtains the sensor data to be processed according to the detection request.

[0043] In some embodiments, point cloud data and image data are collected based on sensors such as vehicle-mounted radars and cameras, and target detection is performed based on the collected sensor data. In actual scenarios, due to bandwidth delays or problems with the sensors themselves, etc., it is easy to cause image and point cloud data to not arrive at the same time or for image data to be missing, etc., which in turn can cause the sensor data to be processed obtained by the target detection device to be in one of the following three situations: only point cloud data, only image data, and including both point cloud data and image data.

[0044] ​Step S120: determining the detection model to be activated according to whether the to-be-processed sensor data comprises image data corresponding to the point cloud data.

[0045] In some embodiments, the detection model comprises a first detection model and a second detection model. The first detection model is trained based on point cloud sample data, and the second detection model is trained based on point cloud sample data and image sample data.

[0046] In some embodiments, the to-be-processed sensor data comprises point cloud data. In a case where the to-be-processed sensor data does not comprise image data corresponding to the point cloud data, the first detection model is taken as the detection model to be activated; in a case where the to-be-processed sensor data further comprises image data corresponding to the point cloud data, the second detection model is taken as the detection model to be activated.

[0047] In some embodiments, in a case where the to-be-processed sensor data is image data, it is considered that the detection data is abnormal, and a detection data abnormality prompt is performed or a record of the detection data abnormality is made.

[0048] Step S130: processing the to-be-processed sensor data based on the detection model to be activated to obtain a detection result of a to-be-identified target.

[0049] In some embodiments, in a case where the detection model to be activated is the first detection model, the to-be-processed point cloud data is processed based on the first detection model to obtain the detection result of the to-be-identified target; in a case where the detection model to be activated is the second detection model, the to-be-processed point cloud data and image data are processed based on the second detection model to obtain the detection result of the to-be-identified target.

[0050] Exemplarily, in an automatic driving scenario, the to-be-identified target is an obstacle in a vehicle driving environment or a traffic light.

[0051] In the embodiments of the present disclosure, through the above steps, a detection model trained based on sample data of the same type can be selected for target detection according to the actually obtained to-be-processed sensor data, so as to improve the accuracy and detection efficiency of the target detection result, solve the problems of reducing the model detection accuracy, even failing to detect, and reducing the model detection efficiency due to the inconsistency between the to-be-processed sensor data and the sample data used for training the detection model, improve the accuracy and detection efficiency of the target detection result, and further help to improve the safety of unmanned driving.

[0052] Figure 2 A flowchart of determining the detection model to be activated according to some embodiments of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the flow of determining the detection model to be activated in the embodiments of the present disclosure comprises: Figure 2 ​

[0053] Step S121: Determine the type of the sensor data to be processed.

[0054] The sensor data to be processed includes at least one of point cloud data and image data.

[0055] In some embodiments, the type of the sensor data to be processed is determined according to different input channels of the sensor data to be processed. For example, when the sensor data to be processed is received from a first input channel, the sensor data to be processed is determined as point cloud data; when the sensor data to be processed is received from a second input channel, the sensor data to be processed is determined as image data; and when the sensor data to be processed is received from the first and second input channels, the sensor data to be processed is determined as point cloud data and image data.

[0056] In some other embodiments, the type of the sensor data to be processed is determined according to different type identifiers carried by the sensor data to be processed. For example, when the sensor data to be processed carries a first type identifier, the sensor data to be processed is determined as point cloud data; when the sensor data to be processed carries a second type identifier, the sensor data to be processed is determined as image data; and when the sensor data to be processed carries the first type identifier and the second type identifier, the sensor data to be processed is determined as point cloud data and image data.

[0057] Step S122: When the sensor data to be processed includes point cloud data and image data, determine whether the time stamps of the point cloud data and the image data are consistent.

[0058] In some embodiments, the sensor data to be processed carries a time stamp of the point cloud data and a time stamp of the image data. In these embodiments, the time stamp of the point cloud data is compared with the time stamp of the image data, and when the absolute value of the difference between the two is less than a preset threshold, it is determined that the time stamps of the two are consistent, and when the absolute value of the difference between the two is greater than or equal to the preset threshold, it is determined that the time stamps of the two are inconsistent.

[0059] In some other embodiments, the time when the target detection device receives the point cloud data is taken as the time stamp of the point cloud data, and the time when the target detection device receives the image data is taken as the time stamp of the image data. In these embodiments, the time when the point cloud data is received is compared with the time when the image data is received, and when the absolute value of the difference between the two is less than a preset threshold, it is determined that the time stamps of the two are consistent, and when the absolute value of the difference between the two is greater than or equal to the preset threshold, it is determined that the time stamps of the two are inconsistent.

[0060] When the time stamps of the point cloud data and the image data are inconsistent, step S123 is performed; and when the time stamps of the point cloud data and the image data are consistent, step S124 is performed.

[0061] Step S123: taking the first detection model as the detection model to be activated.

[0062] The first detection model is a detection model trained based on point cloud sample data.

[0063] Step S124: taking the second detection model as the detection model to be activated.

[0064] The second detection model is a detection model trained based on point cloud sample data and image sample data.

[0065] Step S125: in a case where the sensor data to be processed is point cloud data, taking the first detection model as the detection model to be activated.

[0066] In the embodiments of the present disclosure, through the above steps, the detection model that is more matched with the sensor data to be processed can be determined according to the type of the sensor data to be processed and the difference between the time stamps of the point cloud data and the image data, which helps to improve the accuracy and detection efficiency of subsequent target detection based on the detection model.

[0067] Figure 3 A flowchart of a process of target detection based on the first detection model according to some embodiments of the present disclosure is shown. In a case where the detection model to be activated is the first detection model, the process of target detection based on the first detection model is performed. Figure 3 As shown in the flowchart, the process of target detection based on the first detection model includes: Figure 3

[0068] Step S131: performing feature encoding on the point cloud data to obtain a first feature map.

[0069] In some embodiments, step S131 includes voxelizing the point cloud data to obtain a voxel feature map, generating a bird's eye view feature map according to the voxel feature map, and inputting the bird's eye view feature map into a point cloud feature extraction network model to obtain a point cloud feature map.

[0070] In some embodiments, the point cloud data is voxelized in the following manner: each point cloud point in the point cloud data is assigned to a voxel unit in a voxel grid, and the point cloud points in the voxel unit are encoded for features to obtain voxel features; then, a voxel feature map is determined according to the voxel features. For example, the point cloud data can be voxelized based on the manner proposed by the PointPillar model or the VoxelNet model.

[0071] In some embodiments, the voxel feature map is mapped to a bird's eye view perspective to obtain a bird's eye view feature map. The bird's eye view is a perspective drawing of the ground from a high point of view, drawn from a high point of view.​

[0072] In some embodiments, the point cloud feature extraction network model is a two-dimensional convolutional neural network. The bird's eye view feature map is input into the two-dimensional convolutional neural network to obtain a point cloud feature map.

[0073] In the embodiments of the present disclosure, through the above steps, the features of the point cloud data can be quickly and accurately extracted in the case that the sensor data to be processed is point cloud data, for subsequent target detection.

[0074] Step S132: inputting the first feature map into the first detection model to obtain a detection result of the target to be recognized.

[0075] The first detection model is a detection model trained based on point cloud sample data.

[0076] In the embodiments of the present disclosure, through the above steps, target detection can be quickly and accurately performed based on the detection model matched with the point cloud data in the case that the arrival times of various sensor data are inconsistent or image data is missing, which ensures the target detection effect in the case of missing image data and solves the problems of reduced detection efficiency and reduced detection accuracy caused by inconsistent arrival times of various sensor data or missing image data of the autonomous vehicle in the actual application process.

[0077] Figure 4 FIG. 2 shows a flowchart of a process of target detection based on the second detection model according to some embodiments of the present disclosure. In the case that the detection model to be enabled is the second detection model, the process shown in FIG. 2 is performed. As shown in FIG. 2, the process of target detection based on the second detection model includes: Figure 4 Figure 4

[0078] Step S131': performing feature encoding on the point cloud data to obtain a first feature map.

[0079] In some embodiments, step S131' includes: performing voxelization encoding on the point cloud data to obtain a voxel feature map; generating a bird's eye view feature map based on the voxel feature map; and inputting the bird's eye view feature map into a point cloud feature extraction network model to obtain a point cloud feature map.

[0080] In some embodiments, the point cloud data is voxelized encoded in the following manner: each point cloud point in the point cloud data is assigned to a voxel unit in a voxel grid, and the point cloud points in the voxel unit are encoded to obtain voxel features; then, the voxel feature map is determined based on the voxel features. For example, the point cloud data can be voxelized encoded based on the manner proposed by the PointPillar model or the VoxelNet model.

[0081] ​​In some embodiments, the voxel feature map is mapped to a bird's eye view perspective, thereby obtaining a bird's eye view feature map. The bird's eye view is a perspective drawing of the ground from a high point of view, which is drawn according to the perspective principle.

[0082] In some embodiments, the point cloud feature extraction network model is a two-dimensional convolutional neural network. The bird's eye view feature map is input into the two-dimensional convolutional neural network to obtain a point cloud feature map.

[0083] In the embodiments of the present disclosure, through the above steps, the features of the point cloud data can be quickly and accurately extracted in the case that the sensor data to be processed includes point cloud data, for subsequent target detection.

[0084] Step S132': feature encoding is performed on the image data to obtain a second feature map.

[0085] In some embodiments, step S132' includes: step a, semantic segmentation is performed on the image data to obtain semantic information of each pixel point in the image data; step b, semantic information of a point cloud point corresponding to the pixel point is determined according to the semantic information of each pixel point in the image data and a coordinate system conversion relationship; and step c, feature encoding is performed on the semantic information of the point cloud point to obtain the second feature map.

[0086] In some embodiments, in step a, a two-dimensional image segmentation network, for example, MaskRNN, is used to segment the image data to obtain the semantic information of each pixel point in the image data. Exemplarily, the semantic information of the pixel point is a score of a category to which the pixel point belongs.

[0087] In some embodiments, in step b, the point cloud data is projected into the image coordinate system according to the coordinate system conversion relationship between the camera coordinate system and the radar coordinate system to determine the point cloud point corresponding to each pixel point in the image, and then the semantic information of the point cloud point corresponding to the pixel point is determined according to the semantic information of each pixel point in the image data and the correspondence between the pixel point and the point cloud point. Through the above operations, data alignment between the image data and the point cloud data is realized, and data fusion is performed on the basis of the data alignment, which helps to improve the accuracy of subsequent target detection.

[0088] In some embodiments, in step c, the semantic information of the point cloud point is voxelized and encoded to obtain a voxel feature map; a bird's eye view feature map is generated according to the voxel feature map; and down-sampling is performed on the bird's eye view feature map to obtain a second feature map, wherein the second feature map has the same size as the first feature map.

[0089] In the embodiments of the present disclosure, by performing down-sampling operation on the bird's eye view feature map, the size of the down-sampled feature map is kept consistent with that of the first feature map, thereby realizing feature alignment and facilitating subsequent feature fusion.

[0090] Step S133': fusing the first feature map and the second feature map to obtain a fused feature map.

[0091] In some embodiments, the first feature map and the second feature map are spliced, and the spliced feature map is taken as the fused feature map.

[0092] Step S134': inputting the fused feature map into the second detection model to obtain a detection result of the target to be recognized.

[0093] The second detection model is a detection model trained based on point cloud sample data and image sample data.

[0094] Exemplarily, the fused feature map is input into different detection networks, including a detection network of a category to which the target belongs and a detection network of a position of the target, to obtain a three-dimensional target detection result including the category to which the target belongs and the position of the target.

[0095] In the embodiments of the present disclosure, through the above steps, target detection can be efficiently and accurately performed based on the second detection model when the sensor data to be processed includes point cloud data and image data corresponding to the point cloud data, so that the accuracy of target detection can be improved within the range allowed by the computing power of the vehicle. In the embodiments of the present disclosure, two detection models are supported, one is the first detection model based on point cloud data, and the other is the second detection model based on point cloud data and image data. When the image data in the sensor data to be processed is missing or delayed, the first detection model based on point cloud data is enabled, and when the sensor data to be processed includes point cloud data and corresponding image data, the second detection model based on point cloud and image data is enabled. In this way, the target detection effect in the case of missing image data can be guaranteed, and a high target detection accuracy can be maintained in the case of image data corresponding to the point cloud data.

[0096] Figure 5 FIG. 1 is a structural schematic diagram of a target detection apparatus according to some embodiments of the present disclosure. As shown in FIG. 1, the target detection apparatus according to the embodiments of the present disclosure includes an acquisition module 510, a determination module 520, and a detection module 530. Figure 5

[0097] The acquisition module 510 is configured to acquire sensor data to be processed.

[0098] In some embodiments, the application scenario is an automatic driving scenario, and the target detection apparatus can be arranged in a vehicle-mounted electronic device or a server for controlling vehicle driving.

[0099] ​In some embodiments, the obtaining module 510 obtains the to-be-processed sensor data in a timed manner. For example, the obtaining module 510 pulls the to-be-processed sensor data from the external module in a timed manner.

[0100] In some other embodiments, the obtaining module 510 obtains the to-be-processed sensor data in response to a request from the external module. For example, the obtaining module 510 receives a detection request sent by the external module, and obtains the to-be-processed sensor data according to the detection request.

[0101] In some embodiments, point cloud data and image data are collected based on sensors such as vehicle-mounted radars and cameras, and target detection is performed based on the collected sensor data. In actual scenarios, due to bandwidth delay or problems of the sensors themselves, etc., it is easy to cause the image and point cloud data to not arrive at the same time or the image data to be missing, etc., and thus the to-be-processed sensor data obtained by the target detection device can be in the following three situations: only point cloud data, only image data, and including point cloud data and image data.

[0102] The determining module 520 is configured to determine the to-be-enabled detection model according to whether the to-be-processed sensor data includes image data corresponding to the point cloud data.

[0103] In some embodiments, the detection model includes a first detection model and a second detection model. The first detection model is trained based on point cloud sample data, and the second detection model is trained based on point cloud sample data and image sample data.

[0104] In some embodiments, the to-be-processed sensor data includes point cloud data. In a case where the to-be-processed sensor data does not include image data corresponding to the point cloud data, the determining module 520 determines the first detection model as the to-be-enabled detection model; and in a case where the to-be-processed sensor data further includes image data corresponding to the point cloud data, the determining module 520 determines the second detection model as the to-be-enabled detection model.

[0105] In some embodiments, the determining module 520 determines whether the to-be-processed sensor data includes image data corresponding to the point cloud data according to the following manner: in a case where the to-be-processed sensor data does not include image data, or the image data included in the to-be-processed sensor data and the point cloud data have inconsistent time stamps, the determining module 520 determines that the to-be-processed sensor data does not include image data corresponding to the point cloud data; and in a case where the image data included in the to-be-processed sensor data and the point cloud data have consistent time stamps, the determining module 520 determines that the to-be-processed sensor data includes image data corresponding to the point cloud data.

[0106] In some embodiments, when the sensor data to be processed is image data, the determination module 520 is further configured to confirm that the detection data is abnormal and to provide a detection data abnormality prompt or record the detection data abnormality.

[0107] The detection module 530 is configured to process the sensor data to be processed based on the detection model to be enabled in order to obtain the detection result of the target to be identified.

[0108] In some embodiments, when the detection model to be enabled is a first detection model, the detection module 530 processes the point cloud data to be processed based on the first detection model to obtain the detection result of the target to be identified; when the detection model to be enabled is a second detection model, the detection module 530 processes the point cloud data and image data to be processed based on the second detection model to obtain the detection result of the target to be identified.

[0109] For example, in an autonomous driving scenario, the target to be identified is an obstacle in the vehicle's driving environment, or a traffic light, etc.

[0110] In this embodiment of the disclosure, the above device can select a detection model trained on the same type of sample data to perform target detection based on the actual acquired sensor data to be processed, thereby improving the accuracy and efficiency of target detection results. It solves the problems of reduced model detection accuracy or even failure to detect due to inconsistency between the sensor data to be processed and the sample data used to train the detection model, and reduces model detection efficiency. This improves the accuracy and efficiency of target detection results and helps to improve the safety of autonomous driving.

[0111] Figure 6 This is a schematic diagram of the structure of a target detection device according to other embodiments of the present disclosure.

[0112] like Figure 6 As shown, the target detection apparatus 600 includes a memory 610 and a processor 620 coupled to the memory 610. The memory 610 is used to store instructions for executing embodiments of the target detection method. The processor 620 is configured to execute the target detection method in any of the embodiments of this disclosure based on the instructions stored in the memory 610.

[0113] Figure 7 This is a schematic diagram of the structure of a computer system according to some embodiments of the present disclosure.

[0114] like Figure 7 As shown, the computer system 700 can be represented in the form of a general-purpose computing device. The computer system 700 includes a memory 710, a processor 720, and a bus 930 connecting different system components.

[0115] The memory 710 can include, for example, system memory, non-volatile memory, and / or the like. The system memory can store, for example, an operating system, application programs, a Boot Loader, and other programs. The system memory can include volatile memory, such as random access memory (RAM) and / or cache memory. The non-volatile memory can store, for example, instructions to perform at least one of the target detection methods in corresponding embodiments. The non-volatile memory includes, but is not limited to, magnetic storage, optical storage, flash memory, and / or the like.

[0116] The processor 720 can be implemented with a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or the like. Accordingly, each module, such as the construction module, the bias compensation module, can be implemented by a central processing unit (CPU) running instructions in the memory to perform corresponding steps, or by a dedicated circuit performing the corresponding steps.

[0117] The bus 730 can use any of a variety of bus structures. For example, the bus structure includes, but is not limited to, an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus.

[0118] The computer system 700 can be connected between the interfaces 740, 750, 760, and the memory 710 and the processor 720 through the bus 730. The input / output interface 740 can provide a connection interface for display, mouse, keyboard, and other input / output devices. The network interface 750 provides a connection interface for various networking devices. The storage interface 760 provides a connection interface for external storage devices such as floppy disks, U disks, SD cards, and the like.

[0119] Here, various aspects of the disclosure are described with reference to flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and / or block diagrams can be implemented by computer readable program instructions.

[0120] These computer readable program instructions can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable device to produce a machine, so that the instructions executed by the processor produce the device that implements the functions specified in one or more blocks of the flowcharts and / or block diagrams.

[0121] These computer readable program instructions can also be stored in a computer readable storage medium that causes a computer to work in a specific manner, thereby producing a manufactured product including instructions to implement the functions specified in one or more blocks of the flowcharts and / or block diagrams.

[0122] Figure 8 FIG. 1 is a structural schematic diagram of an unmanned vehicle according to some embodiments of the present disclosure. As shown in FIG. 1, the unmanned vehicle 800 includes a target detection apparatus 810. Figure 8

[0123] The unmanned vehicle 800 further includes various sensors, such as one or more of a laser radar sensor, a millimeter wave sensor, a camera, and the like. The unmanned vehicle 800 collects sensor data required for target detection through the sensors on board.

[0124] The target detection apparatus 810 is configured to obtain to-be-processed sensor data, determine a to-be-activated detection model according to whether the to-be-processed sensor data further includes image data corresponding to point cloud data, and process the to-be-processed sensor data based on the to-be-activated detection model to obtain a detection result of a to-be-identified target.

[0125] Exemplarily, the to-be-identified target is an obstacle in a vehicle driving environment or a traffic light, and the like.

[0126] The detection model includes a first detection model and a second detection model, the first detection model is trained based on point cloud sample data, and the second detection model is trained based on point cloud sample data and image sample data.

[0127] In some embodiments, the to-be-processed sensor data includes point cloud data. In a case where the to-be-processed sensor data does not include image data corresponding to the point cloud data, the first detection model is taken as the to-be-activated detection model; and in a case where the to-be-processed sensor data further includes image data corresponding to the point cloud data, the second detection model is taken as the to-be-activated detection model.

[0128] In some embodiments, in a case where the to-be-processed sensor data is image data, it is considered that the detection data is abnormal, and a detection data abnormality prompt or a record of the detection data abnormality is performed.

[0129] In some embodiments, after obtaining the target detection result through the target detection apparatus 810, the operation of the unmanned vehicle can be further controlled, the driving path of the unmanned vehicle can be planned, and the like based on the target detection result.

[0130] ​In the embodiments of the present disclosure, by means of the above unmanned vehicle, two detection models can be supported, one is a first detection model based on point cloud data, and the other is a second detection model based on point cloud data and image data. When the image data in the to-be-processed sensor data is missing or delayed, the first detection model based on point cloud data is enabled, and when the to-be-processed sensor data includes point cloud data and corresponding image data, the second detection model based on point cloud and image data is enabled. In this way, the target detection effect in the case of missing image data can be guaranteed, and in the case of image data corresponding to the point cloud data, the target detection accuracy can be maintained, thereby improving the safety of autonomous driving.

[0131] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both software and hardware aspects.

[0132] By means of the target detection method, device and unmanned vehicle in the above embodiments, the accuracy and detection efficiency of the target detection result can be improved, which helps to improve the safety of unmanned driving.

[0133] So far, the target detection method, device and unmanned vehicle according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

Claims

1. A target detection method, comprising: obtaining to-be-processed sensor data, wherein the to-be-processed sensor data comprises point cloud data; determining a to-be-activated detection model according to whether the to-be-processed sensor data further comprises image data corresponding to the point cloud data, comprising: in a case where the to-be-processed sensor data does not comprise image data corresponding to the point cloud data, taking a first detection model as the to-be-activated detection model; in a case where the to-be-processed sensor data comprises image data corresponding to the point cloud data, taking a second detection model as the to-be-activated detection model, wherein the detection model comprises the first detection model and the second detection model, the first detection model is trained based on point cloud sample data, and the second detection model is trained based on the point cloud sample data and image sample data; processing the to-be-processed sensor data based on the to-be-activated detection model to obtain a detection result of a to-be-recognized target.

2. The target detection method of claim 1, wherein in a case where the to-be-processed sensor data does not comprise image data corresponding to the point cloud data, taking the first detection model as the to-be-activated detection model comprises: in a case where the to-be-processed sensor data does not comprise image data, or the time stamps of the image data and the point cloud data comprised by the to-be-processed sensor data are inconsistent, taking the first detection model as the to-be-activated detection model.

3. The object detection method of claim 1, wherein, in a case where the to-be-processed sensor data comprises image data corresponding to the point cloud data, taking the second detection model as the to-be-activated detection model comprises: in a case where the time stamps of the image data and the point cloud data comprised by the to-be-processed sensor data are consistent, taking the second detection model as the to-be-activated detection model.

4. The object detection method of claim 1, wherein, the to-be-activated model is the first detection model, and the processing of the to-be-processed sensor data based on the to-be-activated detection model to obtain the detection result of the to-be-recognized target comprises: feature encoding of the point cloud data to obtain a first feature map; inputting the first feature map into the first detection model to obtain the detection result of the to-be-recognized target.

5. The object detection method of claim 4, wherein, the feature encoding of the point cloud data to obtain the point cloud feature map comprises: voxelization encoding of the point cloud data to obtain a voxel feature map; generating a bird's eye view feature map according to the voxel feature map; inputting the bird's eye view feature map into a point cloud feature extraction network model to obtain the point cloud feature map.

6. The object detection method of claim 1, wherein, the to-be-activated model is the second detection model, and the processing of the to-be-processed sensor data based on the to-be-activated detection model to obtain the detection result of the to-be-recognized target comprises: feature encoding of the point cloud data to obtain a first feature map; feature encoding of the image data to obtain a second feature map; fusion of the first feature map and the second feature map to obtain a fused feature map; inputting the fused feature map into the second detection model to obtain the detection result of the to-be-recognized target.

7. The object detection method of claim 6, wherein, the feature encoding of the image data to obtain the second feature map comprises: semantic segmentation of the image data to obtain semantic information of each pixel point in the image data; According to semantic information of each pixel point in image data and a coordinate system conversion relationship, semantic information of a point cloud point corresponding to the pixel point is determined; The semantic information of the point cloud point is feature-encoded to obtain a second feature map.

8. The object detection method of claim 7, wherein, The feature encoding of the semantic information of the point cloud point to obtain the second feature map includes: The semantic information of the point cloud point is voxelized to obtain a voxel feature map; According to the voxel feature map, an aerial view feature map is generated; The aerial view feature map is down-sampled to obtain the second feature map, wherein the second feature map is consistent in size with the first feature map.

9. An object detection apparatus, comprising: an acquisition module configured to acquire to-be-processed sensor data, wherein the to-be-processed sensor data includes point cloud data; a determination module configured to determine a to-be-activated detection model according to whether the to-be-processed sensor data includes image data corresponding to the point cloud data, including: in a case where the to-be-processed sensor data does not include image data corresponding to the point cloud data, taking a first detection model as the to-be-activated detection model; in a case where the to-be-processed sensor data includes image data corresponding to the point cloud data, taking a second detection model as the to-be-activated detection model, wherein the detection model includes the first detection model and the second detection model, the first detection model is trained based on point cloud sample data, and the second detection model is trained based on point cloud sample data and image sample data; a detection module configured to process the to-be-processed sensor data based on the to-be-activated detection model to obtain a detection result of a to-be-identified object.

10. The object detection apparatus according to claim 9, wherein The determination module is configured to: in a case where the to-be-processed sensor data does not include image data or time stamps of image data and point cloud data included in the to-be-processed sensor data are inconsistent, taking the first detection model as the to-be-activated detection model.

11. An object detection apparatus, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute an object detection method according to any one of claims 1 to 8 based on instructions stored in the memory.

12. A computer-readable storage medium having computer program instructions stored thereon, the instructions being executed by a processor to implement an object detection method according to any one of claims 1 to 8.

13. An unmanned vehicle, comprising: an object detection apparatus according to any one of claims 9 to 11.

Images