Three-dimensional target detection method and device, electronic equipment, medium and vehicle

Abstract

The present disclosure provides a three-dimensional target detection method and device, electronic equipment, computer readable storage medium and computer program product, relates to the field of automatic driving, and particularly relates to the fields of perception and 3D point cloud target detection. The implementation scheme is as follows: first point cloud data and second point cloud data are acquired; first image data and second image data associated with the first point cloud data are acquired; the first point cloud data, the first image data and the second image data are fused to obtain first fused point cloud data; third image data and fourth image data associated with the second point cloud data are acquired; the second point cloud data, the third image data and the fourth image data are fused to obtain second fused point cloud data; and the first fused point cloud data and the second fused point cloud data are fused to determine a target detection result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of autonomous driving, in particular to the field of perception and 3D point cloud target detection, and more particularly to a three-dimensional target detection method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development and landing of autonomous driving technology, complex and variable application scenarios put forward higher requirements for the perception ability of autonomous driving vehicles. Target detection, as an important research direction in the field of computer vision, plays a crucial role in the perception field of autonomous driving. In the scenario of autonomous driving, providing only 2D information of objects will not be enough to fully support the perception task, and additional three-dimensional information such as position and orientation needs to be supplemented. Therefore, it is necessary to provide dense 3D point clouds through laser radar to accurately represent the position and shape of objects in 3D space.

[0003] In related technologies, the target objects to be detected are respectively identified and framed from the point cloud data and the corresponding image data, and then point cloud feature extraction and image feature extraction are performed on the target objects to fuse the extracted two types of features as the input of the target detection model. However, this approach loses a large amount of background information, which seriously reduces the accuracy of the target detection result. SUMMARY

[0004] The present disclosure provides a three-dimensional target detection method and device, electronic equipment, computer readable storage medium and computer program product.

[0005] According to an aspect of the present disclosure, a three-dimensional target detection method is provided, comprising: obtaining first point cloud data and second point cloud data, wherein the first point cloud data and the second point cloud data are obtained by collecting a same target scene, and the target scene comprises a target object; obtaining first image data and second image data associated with the first point cloud data, wherein the first image data and the second image data are obtained by collecting the target scene, and the collection time of the first image data and the second image data falls within a first time period, and the first time period is determined based on the collection time of the first point cloud data; fusing the first point cloud data, the first image data and the second image data to obtain first fused point cloud data; obtaining third image data and fourth image data associated with the second point cloud data, wherein the third image data and the fourth image data are obtained by collecting the target scene, and the collection time of the third image data and the fourth image data falls within a second time period, and the second time period is determined based on the collection time of the second point cloud data; fusing the second point cloud data, the third image data and the fourth image data to obtain second fused point cloud data; fusing the first fused point cloud data and the second fused point cloud data to obtain target fused point cloud data; and determining a target detection result for the target object based on the target fused point cloud data.

[0006] According to another aspect of the present disclosure, there is provided a three-dimensional target detection apparatus, comprising: a point cloud acquisition unit configured to acquire first point cloud data and second point cloud data, wherein the first point cloud data and the second point cloud data are obtained by capturing a same target scene, and the target scene comprises a target object; a first image acquisition unit configured to acquire first image data and second image data associated with the first point cloud data, wherein the first image data and the second image data are obtained by capturing the target scene, and a capturing time of the first image data and the second image data falls within a first time period, and the first time period is determined based on a capturing time of the first point cloud data; a first fusion unit configured to fuse the first point cloud data, the first image data and the second image data to obtain first fused point cloud data; a second image acquisition unit configured to acquire third image data and fourth image data associated with the second point cloud data, wherein the third image data and the fourth image data are obtained by capturing the target scene, and a capturing time of the third image data and the fourth image data falls within a second time period, and the second time period is determined based on a capturing time of the second point cloud data; a second fusion unit configured to fuse the second point cloud data, the third image data and the fourth image data to obtain second fused point cloud data; a third fusion unit configured to fuse the first fused point cloud data and the second fused point cloud data to obtain target fused point cloud data; and a target detection unit configured to determine a target detection result for the target object based on the target fused point cloud data.

[0007] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the three-dimensional target detection method as described above.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the three-dimensional target detection method as described above.

[0009] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the three-dimensional target detection method as described above.

[0010] According to another aspect of the present disclosure, there is provided an autonomous vehicle comprising the electronic device as described above.

[0011] According to one or more embodiments of the present disclosure, for each frame of point cloud data, two frames of image data associated with the acquisition time of the frame of point cloud data and acquired for the same scene are obtained, the frame of point cloud data is fused with the two frames of image data, and then the two frames of point cloud data fused with the image data are fused, so that the multi-modal fused point cloud data obtained based on the above fusion manner can retain the global information in the image data and the point cloud data to the greatest extent, the implementation is simple, and the accuracy of the result obtained by target detection based on the fused point cloud data can be effectively improved.

[0012] Further, since there is a correlation between the point cloud data and the image data in the acquisition time, if the acquisition times of any two frames of data are different, time sequence information can be introduced as a supplement in the process of target detection, so that a more accurate target detection result can be obtained.

[0013] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are merely examples and do not limit the scope of the claims. In all the drawings, like reference numerals refer to like elements throughout the various drawings. As used in this specification and the appended drawings, the singular forms "a", "an" and "the" include plural referents unless the context clearly indicates otherwise. Thus, for example, reference to "a component" can include

[0015] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein can be implemented according to embodiments of the present disclosure is shown;

[0016] Figure 2 A flowchart of a three-dimensional target detection method according to embodiments of the present disclosure is shown;

[0017] Figure 3 A flowchart of obtaining first image data and second image data according to embodiments of the present disclosure is shown;

[0018] Figure 4 A flowchart of fusing first point cloud data, first image data and second image data according to embodiments of the present disclosure is shown;

[0019] Figure 5 A flowchart of fusing first fused image data and first point cloud data according to embodiments of the present disclosure is shown;

[0020] Figure 6A structural block diagram of a three-dimensional object detection apparatus according to an embodiment of the disclosure is shown; and

[0021] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the disclosure is shown. DETAILED DESCRIPTION

[0022] Exemplary embodiments of the disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the disclosure to help with understanding. These should be considered in their context only as illustrative. Thus, those of ordinary skill in the art will recognize various changes and modifications of the embodiments described herein, which do not depart from the scope of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0023] In the disclosure, the terms "first", "second", and the like are used to describe various elements only and do not intend to limit the positional relationship, the time relationship, or the importance of the elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0024] The terms used in the description of various described examples in the disclosure are only for the purpose of describing particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, the element can be one or more if the number of elements is not specifically limited. In addition, the term "and / or" used in the disclosure encompasses any one of the listed items and all possible combinations thereof.

[0025] Embodiments of the disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 A schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein can be implemented according to embodiments of the disclosure is shown. Referring to Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0027] In embodiments of the disclosure, the server 120 can run one or more services or software applications that enable the execution of a three-dimensional object detection method.

[0028] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0029] In Figure 1 In the illustrated configuration, the server 120 can include one or more components that implement the functionality performed by the server 120. These components can include software components that are executable by one or more processors, hardware components, or combinations thereof. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by the components. It should be understood that a wide variety of system configurations are possible, which can differ from system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0030] A user can use the client device 101, 102, 103, 104, 105, and / or 106 to implement three-dimensional object detection. The client device can provide an interface that enables the user of the client device to interact with the client device. The client device can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but one of skill in the art will understand that the present disclosure can support any number of client devices.

[0031] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or including various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. Client devices are capable of executing a variety of different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0032] Network 110 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As examples only, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a local area network (LAN), a wide area network (WAN), a wireless network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth network), and / or any combination of these and / or other networks.

[0033] Server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, large mainframe computers, server clusters, or any other appropriate arrangement and / or combination. Server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 can run one or more services or software applications that provide the functionality described below.

[0034] The computing units in the server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0035] In some embodiments, the server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client devices 101, 102, 103, 104, 105, and 106. The server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client devices 101, 102, 103, 104, 105, and 106.

[0036] In some embodiments, the server 120 can be a server of a distributed system, or a server combined with a blockchain. The server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0037] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store point cloud data and image data. The databases 130 can reside in a variety of locations. For example, databases used by the server 120 can reside locally to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network- or application-specific connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0038] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores backed by file systems.

[0039] Figure 1The system 100 can be configured and operated in various ways to enable the various methods and apparatuses described according to the present disclosure to be applied.

[0040] In the scenario of autonomous driving, providing only 2D information of objects will not be sufficient to fully support the target detection perception task, and additional three-dimensional information such as position and orientation needs to be supplemented. Therefore, it is necessary to provide dense 3D point clouds through a laser radar to accurately represent the position and shape of objects in 3D space. In the related art, the target objects to be detected are identified and framed from the point cloud data and the corresponding image data, respectively, and then point cloud feature extraction and image feature extraction are performed on the target objects to fuse the extracted two types of features as the input of the target detection model. However, this way will lose a large amount of background information, which seriously reduces the accuracy of the result of target detection.

[0041] To this end, Figure 2 A schematic flowchart of a three-dimensional target detection method according to an embodiment of the present disclosure is shown. The method can be implemented by a client or a server as shown in Figure 1 The method 200 in Figure 2 The method 200 in

[0042] As shown in Figure 2 According to an embodiment of the present disclosure, a three-dimensional target detection method 200 is provided, comprising: obtaining first point cloud data and second point cloud data, wherein the first point cloud data and the second point cloud data are obtained by collecting the same target scene, and the target scene includes a target object (step 210); obtaining first image data and second image data associated with the first point cloud data, wherein the first image data and the second image data are obtained by collecting the target scene, and the collection time of the first image data and the second image data falls within a first time period, and the first time period is determined based on the collection time of the first point cloud data (step 220); fusing the first point cloud data, the first image data and the second image data to obtain first fused point cloud data (step 230); obtaining third image data and fourth image data associated with the second point cloud data, wherein the third image data and the fourth image data are obtained by collecting the target scene, and the collection time of the third image data and the fourth image data falls within a second time period, and the second time period is determined based on the collection time of the second point cloud data (step 240); fusing the second point cloud data, the third image data and the fourth image data to obtain second fused point cloud data (step 250); fusing the first fused point cloud data and the second fused point cloud data to obtain target fused point cloud data (step 260); and determining a target detection result for the target object based on the target fused point cloud data (step 270).

[0043] According to the three-dimensional target detection method provided in the embodiments of the present disclosure, for each frame of point cloud data, two frames of image data associated with the acquisition time of the frame of point cloud data and acquired for the same scene are obtained, the frame of point cloud data is fused with the two frames of image data, and then the two frames of point cloud data fused with the image data are fused. The multi-modal fused point cloud data obtained based on the above fusion manner can retain the global information in the image data and the point cloud data to the greatest extent, and the implementation is simple and the accuracy of the result obtained by target detection based on the fused point cloud data can be effectively improved.

[0044] Further, since the point cloud data and the image data to be fused are associated in the acquisition time, if the acquisition times of any two frames of data are different, time sequence information can be introduced as a supplement in the process of target detection, so that more accurate target detection results can be obtained.

[0045] In step 210, first point cloud data and second point cloud data are obtained, wherein the first point cloud data and the second point cloud data are obtained by collecting the same target scene, and the target scene includes a target object.

[0046] In some embodiments, the target scene includes an autonomous driving scene, and the target object includes an obstacle in the autonomous driving scene. The first point cloud data and the second point cloud data can be collected by one or more laser radars arranged on an autonomous driving vehicle, or can be collected by a laser radar arranged for road testing, and the present disclosure does not limit this.

[0047] In some embodiments, the acquisition times of the first point cloud data and the second point cloud data can be the same or different. In the case where the acquisition times of the first point cloud data and the second point cloud data are the same, they can be mutually verified and information supplemented in space to more effectively represent the real target scene. In the case where the acquisition times of the first point cloud data and the second point cloud data are different, they can be mutually verified and information supplemented in time to introduce time sequence information into target detection and improve the accuracy of the target detection result.

[0048] In an example, each set of scanning data of the laser radar has a corresponding time stamp to represent the acquisition time of each frame of point cloud data. Moreover, the IMU / GNSS data at the corresponding moment can also be determined through the time stamp, so that the data of different beams of the laser radar can be aligned to the coordinate system at the same moment to avoid motion blur caused by the motion of the autonomous driving vehicle itself.

[0049] In some embodiments, the collection angles of the first point cloud data and the second point cloud data for collecting the target scene can be the same or different. In the case that the collection angles of the first point cloud data and the second point cloud data are different, the scanning data of the target scene can be provided from multiple perspectives to perfect the information represented by the point cloud data.

[0050] It should be understood that although only the first point cloud data and the second point cloud data are described in the present disclosure, the number of point cloud data is not limited thereto. The number of point cloud data can be more than two frames, for example, three frames, ten frames, fifty frames, etc., which is not limited in the present disclosure.

[0051] In step 220, the first image data and the second image data associated with the first point cloud data are obtained, wherein the first image data and the second image data are obtained by collecting the target scene, the collection time of the first image data and the second image data falls within the first time period, and the first time period is determined based on the collection time of the first point cloud data.

[0052] In some embodiments, the first point cloud data and the second point cloud data can be collected by one or more cameras arranged on an autonomous vehicle, or can be collected by cameras arranged for road testing, which is not limited in the present disclosure. In an example, the cameras can include a surround view camera, a binocular camera, a monocular camera, etc., which is not limited in the present disclosure.

[0053] In some embodiments, the first time period can be a time period before or after the collection time of the first point cloud data, or the collection time of the first point cloud data can also fall within the first time period, which is not limited in the present disclosure. In other words, the collection time of the first image data and the second image data can be earlier or later than the collection time of the first point cloud data, or the collection time of the first point cloud data can be between the collection time of the first image data and the collection time of the second image data.

[0054] In some embodiments, the collection time of the first image data and the second image data can be the same or different. In the case that the collection time of the first image data and the second image data is the same, they can mutually verify and supplement information in space to more effectively represent the real target scene. In the case that the collection time of the first image data and the second image data is different, they can mutually verify and supplement information in time to introduce time sequence information into target detection, thereby improving the accuracy of the target detection result.

[0055] In some embodiments, the acquisition angles of the first image data and the second image data for the target scene can be the same or different. In the case where the acquisition angles of the first image data and the second image data are different, image data of the target scene can be provided from multiple perspectives to perfect the information represented by the image data.

[0056] It should be understood that although only the first image data and the second image data are described in the present disclosure, the number of image data is not limited thereto. The number of image data can be more than two frames, for example, three frames, ten frames, fifty frames, etc., which is not limited in the present disclosure.

[0057] Figure 3 A schematic flowchart of acquiring the first image data and the second image data according to an embodiment of the present disclosure is shown.

[0058] According to some embodiments, as shown in Figure 3 The acquisition time of the first point cloud data is the first time, and step 220 includes: acquiring image data acquired at a second time as the first image data, where the second time is earlier than the first time, and the time difference between the second time and the first time is less than a first threshold (step 3210); and acquiring image data acquired at a third time as the second image data, where the third time is earlier than the second time, and the time difference between the third time and the second time is less than a second threshold (step 3220).

[0059] In an example, the first threshold and the second threshold can be set, and the specific values of the above two thresholds are not limited in the present disclosure.

[0060] In an example, a sequence of image data corresponding to the first point cloud data can be acquired, and image data closest in time to the acquisition time of the first point cloud data and earlier than the first point cloud data is selected from the sequence as the second image data, and then image data closest in time to the acquisition time of the second image data and earlier than the second image data is selected as the first image data, so that the first image data and the second image data are the two frames of image data closest in time to the first point cloud data and continuous. In this way, the timing information included in adjacent image frames can be introduced, avoiding detection bias caused by unsynchronized laser radar and camera hardware or frame loss, and further improving the accuracy of target detection.

[0061] It should be understood that the above example is only illustrative and is not limited thereto. The first image data and the second image data can also be two frames of non-continuous image data, which is not limited in the present disclosure.

[0062] In step 230, the first point cloud data, the first image data and the second image data are fused to obtain the first fused point cloud data.

[0063] Figure 4 A schematic flowchart of fusing first point cloud data, first image data and second image data is shown according to an embodiment of the present disclosure.

[0064] According to some embodiments, as shown in Figure 4 Step 230 includes, as shown in the example of FIG. 43, feature extraction on the second image data to obtain first to-be-processed image features (step 4310), generation of a first heat map based on the first to-be-processed image features (step 4320), input of the first image data, the first heat map and the second image data into a first visual feature fusion network model to obtain first fused image data output by the first visual feature fusion network model (step 4330), and fusion of the first fused image data and the first point cloud data to obtain first fused point cloud data (step 4340).

[0065] In the example, the two frames of image data are first fused in features and then fused with the point cloud data, which can reduce the processing difficulty of data fusion. In addition, in the case where the first image data and the second image data are collected at different times, the image frames are extracted and fused in features, which can realize the association and fusion of the time sequence features of the images, and further introduces the time sequence information.

[0066] In the example, for steps 4310 to 4330, the visual feature extraction backbone network model can be used to process the second image data first, and the intermediate feature result of the model can be extracted as the first to-be-processed image features to generate the first heat map. Then, the first image data, the first heat map and the second image data are input into the visual feature extraction backbone network model again for shallow coding and image fusion. Based on this, the context feature information of the image data is fused, so that the first fused image data has shallow feature information, high-resolution location semantic information and deep and fine semantic feature information at the same time, contains richer semantic information and a larger receptive field, and can better represent the real target scene.

[0067] In the example, the visual feature extraction backbone network model is a backbone network model obtained by training a single image data set with a magnitude of ten million, which can have good feature extraction capability and generalization.

[0068] For step 4340, Figure 5 A schematic flowchart of fusing first fused image data and first point cloud data is shown according to an embodiment of the present disclosure.

[0069] According to some embodiments, as shown in Figure 5As shown, step 4340 includes: determining a first mapping relationship between the first fused image data and the first point cloud data (step 5341); for each point in the first point cloud data, performing the following operations: determining at least one first pixel point in the first fused image data corresponding to the point based on the first mapping relationship (step 5342); and splicing a feature vector of the at least one first pixel point with a feature vector of the point (step 5343).

[0070] The fusion of the first fused image data and the first point cloud data is a point-level fusion, that is, the feature vectors of the points in the point cloud data and the feature vectors of the pixel points in the image data are spliced, so that the feature dimension of each point in the obtained first point cloud fused data is more and the information is richer. In addition, such point-to-point feature splicing can more accurately realize feature alignment between camera image data and lidar point cloud data, so that the obtained fused data can better represent the actual scene.

[0071] In step 240, third image data and fourth image data associated with the second point cloud data are obtained, wherein the third image data and the fourth image data are obtained by collecting the target scene, and the collection time of the third image data and the fourth image data falls within a second time period, and the second time period is determined based on the collection time of the second point cloud data.

[0072] According to some embodiments, the collection time of the second point cloud data is a fourth moment, and the third image data and the fourth image data associated with the second point cloud data are obtained, including: obtaining image data collected for the target scene at a fifth moment as the third image data, wherein the fifth moment is earlier than the fourth moment, and the time difference between the fifth moment and the fourth moment is less than a third threshold; and obtaining image data collected for the target scene at a sixth moment as the fourth image data, wherein the sixth moment is earlier than the fifth moment, and the time difference between the sixth moment and the fifth moment is less than a fourth threshold.

[0073] In step 250, the second point cloud data, the third image data and the fourth image data are fused to obtain second fused point cloud data.

[0074] According to some embodiments, step 250 includes: performing feature extraction on the fourth image data to obtain second to-be-processed image features; generating a second heat map based on the second to-be-processed image features; inputting the third image data, the second heat map and the fourth image data into a second visual feature fusion network model to obtain second fused image data output by the second visual feature fusion network model; and fusing the second fused image data and the second point cloud data to obtain second fused point cloud data.

[0075] According to some embodiments, the fusing the second fused image data and the second point cloud data comprises: determining a second mapping relationship between the second fused image data and the second point cloud data; for each point in the second point cloud data, performing the following operations: determining at least one second pixel point in the second fused image data corresponding to the point based on the second mapping relationship; and splicing a feature vector of the at least one second pixel point and a feature vector of the point.

[0076] For steps 240 and 250, reference can be made to the above description of steps 220 and 230, which will not be repeated here. It should be understood that the order of steps 240 and 250 and steps 220 and 230 can be interchanged, or both can be performed simultaneously, and the present disclosure does not limit in this regard.

[0077] In step 260, the first fused point cloud data and the second fused point cloud data are fused to obtain target fused point cloud data.

[0078] In some embodiments, voxelization feature extraction can be performed on the first fused point cloud data and the second fused point cloud data respectively to generate first pseudo image data corresponding to the first fused point cloud data and second pseudo image data corresponding to the second fused point cloud data, and then the first pseudo image data and the second pseudo image data are superimposed to obtain the target fused point cloud data.

[0079] Based on this, compared with directly performing target detection on original single-frame point cloud data, using the above target fused point cloud data can effectively improve the problem of unstable detection of distant target objects, and has good detection ability for target objects that are temporarily occluded.

[0080] In step 270, based on the target fused point cloud data, a target detection result for the target object is determined.

[0081] In some embodiments, the target fused point cloud data can be input into a target detection model to obtain a target detection result for the target object. In an example, the target detection model includes but is not limited to a PointPillar model.

[0082] Figure 6 A structural block diagram of a three-dimensional target detection device according to an embodiment of the present disclosure is shown.

[0083] As Figure 6As shown, according to an embodiment of the present disclosure, a three-dimensional target detection apparatus is provided, comprising: a point cloud acquisition unit 610, configured to acquire first point cloud data and second point cloud data, wherein the first point cloud data and the second point cloud data are obtained by collecting a same target scene, and the target scene comprises a target object; a first image acquisition unit 620, configured to acquire first image data and second image data associated with the first point cloud data, wherein the first image data and the second image data are obtained by collecting the target scene, and the collection time of the first image data and the second image data falls within a first time period, and the first time period is determined based on the collection time of the first point cloud data; a first fusion unit 630, configured to fuse the first point cloud data, the first image data and the second image data to obtain first fused point cloud data; a second image acquisition unit 640, configured to acquire third image data and fourth image data associated with the second point cloud data, wherein the third image data and the fourth image data are obtained by collecting the target scene, and the collection time of the third image data and the fourth image data falls within a second time period, and the second time period is determined based on the collection time of the second point cloud data; a second fusion unit 650, configured to fuse the second point cloud data, the third image data and the fourth image data to obtain second fused point cloud data; a third fusion unit 660, configured to fuse the first fused point cloud data and the second fused point cloud data to obtain target fused point cloud data; and a target detection unit 670, configured to determine a target detection result for the target object based on the target fused point cloud data.

[0084] Here, the operations of the above-mentioned units 610-670 of the three-dimensional target detection apparatus 600 are similar to the operations of the above-mentioned steps 210-270, and will not be described here again.

[0085] According to an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the three-dimensional target detection method as described above.

[0086] According to an embodiment of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the three-dimensional target detection method as described above.

[0087] According to an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the three-dimensional target detection method as described above.

[0088] According to an embodiment of the disclosure, there is provided an autonomous vehicle including the electronic device as described above.

[0089] Reference Figure 7 A block diagram of an electronic device 700, which can be an example of a hardware device that can be applied to aspects of the disclosure, will now be described, which is an example of a hardware device that can be applied to aspects of the disclosure. The electronic device is intended to represent various forms of digital electronic computing devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent various forms of mobile devices such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components, their connections, and their functions, as shown in the figures, and as described in this document, are meant only to be examples and are not intended to limit the implementations of the disclosure described and / or claimed in this document.

[0090] As shown in FIG. 7, the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Figure 7

[0091] Various components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device 700, can receive inputted digital or character information, and can generate key signal inputs related to user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and / or a remote controller. The output unit 707 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.​

[0092] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the method 777. For example, in some embodiments, the method 777 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the method 777 described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the method 777 by any other suitable means, such as by means of firmware.

[0093] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0094] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0095] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0096] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0097] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0098] ​The computer system can include clients and servers. This relationship can be remote or on-site. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0099] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

[0100] Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but only by the granted claims and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples can be combined in various ways. It is important that many of the elements described herein can be replaced by equivalent elements that appear after the present disclosure as technology evolves.

Claims

1. A three-dimensional target detection method, comprising: Acquire first point cloud data and second point cloud data, wherein the first point cloud data and the second point cloud data are obtained by collecting data from the same target scene, and the target scene includes a target object; First image data and second image data associated with the first point cloud data are obtained, wherein the first image data and the second image data are obtained by collecting data from the target scene, and the collection time of the first image data and the second image data falls into a first time period, which is determined based on the collection time of the first point cloud data. The first point cloud data, the first image data, and the second image data are fused to obtain the first fused point cloud data; Acquire third and fourth image data associated with the second point cloud data, wherein the third and fourth image data are obtained by collecting data from the target scene, and the collection time of the third and fourth image data falls within a second time period, which is determined based on the collection time of the second point cloud data; The second point cloud data, the third image data, and the fourth image data are fused to obtain the second fused point cloud data; The first fused point cloud data and the second fused point cloud data are fused to obtain the target fused point cloud data; and Based on the target fusion point cloud data, the target detection result for the target object is determined.

2. The method according to claim 1, wherein, The acquisition time of the first point cloud data is a first moment. The acquisition of the first image data and the second image data associated with the first point cloud data includes: The image data acquired at the second time point is used as the first image data, wherein the second time point is earlier than the first time point, and the time difference between the second time point and the first time point is less than a first threshold. Image data acquired at a third time point is used as the second image data, wherein the third time point is earlier than the second time point, and the time difference between the third time point and the second time point is less than a second threshold.

3. The method according to claim 1, wherein, The fusion of the first point cloud data, the first image data, and the second image data includes: Feature extraction is performed on the second image data to obtain the first image features to be processed; A first heatmap is generated based on the features of the first image to be processed; The first image data, the first heatmap, and the second image data are input into a first visual feature fusion network model to obtain the first fused image data output by the first visual feature fusion network model; and The first fused image data and the first point cloud data are fused together to obtain the first fused point cloud data.

4. The method according to claim 3, wherein, The step of fusing the first fused image data and the first point cloud data includes: Determine a first mapping relationship between the first fused image data and the first point cloud data; For each point in the first cloud data, perform the following operations: Based on the first mapping relationship, at least one first pixel in the first fused image data corresponding to that point is determined; and The feature vector of the at least one first pixel is concatenated with the feature vector of that pixel.

5. The method according to claim 1, wherein, The acquisition time of the second point cloud data is the fourth moment. The acquisition of the third and fourth image data associated with the second point cloud data includes: Image data acquired at a fifth time point targeting the target scene is used as the third image data, wherein the fifth time point is earlier than the fourth time point, and the time difference between the fifth time point and the fourth time point is less than a third threshold; and The image data acquired at the sixth moment for the target scene is used as the fourth image data, wherein the sixth moment is earlier than the fifth moment, and the time difference between the sixth moment and the fifth moment is less than a fourth threshold.

6. The method according to any one of claims 1-5, wherein, The process of fusing the second point cloud data, the third image data, and the fourth image data includes: Feature extraction is performed on the fourth image data to obtain the second image features to be processed; A second heatmap is generated based on the features of the second image to be processed; The third image data, the second heatmap, and the fourth image data are input into the second visual feature fusion network model to obtain the second fused image data output by the second visual feature fusion network model; and The second fused image data and the second point cloud data are fused together to obtain the second fused point cloud data.

7. The method according to claim 6, wherein, The process of fusing the second fused image data and the second point cloud data includes: Determine a second mapping relationship between the second fused image data and the second point cloud data; For each point in the second cloud data, perform the following operations: Based on the second mapping relationship, at least one second pixel point in the second fused image data corresponding to that point is determined; The feature vector of the at least one second pixel is concatenated with the feature vector of that pixel.

8. A three-dimensional target detection device, comprising: A point cloud acquisition unit is used to acquire first point cloud data and second point cloud data, wherein the first point cloud data and the second point cloud data are obtained by collecting data from the same target scene, and the target scene includes a target object; The first image acquisition unit is used to acquire first image data and second image data associated with the first point cloud data, wherein the first image data and the second image data are obtained by collecting data from the target scene, and the acquisition time of the first image data and the second image data falls into a first time period, which is determined based on the acquisition time of the first point cloud data. The first fusion unit is used to fuse the first point cloud data, the first image data and the second image data to obtain the first fused point cloud data; The second image acquisition unit is used to acquire the third image data and the fourth image data associated with the second point cloud data. The third image data and the fourth image data are obtained by collecting data from the target scene. The acquisition time of the third image data and the fourth image data falls into a second time period, which is determined based on the acquisition time of the second point cloud data. The second fusion unit is used to fuse the second point cloud data, the third image data, and the fourth image data to obtain the second fused point cloud data; The third fusion unit is used to fuse the first fused point cloud data and the second fused point cloud data to obtain the target fused point cloud data; and The target detection unit is used to determine the target detection result for the target object based on the target fused point cloud data.

9. The apparatus according to claim 8, wherein, The acquisition time of the first point cloud data is the first moment, and the first image acquisition unit includes: The first image acquisition subunit is used to acquire image data collected at a second time as the first image data, wherein the second time is earlier than the first time, and the time difference between the second time and the first time is less than a first threshold. The second image acquisition subunit is used to acquire image data collected at a third time as the second image data, wherein the third time is earlier than the second time, and the time difference between the third time and the second time is less than a second threshold.

10. The apparatus according to claim 8, wherein, The first fusion unit includes: The first feature extraction subunit is used to extract features from the second image data to obtain the first image features to be processed; The first feature processing subunit is used to generate a first heatmap based on the features of the first image to be processed. A first image fusion subunit is configured to input the first image data, the first heatmap, and the second image data into a first visual feature fusion network model to obtain the first fused image data output by the first visual feature fusion network model; and The first fusion subunit is used to fuse the first fused image data and the first point cloud data to obtain the first fused point cloud data.

11. The apparatus according to claim 10, wherein, The first fusion subunit includes: The first mapping subunit is used to determine the first mapping relationship between the first fused image data and the first point cloud data; The first feature splicing subunit is used to perform the following operations for each point in the first point cloud data: Based on the first mapping relationship, at least one first pixel in the first fused image data corresponding to that point is determined; and The feature vector of the at least one first pixel is concatenated with the feature vector of that pixel.

12. The apparatus according to claim 9, wherein, The second point cloud data is acquired at the fourth moment, and the second image acquisition unit includes: The third image acquisition subunit is configured to acquire image data collected from the target scene at a fifth time point as the third image data, wherein the fifth time point is earlier than the fourth time point, and the time difference between the fifth time point and the fourth time point is less than a third threshold; and The fourth image acquisition subunit is used to acquire image data collected from the target scene at the sixth time as the fourth image data, wherein the sixth time is earlier than the fifth time, and the time difference between the sixth time and the fifth time is less than the fourth threshold.

13. The apparatus according to any one of claims 8-12, wherein, The second fusion unit includes: The second feature extraction subunit is used to extract features from the fourth image data to obtain the second image features to be processed. The second feature processing subunit is used to generate a second heatmap based on the features of the second image to be processed. The second image fusion subunit is used to input the third image data, the second heatmap, and the fourth image data into the second visual feature fusion network model to obtain the second fused image data output by the second visual feature fusion network model; and The second fusion subunit is used to fuse the second fused image data and the second point cloud data to obtain the second fused point cloud data.

14. The apparatus according to claim 13, wherein, The second fusion subunit includes: The second mapping subunit is used to determine the second mapping relationship between the second fused image data and the second point cloud data; The second feature splicing subunit is used to perform the following operations for each point in the second point cloud data: Based on the second mapping relationship, at least one second pixel point in the second fused image data corresponding to that point is determined; The feature vector of the at least one second pixel is concatenated with the feature vector of that pixel.

15. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-7.

18. An autonomous vehicle, including the electronic equipment as claimed in claim 15.

Images