Three-dimensional target detection method, device, system and storage medium

Abstract

A three-dimensional target detection method, apparatus, system, and storage medium are disclosed. The method includes: acquiring a set of point clouds acquired by a point cloud acquisition device for a current scene and a set of first images acquired by an image acquisition device for the current scene; projecting the point clouds onto the viewpoint of the image acquisition device to obtain a set of second images; extracting features from the first images and their corresponding second images to obtain first deep features of the first images and second deep features of the second images, wherein, during the feature extraction process, first shallow features of the first images and second shallow features of the second images are acquired, and the first shallow features and second shallow features are fused to obtain fused shallow features; obtaining a first feature map based on the fused shallow features and the first deep features, and obtaining a second feature map based on the fused shallow features and the second deep features; performing three-dimensional target detection based on the first feature map and the second feature map, and outputting a three-dimensional target detection result.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and more specifically to a three-dimensional target detection method, apparatus, system and storage medium. Background Technology

[0002] Accurate and real-time target detection is a crucial function for autonomous vehicles and intelligent robots to accurately perceive their complex surroundings. How to accurately determine the position, size, distance, and pose of surrounding objects in three dimensions (3D) is a classic challenge in 3D target detection. Currently, 3D target detection methods mainly include single-modal methods based on camera, LiDAR, or millimeter-wave radar sensors, as well as multi-modal fusion methods using multiple sensors.

[0003] In practical industrial and academic fields, 3D target detection methods based on cameras and LiDAR are the most popular. Camera-based detection methods are inexpensive to deploy and use images as input, which contain rich information such as texture and color. However, they are susceptible to object occlusion, viewpoint changes, and scale changes, leading to poor detection accuracy and robustness. LiDAR-based 3D target detection methods use point cloud information as input, which can contain accurate target position and distance information. However, the point cloud is relatively sparse and not sensitive to the target's texture information.

[0004] Current camera-based and LiDAR-based object detection methods differ significantly in their representation, and single-modal methods struggle to overcome their respective performance limitations. Therefore, academia and industry have begun exploring multimodal 3D object detection methods. Existing multimodal 3D object detection methods fuse multimodal features within a unified bird's-eye view (BEV) space. This fusion approach only utilizes deep semantic features and fails to fully leverage the capabilities of point clouds and images. Summary of the Invention

[0005] This application is proposed to address the aforementioned problems. According to one aspect of this application, a three-dimensional target detection method is provided, comprising: acquiring a set of point clouds acquired by a point cloud acquisition device for a current scene and a set of first images acquired by an image acquisition device for the current scene; projecting the point clouds onto the viewpoint of the image acquisition device to obtain a set of second images, wherein each second image in the set of second images corresponds to a first image in the set of first images; performing feature extraction on the first images and their corresponding second images respectively to obtain first deep features of the first images and second deep features of the second images, wherein, during the feature extraction process, first shallow features of the first images and second shallow features of the second images are acquired, and the first shallow features and second shallow features are fused to obtain fused shallow features; obtaining a first feature map of the first image based on the fused shallow features and the first deep features, and obtaining a second feature map of the second image based on the fused shallow features and the second deep features; performing three-dimensional target detection based on the first feature map of each first image in the set of first images and the second feature map of each second image in the set of second images, and outputting a three-dimensional target detection result.

[0006] According to another aspect of this application, a three-dimensional target detection device is provided. The device includes a point cloud projection module, a feature extraction module, a shallow feature fusion module, and a target detection module. The point cloud projection module acquires point clouds collected by a point cloud acquisition device for the current scene and projects the point clouds onto the viewpoint of an image acquisition device to obtain a set of second images. The image acquisition device acquires a set of first images for the current scene, and each second image in the set of second images corresponds to a first image in the set of first images. The feature extraction module performs feature extraction on the first image and its corresponding second image to obtain first deep features of the first image and second deep features of the second image. The shallow feature fusion module is used to obtain the first shallow features of the first image and the second shallow features of the second image from the feature extraction module during the feature extraction process, and fuse the first shallow features and the second shallow features to obtain the fused shallow features; the feature extraction module is also used to obtain the first feature map of the first image based on the fused shallow features and the first deep features, and to obtain the second feature map of the second image based on the fused shallow features and the second deep features; the target detection module is used to perform three-dimensional target detection based on the first feature map of each first image in the set of first images and the second feature map of each second image in the set of second images, and output the three-dimensional target detection result.

[0007] According to another aspect of this application, a three-dimensional target detection device is provided, which includes a memory and a processor. The memory stores a computer program that is executed by the processor. When the computer program is executed by the processor, it causes the processor to perform the above-described three-dimensional target detection method.

[0008] According to another aspect of this application, a three-dimensional target detection system is provided, the system comprising: a point cloud acquisition device for acquiring point clouds for the current scene; an image acquisition device for acquiring images for the current scene; and the aforementioned three-dimensional target detection device for performing three-dimensional target detection on the current scene based on the point cloud acquired by the point cloud acquisition device and the image acquired by the image acquisition device.

[0009] According to another aspect of this application, a storage medium is provided, on which a computer program is stored, and when the computer program is run, it causes a processor to execute the above-described three-dimensional target detection method.

[0010] According to another aspect of this application, a computer program product is provided, which includes a computer program that, when run by a processor, causes the processor to perform the above-described three-dimensional target detection method.

[0011] The 3D target detection method, apparatus, and system of this application generate a point cloud pseudo-image by projecting a point cloud acquired by a point cloud acquisition device onto the viewpoint of an image acquisition device. Shallow and deep features are extracted from both the point cloud pseudo-image and the image captured by the image acquisition device. The shallow features of both the point cloud pseudo-image and the image captured by the image acquisition device are fused together, and then further fused with their respective deep features to obtain the final feature extraction result for 3D target detection. This process relies not only on multimodal shallow features (shallow features of the point cloud pseudo-image and the image captured by the image acquisition device) but also on multimodal deep information, enabling 3D target detection to combine rich texture and color information with high-level semantic information, thus greatly improving the accuracy of the 3D target detection results. Attached Figure Description

[0012] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 A schematic block diagram of an example electronic device for implementing a three-dimensional target detection method and apparatus according to embodiments of the present invention is shown.

[0014] Figure 2 A schematic flowchart of a three-dimensional target detection method according to an embodiment of this application is shown.

[0015] Figure 3 A schematic flowchart of a three-dimensional target detection method according to another embodiment of this application is shown.

[0016] Figure 4 A schematic flowchart of a three-dimensional target detection method according to another embodiment of this application is shown.

[0017] Figure 5 A schematic flowchart of a three-dimensional target detection method according to yet another embodiment of this application is shown.

[0018] Figure 6 A schematic structural block diagram of a three-dimensional target detection device according to an embodiment of this application is shown.

[0019] Figure 7 A schematic structural block diagram of a three-dimensional target detection apparatus according to another embodiment of this application is shown.

[0020] Figure 8 A schematic structural block diagram of a three-dimensional target detection device according to another embodiment of this application is shown.

[0021] Figure 9 A schematic structural block diagram of a three-dimensional target detection device according to yet another embodiment of this application is shown.

[0022] Figure 10 A schematic structural block diagram of a three-dimensional target detection system according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0024] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0025] Below, refer to Figure 1 This describes an example electronic device 100 for implementing the three-dimensional target detection method and apparatus of the present invention.

[0026] like Figure 1 As shown, the electronic device 100 includes one or more processors 102, one or more storage devices 104, input devices 106, and output devices 108, which are interconnected via a bus system 110 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0027] The processor 102 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0028] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present invention described below, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0029] The input device 106 can be a device used by a user to input commands, and may include one or more of a keyboard, mouse, microphone, and touchscreen. Furthermore, the input device 106 can also be any interface for receiving information.

[0030] The output device 108 can output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. Furthermore, the output device 108 can also be any other device with output functionality.

[0031] For example, the example electronic device used to implement the three-dimensional target detection method and apparatus according to embodiments of the present invention can be implemented in terminals such as smartphones, tablets, cameras, vehicle-mounted devices, and intelligent robots.

[0032] Below, we will refer to Figure 2 A three-dimensional target detection method 200 according to an embodiment of this application is described. For example... Figure 2 As shown, the three-dimensional target detection method 200 may include the following steps:

[0033] In step S210, a set of point clouds acquired by the point cloud acquisition device for the current scene and a first image acquired by the image acquisition device for the current scene is obtained.

[0034] In step S220, the point cloud is projected onto the view of the image acquisition device to obtain a set of second images, and each second image in the set of second images corresponds to a first image in the set of first images.

[0035] In step S230, feature extraction is performed on the first image and its corresponding second image respectively to obtain the first deep feature of the first image and the second deep feature of the second image. During the feature extraction process, the first shallow feature of the first image and the second shallow feature of the second image are obtained. The first shallow feature and the second shallow feature are fused to obtain the fused shallow feature.

[0036] In step S240, a first feature map of the first image is obtained based on the fused shallow features and the first deep features, and a second feature map of the second image is obtained based on the fused shallow features and the second deep features.

[0037] In step S250, three-dimensional target detection is performed based on the first feature map of each first image in the set of first images and the second feature map of each second image in the set of second images, and the three-dimensional target detection result is output.

[0038] In the embodiments of this application, in order to perform three-dimensional target detection on the current scene, a point cloud acquisition device (such as radar) acquires point clouds on the current scene, and an image acquisition device (such as a camera) acquires images on the current scene.

[0039] After acquiring the point cloud and image of the current scene, the point cloud is projected onto the viewpoint of the image acquisition device to convert it into an image. To distinguish them, the image captured by the image acquisition device for the current scene is called the first image, and the image generated by projecting the point cloud onto the viewpoint of the image acquisition device is called the second image, which can also be called a point cloud pseudo-image. Generally, one point cloud acquisition device is used to acquire the point cloud for the current scene, and multiple image acquisition devices are used to acquire the first image for the current scene. That is, different image acquisition devices acquire the first image from different viewpoints, and the first images from all viewpoints are used together as the image acquisition result for the current scene (i.e., the set of first images). The point cloud needs to be projected onto the viewpoints of multiple image acquisition devices to obtain multiple second images (i.e., multi-view point cloud pseudo-images, i.e., the set of second images). Each second image corresponds to a first image acquired by one image acquisition device. Based on this correspondence, feature extraction, shallow feature fusion, and feature map fusion in subsequent steps are all performed separately for each pair of corresponding first and second images. This results in multiple first feature maps and multiple second feature maps, which are then fused together for 3D object detection. Since the second image is generated by projecting the point cloud onto the viewpoint of the image acquisition device, the second image and the first image have highly similar feature representations. Based on this, feature extraction can be performed on the first and second images separately to obtain their respective deep features. Deep features (also called high-level features) are features extracted by the deep networks of the feature extraction network (such as one or more convolutional layers near the output). Deep features are closer to the output and contain coarse-grained, more abstract information, i.e., semantic information. Furthermore, deep features have a larger receptive field and a larger overlapping area, compressing image information and obtaining information about the overall image. To distinguish them, the deep features extracted from the first image are called first deep features, and the deep features extracted from the second image are called second deep features. Furthermore, during feature extraction from the first and second images, shallow features can be extracted before the deep features are extracted. Shallow features are those extracted by the shallow layers of the feature extraction network (such as one or more convolutional layers close to the input). Shallow features are closer to the input image, containing more pixel information and finer-grained information such as color, texture, edges, and corner details. In addition, shallow features have smaller receptive fields and less overlap, thus capturing more detail. To distinguish them, the shallow features extracted from the first image are called the first shallow features, and the shallow features extracted from the second image are called the second shallow features. After obtaining the shallow features of the first and second images respectively, they are fused to obtain the fused shallow features.The fused shallow features are then fused with the first and second deep features respectively to obtain the final feature extraction results of the first and second images, namely the first feature map and the second feature map. Finally, 3D target detection can be performed based on the first and second feature maps.

[0040] Therefore, the three-dimensional target detection method according to the embodiments of this application generates a point cloud pseudo-image by projecting the point cloud acquired by the point cloud acquisition device onto the view of the image acquisition device. Shallow features and deep features are extracted from the point cloud pseudo-image and the image captured by the image acquisition device, respectively. The shallow features of the point cloud pseudo-image and the image captured by the image acquisition device are fused together and then fused with their respective deep features to obtain the final feature extraction result for three-dimensional target detection. This makes the three-dimensional target detection process not only dependent on multimodal shallow features (shallow features of the point cloud pseudo-image and shallow features of the image captured by the image acquisition device) but also dependent on multimodal deep information. This allows the three-dimensional target detection to combine rich texture and color information and high-level semantic information, which greatly improves the accuracy of the three-dimensional target detection result.

[0041] In the embodiments of this application, step S220, which projects the point cloud onto the viewpoint of the image acquisition device, specifically includes projecting the point cloud onto the coordinate system of the image acquisition device. Because the point cloud is projected onto the viewpoint of the image acquisition device, the problems of sparse point clouds being unable to be used for feature extraction using convolutional neural networks and inaccurate point cloud height information are solved. The coordinate transformation of the point cloud, the association of the point cloud with target information in the image, point cloud expansion, and information extraction operations enable precise matching and association between point cloud information and image information, ultimately converting the sparse point cloud into a pseudo-image format acceptable to convolutional neural networks.

[0042] In the embodiments of this application, the feature extraction of the first image in step S230 is performed by a first feature extractor, and the feature extraction of the second image is performed by a second feature extractor. The first and second feature extractors are the same feature extractor. Since the first and second images have similar representations, the first feature extractor for the first image and the second feature extractor for the second image can be the same feature extractor (backbone network). For example, the first and second feature extractors can be Residual Networks (ResNet), Convolutional Networks (ConvNeXt), MobileNet, Visual Geometry Groups (VGG), etc. Using the same feature extractor for both the first and second images simplifies the overall network model for 3D object detection.

[0043] Since the first feature extractor for extracting features from the first image and the second feature extractor for extracting features from the second image can be the same feature extractor, the extracted first and second shallow features can also have the same size, which is very convenient for subsequent fusion. In the embodiments of this application, the fusion of the first and second shallow features in step S230 can include: adding or concatenating the first and second shallow features; or, passing the first and second shallow features through one or more convolutional layers respectively and then adding them together. The fusion of shallow features from point cloud pseudo-images and images captured by image acquisition devices can solve the problem of not utilizing shallow information caused by existing methods that only fuse deep information from both in bird's-eye view space, thereby leveraging the capabilities of point clouds and images.

[0044] Furthermore, the first image captured by the image capturing device can be a two-dimensional image. Correspondingly, the second image generated by projecting the point cloud captured by the point cloud acquisition device onto the viewpoint of the image acquisition device can also be a two-dimensional image. In this way, the feature extractors for extracting features from the first and second images can both be two-dimensional feature extractors. Compared with extracting features from point clouds by relying on voxels and three-dimensional convolution, the required computing power is reduced, thus improving the inference speed and reducing deployment limitations.

[0045] In the embodiments of this application, the step S240, which involves obtaining a first feature map of a first image based on the fused shallow features and the first deep features, may include: adding or concatenating the fused shallow features and the first deep features to obtain the first feature map. Similarly, the step S240, which involves obtaining a second feature map of a second image based on the fused shallow features and the second deep features, may include: adding or concatenating the fused shallow features and the second deep features to obtain the second feature map. Step S230 achieves the fusion of shallow features, and step S240 achieves the fusion of deep features and the fused shallow features, enabling flexible multi-level fusion in 3D object detection, which is beneficial for improving the accuracy of 3D object detection results.

[0046] In one embodiment of this application, step S250, which involves performing 3D object detection based on the first feature map of each first image in the set of first images and the second feature map of each second image in the set of second images, and outputting the 3D object detection result, may include: directly performing 3D object detection based on the first feature map of each first image and the second feature map of its corresponding second image. In this embodiment, 3D object detection can be performed based on two-dimensional features such as the first feature map of each first image and the second feature map of its corresponding second image. As mentioned above, since the first feature map and the second feature map are obtained by fusing the shallow features of the point cloud pseudo-image and the image captured by the image acquisition device, respectively, and then fusing them with their respective deep features, the 3D object detection process relies not only on multimodal shallow features but also on multimodal deep information. This allows 3D object detection to combine rich texture and color information with high-level semantic information, greatly improving the accuracy of the 3D object detection result.

[0047] The above exemplarily illustrates a three-dimensional target detection method 200 according to an embodiment of this application. Based on the above description, the three-dimensional target detection method 200 according to the embodiment of this application generates a point cloud pseudo-image by projecting a point cloud acquired by a point cloud acquisition device onto the viewpoint of an image acquisition device. Shallow and deep features are extracted from both the point cloud pseudo-image and the image captured by the image acquisition device. The shallow features of the point cloud pseudo-image and the image captured by the image acquisition device are fused together, and then further fused with their respective deep features to obtain the final feature extraction result for three-dimensional target detection. This makes the three-dimensional target detection process rely not only on multimodal shallow features (shallow features of the point cloud pseudo-image and shallow features of the image captured by the image acquisition device) but also on multimodal deep information, enabling three-dimensional target detection to combine rich texture and color information with high-level semantic information, greatly improving the accuracy of the three-dimensional target detection results.

[0048] In another embodiment of this application, step S250 above, which involves performing 3D target detection based on the first feature map of each first image and the second feature map of its corresponding second image, and outputting the 3D target detection result, may also include: projecting the first feature map of each first image and the second feature map of its corresponding second image into a 3D space and then projecting them into a bird's-eye view space to obtain first bird's-eye view features and second bird's-eye view features; fusing the first bird's-eye view features and the second bird's-eye view features to obtain fused bird's-eye view features; and performing 3D target detection based on the fused bird's-eye view features of each first image and its corresponding second image. In this embodiment, the first feature map and the second feature map can be projected into a 3D space and then projected into a bird's-eye view (BEV) space to perform 3D target detection. The following is in conjunction with... Figure 3 The following describes this embodiment in more detail.

[0049] Figure 3 A three-dimensional target detection method 300 according to another embodiment of this application is shown. For example... Figure 3 As shown, the three-dimensional target detection method 300 may include the following steps:

[0050] In step S310, a set of point clouds acquired by the point cloud acquisition device for the current scene and a first image acquired by the image acquisition device for the current scene is obtained.

[0051] In step S320, the point cloud is projected onto the view of the image acquisition device to obtain a set of second images, and each second image in the set of second images corresponds to a first image in the set of first images.

[0052] In step S330, feature extraction is performed on the first image and its corresponding second image respectively to obtain the first deep feature of the first image and the second deep feature of the second image. During the feature extraction process, the first shallow feature of the first image and the second shallow feature of the second image are obtained. The first shallow feature and the second shallow feature are fused to obtain the fused shallow feature.

[0053] In step S340, a first feature map of the first image is obtained based on the fused shallow features and the first deep features, and a second feature map of the second image is obtained based on the fused shallow features and the second deep features.

[0054] In step S350, the first feature map and the second feature map are projected into three-dimensional space and then projected into bird's-eye view space to obtain the first bird's-eye view feature and the second bird's-eye view feature.

[0055] In step S360, the first bird's-eye view feature and the second bird's-eye view feature are fused to obtain the fused bird's-eye view feature.

[0056] In step S370, three-dimensional target detection is performed based on the fused bird's-eye view features of each first image and its corresponding second image, and the three-dimensional target detection result is output.

[0057] The 3D target detection method 300 according to the embodiments of this application is generally similar to the 3D target detection method 200 according to the embodiments of this application described above, with only some differences. Specifically, steps S310 to S340 of the 3D target detection method 300 are the same as steps S210 to S240 of the 3D target detection method 200, respectively. Steps S350 to S370 of the 3D target detection method 300 can be understood as an implementation of step S250 of the 3D target detection method 200 (assuming that step S250 is understood as 3D target detection based on the first feature map and the second feature map), or it can be understood as a different implementation of step S250 of the 3D target detection method 200 (assuming that step S250 is understood as 3D target detection directly based on the first feature map and the second feature map). For simplicity, only the differences between the 3D target detection method 300 and the 3D target detection method 200 are described here, that is, only steps S350 to S370 are described. Those skilled in the art can understand the detailed operation and specific details of steps S310 to S340 in conjunction with the foregoing description, and this part will not be repeated here.

[0058] In embodiments of this application, in steps S350 to S370, by projecting the two-dimensional first feature map and the second feature map into a three-dimensional space and then into a bird's-eye view space, their respective bird's-eye view features are obtained. These bird's-eye view features are then fused, allowing three-dimensional object detection to be performed based on the fused bird's-eye view features. In one embodiment of this application, projecting the first feature map and the second feature map into the three-dimensional space in step S350 may include: directly projecting the first feature map and the second feature map into the three-dimensional space to obtain a first frustum feature and a second frustum feature, respectively; and then projecting the first frustum feature and the second frustum feature into the bird's-eye view space to obtain a first bird's-eye view feature and a second bird's-eye view feature. Compared to three-dimensional object detection directly based on the first feature map and the second feature map, the calculation of three-dimensional object detection based on the fused bird's-eye view features is simpler, and because the features are further fused in the bird's-eye view space, it is more conducive to obtaining higher accuracy object detection results.

[0059] Therefore, the three-dimensional target detection method 300 according to the embodiments of this application generates a point cloud pseudo-image by projecting the point cloud acquired by the point cloud acquisition device onto the view of the image acquisition device. Shallow features and deep features are extracted from the point cloud pseudo-image and the image captured by the image acquisition device, respectively. The shallow features of the point cloud pseudo-image and the image captured by the image acquisition device are fused together and then fused with their respective deep features to obtain the final feature extraction result. The obtained features are then projected into three-dimensional space and then into bird's-eye view space to obtain their respective bird's-eye view features, and feature fusion is performed again. The fused features are used for three-dimensional target detection. This makes the three-dimensional target detection process not only dependent on multimodal shallow features but also on multimodal deep information. This allows the three-dimensional target detection to combine rich texture and color information and high-level semantic information, and performs multi-level feature fusion, which greatly improves the accuracy of the three-dimensional target detection result.

[0060] In another embodiment of this application, step S350, projecting the first feature map and the second feature map into three-dimensional space, may include: performing depth estimation on the first feature map and the second feature map respectively to obtain depth information corresponding to each point on the first feature map and the second feature map; projecting the first feature map into three-dimensional space based on the depth information corresponding to each point on the first feature map to obtain a first frustum feature; and projecting the second feature map into three-dimensional space based on the depth information corresponding to each point on the second feature map to obtain a second frustum feature; wherein the first frustum feature and the second frustum feature are projected onto a bird's-eye view space respectively to obtain a first bird's-eye view feature and a second bird's-eye view feature. In this embodiment, instead of directly projecting the first feature map and the second feature map into three-dimensional space, depth estimation is performed on them first, and then projection is performed based on the depth information. Compared to direct projection, projection based on depth information can obtain a more accurate projection result. The following is in conjunction with... Figure 4 To describe.

[0061] Figure 4 A three-dimensional target detection method 400 according to another embodiment of this application is shown. Figure 4 As shown, the three-dimensional target detection method 400 may include the following steps:

[0062] In step S410, a set of point clouds acquired by the point cloud acquisition device for the current scene and a first image acquired by the image acquisition device for the current scene is obtained.

[0063] In step S420, the point cloud is projected onto the view of the image acquisition device to obtain a set of second images, and each second image in the set of second images corresponds to a first image in the set of first images.

[0064] In step S430, feature extraction is performed on the first image and its corresponding second image respectively to obtain the first deep feature of the first image and the second deep feature of the second image. During the feature extraction process, the first shallow feature of the first image and the second shallow feature of the second image are obtained. The first shallow feature and the second shallow feature are fused to obtain the fused shallow feature.

[0065] In step S440, a first feature map of the first image is obtained based on the fused shallow features and the first deep features, and a second feature map of the second image is obtained based on the fused shallow features and the second deep features.

[0066] In step S450, depth estimation is performed on the first feature map and the second feature map respectively to obtain the depth information corresponding to each point on the first feature map and the second feature map.

[0067] In step S460, the first feature map is projected into three-dimensional space based on the depth information corresponding to each point on the first feature map to obtain the first frustum feature, and the second feature map is projected into three-dimensional space based on the depth information corresponding to each point on the second feature map to obtain the second frustum feature.

[0068] In step S470, the first view cone feature and the second view cone feature are projected onto the bird's-eye view space to obtain the first bird's-eye view feature and the second bird's-eye view feature.

[0069] In step S480, the first bird's-eye view feature and the second bird's-eye view feature are fused to obtain the fused bird's-eye view feature.

[0070] In step S490, three-dimensional target detection is performed based on the fused bird's-eye view features of each first image and its corresponding second image, and the three-dimensional target detection result is output.

[0071] The three-dimensional target detection method 400 according to the embodiments of this application is generally similar to the three-dimensional target detection method 300 according to the embodiments of this application described above, with only some differences. Specifically, steps S410 to S440 of the three-dimensional target detection method 400 are the same as steps S310 to S340 of the three-dimensional target detection method 300, and steps S480 to S490 of the three-dimensional target detection method 400 are the same as steps S360 to S370 of the three-dimensional target detection method 300. Steps S450 to S460 of the three-dimensional target detection method 400 can be understood as an implementation of step S350 of the three-dimensional target detection method 300 (assuming that step S350 is understood as projecting the first feature map and the second feature map onto three-dimensional space respectively), or they can be understood as a different implementation from step S350 of the three-dimensional target detection method 300 (assuming that step S350 is understood as directly projecting the first feature map and the second feature map onto three-dimensional space respectively). For the sake of brevity, only the differences between the 3D target detection method 400 and the 3D target detection method 300 are described here, that is, only steps S450 to S460 are described. Those skilled in the art can understand the detailed operation and specific details of steps S410 to S440 and steps S480 to S490 in conjunction with the foregoing description, and these parts will not be repeated here.

[0072] In the embodiments of this application, in steps S450 to S460, depth information corresponding to each point on the feature map is obtained by performing depth estimation on the first feature map and the second feature map. Then, based on this depth information, the first feature map and the second feature map are projected into three-dimensional space. Compared with projection without combining depth information, more accurate frustum features can be obtained for each of the first and second feature maps. Thus, after the frustum features are projected into the bird's-eye view space, more accurate bird's-eye view features are obtained accordingly. In this way, the three-dimensional target detection based on the fused bird's-eye view features will also obtain more accurate detection results.

[0073] In embodiments of this application, the depth estimation of the first feature map and the second feature map in step S450 may include: performing depth estimation of the first feature map and the second feature map based on the intrinsic parameters of the image acquisition device; or, performing depth estimation of the first feature map and the second feature map based on the intrinsic parameters of the image acquisition device and the second image. In this embodiment, depth estimation can be performed based on the intrinsic parameters of the image acquisition device, or it can be performed by combining the intrinsic parameters of the image acquisition device and the second image (i.e., the point cloud pseudo-image). Unlike the first image, which stores RGB information in its three channels, the second image stores the actual depth information of the point cloud, the reflection information of the point cloud acquisition device, and the timestamp information in its three channels, respectively. Therefore, the second image contains three-dimensional geometric information. Compared to depth estimation based solely on the intrinsic parameters of the image acquisition device, depth estimation by combining the second image and the intrinsic parameters of the image acquisition device can improve the accuracy of depth estimation by 40%. Exemplarily, depth estimation can be implemented by a network composed of one or more convolutional and fully connected operators.

[0074] Based on the depth information obtained from depth estimation, the view frustum feature can be obtained by taking the outer product of the depth information and the feature map. Specifically, the depth information of the first feature map can be outer-productted with the first feature map to obtain the first view frustum feature; the depth information of the second feature map can be outer-productted with the second feature map to obtain the second view frustum feature. The first and second view frustum features are then projected onto the bird's-eye view space to obtain the first and second bird's-eye view features, respectively. After the two are fused, 3D object detection is performed, as described above. For example, 3D object detection here can be implemented by a CenterPoint Head network.

[0075] Based on the above description, the target detection algorithm 400 according to the embodiments of this application generates a point cloud pseudo-image by projecting the point cloud acquired by the point cloud acquisition device onto the view of the image acquisition device. Shallow and deep features are extracted from both the point cloud pseudo-image and the image captured by the image acquisition device. The shallow features of the point cloud pseudo-image and the image captured by the image acquisition device are fused together, and then fused with their respective deep features to obtain the final feature extraction result. Then, depth estimation is performed on the obtained feature map. The feature map is then projected into a three-dimensional space and then into a bird's-eye view space to obtain their respective bird's-eye view features, which are then fused again. The fused features are used for three-dimensional target detection. This process relies not only on multimodal shallow features but also on multimodal deep information, enabling three-dimensional target detection to combine rich texture and color information with high-level semantic information. Furthermore, multi-level feature fusion is performed, and the final feature fusion is based on frustum features estimated with depth information, making the frustum features more accurate. This improves the accuracy of the obtained bird's-eye view features, thereby further enhancing the accuracy of the three-dimensional target detection result.

[0076] The following is combined Figure 5 This describes a schematic flowchart of a three-dimensional target detection method 500 according to yet another embodiment of this application. The three-dimensional target detection method 500 is similar to the three-dimensional target detection method described above, except that it presents the three-dimensional target detection process more intuitively in the form of a flowchart. Figure 5 As shown, the image acquisition device acquires a set of first images for the current scene; the point cloud acquisition device acquires a point cloud for the current scene; the point cloud is projected onto the viewpoint of the image acquisition device to generate a set of second images; the first image is processed by feature extraction to output a first deep feature, and a first shallow feature is also output during the feature extraction process; the second image is processed by feature extraction to output a second deep feature, and a second shallow feature is also output during the feature extraction process; the first and second shallow features are fused together, and then fused with the first deep feature to obtain a first feature map, and fused with the second deep feature to obtain a second feature map; the first feature map is depth-estimated, projected into 3D space, and then projected onto the bird's-eye view control to obtain a first bird's-eye view feature; the second feature map is depth-estimated, projected into 3D space, and then projected onto the bird's-eye view control to obtain a second bird's-eye view feature; the first bird's-eye view feature and the second bird's-eye view feature are fused (cross-attention) to perform 3D object detection, and the 3D object detection result is output.

[0077] The above exemplarily illustrates a three-dimensional target detection method according to an embodiment of this application. Based on the above description, the three-dimensional target detection method according to an embodiment of this application generates a point cloud pseudo-image by projecting a point cloud acquired by a point cloud acquisition device onto the viewpoint of an image acquisition device. Shallow and deep features are extracted from both the point cloud pseudo-image and the image captured by the image acquisition device. The shallow features of the point cloud pseudo-image and the image captured by the image acquisition device are fused together, and then further fused with their respective deep features to obtain the final feature extraction result for three-dimensional target detection. This makes the three-dimensional target detection process rely not only on multimodal shallow features (shallow features of the point cloud pseudo-image and shallow features of the image captured by the image acquisition device) but also on multimodal deep information. This allows the three-dimensional target detection to combine rich texture and color information with high-level semantic information, greatly improving the accuracy of the three-dimensional target detection results.

[0078] The following is combined Figures 6 to 9 This application describes a three-dimensional target detection apparatus that can be used to perform the three-dimensional target detection method described above according to the embodiments of this application. The specific operation process of the three-dimensional target detection method has already been described in detail above; therefore, for the sake of brevity, the specific details will not be described again below, only some main operations will be described.

[0079] Figure 6 A schematic structural block diagram of a three-dimensional target detection apparatus 600 according to an embodiment of this application is shown. The three-dimensional target detection apparatus 600 according to this application embodiment can be used to perform the three-dimensional target detection method 200 described above according to an embodiment of this application. The three-dimensional target detection method 200 has been described in detail above; for the sake of brevity, only the structure and main operation of the three-dimensional target detection apparatus 600 are described here, and other details are not repeated.

[0080] like Figure 6As shown, the 3D target detection device 600 may include a point cloud projection module 610, a feature extraction module 620, a shallow feature fusion module 630, and a target detection module 640. Specifically: the point cloud projection module 610 acquires the point cloud collected by the point cloud acquisition device for the current scene and projects the point cloud onto the viewpoint of the image acquisition device to obtain a set of second images. The image acquisition device is used to acquire a set of first images for the current scene, and each second image in the set of second images corresponds to a first image in the set of first images. The feature extraction module 620 performs feature extraction on each first image and its corresponding second image to obtain first deep features of the first image and second deep features of the second image. The shallow feature fusion module 630 performs feature extraction in the feature extraction module 620. During the process, the first shallow features of the first image and the second shallow features of the second image are obtained from the feature extraction module 620. The first shallow features and the second shallow features are fused to obtain the fused shallow features. The feature extraction module 620 is also used to obtain the first feature map of the first image based on the fused shallow features and the first deep features, and to obtain the second feature map of the second image based on the fused shallow features and the second deep features. The target detection module 640 is used to perform three-dimensional target detection based on the first feature map of each first image in the set of first images and the second feature map of each second image in the set of second images, and output the three-dimensional target detection result.

[0081] Figure 7 A schematic structural block diagram of a three-dimensional target detection apparatus 700 according to another embodiment of this application is shown. The three-dimensional target detection apparatus 700 according to this application embodiment can be used to perform the three-dimensional target detection method 300 described above according to this application embodiment. The three-dimensional target detection method 300 has been described in detail above; for brevity, only the structure and main operation of the three-dimensional target detection apparatus 700 are described here, and other details are not repeated.

[0082] like Figure 7As shown, the 3D target detection device 700 may include a point cloud projection module 710, a feature extraction module 720, a shallow feature fusion module 730, a target detection module 740, a spatial projection module 750, and a bird's-eye view feature fusion module 760. Specifically: the point cloud projection module 710 acquires the point cloud collected by the point cloud acquisition device for the current scene and projects the point cloud onto the viewpoint of the image acquisition device to obtain a set of second images. The image acquisition device acquires a set of first images for the current scene, and each second image in the set of second images corresponds to a first image in the set of first images. The feature extraction module 720 extracts features from the first image and its corresponding second image to obtain the first deep features of the first image and the second deep features of the second image. The shallow feature fusion module 730, during the feature extraction process of the feature extraction module 720, acquires the first shallow features of the first image and the second shallow features of the second image from the feature extraction module 720, and integrates the first shallow features... The first and second shallow features are fused to obtain fused shallow features; the feature extraction module 720 is also used to obtain a first feature map of the first image based on the fused shallow features and the first deep features, and to obtain a second feature map of the second image based on the fused shallow features and the second deep features; the spatial projection module 750 is used to project the first feature map and the second feature map into three-dimensional space and then into bird's-eye view space respectively to obtain first bird's-eye view features and second bird's-eye view features; the bird's-eye view feature fusion module 760 is used to fuse the first bird's-eye view features and the second bird's-eye view features to obtain fused bird's-eye view features; the target detection module 740 is used to perform three-dimensional target detection based on the fused bird's-eye view features of each first image and its corresponding second image, and output the three-dimensional target detection results.

[0083] Figure 8 A schematic structural block diagram of a three-dimensional target detection apparatus 800 according to another embodiment of this application is shown. The three-dimensional target detection apparatus 800 according to this application embodiment can be used to perform the three-dimensional target detection method 400 described above according to this application embodiment. The three-dimensional target detection method 400 has been described in detail above; for brevity, only the structure and main operation of the three-dimensional target detection apparatus 800 are described here, and other details will not be repeated.

[0084] like Figure 8As shown, the three-dimensional target detection device 800 may include a point cloud projection module 810, a feature extraction module 820, a shallow feature fusion module 830, a target detection module 840, a spatial projection module 850, a bird's-eye view feature fusion module 860, and a depth estimation module 870. The system comprises: a point cloud projection module 810 for acquiring point clouds collected by a point cloud acquisition device for the current scene and projecting the point clouds onto the viewpoint of an image acquisition device to obtain a set of second images, wherein the image acquisition device is used to acquire a set of first images for the current scene, and each second image in the set of second images corresponds to a first image in the set of first images; a feature extraction module 820 for extracting features from each first image and its corresponding second image to obtain first deep features of the first image and second deep features of the second image; a shallow feature fusion module 830 for acquiring first shallow features of the first image and second shallow features of the second image from the feature extraction module 820 during the feature extraction process, fusing the first shallow features and the second shallow features to obtain fused shallow features; and a feature extraction module 820 for obtaining a first feature map of the first image based on the fused shallow features and the first deep features. The system obtains a second feature map of the second image based on the fused shallow features and the second deep features; the depth estimation module 870 is used to estimate the depth of the first feature map and the second feature map respectively, to obtain the depth information corresponding to each point on the first feature map and the second feature map; the spatial projection module 850 is used to project the first feature map onto the three-dimensional space based on the depth information corresponding to each point on the first feature map to obtain the first frustum feature, and to project the second feature map onto the three-dimensional space based on the depth information corresponding to each point on the second feature map to obtain the second frustum feature, and to project the first frustum feature and the second frustum feature onto the bird's-eye view space respectively, to obtain the first bird's-eye view feature and the second bird's-eye view feature; the bird's-eye view feature fusion module 860 is used to fuse the first bird's-eye view feature and the second bird's-eye view feature to obtain the fused bird's-eye view feature; and the target detection module 840 is used to perform three-dimensional target detection based on the fused bird's-eye view feature of each first image and its corresponding second image, and output the three-dimensional target detection result.

[0085] Figure 9 A schematic block diagram of a three-dimensional target detection apparatus 900 according to yet another embodiment of this application is shown. Figure 9As shown, the three-dimensional target detection device 900 according to an embodiment of this application may include a memory 910 and a processor 920. The memory 910 stores a computer program executed by the processor 920. When the computer program is executed by the processor 920, the processor 920 performs the three-dimensional target detection methods 200, 300, 400, or 500 described above according to embodiments of this application. Those skilled in the art can understand the specific operation of the three-dimensional target detection device 900 according to embodiments of this application in conjunction with the foregoing description. For the sake of brevity, specific details will not be repeated here.

[0086] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When executed by a computer or processor, these program instructions are used to perform corresponding steps of the three-dimensional target detection method of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0087] Furthermore, according to embodiments of this application, a computer program product is also provided, the computer program product including a computer program, which, when run by a processor, causes the processor to execute the three-dimensional target detection method of embodiments of this application.

[0088] According to another aspect of this application, a three-dimensional target detection system is also provided. The following is in conjunction with... Figure 10 To describe. For example... Figure 10 As shown, the 3D target detection system 1000 includes: a point cloud acquisition device 1010 for acquiring point clouds of the current scene; an image acquisition device 1020 for acquiring images of the current scene; and a 3D target detection device 1030 for performing 3D target detection on the current scene based on the point cloud acquired by the point cloud acquisition device 1010 and the image acquired by the image acquisition device 1020. The 3D target detection device 1030 can be any of the 3D target detection devices 600, 700, 800, or 900 described above according to the embodiments of this application.

[0089] Based on the above description, the three-dimensional target detection method, apparatus, and system according to the embodiments of this application generate a point cloud pseudo-image by projecting a point cloud acquired by a point cloud acquisition device onto the viewpoint of an image acquisition device. Shallow and deep features are extracted from both the point cloud pseudo-image and the image captured by the image acquisition device. The shallow features of the point cloud pseudo-image and the image captured by the image acquisition device are fused together, and then further fused with their respective deep features to obtain the final feature extraction result for three-dimensional target detection. This process relies not only on multimodal shallow features (shallow features of the point cloud pseudo-image and shallow features of the image captured by the image acquisition device) but also on multimodal deep information, enabling three-dimensional target detection to combine rich texture and color information with high-level semantic information, greatly improving the accuracy of the three-dimensional target detection results.

[0090] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

Claims

1. A three-dimensional target detection method, characterized in that, The method includes: The set of point cloud data acquired by the point cloud acquisition device for the current scene and the first image acquired by the image acquisition device for the current scene is obtained; The point cloud is projected onto the view of the image acquisition device to obtain a set of second images. Each second image in the set of second images corresponds to a first image in the set of first images. The second image is a pseudo-image of the point cloud. Feature extraction is performed on the first image and its corresponding second image to obtain the first deep feature of the first image and the second deep feature of the second image. During the feature extraction process, the first shallow feature of the first image and the second shallow feature of the second image are obtained. The first shallow feature and the second shallow feature are then fused to obtain the fused shallow feature. The first feature map of the first image is obtained based on the fused shallow features and the first deep features, and the second feature map of the second image is obtained based on the fused shallow features and the second deep features. Three-dimensional object detection is performed based on the first feature map of each first image in the set of first images and the second feature map of each second image in the set of second images, and the three-dimensional object detection result is output.

2. The method according to claim 1, characterized in that, The three-dimensional target detection based on the first feature map of each first image in the set of first images and the second feature map of each second image in the set of second images includes: 3D target detection is performed directly based on the first feature map of each first image and the second feature map of each second image; or The first feature map of each first image and the second feature map of the corresponding second image are projected into a three-dimensional space and then projected into a bird's-eye view space to obtain a first bird's-eye view feature and a second bird's-eye view feature. The first bird's-eye view feature and the second bird's-eye view feature are fused to obtain a fused bird's-eye view feature. Three-dimensional target detection is performed based on the fused bird's-eye view feature of each first image and the corresponding second image.

3. The method according to claim 2, characterized in that, The step of projecting the first feature map of each first image and the second feature map of the corresponding second image into three-dimensional space includes: The first feature map of each first image and the second feature map of the corresponding second image are directly projected into three-dimensional space to obtain the first frustum feature and the second frustum feature, respectively; or Depth estimation is performed on the first feature map of each first image and the second feature map of the corresponding second image to obtain depth information corresponding to each point on the first feature map and the second feature map. Based on the depth information corresponding to each point on the first feature map, the first feature map is projected into three-dimensional space to obtain a first frustum feature. Based on the depth information corresponding to each point on the second feature map, the second feature map is projected into the three-dimensional space to obtain a second frustum feature. The first and second view cone features are projected onto the bird's-eye view space to obtain the first bird's-eye view feature and the second bird's-eye view feature, respectively.

4. The method according to claim 3, characterized in that, The step of performing depth estimation on the first feature map of each first image and the second feature map of the corresponding second image includes: Depth estimation is performed on the first feature map of each first image and the second feature map of the corresponding second image based on the intrinsic parameters of the image acquisition device; or Depth estimation is performed on the first feature map of each first image and the second feature map of the corresponding second image based on the intrinsic parameters of the image acquisition device and the second image.

5. The method according to any one of claims 1-4, characterized in that, Feature extraction of the first image is performed by a first feature extractor, and feature extraction of the second image is performed by a second feature extractor. The first feature extractor and the second feature extractor are the same feature extractor.

6. The method according to any one of claims 1-4, characterized in that, The step of fusing the first shallow feature and the second shallow feature includes: Add or concatenate the first shallow feature and the second shallow feature; or The first shallow feature and the second shallow feature are each passed through one or more convolutional layers and then added together.

7. A three-dimensional target detection device, characterized in that, The device includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the three-dimensional target detection method as described in any one of claims 1-6.

8. A three-dimensional target detection system, characterized in that, The system includes: Point cloud acquisition equipment, used to acquire point clouds for the current scene; Image acquisition device, used to acquire images of the current scene; The three-dimensional target detection device as described in claim 7 is used to perform three-dimensional target detection on the current scene based on the point cloud acquired by the point cloud acquisition device and the image acquired by the image acquisition device.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when run, causes the processor to perform the three-dimensional target detection method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when run by a processor, causes the processor to perform the three-dimensional target detection method as described in any one of claims 1-6.

Images