3D target detection method and device

Abstract

The embodiment of the invention discloses a 3D target detection method and device. The specific embodiment of the method comprises the steps of performing semantic segmentation on an image of a target scene to obtain a semantic tag of each pixel; converting the point cloud data of the target scene into a point cloud aerial view under an aerial coordinate system; converting the image into an image aerial view under an aerial view coordinate system; for each pixel point in the point cloud aerial view, searching a semantic tag at a position corresponding to the pixel point in the image aerial view; and performing 3D target detection according to the point cloud aerial view and the semantic tag of each pixel point in the point cloud aerial view. According to the embodiment, the accuracy and speed of 3D target detection can be improved, and the calculation complexity is reduced.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to 3D target detection methods and apparatus. Background Technology

[0002] 3D object detection is a crucial module for autonomous driving, as the vehicle uses the detection results to make subsequent obstacle avoidance decisions and path planning. Commonly used sensors include radar, cameras, and millimeter-wave radar. Currently, the perception capabilities of single radar sensors have reached industry-leading levels, achieving excellent detection results on multiple large-scale autonomous driving datasets.

[0003] The current mainstream method uses single radar for detection. However, due to the sparse nature of radar data, it performs poorly in some scenarios. For example, for distant objects, the radar can only detect a very small number of points on the object, or for small objects, the number of points that can be scanned is also relatively small. This is a challenge brought about by the detection task. Monocular vision detection has unique advantages in this regard, so in recent years, researchers have been studying the fusion detection of radar and camera data, which are complementary data types. The fusion of the two modalities faces two major challenges: one is the alignment of sensor data, and the other is the computational power of the model. To solve these two problems, existing methods use predicted image category labels and then fuse them with point clouds for detection. In existing methods, each point in the point cloud needs to be assigned semantic information, which is a significant workload. Furthermore, in the subsequent point cloud voxelization process and conversion to BEV (bird's-eye view), the position of each pixel only takes information from one point. If the semantics of this point are incorrect, it will result in incorrect information for that pixel. Summary of the Invention

[0004] The embodiments of this disclosure present a 3D target detection method and apparatus.

[0005] In a first aspect, embodiments of this disclosure provide a 3D target detection method, comprising: performing semantic segmentation on an image of a target scene to obtain a semantic label for each pixel; converting point cloud data of the target scene into a point cloud bird's-eye view in a bird's-eye coordinate system; converting the image into an image bird's-eye view in a bird's-eye coordinate system; for each pixel in the point cloud bird's-eye view, searching for the semantic label at the corresponding position of the pixel in the image bird's-eye view; and performing 3D target detection based on the point cloud bird's-eye view and the semantic label of each pixel in the point cloud bird's-eye view.

[0006] In some embodiments, converting the image into a bird's-eye view in a bird's-eye coordinate system includes: projecting the point cloud data onto a 2D image coordinate system; obtaining the image depth of the corresponding pixel in the image based on the projection position and depth of each point cloud; and converting the image into a bird's-eye coordinate system based on the image depth of each pixel in the image to obtain a bird's-eye view.

[0007] In some embodiments, obtaining the image depth of a pixel at a corresponding position in the image based on the projection position and depth of each point cloud includes: for each pixel in the image, taking the depth of the point cloud whose projection position is closest to that pixel as the image depth of that pixel.

[0008] In some embodiments, converting point cloud data of a target scene into a point cloud bird's-eye view in a bird's-eye coordinate system includes: discretizing the point cloud data into a grid with uniform intervals on a top-view plane to obtain a columnar set; extracting features from the columnar set and mapping them into a point cloud bird's-eye view.

[0009] In some embodiments, the step of finding the semantic label of the corresponding position of each pixel in the point cloud bird's-eye view in the image bird's-eye view includes: finding the semantic label that appears most frequently at the corresponding position of each pixel in the point cloud bird's-eye view in the image bird's-eye view.

[0010] In some embodiments, the step of performing 3D target detection based on the point cloud bird's-eye view and the semantic labels of each pixel in the point cloud bird's-eye view includes: concatenating the semantic labels of each pixel in the point cloud bird's-eye view as new channel information into the features of the point cloud bird's-eye view to obtain fused features; and inputting the fused features into a 3D target detection model to obtain 3D target detection results.

[0011] Secondly, embodiments of this disclosure provide a 3D target detection apparatus, comprising: a segmentation unit configured to perform semantic segmentation on an image of a target scene to obtain a semantic label for each pixel; a point cloud conversion unit configured to convert point cloud data of the target scene into a point cloud bird's-eye view in a bird's-eye coordinate system; an image conversion unit configured to convert the image into an image bird's-eye view in a bird's-eye coordinate system; a rendering unit configured to, for each pixel in the point cloud bird's-eye view, find the semantic label at the corresponding position of the pixel in the image bird's-eye view; and a detection unit configured to perform 3D target detection based on the point cloud bird's-eye view and the semantic label of each pixel in the point cloud bird's-eye view.

[0012] In some embodiments, the image conversion unit is further configured to: project the point cloud data onto a 2D image coordinate system; obtain the image depth of the corresponding pixel in the image based on the projection position and depth of each point cloud; and convert the image to a bird's-eye view coordinate system based on the image depth of each pixel in the image to obtain a bird's-eye view of the image.

[0013] In some embodiments, the image conversion unit is further configured to: for each pixel in the image, use the depth of the point cloud whose projection location is closest to that pixel as the image depth of that pixel.

[0014] In some embodiments, the point cloud conversion unit is further configured to: discretize the point cloud data into a grid with uniform spacing on a top-view plane to obtain a columnar set; extract features from the columnar set and map them into a point cloud bird's-eye view.

[0015] In some embodiments, the rendering unit is further configured to: for each pixel in the point cloud bird's-eye view, find the semantic tag that appears most frequently at the corresponding position of the pixel in the image bird's-eye view.

[0016] In some embodiments, the detection unit is further configured to: concatenate the semantic label of each pixel in the point cloud bird's-eye view as new channel information into the features of the point cloud bird's-eye view to obtain fused features; and input the fused features into a 3D target detection model to obtain 3D target detection results.

[0017] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors perform the method as described in any one of the first aspects.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first aspects.

[0019] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0020] The 3D object detection method and apparatus provided in this disclosure use a 2D semantic segmenter to obtain semantic results from an image. The point cloud is transformed from the 3D coordinate system to the BEV coordinate system, and the 2D image is also transformed to the BEV coordinate system. The point cloud and the image are overlaid to obtain the semantic information of the point cloud. This semantic information, as additional information of the point cloud in the BEV coordinate system, is added as an extra channel and enters the next convolutional process to complete the subsequent detection task. This disclosure reduces the workload by assigning semantics to the point cloud from the BEV perspective. It also avoids false detections of the point cloud caused by semantic segmentation errors of a pixel in the 2D image, thus improving the accuracy of 3D object detection.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0023] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0024] Figure 2 This is a flowchart of an embodiment of the 3D target detection method according to the present disclosure;

[0025] Figure 3 This is a schematic diagram of an application scenario of the 3D target detection method according to this disclosure;

[0026] Figure 4 This is a schematic diagram of the structure of an embodiment of the 3D target detection device according to the present disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0028] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Figure 1 An exemplary system architecture 100 is shown that can be applied to embodiments of the 3D target detection method or 3D target detection apparatus of this disclosure.

[0031] like Figure 1 As shown, the system architecture 100 may include driverless vehicles (also known as autonomous vehicles) 101 and 102, a network 103, a database server 104, and a server 105. The network 103 serves as the medium for providing communication links between driverless vehicles 101 and 102, the database server 104, and the server 105. The network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0032] Unmanned vehicles 101 and 102 can be equipped with driving control equipment and devices such as LiDAR and millimeter-wave radar to collect point cloud data. They are also equipped with cameras to capture panoramic images. The driving control equipment (also known as the vehicle's brain) is responsible for the intelligent control of the unmanned vehicle. The driving control equipment can be a standalone controller, such as a Programmable Logic Controller (PLC), a microcontroller, or an industrial control computer; it can also be a device composed of other electronic components with input / output ports and computational control functions; or it can be a computer device equipped with vehicle driving control applications.

[0033] It should be noted that in practice, autonomous vehicles can also be equipped with at least one sensor, such as a gravity sensor or wheel speed sensor. In some cases, autonomous vehicles can also be equipped with GNSS (Global Navigation Satellite System) equipment and SINS (Strap-down Inertial Navigation System), etc.

[0034] Database server 104 can be a database server that provides various services. For example, the database server can store sample sets and 3D object detection models. In this way, users can collect environmental images and point cloud data as samples using unmanned vehicles 101 and 102, store the samples in database server 104, and also use the samples to train 3D object detection models.

[0035] Server 105 can also be a server providing various services. It can receive scene images and point cloud data collected by autonomous vehicles and perform 3D object detection based on this data. First, a 2D semantic segmenter is used to obtain the semantic results on the image. Then, the point cloud is transformed from the 3D coordinate system to the BEV coordinate system, and the 2D image is also transformed to the BEV coordinate system. Then, the point cloud and the image are overlaid to obtain the semantic information of the point cloud. Specifically, the label with the most semantic occurrences in a pixel is selected as the semantic label of that pixel and assigned to the point cloud of that pixel. Then, this semantic information is used as additional information in the BEV point cloud as a channel for subsequent processing. For example, if the PointPillar voxelization method is used, after the point cloud is transformed to BEV and the corresponding semantic information is obtained, it will be added as an additional channel and enter the next convolutional processing to complete the subsequent detection task.

[0036] The database servers 104 and 105 here can be either hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here. Database servers 104 and 105 can also be servers in a distributed system, or servers integrated with blockchain. Database servers 104 and 105 can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0037] It should be noted that the 3D target detection method provided in the embodiments of this disclosure is generally executed by server 105, and correspondingly, the 3D target detection device is generally set in server 105.

[0038] It should be noted that if server 105 can perform the relevant functions of database server 104, database server 104 may not be set up in system architecture 100.

[0039] It should be understood that Figure 1 The number of driverless vehicles, networks, database servers, and servers shown is merely illustrative. Depending on implementation needs, any number of driverless vehicles, networks, database servers, and servers can be included.

[0040] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a 3D object detection method according to the present disclosure. The 3D object detection method includes the following steps:

[0041] Step 201: Perform semantic segmentation on the image of the target scene to obtain the semantic label of each pixel.

[0042] In this embodiment, the execution entity of the 3D target detection method (e.g. Figure 1 The server shown can acquire images and point cloud data of the target scene from the autonomous vehicle via wired or wireless connections. The image can be a panoramic image directly captured by the autonomous vehicle, or a panoramic image fused from images captured by multiple cameras, all acquired simultaneously at the same location. A 2D semantic segmenter is used to perform semantic segmentation on the target scene image, obtaining a semantic label for each pixel. The semantic label represents the category of the pixel, such as pedestrian, vehicle, or building.

[0043] Step 202: Convert the point cloud data of the target scene into a point cloud bird's-eye view in a bird's-eye coordinate system.

[0044] In this embodiment, point cloud data of the target scene can be converted into a bird's-eye view of the point cloud in a bird's-eye coordinate system using some existing software tools, such as PointPillar based on BEV view, and VoxelNet that directly processes the original point cloud using 3D sparse convolution.

[0045] In some optional implementations of this embodiment, converting the point cloud data of the target scene into a point cloud bird's-eye view in a bird's-eye coordinate system includes: discretizing the point cloud data into a uniformly spaced grid on a top-view plane to obtain a columnar set; extracting features from the columnar set and mapping them to a point cloud bird's-eye view. On the xy-plane of the top view, a uniform grid of H×W is divided, and the H×W uniform grids are stretched along the Z-axis to form a columnar set. The original point cloud data (point) has four dimensions (x, y, z, r), where r represents reflectivity. The data dimensions are expanded. PointNet is used to extract the columnar features. PointNet consists of a BatchNorm layer, a ReLU layer, and a fully connected layer. Finally, the tensor obtained by max pooling is exactly the dimension of the image.

[0046] Step 203: Convert the image into a bird's-eye view in a bird's-eye coordinate system.

[0047] In this embodiment, the image from each camera is projected onto a pre-set BEV grid according to intrinsic and extrinsic parameters, and then spatial and temporal features are fused using a transformer. The BEV feature map is output. Simply put, each point in the pre-set BEV grid is used as a query, 3D projected onto the 2D position of each image feature, and then deformable attention is used to learn and sample some important point features in the vicinity, obtaining attention weights and the features at that point in the BEV grid.

[0048] In some optional implementations of this embodiment, converting the image into a bird's-eye view in a bird's-eye coordinate system includes: projecting the point cloud data onto a 2D image coordinate system; obtaining the image depth of the corresponding pixel in the image based on the projection position and depth of each point cloud; and converting the image into a bird's-eye coordinate system based on the image depth of each pixel in the image to obtain a bird's-eye view.

[0049] The point cloud coordinate system is projected onto the 2D image coordinate system using calibration information. Based on the position of the point cloud falling on the pixel, the image depth z corresponding to that position is obtained. c After obtaining the depth of the image, this depth is used to transform the image to the BEV coordinate system. Specifically:

[0050]

[0051] P x ,P y ,P z U, V, Z represent the coordinates of the point cloud, K represents the camera's intrinsic parameters, and R and T represent the translation and rotation transformation matrices from the point cloud coordinate system to the image coordinate system, respectively. These two transformation matrices are determined based on the calibration extrinsic parameters of the camera and the point cloud. c These represent the coordinate positions after the image is projected. By mapping 2D images to point cloud data, depth information can be assigned to pixels in 2D images that do not inherently possess depth information, thus solving the problem that 2D images cannot be converted into bird's-eye views due to the lack of depth information.

[0052] In some optional implementations of this embodiment, obtaining the image depth of a pixel at a corresponding position in the image based on the projection position and depth of each point cloud includes: for each pixel in the image, taking the depth of the point cloud whose projection position is closest to that pixel as the image depth of that pixel. Because point clouds are sparse and images are dense, it is impossible to obtain a one-to-one correspondence between pixels and point clouds. In this case, clustering methods or the idea of ​​nearest neighbors can be used to select the depth of the point cloud closest to that pixel as the depth of that pixel. This method is more accurate than calculating the average depth of multiple point clouds as the pixel depth, avoiding false detections of point clouds caused by incorrect depth of a single pixel in a 2D image, and improving the accuracy of 3D object detection.

[0053] Step 204: For each pixel in the point cloud bird's-eye view, find the semantic label of the corresponding position of the pixel in the image bird's-eye view.

[0054] In this embodiment, for each pixel in the point cloud BEV view, the corresponding semantic label is found in the image BEV view. For example, to determine the semantic label at (1,1) in the point cloud BEV view, it is necessary to find the semantic label at (1,1) in the image BEV view. If the semantic label at (1,1) in the image BEV view is unique, it is directly assigned to (1,1) in the point cloud BEV view. This process is also called point cloud rendering.

[0055] In some optional implementations of this embodiment, for each pixel in the point cloud bird's-eye view, the semantic label that appears most frequently at the corresponding position of that pixel is searched in the image bird's-eye view. Theoretically, the image BEV view is relatively large and has not undergone downsampling processing, retaining the original semantic label information. The point cloud BEV is smaller than the image BEV, so a one-to-many phenomenon may occur when searching for image labels in the point cloud. In this case, the semantic label that appears most frequently in that pixel is selected as the semantic label of the point cloud. This improves the accuracy of semantic labels, avoids false detections of the point cloud caused by semantic segmentation errors of a certain pixel in the 2D image, and improves the accuracy of 3D object detection.

[0056] Step 205: Perform 3D target detection based on the point cloud bird's-eye view and the semantic label of each pixel in the point cloud bird's-eye view.

[0057] In this embodiment, the 3D object detection model is an extension of the existing model that is based solely on point cloud bird's-eye view detection. The model's input has been increased by one dimension of features (semantic labels), and the output 3D object detection result is still the 3D position of the target object.

[0058] In some optional implementations of this embodiment, the step of performing 3D target detection based on the point cloud bird's-eye view and the semantic label of each pixel in the point cloud bird's-eye view includes: stitching the semantic label of each pixel in the point cloud bird's-eye view as new channel information into the features of the point cloud bird's-eye view to obtain fused features; and inputting the fused features into a 3D target detection model to obtain 3D target detection results.

[0059] In the process of generating a point cloud BEV view using the PointPillar method described above, the point cloud data P... i The dimension was expanded from the original 4 dimensions (x) i ,y i ,z i ,r i It has been expanded to 8 dimensions:

[0060] P i =(x i ,y i ,z i ,ri ,C,x c ,y c ,z c )

[0061] x i ,y i ,z i This represents the original information of the point cloud, r i The reflectance of the point cloud is represented by x, where C represents the category (semantic label) to which the point belongs, and x represents the reflectance of the point cloud. c ,y c ,z c This indicates the location of the center point of the instance to which this point belongs.

[0062] The feature fusion method described above improves the comprehensiveness of features, thereby enhancing the accuracy of 3D target detection.

[0063] The method provided in the above embodiments of this disclosure assigns semantics to point clouds from a BEV perspective, reducing workload. It also avoids false detections of point clouds caused by semantic segmentation errors of a pixel in a 2D image, thus improving the accuracy of 3D object detection.

[0064] See also Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of the 3D target detection method according to this embodiment. Figure 3 In this application scenario, a 2D image semantic segmentation model is first used to process the image. Then, the image is projected onto the BEV (Browser Active Volume) view using the calibration coefficients of the camera and radar. Next, the point cloud is voxelized using the PointPillar method to obtain the BEV view of the point cloud. For each pixel in the BEV view, the corresponding semantic label from the image BEV view is found (theoretically, the image BEV view is larger and has not undergone downsampling, preserving the original semantic label information). Since the point cloud BEV is smaller than the image BEV, a one-to-many phenomenon occurs when searching for image labels in the point cloud. In this case, the semantic label that appears most frequently in that pixel is selected as the semantic label of the point cloud. Then, the label information is used as new feature information of the point cloud and fused with it, i.e., an additional one-dimensional channel carries the label information, which is then used in subsequent convolutional processing and subsequent detection tasks.

[0065] The specific steps are as follows:

[0066] 1. Semantic Segmentation: First, the panoramic images from different angles are fused to obtain a 2D image of the target scene. For the 2D image, any 2D semantic segmentation model is used to perform semantic segmentation on the image. The model input is the RGB information of the image, and the output is the instance category of each pixel.

[0067] 2. Coordinate Transformation: Project the point cloud coordinate system onto the 2D image coordinate system using calibration information. Based on the position of the point cloud at a pixel, obtain the image depth corresponding to that position. Because the point cloud is sparse and the image is dense, a one-to-one correspondence between pixels and point clouds cannot be obtained. In this case, we can use clustering methods or the nearest neighbor approach, selecting the depth of the point cloud closest to the pixel as the depth z of that pixel. c After obtaining the depth of the image, this depth is used to transform the image into the BEV coordinate system, resulting in a BEV view of the image. Specifically:

[0068]

[0069] P x ,P y ,P z U, V, Z represent the coordinates of the point cloud, K represents the camera's intrinsic parameters, and R and T represent the translation and rotation transformation matrices from the point cloud coordinate system to the image coordinate system, respectively. These two transformation matrices are determined based on the calibration extrinsic parameters of the camera and the point cloud. c These represent the coordinate positions of the projected image.

[0070] 3. Generate point cloud BEV view: The point cloud is voxelized using a convolutional neural network, and the BEV view of the point cloud is obtained by referring to the PointPillar method.

[0071] P i =(x i ,y i ,z i ,r i ,C,x c ,y c ,z c )

[0072] x i ,y i ,z i This represents the original information of the point cloud, r i Let x represent the reflectance of the point cloud, C represent the category to which the point belongs, and x represent the reflectance of the point cloud. c ,y c ,z c This indicates the location of the center point of the instance to which this point belongs.

[0073] 4. Point Cloud Rendering: Based on steps 2 and 3, we obtain the image BEV view and the point cloud BEV view. Since the image BEV view contains the raw information of the semantic labels, its resolution is higher than that of the point cloud BEV view. To obtain the semantic meaning corresponding to each pixel in the point cloud, the semantic label that appears most frequently in that pixel is selected as the semantic label for that point cloud pixel. The new label is used as new channel information for the point cloud BEV and fed into subsequent convolutions and subsequent detection tasks.

[0074] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a 3D target detection device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0075] like Figure 4 As shown, the 3D target detection device 400 of this embodiment includes: a segmentation unit 401, a point cloud conversion unit 402, an image conversion unit 403, a rendering unit 404, and a detection unit 405. The segmentation unit 401 is configured to perform semantic segmentation on an image of a target scene to obtain a semantic label for each pixel; the point cloud conversion unit 402 is configured to convert the point cloud data of the target scene into a point cloud bird's-eye view in a bird's-eye coordinate system; the image conversion unit 403 is configured to convert the image into an image bird's-eye view in a bird's-eye coordinate system; the rendering unit 404 is configured to, for each pixel in the point cloud bird's-eye view, find the semantic label corresponding to that pixel in the image bird's-eye view; and the detection unit 405 is configured to perform 3D target detection based on the point cloud bird's-eye view and the semantic labels of each pixel in the point cloud bird's-eye view.

[0076] In this embodiment, the specific processing of the segmentation unit 401, point cloud conversion unit 402, image conversion unit 403, rendering unit 404, and detection unit 405 of the 3D target detection device 400 can be referred to Figure 2 The corresponding steps are 201, 202, 203, 204 and 205 in the embodiment.

[0077] In some optional implementations of this embodiment, the image conversion unit 403 is further configured to: project the point cloud data onto a 2D image coordinate system; obtain the image depth of the corresponding pixel in the image based on the projection position and depth of each point cloud; and convert the image to a bird's-eye view coordinate system based on the image depth of each pixel in the image to obtain a bird's-eye view of the image.

[0078] In some optional implementations of this embodiment, the image conversion unit 403 is further configured to: for each pixel in the image, use the depth of the point cloud whose projection position is closest to the pixel as the image depth of the pixel.

[0079] In some optional implementations of this embodiment, the point cloud conversion unit 402 is further configured to: discretize the point cloud data into a grid with uniform intervals on a top-view plane to obtain a columnar set; extract features from the columnar set and map them into a point cloud bird's-eye view.

[0080] In some optional implementations of this embodiment, the rendering unit 404 is further configured to: for each pixel in the point cloud bird's-eye view, find the semantic tag that appears most frequently at the corresponding position of the pixel in the image bird's-eye view.

[0081] In some optional implementations of this embodiment, the detection unit 405 is further configured to: stitch the semantic label of each pixel in the point cloud bird's-eye view as new channel information into the features of the point cloud bird's-eye view to obtain fused features; and input the fused features into a 3D target detection model to obtain 3D target detection results.

[0082] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0083] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0084] An electronic device includes: one or more processors; and a storage device having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method described in process 200.

[0085] A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in process 200.

[0086] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0087] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0088] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0089] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as road planning methods. For example, in some embodiments, the road planning method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the road planning method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the road planning method by any other suitable means (e.g., by means of firmware).

[0090] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0094] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0095] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be servers in distributed systems or servers incorporating blockchain technology. Servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0096] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A 3D object detection method, comprising: performing semantic segmentation on an image of a target scene to obtain a semantic label of each pixel; converting point cloud data of the target scene into a point cloud bird's eye view in a bird's eye coordinate system; converting the image into an image bird's eye view in the bird's eye coordinate system; for each pixel point in the point cloud bird's eye view, searching for a semantic label of a corresponding position of the pixel point in the image bird's eye view; performing 3D object detection according to the point cloud bird's eye view and the semantic label of each pixel point in the point cloud bird's eye view.

2. The method of claim 1, wherein, The conversion of the image into an image bird's eye view in a bird's eye coordinate system comprises: projecting the point cloud data onto a 2D image coordinate system; obtaining an image depth of a pixel at a corresponding position in the image according to a projection position of each point cloud and a depth of the point cloud; converting the image into an image bird's eye view in the bird's eye coordinate system according to the image depth of each pixel in the image.

3. The method of claim 2, wherein, The obtaining of the image depth of the pixel at the corresponding position in the image according to the projection position of each point cloud and the depth of the point cloud comprises: for each pixel in the image, taking the depth of the point cloud closest to the pixel in the projection position as the image depth of the pixel.

4. The method of claim 1, wherein, The conversion of the point cloud data of the target scene into a point cloud bird's eye view in a bird's eye coordinate system comprises: discretizing the point cloud data into a uniform grid on a top view plane to obtain a columnar set; extracting features from the columnar set and mapping the features into the point cloud bird's eye view.

5. The method of claim 1, wherein, The searching for the semantic label of the corresponding position of the pixel point in the image bird's eye view comprises: for each pixel point in the point cloud bird's eye view, searching for a semantic label of a position appearing most frequently in the image bird's eye view.

6. The method of claim 1, wherein, The 3D object detection according to the point cloud bird's eye view and the semantic label of each pixel point in the point cloud bird's eye view comprises: splicing the semantic label of each pixel point in the point cloud bird's eye view as new channel information into features of the point cloud bird's eye view to obtain fused features; inputting the fused features into a 3D object detection model to obtain a 3D object detection result. 7.A 3D object detection apparatus, comprising: a segmentation unit configured to perform semantic segmentation on an image of a target scene to obtain a semantic label of each pixel; a point cloud conversion unit configured to convert point cloud data of the target scene into a point cloud bird's eye view in a bird's eye coordinate system; an image conversion unit configured to convert the image into an image bird's eye view in the bird's eye coordinate system; a rendering unit configured to, for each pixel point in the point cloud bird's eye view, search for a semantic label of a corresponding position of the pixel point in the image bird's eye view; a detection unit configured to perform 3D object detection according to the point cloud bird's eye view and the semantic label of each pixel point in the point cloud bird's eye view. 8.An electronic device, comprising: one or more processors; a storage device having one or more computer programs stored thereon. When the one or more computer programs are executed by the one or more processors, the one or more processors implement a method as claimed in any of claims 1-6.

9. A computer readable medium having stored thereon a computer program, wherein, The computer program, which when executed by a processor implements a method as claimed in any of claims 1-6.

10. A computer program product comprising a computer program which, when executed by a processor, implements a method according to any of claims 1-6.

Images