Three-dimensional target detection method, electronic device, and storage medium

Abstract

The embodiment of the application provides a three-dimensional target detection method and device, the method comprises: acquiring a plurality of target images; inputting the plurality of target images into a three-dimensional target detection network to obtain a three-dimensional detection result, the three-dimensional target detection network is configured to: for each target image, extracting target two-dimensional features of the target image; generating three-dimensional position information corresponding to the target image based on the position of a pixel point in the target image, a candidate depth value and a projection transformation matrix; generating three-dimensional position features corresponding to the target image based on the three-dimensional position information corresponding to the target image; generating features of the target image for generating a three-dimensional detection result based on the target two-dimensional features of the target image and the three-dimensional position features corresponding to the target image; and generating a three-dimensional detection result based on the features of each target image for generating a three-dimensional detection result.

Description

Technical Field

[0001] This application relates to the field of computer vision, specifically to three-dimensional target detection methods, electronic devices, and storage media. Background Technology

[0002] Multi-view vision 3D target detection technology is a computer vision technology that obtains the 3D detection result of the target by acquiring multiple images with different acquisition perspectives.

[0003] Current multi-view vision 3D target detection technology extracts two-dimensional features from each of multiple images with different acquisition perspectives to obtain two-dimensional features (2D features) for each image. The two-dimensional features of each image are then transformed to obtain three-dimensional features (3D features) for each image. These three-dimensional features are used to generate three-dimensional detection results.

[0004] However, the 2D features of images involve a large amount of data. For each image, a significant amount of data needs to be converted, resulting in a high computational load for obtaining the 3D features used to generate the 3D detection results—that is, the 3D features corresponding to each image. This slows down the process of generating the 3D detection results. For intelligent vehicles, such as autonomous vehicles, which have high real-time requirements for environmental perception using multi-view 3D object detection technology, it is difficult to meet the speed requirements for obtaining 3D detection results. Summary of the Invention

[0005] This application provides a three-dimensional target detection method, apparatus, electronic device, and storage medium.

[0006] This application provides a three-dimensional target detection method, including:

[0007] Multiple target images are acquired, each of which is captured by one of a plurality of cameras, and the plurality of cameras have different viewing angles.

[0008] The multiple target images are input into a 3D target detection network to obtain 3D detection results. The 3D target detection network is configured as follows:

[0009] For each target image, extract the target two-dimensional features of the target image; based on the position and candidate depth values ​​of the pixels in the target image and the projection transformation matrix, generate the corresponding three-dimensional position information of the target image, wherein the projection transformation matrix indicates the transformation between the image coordinate system and the world coordinate system; based on the corresponding three-dimensional position information of the target image, generate the corresponding three-dimensional position features of the target image; based on the target two-dimensional features and the corresponding three-dimensional position features of the target image, generate the features of the target image used to generate the three-dimensional detection results;

[0010] The three-dimensional detection result is generated based on the features of each target image used to generate the three-dimensional detection result.

[0011] This application provides a three-dimensional target detection device, including:

[0012] The acquisition unit is configured to acquire multiple target images, each of which is captured by one of a plurality of cameras, and the plurality of cameras have different acquisition angles.

[0013] A detection unit is configured to input the plurality of target images into a 3D target detection network to obtain a 3D detection result. The 3D target detection network is configured to: extract target 2D features for each target image; extract target 2D features for each target image; generate 3D position information corresponding to the target image based on the position and candidate depth values ​​of pixels in the target image and a projection transformation matrix, wherein the projection transformation matrix indicates the transformation between the image coordinate system and the world coordinate system; generate 3D position features corresponding to the target image based on the 3D position information corresponding to the target image; generate features of the target image for generating the 3D detection result based on the target 2D features and the 3D position features corresponding to the target image; and generate the 3D detection result based on the features of each target image for generating the 3D detection result.

[0014] This application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described three-dimensional target detection method.

[0015] This application provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the above-described three-dimensional target detection method.

[0016] This application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the above-described three-dimensional target detection method.

[0017] The 3D target detection method provided in this application generates 3D detection results based on the target image's 2D features and corresponding 3D position features. These features simultaneously express both the target image's characteristics and the positional features of the 3D points corresponding to the pixels within the target image. 3D detection can be performed using these features for each target image to obtain the 3D detection result. For each target image, there is no need to convert the corresponding 2D features into 3D features, allowing for rapid acquisition of 3D features and 3D detection results. This method is suitable for intelligent vehicles with high real-time requirements for environmental perception using multi-view 3D target detection technology. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 A flowchart of the three-dimensional target detection method provided in an embodiment of this application is shown;

[0020] Figure 2 A structural block diagram of the three-dimensional target detection device provided in an embodiment of this application is shown. Detailed Implementation

[0021] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 A flowchart of a three-dimensional target detection method provided in an embodiment of this application is shown. The method includes:

[0024] Step 101: Acquire multiple target images.

[0025] The multiple images targeted in step 101 can be acquired by multiple cameras on a vehicle to which the 3D target detection method provided in this application can be applied.

[0026] Each of the multiple target images targeted in step 101 is captured by one of the multiple cameras, and each of the multiple cameras has a different acquisition angle.

[0027] For example, multiple cameras include: a front camera, a front left camera, a front right camera, a rear left camera, a rear right camera, and a rear camera. At any given moment, all cameras simultaneously acquire images. For each camera, the image acquired at that moment is considered one of the multiple target images targeted in step 101. In other words, the multiple target images targeted in step 101 consist of the target image acquired by the front camera, the front left camera, the front right camera, the rear left camera, the rear right camera, and the rear camera.

[0028] The three-dimensional target detection method provided in this application can be applied to vehicles during driving to perform three-dimensional target detection using multiple target images targeted in step 101, and obtain three-dimensional detection results.

[0029] In some embodiments, multiple cameras are mounted on an autonomous vehicle. The three-dimensional target detection method provided in this application can be applied to autonomous vehicles.

[0030] Step 102: Input multiple target images into the 3D target detection network to obtain 3D detection results.

[0031] In this application, the 3D target detection network is configured to: for each target image, extract the target's two-dimensional features; generate the target image's corresponding 3D position information based on the pixel positions and candidate depth values ​​in the target image and the projection transformation matrix, wherein the projection transformation matrix indicates the transformation between the image coordinate system and the world coordinate system; generate the target image's corresponding 3D position features based on the target image's corresponding 3D position information; generate the target image's features for generating 3D detection results based on the target image's two-dimensional features and the target image's corresponding 3D position features; and generate 3D detection results based on the features of each target image for generating 3D detection results.

[0032] In this application, a point in three-dimensional space is referred to as a three-dimensional point.

[0033] The 3D object detection network in this application includes a network for extracting 2D features. The structure of the network for extracting 2D features can be the same as that of existing backbone networks for extracting 2D features from images, such as ResNet and Swin Transformer.

[0034] For each target image, the target image can be input into a network for extracting two-dimensional features, and the output of the network for extracting two-dimensional features can be used as the target two-dimensional features of the target image.

[0035] For each target image, the target image can also be input into a network for extracting two-dimensional features. The output of the network for extracting two-dimensional features is input into a convolutional layer for generating target two-dimensional features. The convolutional layer for generating target two-dimensional features outputs the target two-dimensional features of the target image. The convolutional layer for generating target two-dimensional features is a 1x1 convolutional layer, that is, the size of the convolutional kernel in the first convolutional layer is 1x1.

[0036] For each target image, the target image's two-dimensional features include multiple channels, and the channels among these multiple channels are feature maps.

[0037] In this application, for each target image, the three-dimensional position information corresponding to the target image is generated based on the position of the pixels in the target image, the candidate depth value, and the projection transformation matrix.

[0038] Candidate depth values ​​can be obtained by uniformly sampling within a preset depth value range. For example, uniform sampling within a preset depth value range can yield 64 candidate depth values. The left endpoint of the preset depth value range can be the minimum possible depth value corresponding to a pixel, and the right endpoint can be the maximum possible depth value corresponding to a pixel.

[0039] In this application, for each target image, the corresponding three-dimensional position information can be composed of the three-dimensional position information corresponding to each pixel in the target image. For each pixel in the target image, the three-dimensional position information corresponding to the pixel includes the coordinates of the position of each three-dimensional point corresponding to the pixel. Each three-dimensional point corresponding to the pixel corresponds to a candidate depth value among all sampled candidate depths. The candidate depth values ​​corresponding to each three-dimensional point of the pixel are different. For each three-dimensional point corresponding to the pixel, the pixel is projected into three-dimensional space through a projection transformation matrix and the candidate depth value corresponding to the three-dimensional point.

[0040] For a pixel, its coordinates and candidate depth value are denoted as (uj, vj, dj), where uj represents the u-axis coordinate of the pixel (i.e., the u-axis coordinate in the image coordinate system), vj represents the v-axis coordinate of the pixel (i.e., the v-axis coordinate in the image coordinate system), and dj is the depth value of the pixel. The coordinates of the corresponding 3D point are denoted as (xj, yj, zj), where xj represents the x-axis coordinate of the corresponding 3D point in the world coordinate system, yj represents the y-axis coordinate of the corresponding 3D point in the world coordinate system, and zj represents the z-axis coordinate of the corresponding 3D point in the world coordinate system. The projection transformation matrix is ​​denoted as T. Based on the pixel's position, candidate depth value, and projection transformation matrix, the coordinates of the corresponding 3D point can be expressed as:

[0041]

[0042] If the image data of the target image includes the depth value corresponding to each pixel in the target image (e.g., the camera capturing the target image is an RGB-D camera), then the depth value corresponding to each pixel in the target image can be directly obtained from the image data. If the image data of the target image does not include the depth value corresponding to each pixel in the target image, then an existing neural network for predicting the depth value corresponding to each pixel in the target image can be used to predict the depth value corresponding to each pixel in the target image.

[0043] In some embodiments, for each target image, the three-dimensional position information corresponding to the target image includes: the coordinates of the positions of the three-dimensional points corresponding to the grid pixels in the grid image of the target image; and for each target image, generating the three-dimensional position information corresponding to the target image based on the positions of the pixels in the target image, candidate depth values, and a projection transformation matrix includes: generating multiple grid images of the target image, wherein, for each of the multiple grid images, the grid pixels in the grid image correspond one-to-one with the pixels in the target image, the position of the grid pixel is the same as the position of the pixel corresponding to the grid pixel in the target image, each grid image corresponds to a candidate depth value, and the depth value corresponding to each grid pixel in the grid image is the candidate depth value corresponding to the grid image; for each of the multiple grid images, based on the projection transformation matrix and the candidate depth value corresponding to each grid pixel in the grid image, the position of each grid pixel in the grid image is transformed to obtain the position of the three-dimensional point corresponding to each grid pixel.

[0044] In this application, for each target image, the corresponding three-dimensional position information can be composed of a set of coordinate values ​​of each grid image of the target image.

[0045] For each target image, the number of grid images in the target image is equal to the number of candidate depth values. Each grid image in the target image corresponds to one of the multiple candidate depth values, and the candidate depth values ​​corresponding to each grid image in the target image are different.

[0046] Multiple candidate depth values ​​can be obtained by uniformly sampling within a preset depth value range. For example, uniform sampling within a preset depth value range can yield 64 candidate depth values. The left endpoint of the preset depth value range can be the minimum possible depth value corresponding to a pixel, and the right endpoint can be the maximum possible depth value corresponding to a pixel.

[0047] In this application, for each grid image of a target image, the size of the grid image is the same as the size of the target image.

[0048] In this application, for each grid image of a target image, the grid pixels in the grid image correspond one-to-one with the pixels in the target image, and for each grid pixel in the grid image, the position of the grid pixel is the same as the position of a corresponding pixel in the target image.

[0049] In this application, for each grid image of a target image, the depth value corresponding to each grid pixel in the grid image is a candidate depth value corresponding to the grid image.

[0050] In this application, for any grid image, the set of coordinate value maps of the grid image includes: the X-axis coordinate value map of the grid image, the Y-axis coordinate value map of the grid image, and the Z-axis coordinate value map of the grid image.

[0051] For any given grid image, the X-axis coordinate map of the grid image includes multiple X-axis coordinate map pixels. For each X-axis coordinate map pixel in the grid image, the X-axis coordinate map pixel corresponds to a grid pixel point in the grid image. The position of the X-axis coordinate map pixel is the same as the position of the corresponding grid pixel point in the grid image. The pixel value of the X-axis coordinate map pixel is the X-axis coordinate value in the world coordinate system of the coordinates of the position of the three-dimensional point corresponding to the X-axis coordinate map pixel point in the grid image.

[0052] For any given grid image, the Y-axis coordinate map of the grid image includes multiple Y-axis coordinate map pixels. For each Y-axis coordinate map pixel in the grid image, the Y-axis coordinate map pixel corresponds to a grid pixel point in the grid image. The position of the Y-axis coordinate map pixel is the same as the position of the corresponding grid pixel point in the grid image. The pixel value of the Y-axis coordinate map pixel is the Y-axis coordinate value in the world coordinate system of the coordinates of the position of the three-dimensional point corresponding to the Y-axis coordinate map pixel point in the grid image.

[0053] For any given grid image, the Z-axis coordinate map of the grid image includes multiple Z-axis coordinate map pixels. For each Z-axis coordinate map pixel in the grid image, the Z-axis coordinate map pixel corresponds to a grid pixel point in the grid image. The position of the Z-axis coordinate map pixel is the same as the position of the corresponding grid pixel point in the grid image. The pixel value of the Z-axis coordinate map pixel is the Z-axis coordinate value in the world coordinate system of the coordinates of the position of the three-dimensional point corresponding to the Z-axis coordinate map pixel point in the grid image.

[0054] In this application, for each target image, a three-dimensional position feature corresponding to the target image is generated based on the three-dimensional position information corresponding to the target image.

[0055] For each target image, when generating the three-dimensional position features corresponding to the target image based on the three-dimensional position information corresponding to the target image, the three-dimensional position information corresponding to the target image can be input into the convolutional layer used to generate the three-dimensional position features, and the convolutional layer used to generate the three-dimensional position features outputs the three-dimensional position features corresponding to the target image. The convolutional layer used to generate the three-dimensional position features can be a 1x1 convolutional layer.

[0056] In this application, for each target image, the three-dimensional position features corresponding to the target image include multiple channels, and the channels in the three-dimensional position features corresponding to the target image are feature maps.

[0057] In this application, for each target image, the number of channels in the three-dimensional position features corresponding to the target image is equal to the number of channels in the target two-dimensional features of the target image.

[0058] In some embodiments, for each target image, generating the three-dimensional position features corresponding to the target image based on the three-dimensional position information corresponding to the target image includes: convolving the three-dimensional position information corresponding to the target image using a first convolutional layer to obtain the features of the three-dimensional position information corresponding to the target image, wherein the first convolutional layer is a 1x1 convolutional layer; performing activation processing on the features of the three-dimensional position information corresponding to the target image to obtain the activated features of the three-dimensional position information corresponding to the target image; and convolving the activated features of the three-dimensional position information corresponding to the target image using a second convolutional layer to obtain the three-dimensional position features corresponding to the target image, wherein the second convolutional layer is a 1x1 convolutional layer.

[0059] For each target image, when the three-dimensional position information corresponding to the target image is convolved using the first convolutional layer to obtain the feature of the three-dimensional position information corresponding to the target image, the three-dimensional position information corresponding to the target image is input into the first convolutional layer, and the first convolutional layer outputs the feature of the three-dimensional position information corresponding to the target image.

[0060] For each target image, when the features of the three-dimensional position information corresponding to the target image are activated to obtain the activated features of the three-dimensional position information corresponding to the target image, the features of the three-dimensional position information corresponding to the target image can be input into the activation layer using the ReLU activation function to obtain the activated features of the three-dimensional position information corresponding to the target image output by the activation layer using the ReLU activation function.

[0061] For each target image, when the activated features of the three-dimensional position information corresponding to the target image are convolved using the second convolutional layer to obtain the three-dimensional position features corresponding to the target image, the activated features of the three-dimensional position information corresponding to the target image are input into the second convolutional layer to obtain the three-dimensional position features corresponding to the target image output by the second convolutional layer.

[0062] In this application, for each target image, features for generating three-dimensional detection results are generated based on the target two-dimensional features of the target image and the corresponding three-dimensional position features of the target image.

[0063] For each target image, when generating features for generating 3D detection results based on the target image's 2D features and corresponding 3D position features, the features for generating 3D detection results can be determined according to the target image's 2D features, corresponding 3D position features, and prior relationships. Prior relationships indicate the relationship between the given image's 2D features, corresponding 3D position features, and features for generating 3D detection results.

[0064] In some embodiments, for each target image, generating features for generating a three-dimensional detection result for the target image based on the target two-dimensional features of the target image and the corresponding three-dimensional position features of the target image includes: adding the target two-dimensional features of the target image and the corresponding three-dimensional position features of the target image to obtain the features for generating a three-dimensional detection result for the target image.

[0065] In this application, for each target image, the number of channels in the three-dimensional position features corresponding to the target image is equal to the number of channels in the target two-dimensional features of the target image. The channels in the three-dimensional position features of the target image constitute a feature map, and the channels in the target two-dimensional features of the target image also constitute a feature map. When adding the target two-dimensional features and the target three-dimensional position features of the target image, two channels of the same channel order in both the target two-dimensional features and the target three-dimensional position features can be added. The channel order indicates which channel it is.

[0066] In this application, a 3D detection result can be generated based on the features of each target image used to generate the 3D detection result. When generating the 3D detection result using the features of each target image, the features used to generate the 3D detection result can be concatenated into the input vector of the detector in the 3D target detection network. The features of each target image used to generate the 3D detection result are used as a component in the input vector of the detector in the 3D target detection network. The input vector of the detector is then fed into the detector, and the detector outputs the 3D detection result.

[0067] The 3D detection results include: the detection results for each detected target. The detection results for each detected target may include: the type of the detected target, the coordinates of the center point of the 3D bounding box of the detected target in 3D space along the x-axis of the world coordinate system, the coordinates of the center point of the 3D bounding box of the detected target in 3D space along the y-axis of the world coordinate system, the coordinates of the center point of the 3D bounding box of the detected target in 3D space along the z-axis of the world coordinate system, the length of the 3D bounding box of the detected target, the width of the 3D bounding box of the detected target, the height of the 3D bounding box of the detected target, and the heading angle of the detected target.

[0068] The detected targets include pedestrians, vehicles, and obstacles in the vehicle's environment. The 3D bounding box of the detected target is the bounding box that surrounds the detected target in 3D space.

[0069] The detector structure in the 3D object detection network of this application can be the same as that of existing detectors that perform object detection based on features extracted from the corresponding images, such as the detector in FastR-CNN and the detector in YOLO.

[0070] In some embodiments, generating a three-dimensional detection result based on the features of each target image used to generate the three-dimensional detection result includes: concatenating the features of each target image used to generate the three-dimensional detection result into an input vector of the DETR detector; inputting the input vector of the DETR detector into the DETR detector to obtain the three-dimensional detection result output by the DETR detector.

[0071] The structure of the DETR detector in the 3D target detection network of this application is the same as the structure of the detector in the end-to-end target detection network DETR network.

[0072] When generating a 3D detection result based on the features of each target image, the features of each target image used to generate the 3D detection result are concatenated into the input vector of the DETR detector, and the features of each target image used to generate the 3D detection result are used as a component in the input vector of the DETR detector.

[0073] Please refer to Figure 2 The diagram illustrates the structural block diagram of the three-dimensional target detection device provided in an embodiment of this application. The three-dimensional target detection device includes: an acquisition unit 201 and a detection unit 302.

[0074] The acquisition unit is configured to acquire multiple target images, each of which is captured by one of a plurality of cameras, and the plurality of cameras have different acquisition angles.

[0075] A detection unit is configured to input the plurality of target images into a 3D target detection network to obtain a 3D detection result. The 3D target detection network is configured to: extract target 2D features for each target image; extract target 2D features for each target image; generate 3D position information corresponding to the target image based on the position and candidate depth values ​​of pixels in the target image and a projection transformation matrix, wherein the projection transformation matrix indicates the transformation between the image coordinate system and the world coordinate system; generate 3D position features corresponding to the target image based on the 3D position information corresponding to the target image; generate features of the target image for generating the 3D detection result based on the target 2D features and the 3D position features corresponding to the target image; and generate the 3D detection result based on the features of each target image for generating the 3D detection result.

[0076] In some embodiments, for each target image, the three-dimensional position information corresponding to the target image includes: the coordinates of the positions of the three-dimensional points corresponding to the grid pixels in the grid image of the target image; and for each target image, generating the three-dimensional position information corresponding to the target image based on the positions of the pixels in the target image, candidate depth values, and a projection transformation matrix includes: generating multiple grid images of the target image, wherein, for each of the multiple grid images, the grid pixels in the grid image correspond one-to-one with the pixels in the target image, the position of the grid pixel is the same as the position of the pixel corresponding to the grid pixel in the target image, each grid image corresponds to a candidate depth value, and the depth value corresponding to each grid pixel in the grid image is the candidate depth value corresponding to the grid image; for each of the multiple grid images, based on the projection transformation matrix and the candidate depth value corresponding to each grid pixel in the grid image, the position of each grid pixel in the grid image is transformed to obtain the position of the three-dimensional point corresponding to each grid pixel.

[0077] In some embodiments, generating three-dimensional position features corresponding to the target image based on the three-dimensional position information corresponding to the target image includes: convolving the three-dimensional position information corresponding to the target image using a first convolutional layer to obtain features of the three-dimensional position information corresponding to the target image, wherein the first convolutional layer is a 1x1 convolutional layer; performing activation processing on the features of the three-dimensional position information corresponding to the target image to obtain activated features of the three-dimensional position information corresponding to the target image; and convolving the activated features of the three-dimensional position information corresponding to the target image using a second convolutional layer to obtain the three-dimensional position features corresponding to the target image, wherein the second convolutional layer is a 1x1 convolutional layer.

[0078] In some embodiments, generating features of the target image for generating a three-dimensional detection result based on the target two-dimensional features of the target image and the corresponding three-dimensional position features of the target image includes: adding the target two-dimensional features of the target image and the corresponding three-dimensional position features of the target image to obtain the features of the target image for generating a three-dimensional detection result.

[0079] In some embodiments, generating the three-dimensional detection result based on the features of each target image used to generate the three-dimensional detection result includes: concatenating the features of each target image used to generate the three-dimensional detection result into an input vector of the DETR detector; inputting the input vector of the DETR detector into the DETR detector to obtain the three-dimensional detection result output by the DETR detector.

[0080] In some embodiments, the plurality of cameras are mounted on an autonomous vehicle.

[0081] This application also provides an electronic device that may be configured with one or more processors; and a memory for storing one or more programs, which may include instructions for performing the operations described in the above embodiments. When the one or more programs are executed by the one or more processors, the one or more processors perform the instructions for performing the operations described in the above embodiments of the three-dimensional target detection method.

[0082] This application also provides a storage medium, which may be included in an electronic device or exist independently, not assembled into an electronic device. The storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the operations described in the embodiments of the three-dimensional target detection method.

[0083] It should be noted that the storage medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, including but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium that includes or stores a program that can be used by or in conjunction with a message execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signaling media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with a message execution system, apparatus, or device. Program code included on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable messages for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer messages.

[0085] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical embodiments formed by specific combinations of the above-described technical features, but should also cover other technical embodiments formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical embodiments formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A three-dimensional target detection method, characterized in that, The method includes: Multiple target images are acquired, each of which is captured by one of a plurality of cameras, and the plurality of cameras have different viewing angles. The multiple target images are input into a 3D target detection network to obtain 3D detection results. The 3D target detection network is configured as follows: For each target image, extract the target two-dimensional features of the target image; based on the position and candidate depth values ​​of the pixels in the target image and the projection transformation matrix, generate the corresponding three-dimensional position information of the target image, wherein the projection transformation matrix indicates the transformation between the image coordinate system and the world coordinate system; based on the corresponding three-dimensional position information of the target image, generate the corresponding three-dimensional position features of the target image; based on the target two-dimensional features and the corresponding three-dimensional position features of the target image, generate features of the target image for generating three-dimensional detection results; the corresponding three-dimensional position information of the target image includes: the coordinates of the three-dimensional points corresponding to the grid pixels in the grid image of the target image; the target two-dimensional features of the target image include multiple channels, the corresponding three-dimensional position features of the target image include multiple channels, and the number of channels in the corresponding three-dimensional position features of the target image is equal to the number of channels in the target two-dimensional features of the target image; The three-dimensional detection result is generated based on the features of each target image used to generate the three-dimensional detection result; Based on the target image's two-dimensional features and the corresponding three-dimensional position features, the features used to generate the three-dimensional detection results for the target image include: The target two-dimensional features and the corresponding three-dimensional position features of the target image are added together to obtain the features of the target image used to generate the three-dimensional detection result; wherein, when adding the target two-dimensional features and the corresponding three-dimensional position features of the target image, two channels of the same channel order in the target two-dimensional features and the corresponding three-dimensional position features of the target image are added together, and the channel order indicates which channel it is.

2. The method according to claim 1, characterized in that, Based on the pixel positions and candidate depth values ​​in the target image, as well as the projection transformation matrix, the three-dimensional position information corresponding to the target image is generated, including: Multiple grid images of the target image are generated, wherein for each of the multiple grid images, the grid pixels in the grid image correspond one-to-one with the pixels in the target image, the position of the grid pixels is the same as the position of the corresponding pixel in the target image, each grid image corresponds to a candidate depth value, and the depth value corresponding to each grid pixel in the grid image is the candidate depth value corresponding to the grid image. For each of the plurality of grid images, based on the projection transformation matrix and the candidate depth value corresponding to each grid pixel in the grid image, the position of each grid pixel in the grid image is transformed to obtain the position of the three-dimensional point corresponding to each grid pixel.

3. The method according to claim 1, characterized in that, Based on the three-dimensional position information corresponding to the target image, generating the three-dimensional position features corresponding to the target image includes: The first convolutional layer is used to convolve the three-dimensional position information corresponding to the target image to obtain the features of the three-dimensional position information corresponding to the target image. The first convolutional layer is a 1x1 convolutional layer. The features of the three-dimensional position information corresponding to the target image are activated to obtain the activated features of the three-dimensional position information corresponding to the target image. The activated features of the three-dimensional position information corresponding to the target image are convolved using the second convolutional layer to obtain the three-dimensional position features corresponding to the target image. The second convolutional layer is a 1x1 convolutional layer.

4. The method according to claim 1, characterized in that, Based on the features of each target image used to generate the 3D detection result, the 3D detection result is generated as follows: The features of each target image used to generate the 3D detection result are concatenated into the input vector of the DETR detector; The input vector of the DETR detector is input into the DETR detector to obtain the three-dimensional detection result output by the DETR detector.

5. The method according to any one of claims 1-4, characterized in that, The multiple cameras are installed on the autonomous vehicle.

6. An electronic device, comprising: A memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the method of any one of claims 1-5.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-5.

Images