Method and apparatus for detecting a target in three-dimensional space

Abstract

The present application relates to a method for detecting a target in a three-dimensional space. The method comprises obtaining a two-dimensional feature representation based on an image captured by an image sensor; selecting, based on the two-dimensional feature representation, a depth prediction network to be used for the image from a set of depth prediction networks, wherein the set of depth prediction networks comprises a plurality of depth prediction networks; determining a depth distribution of the image by the selected depth prediction network; obtaining a three-dimensional feature representation associated with a position in a three-dimensional space corresponding to the image based on the two-dimensional feature representation and the depth distribution; and detecting a target contained in the image based on the three-dimensional feature representation. Other aspects are also disclosed.

Description

Technical Field

[0001] This application relates to computer vision, and more specifically, to methods, apparatus, systems, computer-readable storage media, and computer program products for detecting targets in three-dimensional space. Background Technology

[0002] With the continuous development of computer vision technology, object detection methods have gradually evolved from two-dimensional (2D) space to three-dimensional (3D) space. Compared to 2D object detection, 3D object detection can provide more accurate object information, and therefore has been widely used in many fields, such as autonomous driving, robot navigation, and augmented reality. Typically, 3D object detection can be based on 2D images acquired from single or multiple viewpoints. Therefore, how to link 3D spatial locations to 2D image features has a significant impact on the accuracy of object detection.

[0003] In existing technologies, 2D image features can be converted into 3D image features based on depth estimation of 2D images. Current object detection methods typically apply a fixed distance range, such as 0 to 50 meters, to all images. However, in practical applications, the depth range can vary significantly depending on the object detection scenario. Furthermore, subsequent object prediction is generally based on uniformly sampled location points in 3D space. As the depth range increases, maintaining the original sampling density requires increasing the number of sampling points, which consumes additional computational resources and storage space; conversely, maintaining the original number of sampling points to save computational resources requires increasing the sampling interval, which may lead to erroneous object detection.

[0004] Therefore, we look forward to an efficient and accurate method for detecting targets in three-dimensional space. Summary of the Invention

[0005] The following brief introduction is provided to present some of the selected concepts in a simplified manner, which will be further described in the detailed description that follows. This brief introduction is not intended to highlight the key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] 3D object detection refers to the technology of detecting and recognizing objects in three-dimensional space. Compared with 2D object detection, 3D object detection can provide more accurate target position and pose information, making it suitable for many real-world scenarios such as autonomous driving, robot navigation, and augmented reality. For example, one can first acquire single-view or multi-view images using an image sensor, obtain 2D image features based on the images, obtain a depth estimate based on the 2D image features, and then perform 3D object detection based on this depth estimate.

[0007] In some 3D object detection methods, depth estimation of an image is typically performed based on a single depth prediction network, which may be trained for a fixed distance range, such as within 50 meters. However, in practical applications, the depth range of the target object in an image can vary considerably. For example, in autonomous driving scenarios, when a vehicle is driving on ordinary city roads, it may focus more on targets within a shorter distance, such as within 50 meters; while on highways, it needs to focus on targets at greater distances, such as 50 to 150 meters. Applying a single depth prediction network in this situation can present problems. Traditionally, feature extraction and subsequent object prediction can be based on uniformly sampled location points in 3D space. As the distance range increases, maintaining the original sampling density requires increasing the number of sampling points, which consumes additional computational resources and storage space; or, if the original number of sampling points is maintained to save computational resources, the sampling interval needs to be increased, which may lead to incorrect object detection.

[0008] Therefore, a method for detecting targets in three-dimensional space is desired. This method can provide a set of depth prediction networks, including multiple depth prediction networks. During target detection, the appropriate depth prediction network can be dynamically determined from the set of depth prediction networks based on the features of the 2D image to obtain a depth estimate of the image. Subsequent target detection is then performed based on this depth estimate. The method of this disclosure allows for flexible adaptation to various target detection scenarios, especially when the depth range changes, without excessively increasing computational overhead. The obtained depth information ensures the accuracy of subsequent target detection.

[0009] On one hand, embodiments of this disclosure provide a method for detecting a target in three-dimensional space. The method includes: obtaining a two-dimensional feature representation based on an image; selecting a depth prediction network to be used for the image from a set of depth prediction networks based on the two-dimensional feature representation, wherein the set of depth prediction networks includes multiple depth prediction networks; determining a depth distribution of the image using the selected depth prediction network; and detecting a target contained in the image in the three-dimensional space based at least on the depth distribution.

[0010] On the other hand, embodiments of this disclosure provide a system for detecting targets in three-dimensional space, comprising: a feature extraction module configured to obtain a two-dimensional feature representation based on an image; a network selection module configured to select a depth prediction network to be used for the image from a set of depth prediction networks based on the two-dimensional feature representation, wherein the set of depth prediction networks includes a plurality of depth prediction networks; a depth prediction module configured to determine a depth distribution of the image through the selected depth prediction network; and a target detection module configured to detect targets contained in the image in the three-dimensional space based at least on the depth distribution.

[0011] On the other hand, embodiments of this disclosure provide a system comprising: at least one processor; and a memory coupled to the at least one processor having executable instructions stored thereon, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any embodiment of this disclosure.

[0012] On the other hand, embodiments of this disclosure provide a computer-readable medium storing a computer program including instructions that, when executed by a processor, cause one or more units to perform the method described according to any embodiment of this disclosure.

[0013] On the other hand, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the methods described according to any embodiment of this disclosure. Attached Figure Description

[0014] A further understanding of the nature and advantages of this disclosure can be achieved by referring to the accompanying drawings. In the drawings, similar components or features may have the same reference numerals.

[0015] Figure 1 A schematic diagram of a system for detecting a target in three-dimensional space according to an embodiment of the present disclosure is shown.

[0016] Figure 2 A schematic diagram of a system for detecting a target in three-dimensional space according to an embodiment of the present disclosure is shown.

[0017] Figure 3 A schematic diagram of a feature extraction process according to an embodiment of the present disclosure is shown.

[0018] Figure 4 A schematic diagram of a foreground masking process according to an embodiment of the present disclosure is shown.

[0019] Figure 5A schematic diagram of a network selection process according to an embodiment of the present disclosure is shown.

[0020] Figure 6 A schematic diagram of a depth prediction process according to an embodiment of the present disclosure is shown.

[0021] Figure 7 A schematic diagram of a target detection process according to an embodiment of the present disclosure is shown.

[0022] Figure 8 An exemplary flowchart for detecting a target in three-dimensional space according to an embodiment of the present disclosure is shown.

[0023] Figure 9 An exemplary system diagram according to an embodiment of this disclosure is shown.

[0024] Specific implementation method

[0025] The subject matter described herein will now be discussed with reference to exemplary implementations. It should be understood that these implementations are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as needed in the examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0026] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Unless explicitly indicated by the context, the definition of a term remains consistent throughout the specification.

[0027] It is important to note that applying the methods disclosed herein may involve using user-related information, such as information associated with driving routes and road conditions. It is crucial to remember that the use of information involving user privacy requires user authorization, and its use must not exceed the scope of that authorization.

[0028] With the continuous development of computer vision technology, object detection methods have gradually evolved from two-dimensional (2D) space to three-dimensional (3D) space. Compared with 2D object detection, 3D object detection can provide more accurate object information, and therefore has been widely used in many fields, such as autonomous driving, robot navigation, and augmented reality.

[0029] Figure 1 A schematic diagram of a system for detecting targets in three-dimensional space according to an embodiment of the present disclosure is shown. The various modules provided in this embodiment can be implemented by software and / or hardware, and can be integrated into a computing device capable of executing this embodiment, or distributed among multiple computing devices capable of executing this embodiment. It should be understood that the following examples are only for the purpose of helping to better understand the present disclosure and do not constitute any limitation on the scope of the present disclosure.

[0030] Typically, 3D object detection can be based on 2D images, which can be acquired using single or multiple image sensors at single or multiple viewpoints. Taking autonomous driving scenarios as an example, such as... Figure 1 As shown in 110, multi-view 2D images of the vehicle's surroundings can be acquired using six image sensors mounted on the vehicle body, and used as input to the 3D target detection system 120. For example, the image sensors can be cameras or any image sensor suitable in the art. It is anticipated that the 3D target detection system 120 can output a prediction 130 of 3D targets contained in the images based on features from the 2D images from input 110. For example, the prediction of 3D targets 130 can include predictions of multiple 3D targets in the acquired images. For example, the prediction of each 3D target 130 can include information such as the target's position, type, orientation, etc. For example, the position of the target to be detected can be represented by the coordinates of its bounding box in three-dimensional space. For example, the type of the target to be detected can include vehicles, pedestrians, trees, railings, etc., and additionally, the type of the target to be detected can include lane markings and traffic sign information, such as solid lines, dashed lines, turns, straight ahead, no-stopping, etc. For example, the orientation of the target to be detected can be represented by an angle of deviation relative to various reference points or reference planes. In one embodiment, the prediction 130 can be used for various downstream tasks, such as path planning, automatic parking, collision avoidance, etc. It should be noted that although an autonomous driving scenario is used as the primary example in this description, the 3D object detection method of this disclosure is applicable to various practical applications and is not limited to this example. The specific implementation of the system for detecting objects in three-dimensional space according to this disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Figure 2A schematic diagram of a system for detecting targets in three-dimensional space according to an embodiment of the present disclosure is shown. The various modules provided in this embodiment can be implemented by software and / or hardware, and can be integrated into a computing device capable of executing this embodiment, or distributed among multiple computing devices capable of executing this embodiment. For example, the computing device may be an electronic control unit (ECU) installed in a vehicle, a central processing unit (CPU) installed in a roadside unit, or a central processing unit installed in a cloud server, etc. It should be understood that the following examples are only for the purpose of helping to better understand the present disclosure and do not constitute any limitation on the scope of the present disclosure.

[0032] As described above, one or more image sensors can be used to acquire single-view or multi-view 2D images of the surrounding environment for use as input to a 3D object detection system. Figure 2 As shown in 210, 2D images from three perspectives can be acquired using three image sensors to serve as input to the 3D object detection system 220. It should be noted that other image sensors and the number of images are possible. In one embodiment, the 3D object detection system 220 may include a feature extraction module 201, a foreground masking module 202 (optional), a network selection module 203, a depth prediction module 204, and an object prediction module 205. Figure 2 Although not fully shown, each of these modules can communicate with each other directly or indirectly. As disclosed herein, the 3D object detection system 220 is able to output a prediction 230 of a 3D object based on features from a 2D image from input 210.

[0033] For example, a 2D image can be used as input, and the corresponding two-dimensional feature representation can be obtained through the feature extraction module 201. Combined with... Figure 3 A schematic diagram of a feature extraction process that can be implemented by the feature extraction module 201 is shown. The various steps provided in this embodiment can be implemented by one or more computing-capable devices through software and / or hardware.

[0034] like Figure 3 As shown, 2D images from three perspectives acquired by three image sensors can be used as input 310, and the corresponding two-dimensional feature representations 330 can be obtained through a feature extraction network 320. In one embodiment, the feature extraction network 320 can be implemented based on a convolutional neural network (CNN) model, such as ResNet. Alternatively, the feature extraction network 320 can be implemented based on a transformer model, such as Transformer Swing. Furthermore, the feature extraction network 320 can be implemented based on any image feature extraction method applicable in the art.

[0035] In one embodiment, the two-dimensional feature representation 330 can be represented as a feature map, which may have dimensions of H×W×C×Cam_Num, where H represents the height of the image, W represents the width of the image, C represents the number of feature channels per pixel, and Cam_Num can represent the number of images. In this embodiment, Cam_Num is equal to the number of image sensors, and Cam_Num = 3. Alternatively, the two-dimensional feature representation 330 can be represented as a feature vector, feature matrix, or feature tensor, or other suitable forms.

[0036] In practical applications, the depth range of targets to be detected in an image can vary considerably and changes with time and scene. Furthermore, the depth range of targets may differ in images acquired from different viewpoints. Taking autonomous driving scenarios as an example, when a vehicle is driving on ordinary city roads, it generally focuses on targets that are closer, such as within 50 meters. When the vehicle is driving on a highway, it needs to focus on targets that are farther away, such as 50 to 200 meters. While driving on a highway, most targets to be detected may be far from the vehicle, such as more than 50 meters away, while other vehicles merging into the vehicle may be closer, such as within 10 meters. These merging vehicles may be captured first by the image sensors on the side of the vehicle, and then appear in the image acquired by the image sensor on the front of the vehicle. In this situation, using a single depth prediction network may result in limited detection range and incorrect target detection. Therefore, this disclosure provides a depth prediction network ensemble that includes multiple depth prediction networks. Applicable depth prediction networks can be dynamically selected from the ensemble based on image features to obtain the depth distribution of the image.

[0037] In one embodiment, the depth prediction network set may include multiple depth prediction networks. For example, each depth prediction network may be applicable to different distance ranges: depth prediction network 1 is applicable to 0 to 50 meters, depth prediction network 2 is applicable to 50 to 100 meters, and depth prediction network 3 is applicable to 100 to 200 meters. Alternatively, the distance ranges applicable to each depth prediction network may partially overlap. Furthermore, each depth prediction network may be applicable to different application scenarios. Taking autonomous driving as an example, depth prediction network 1 may be applicable to "urban driving," depth prediction network 2 may be applicable to "suburban driving," and depth prediction network 3 may be applicable to "highway driving."

[0038] return Figure 2 For example, the two-dimensional feature representation output by the feature extraction module 201 can be used as input, and the network selection module 203 can determine the applicable deep prediction network to be selected from the set of deep prediction networks.

[0039] Optionally, the two-dimensional feature representation output by the feature extraction module 201 can be used as input to first identify regions in the image associated with one or more targets to be detected using the foreground mask module 202. Then, the portion of the two-dimensional feature representation and the region associated with one or more targets to be detected can be used as input to determine the appropriate depth prediction network to be selected from the depth prediction network set by the network selection module 203. Since a target detection task will be performed subsequently, selecting a depth prediction network based on the features corresponding to the targets to be detected is more beneficial than selecting features from the entire image. Figure 4 A schematic diagram of a foreground masking process that can be implemented by the foreground masking module 202 is shown. The various steps provided in this embodiment can be implemented by one or more computing-capable devices via software and / or hardware.

[0040] like Figure 4 As shown, a two-dimensional feature representation 410 obtained from an image can be used as input, and a foreground masking network 420 can be used to identify regions in the image associated with one or more targets to be detected, thereby obtaining a foreground mask 430 corresponding to each image. The two-dimensional feature representation 410 can be... Figure 3 An example of the two-dimensional feature representation 330 is shown. In one embodiment, the foreground masking network 420 may be implemented based on a convolutional neural network (CNN) model, such as a fully convolutional (FCN) network. Alternatively, the foreground masking network 420 may be implemented based on any object segmentation method applicable in the art.

[0041] In one embodiment, the foreground mask 430 may include Cam_Num mask matrices corresponding to Cam_Num images, such as Figure 4 As shown, Cam_Num = 3. Each mask matrix can have the same length and width as the image to be applied, such as H×W. Alternatively, the mask matrix can have a smaller length and width than the image to be applied. For example, the mask matrix can be a Boolean matrix, where the matrix elements corresponding to regions where the target to be detected is 1, and the matrix elements corresponding to regions where the target to be detected is 0. Additionally, the foreground mask 430 can be represented in any other suitable form capable of identifying the target region.

[0042] like Figure 4As shown, the foreground mask 430 can be applied to the two-dimensional feature representation 410 to obtain a masked two-dimensional feature representation 440 corresponding to each image. The blank areas in the two-dimensional feature representation 440 represent the feature representation portion corresponding to the image region where the target to be detected exists, and the lined areas represent the feature representation portion corresponding to the image region where the target to be detected does not exist. In one embodiment, applying the foreground mask 430 may include multiplying the mask matrix by the two-dimensional feature representation of the corresponding image.

[0043] Continue to refer to the above text. Figure 2 The description states that the two-dimensional feature representation 330 or the masked two-dimensional feature representation 440 can be used as input to the network selection module 203 to determine the applicable depth prediction network to be selected from the set of depth prediction networks. Combined with... Figure 5 A schematic diagram of a network selection process that can be implemented by the network selection module 203 is shown. The various steps provided in this embodiment can be implemented by one or more computing devices through software and / or hardware.

[0044] like Figure 5 As shown, for example, the masked two-dimensional feature representation 510 can be input into the network selection module 520 to select a depth prediction network. In one embodiment, the network selection module 520 can uniformly determine the depth prediction network to be selected for one or more images in a multi-view image. Alternatively, the network selection module 520 can determine the depth prediction network to be selected for each image separately. For example, the network selection module 520 can output the index of the depth prediction network to be selected. Alternatively, as... Figure 5 As shown, the network selection module 520 can output weights 530 for the depth prediction network to be selected, wherein the weights for the unselected depth prediction network can be set to zero. The masked two-dimensional feature representation 510 can be... Figure 4 An example of a masked two-dimensional feature representation 440 is shown. Alternatively, as described above, two-dimensional feature representations 330 or 410 can also be directly input into the network selection module 520.

[0045] In one embodiment, the network selection module 520 may be implemented based on a convolutional neural network (CNN) model. For example, the network selection module 520 may be implemented based on a structure of convolutional layers, flattening layers, fully connected layers, and activation layers (such as the softmax function). Alternatively, the network selection module 520 may be implemented based on any method applicable in the art.

[0046] In one embodiment, the number of deep prediction networks to be selected from the set of deep prediction networks can be preset. For example... Figure 5As shown, the depth prediction network set contains 3 depth prediction networks. The number of depth prediction networks to be selected can be preset to 2. Therefore, the index and / or weights used for the depth prediction networks to be selected can include 2 valid values. The sum of the valid weight values ​​for a single image is 1, and the weights for unselected depth prediction networks can be set to zero.

[0047] In another embodiment, the number of depth prediction networks to be selected can be determined by the network selection module 520. For example, if the depth prediction network set contains three depth prediction networks, the network selection module 520 can determine, based on the two-dimensional feature representation 510, to select two depth prediction networks for each image. Alternatively, the network selection module 520 can determine the number of depth prediction networks to be selected separately based on the two-dimensional feature representations corresponding to each image. For example, the network selection module 520 can determine to use one depth prediction network for image 1 and two depth prediction networks for images 2 and 3.

[0048] return Figure 2 The depth prediction network selected by the network selection module 203 and the two-dimensional feature representation of each image can be used as inputs, and the depth prediction module 204 can determine the depth distribution of each image. Figure 6 A schematic diagram of a depth prediction process that can be implemented by the depth prediction module 204 is shown. The various steps provided in this embodiment can be implemented by one or more computing-capable devices via software and / or hardware.

[0049] like Figure 6 As shown, the two-dimensional feature representation 620 of an image can be fed into the corresponding depth prediction network based on the depth prediction network index 610 selected for each image to obtain depth distribution prediction. The two-dimensional feature representation 620 can be... Figure 4 and Figure 5 Examples of masked 2D feature representations in the text, such as 440 or 510, or possibly... Figure 3 and Figure 4 The two-dimensional feature representation in the example is 330 or 410.

[0050] As shown in 630, for image 1, a first depth prediction network 640 and a third depth prediction network 660 are selected. Therefore, only the first and third rows in the first column contain valid values ​​corresponding to the two-dimensional feature representation of image 1. For image 2, a second depth prediction network 650 and a third depth prediction network 660 are selected. In the second column, the second and third rows contain valid values ​​corresponding to the two-dimensional feature representation of image 2. Similarly, for image 3, a first depth prediction network 640 and a second depth prediction network 650 are selected. In the third column, the first and second rows contain valid values ​​corresponding to the two-dimensional feature representation of image 3. In a specific implementation, to ensure the order of input and output, all two-dimensional feature representation elements that are not fed into the depth prediction network can be set to zero, as shown in the horizontal line area in 630.

[0051] Subsequently, each depth prediction network can output depth distribution predictions based on the received two-dimensional feature representations. As shown in 670, the first row is the depth distribution predictions for each image output by the first depth prediction network 640, the second row is the depth distribution predictions for each image output by the second depth prediction network 650, and the third row is the depth distribution predictions for each image output by the third depth prediction network 660.

[0052] In one embodiment, the deep prediction network may be implemented based on a fully connected network model or any method applicable in the art.

[0053] As described above, the individual depth prediction networks in the depth prediction network ensemble can be pre-trained for different distance ranges or different application scenarios. In one embodiment, when the depth prediction networks are configured for different distance ranges, the output depth distribution predictions can be normalized according to the distance configuration. For example, if depth prediction network 1 is suitable for 0 to 50 meters, depth prediction network 2 is suitable for 50 to 100 meters, and depth prediction network 3 is suitable for 100 to 200 meters, then the depth distribution predictions output by each depth prediction network can be normalized to [0, 0.25], [0.25, 0.5], and [0.5, 1] ​​respectively according to the distance configuration.

[0054] The method proposed in this disclosure allows for the dynamic determination of an appropriate depth prediction network based on the features of the input image, thereby generating more accurate depth distribution predictions.

[0055] return Figure 2 The depth distribution prediction determined by the depth prediction module 204 can be used as input to detect targets contained in each image in three-dimensional space through the target detection module 205. Alternatively, the two-dimensional feature representation of the image determined by the feature extraction module 202 can also be input to the target detection module 205.

[0056] It should be understood that the object detection module 205 can be implemented based on any object detection algorithm or model applicable in the art, such as the Transformer model or the Mamba model. For example, an object detection module based on a Transformer model can link 3D spatial locations to 2D image features and depth information, thereby enabling direct prediction in 3D space. Furthermore, the number of query vectors can be set to the maximum number of potentially detectable targets, without needing to increase sampling points even if the depth range changes, thus saving computational overhead. Figure 7 A schematic diagram of a target detection process that can be implemented by the target detection module 205 is shown. The various steps provided in this embodiment can be implemented by one or more computing-capable devices through software and / or hardware.

[0057] To detect targets in three-dimensional space, such as Figure 7 As shown, a 3D target query set 710 can be constructed and initialized first, which can include N 3D query vectors, where N can correspond to the maximum number of targets that may be detected in 3D space, such as 100. Each query vector can correspond to a predicted bounding box in 3D space, and the center point of the predicted bounding box can be used as a 3D position reference point corresponding to the query vector.

[0058] In one embodiment, the three-dimensional space can be described by a bird's-eye view (BEV). A BEV view converts two-dimensional planar data into a three-dimensional view from above, eliminating the occlusion or scaling issues present in 2D detection and providing more comprehensive and accurate target detection.

[0059] Subsequently, in order to link the 3D query vectors in 710 to the corresponding 2D image features, the 3D position reference points of each query vector can be projected onto pixels on the 2D plane using a calibration matrix (e.g., determined based on the camera's intrinsic and extrinsic parameters), and then the 3D feature representation corresponding to the query vector can be updated and refined based on the 2D image features of that pixel.

[0060] Taking query vector 720 as an example, the 3D position reference point corresponding to query vector 720 can be projected onto the corresponding pixel on the 2D plane using a calibration matrix. Based on the corresponding pixel to which the 3D position reference point is projected, the 2D image features of that pixel can be obtained. In one embodiment, when acquiring multi-view images, the 3D position reference point corresponding to query vector 720 may not be projected into images from some views, or it may be projected onto corresponding pixels in multiple images from different views simultaneously. For example, the 2D image features corresponding to the 3D position reference point can be a weighted average of the 2D image features obtained based on the corresponding pixels in all the images to which it is projected. It should be understood that other applicable processing methods in the art are possible.

[0061] The 2D image features of a pixel may include two-dimensional image features 740 and depth information 760. The two-dimensional image features 740 may be obtained based on a two-dimensional image feature representation 730 according to the pixel's position. The two-dimensional image feature representation 730 may be obtained by sampling from a single-view or multi-view image based on the pixel's position. The two-dimensional image feature representation 730 may be... Figure 3 Two-dimensional image feature representation 330 and Figure 4 An example of a two-dimensional image feature representation 410. Depth information 760 can be obtained based on a depth distribution prediction 750 based on the pixel's location. Specifically, depth information 760 can be obtained based on depth distribution prediction sampling of a single-view or multi-view image based on the pixel's location. Depth distribution prediction 750 can be... Figure 6 Example of depth distribution prediction in 670.

[0062] In one embodiment, depth information at a corresponding location can be obtained by sampling based on depth distribution prediction 750 according to the location of the pixel. Alternatively, as disclosed herein, when multiple depth prediction networks are selected for a single image, resulting in multiple depth distribution predictions, the sampled depth information can be obtained based on each of the multiple depth distribution predictions. For example, the multiple depth information can be weighted according to the weights 530 used for the depth prediction networks described above, and the weighted depth information can be used as the depth information of the pixel. As another example, when the network selection module has not determined the weights for different depth prediction networks, the multiple depth information can be averaged to obtain the depth information of the pixel.

[0063] Subsequently, after multiple updates and iterations of the 3D feature representation of the query vector, a prediction 780 corresponding to the query vector 720 can be obtained through a prediction network 770 based on the updated 3D feature representation. In one embodiment, the prediction network 770 can be implemented based on a Transformer model decoder. In another embodiment, the prediction network 770 can be implemented based on a regression algorithm and / or a classification algorithm. In one embodiment, the prediction 780 may include the target's location, type, and orientation.

[0064] return Figure 2 The target prediction module 205 can output a prediction 230 for a set of query vectors. In one embodiment, the prediction 780 may include the target's location, type, and orientation. Additionally, for query vectors where no corresponding target is detected, the target type can be output as "no target".

[0065] As described above, the method proposed in this disclosure can first dynamically determine the applicable depth prediction network based on the features of the input image to obtain more accurate depth information of pixels on the 2D plane. Furthermore, 3D object prediction based on depth information can be flexibly adapted to various object detection scenarios, especially when the depth range changes, without excessively increasing computational overhead and ensuring the accuracy of object detection.

[0066] Figure 8 An exemplary flowchart for detecting a target in three-dimensional space according to an embodiment of this disclosure is shown. The various steps provided in this embodiment can be implemented by one or more computing-capable devices via software and / or hardware.

[0067] At step 810, a two-dimensional feature representation can be obtained based on the image. For example, step 810 can be achieved by combining... Figure 2 and Figure 3 The feature extraction module 201 described above is used to implement this.

[0068] In one embodiment, the two-dimensional feature representation may be obtained based on one or more of the following: a convolutional neural network (CNN) model, such as ResNet; a transformer model, such as Swin.

[0069] In one embodiment, multiple images of the surrounding environment can be acquired from multiple perspectives using an image sensor, and a corresponding two-dimensional feature representation can be obtained based on each of the multiple images.

[0070] In one embodiment, the two-dimensional feature representation can be represented in various applicable forms such as feature map, feature vector, feature matrix, and feature tensor.

[0071] At step 820, a depth prediction network to be used for the image can be selected from a set of depth prediction networks based on two-dimensional feature representations, wherein the set of depth prediction networks includes multiple depth prediction networks. For example, step 820 can be achieved by combining... Figure 2 and Figure 5 The network selection module 203 described above is used to implement this.

[0072] In one embodiment, image regions associated with one or more targets to be detected in an image can be identified based on two-dimensional feature representations. Furthermore, a depth prediction network to be used for the image can be selected from a set of depth prediction networks based on the portion of the two-dimensional feature representation corresponding to the identified image regions. For example, image regions associated with one or more targets to be detected in an image can be identified by a masking matrix, which can be a Boolean matrix where matrix elements corresponding to regions where targets to be detected are 1, and matrix elements corresponding to regions where targets to be detected are 0.

[0073] In one embodiment, the image region associated with one or more targets to be detected contained in the image can be identified based on a convolutional neural network (CNN) model, such as a fully convolutional network.

[0074] In one embodiment, each of the multiple depth prediction networks can be applicable to different distance ranges or different application scenarios. For example, each depth prediction network can be applicable to different distance ranges: depth prediction network 1 is applicable to 0 to 50 meters, depth prediction network 2 is applicable to 50 to 100 meters, and depth prediction network 3 is applicable to 100 to 200 meters. Alternatively or additionally, the distance ranges applicable to each depth prediction network can partially overlap. As another example, each depth prediction network can be applicable to different application scenarios. Taking autonomous driving as an example, depth prediction network 1 can be applicable to "urban driving," depth prediction network 2 can be applicable to "suburban driving," and depth prediction network 3 can be applicable to "highway driving."

[0075] In one embodiment, the depth prediction network to be selected can be determined uniformly for one or more images. Alternatively, the depth prediction network to be selected can be determined separately for each image.

[0076] In one embodiment, the number of deep prediction networks to be selected from the set of deep prediction networks can be preset. Alternatively, the number of deep prediction networks to be selected from the set of deep prediction networks can be determined based on two-dimensional feature representations.

[0077] In one embodiment, the index of the depth prediction network to be selected can be determined based on the two-dimensional feature representation.

[0078] In one embodiment, the weights for the selected depth prediction network can be determined based on the two-dimensional feature representation. The weights for unselected depth prediction networks can be set to zero.

[0079] In one embodiment, the deep prediction network can be selected from a set of deep prediction networks based on a convolutional neural network (CNN) model. For example, a CNN model can have convolutional layers, flattening layers, fully connected layers, and activation layers (such as the softmax function).

[0080] At step 830, the depth distribution of the image can be determined using the selected depth prediction network. For example, step 830 can be achieved by combining... Figure 2 and Figure 6 The depth prediction module 204 described is used to implement this.

[0081] In one embodiment, the deep prediction network is implemented based on a fully connected network model.

[0082] In one embodiment, multiple selected depth prediction networks output depth distribution predictions of an image based on two-dimensional feature representations of the image.

[0083] In one embodiment, the deep prediction network can be implemented based on a fully connected network model.

[0084] In one embodiment, when each depth prediction network is configured for different distance ranges, the output depth distribution prediction can be normalized according to the distance configuration.

[0085] At step 840, targets contained in the image can be detected in three-dimensional space, at least based on the depth distribution. For example, step 840 can be achieved by combining... Figure 2 and Figure 7 The target detection module 205 described above is used to implement this.

[0086] In one embodiment, a three-dimensional feature representation associated with a position in a three-dimensional space corresponding to an image can be obtained based on depth distribution and two-dimensional feature representation, and a target contained in the image can be detected based on the three-dimensional feature representation.

[0087] In one embodiment, a location reference point in three-dimensional space can be projected onto a pixel on a two-dimensional plane; depth information associated with the pixel is obtained based on depth distribution predictions determined by each depth prediction network; the depth information associated with the pixel is weighted based on weights used for the selected depth prediction networks; and a three-dimensional feature representation of the location reference point in three-dimensional space is obtained based on the weighted depth information associated with the pixel and the two-dimensional feature representation. Alternatively, when the weights used for different depth prediction networks are not determined, multiple depth information can be averaged to obtain the depth information for the pixel.

[0088] In one embodiment, the three-dimensional space is described by a bird's-eye view (BEV).

[0089] In one embodiment, the type, location, and / or orientation of targets contained in the image are detected. For example, the type of target to be detected may include vehicles, pedestrians, trees, railings, etc. Additionally, the type of target to be detected may include lane lines and traffic sign information, such as solid lines, dashed lines, turns, straight ahead, no-stopping signs, etc. For example, the location of the target to be detected can be represented by the coordinates of its bounding box in three-dimensional space. For example, the orientation of the target to be detected can be represented by the angle of deviation relative to various reference points or reference planes.

[0090] In one embodiment, information associated with the detected target can be used for various downstream tasks, such as path planning, automatic parking, and collision avoidance in autonomous driving scenarios.

[0091] In one embodiment, the target is detected based on a Transformer model. In another embodiment, the target is detected based on a Mamba model.

[0092] Figure 9 A block diagram of a system according to an embodiment of the present disclosure is shown.

[0093] System 900 may include one or more processors 910 and memory 920. Memory 920 may store executable instructions. Processor 910 may execute executable instructions stored or encoded in memory 920, thereby achieving the above-mentioned combination. Figures 1 to 8 The various operations and / or functions described. Although not in Figure 9 As shown, but those skilled in the art will understand, system 900 may include various other components, such as various communication modules, bus modules, and possibly user interface modules. In one embodiment, memory 920 may also store one or more of the following: training dataset, unlabeled dataset, or labeled dataset.

[0094] Embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium may store executable instructions, which, when executed by a processor, can achieve the above-described combinations. Figures 1 to 8 The various operations and / or functions described. For example, computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), hard disk, flash memory, etc.

[0095] Embodiments of this disclosure also provide a computer program product. The computer program product may include a computer program. When executed by a processor, the computer program can achieve the above-described combinations. Figures 1 to 8 The various operations and / or functions described.

[0096] Specific embodiments of this disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0097] Not all steps and units in the above process and system structure diagrams are necessary; some steps or units can be omitted according to actual needs. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical entity, some units may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0098] The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply "preferred" or "advantageous" compared to other embodiments. Specific implementations are included in detail for the purpose of providing an understanding of the described techniques. However, these techniques can be implemented without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0099] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A method for detecting a target in three-dimensional space, comprising: Obtaining two-dimensional feature representations based on images; The depth prediction network to be used for the image is selected from a set of depth prediction networks based on the two-dimensional feature representation, wherein the set of depth prediction networks includes multiple depth prediction networks; The depth distribution of the image is determined by the selected depth prediction network; The target contained in the image is detected in the three-dimensional space based at least on the depth distribution.

2. The method according to claim 1, further comprising: Based on the two-dimensional feature representation, image regions associated with one or more targets to be detected contained in the image are identified. The step of selecting a depth prediction network for the image from the set of depth prediction networks based on the two-dimensional feature representation includes: selecting a depth prediction network for the image from the set of depth prediction networks based on the portion of the two-dimensional feature representation corresponding to the identified image region.

3. The method according to claim 2, wherein, The image regions associated with one or more targets to be detected contained in the image are identified using a convolutional neural network (CNN) model.

4. The method according to claim 1, wherein, The two-dimensional feature representation is obtained based on one or more of the following: Convolutional Neural Network (CNN) model, Transformer model; The deep prediction network is selected from the set of deep prediction networks based on a convolutional neural network (CNN) model; and / or The deep prediction network is implemented based on a fully connected network model.

5. The method according to claim 1, wherein, Each of the multiple depth prediction networks is suitable for different distance ranges or different application scenarios.

6. The method according to claim 1, wherein, The step of selecting a depth prediction network from the set of depth prediction networks for the image based on the two-dimensional feature representation includes: The number of deep prediction networks to be selected from the set of deep prediction networks is determined based on the two-dimensional feature representation; and / or The weights for the selected deep prediction network are determined based on the two-dimensional feature representation.

7. The method of claim 6, wherein detecting a target contained in the image in three-dimensional space based at least on the depth distribution comprises: Based on the depth distribution and the two-dimensional feature representation, a three-dimensional feature representation associated with the position in the three-dimensional space corresponding to the image is obtained; as well as The target contained in the image is detected based on the three-dimensional feature representation.

8. The method according to claim 7, wherein, The process of obtaining a three-dimensional feature representation associated with the position in the three-dimensional space corresponding to the image based on the depth distribution and the two-dimensional feature representation includes: Project the position reference point in the three-dimensional space onto the pixel point on the two-dimensional plane; Depth information associated with the pixel is obtained based on the depth distribution determined by each selected depth prediction network; The depth information associated with the pixel is weighted based on the weights used in the selected depth prediction network. The three-dimensional feature representation of the location reference point in the three-dimensional space is obtained based on the weighted depth information associated with the pixel and the two-dimensional feature representation.

9. The method according to claim 1, wherein, The three-dimensional space is described using a bird's-eye view (BEV).

10. The method according to claim 1, wherein, The detection of targets contained in the image in the three-dimensional space includes: detecting the type, location, and / or orientation of targets contained in the image, wherein the targets are detected based on a Transformer model or a Mamba model.

11. The method according to claim 1, wherein, The method of obtaining two-dimensional feature representations based on images includes: Multiple images of the surrounding environment are acquired from multiple perspectives using an image sensor; and A corresponding two-dimensional feature representation is obtained based on each of the plurality of images; Furthermore, the step of selecting a depth prediction network from the set of depth prediction networks to be used for the image based on the two-dimensional feature representation includes: The depth prediction network to be used for the corresponding image is selected from the set of depth prediction networks based on the two-dimensional feature representation corresponding to each of the plurality of images.

12. A system for detecting a target in three-dimensional space, comprising: The feature extraction module is configured to obtain two-dimensional feature representations based on images; A network selection module is configured to select a depth prediction network to be used for the image from a set of depth prediction networks based on the two-dimensional feature representation, wherein the set of depth prediction networks includes multiple depth prediction networks. A depth prediction module, configured to determine the depth distribution of the image using a selected depth prediction network; and A target detection module is configured to detect targets contained in the image in the three-dimensional space based at least on the depth distribution.

13. A system comprising: At least one processor; A memory coupled to the at least one processor, thereon storing executable instructions that, when executed by the at least one processor, cause the at least one processor to implement the method according to any one of claims 1 to 11.

14. A computer-readable medium storing a computer program comprising instructions that, when executed by a processor, cause one or more units to perform the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images