Target detection method and device

Abstract

The invention provides a target detection method and device. The method comprises the steps of obtaining a feature map of an image based on the image shot by a camera, and obtaining features of multiple voxels based on point cloud data which is obtained through a laser radar and corresponds to the image; pixels corresponding to the plurality of voxels in the image are determined, and the coordinate of the pixel corresponding to each voxel in the plurality of voxels in the coordinate system of the camera corresponds to the coordinate of each voxel in the coordinate system of the laser radar; determining features of pixels corresponding to the plurality of voxels in the image based on the feature map; fusing the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain fused features; and performing target detection based on the fused features.

Description

Target detection methods and devices Technical Field

[0001] This application relates to the field of target detection technology, and more specifically, to a target detection method and apparatus. Background Technology

[0002] With the development of autonomous driving technology, accurate perception of the surrounding environment has become one of the key technologies for realizing autonomous driving. Among them, object detection, as an important part of the perception task, is crucial for identifying and accurately locating objects such as vehicles, pedestrians, and traffic signs on the road.

[0003] Currently, object detection primarily relies on advanced sensor technologies, computer vision algorithms, and deep learning models. Multimodal combinations of LiDAR (LiDAR) and cameras play a crucial role in object detection because they can balance positional and semantic information. Typically, LiDAR collects sparse point cloud data reflecting the geometric and positional information of objects, while cameras collect image data providing semantic information such as color and texture. These data representations are misaligned, necessitating alignment and fusion for use by deep learning models.

[0004] Among related technologies, a method for fusing image features and LiDAR features based on bird's-eye view (BEV) is proposed. First, image features and LiDAR features are converted into a unified BEV view, and then the converted features are fused under the BEV view. This approach compresses point cloud features to the BEV view, resulting in the loss of Z-axis information. This loss of Z-axis information can make some easily distinguishable objects difficult to detect. Simultaneously, projecting image features into the BEV coordinate system causes semantic loss; for example, some objects easily identifiable in the image may become difficult to distinguish after being projected into BEV coordinates.

[0005] Therefore, how to improve the accuracy of target detection is a problem that this invention urgently needs to solve. Summary of the Invention

[0006] The purpose of this invention is to provide a target detection method and apparatus to at least solve the problems in the aforementioned related technologies, or to solve none of the aforementioned problems.

[0007] According to an exemplary embodiment of the present invention, a target detection method is provided. The method may include: acquiring a feature map of an image captured by a camera and acquiring features of a plurality of voxels based on point cloud data corresponding to the image acquired by a lidar, wherein the plurality of voxels are obtained by performing voxelization on the point cloud data; determining pixels in the image corresponding to the plurality of voxels, wherein the coordinates of the pixels corresponding to each of the plurality of voxels in the coordinate system of the camera correspond to the coordinates of each voxel in the coordinate system of the lidar; determining features of the pixels corresponding to the plurality of voxels in the image based on the feature map; fusing the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain fused features; and performing target detection based on the fused features.

[0008] According to embodiments of this disclosure, since no BEV viewpoint transformation is performed, there is no geometric loss (Z-axis information loss) or semantic loss. Furthermore, since only image features associated with LiDAR features are fused with LiDAR features, target detection accuracy can be maintained while reducing computational load.

[0009] Optionally, the step of fusing the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels includes: concatenating the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain concatenated features; and fusing the concatenated features to obtain the fused features.

[0010] Optionally, the stitching step includes: stitching the features of the pixels corresponding to the multiple voxels together with the features of the multiple voxels along the dimension of the feature channels of the multiple voxels.

[0011] Optionally, the fusion step includes: inputting the concatenated features into a convolutional layer to obtain initial fused features; and fusing the initial fused features based on an attention mechanism to obtain the fused features.

[0012] According to embodiments of this disclosure, the number of channels in the concatenated features can be reduced through convolution operations, thereby reducing the computational load of subsequent attention-based mechanisms.

[0013] Optionally, the step of fusing the initial fusion features based on the attention mechanism to obtain the fused features includes: inputting the initial fusion features into a deep fusion module composed of at least one sparse transform block to obtain the fused features.

[0014] According to embodiments of this disclosure, since voxel features are sparse features, the mapped image features are also sparse features, and performing deep fusion based on a sparse transformer can reduce the amount of fusion computation.

[0015] Optionally, the step of performing target detection based on the fused features includes: determining the confidence level of each feature in the fused features; selecting the top K features with high confidence levels as foreground features; and performing target detection based on the foreground features.

[0016] Optionally, the feature map of the image is a pyramid feature map of the image.

[0017] According to an exemplary embodiment of the present disclosure, a target detection apparatus is provided, the target detection apparatus comprising: a feature acquisition unit configured to acquire a feature map of an image captured by a camera and acquire features of a plurality of voxels based on point cloud data corresponding to the image acquired by a lidar; a mapping unit configured to determine pixels in the image corresponding to the plurality of voxels, wherein the coordinates of the pixels corresponding to each of the plurality of voxels in the coordinate system of the camera correspond to the coordinates of each voxel in the coordinate system of the lidar; a feature determination unit configured to determine features of the pixels corresponding to the plurality of voxels in the image based on the feature map; a fusion unit configured to fuse the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain fused features; and a target detection unit configured to perform target detection based on the fused features.

[0018] Optionally, the fusion unit is configured to: concatenate the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain concatenated features; and fuse the concatenated features to obtain the fused features.

[0019] Optionally, the fusion unit is configured to: concatenate the features of the pixels corresponding to the multiple voxels with the features of the multiple voxels along the dimension of the feature channels of the multiple voxels.

[0020] Optionally, the fusion unit is configured to: input the concatenated features into a convolutional layer to obtain initial fused features; and fuse the initial fused features based on an attention mechanism to obtain the fused features.

[0021] Optionally, the fusion unit is configured to input the initial fusion features into a deep fusion module consisting of at least one sparse transform block to obtain the fused features.

[0022] Optionally, the target detection unit is configured to: determine the confidence level of each feature in the fused features; select the top K features with high confidence levels as foreground features; and perform target detection based on the foreground features.

[0023] Optionally, the feature map of the image is a pyramid feature map of the image.

[0024] According to exemplary embodiments of the present disclosure, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one processor, the at least one processor causes the at least one processor to perform the target detection method as described above.

[0025] According to embodiments of this disclosure, by retaining foreground features and discarding background features, the computational load of target detection can be reduced while ensuring the accuracy of target detection. Attached Figure Description

[0026] The above and other objects and features of the present invention will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate an example, in which: FIG1 shows a flowchart of a target detection method according to an embodiment of the present disclosure; FIG2 shows an example flowchart of a target detection method according to an embodiment of the present disclosure; and FIG3 is a block diagram showing the structure of a target detection apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0027] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings, wherein the same reference numerals are used to denote the same or similar elements, features, and structures. However, it is not intended that the present disclosure be limited to the specific embodiments described herein, and it is intended that the present disclosure cover all modifications, equivalents, and / or substitutions of the present disclosure, provided they fall within the scope of the appended claims and their equivalents. The terms and words used in the following description and claims are not limited to their dictionary meanings, but are used only to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the appended claims and their equivalents.

[0028] It should be understood that, unless the context clearly indicates otherwise, the singular form includes the plural form. The terms “comprising,” “including,” and “having” as used herein indicate the presence of a disclosed function, operation, or element, but do not exclude other functions, operations, or elements.

[0029] For example, the expression “A or B” or “at least one of A and / or B” can indicate A and B, or A or B. For example, the expression “A or B” or “at least one of A and / or B” can indicate (1) A, (2) B or (3) both A and B.

[0030] In various embodiments of this disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to, or being "coupled" or "connected" to, another component (e.g., a second component), the component may be directly connected to, or may be connected via, another component (e.g., a third component). Conversely, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to, or being directly coupled to or directly connected to, another component (e.g., a second component), there is no other component (e.g., a third component) between the component and the other component.

[0031] The expression “configured as” used in describing the various embodiments of this disclosure may be used interchangeably, for example, with expressions such as “suitable for,” “capable of,” “designed to,” “suitable for,” “manufactured as,” and “capable,” depending on the context. The term “configured as” may not necessarily indicate that the hardware is “specifically designed for.” Rather, in some cases, the expression “a device configured as…” may indicate that the device and another device or part of it are “capable of…”. For example, the expression “a processor configured to perform A, B, and C” may indicate a dedicated processor (e.g., an embedded processor) for performing the respective operations or a general-purpose processor (e.g., a central processing unit CPU or application processor (AP)) for performing the respective operations by executing at least one software program stored in a memory device.

[0032] The terminology used herein is intended to describe certain embodiments of this disclosure but is not intended to limit the scope of other embodiments. Unless otherwise stated herein, all terms used herein (including technical or scientific terms) are to have the same meaning as commonly understood by one of ordinary skill in the art. Generally, terms as defined in dictionaries should be considered to have the same meaning as in the context of the relevant field and should not be interpreted differently or as having an overly formal meaning unless expressly defined herein. In no event should the terminology defined in this disclosure be construed as excluding embodiments of this disclosure.

[0033] Figure 1 shows a flowchart of a target detection method according to an embodiment of the present disclosure.

[0034] Referring to Figure 1, in step S101, a feature map of the image is obtained based on the image captured by the camera, and features of multiple voxels are obtained based on the point cloud data corresponding to the image acquired by the LiDAR.

[0035] As an example, the plurality of voxels are obtained by performing voxelization on the point cloud data.

[0036] The multiple voxels may refer to non-empty voxels.

[0037] As an example, the images mentioned in this article may refer to RGB images.

[0038] As an example, an image can refer to an image of the surrounding environment taken by a camera, and point cloud data can refer to point cloud data of the surrounding environment obtained through LiDAR.

[0039] As an example, image feature extraction modules (such as ResNet, SwinTransformer, VovNet, etc.) can be used to obtain feature maps of images.

[0040] As an example, a feature map can be a multi-level feature map of an image.

[0041] As an example, multi-level feature maps can be obtained by extracting pyramid features from an image.

[0042] As an example, features of multiple voxels can be obtained based on pixel feature extraction modules (e.g., Voxelnet, SparseEncoder, etc.).

[0043] As an example, a single-level feature map can be represented as M and C represent the number of features (or elements) and the dimension or number of channels (or feature maps) in a single-level feature map, respectively. Features (or voxel features) of multiple voxels can be represented as follows: ,in, and These represent the number of voxel features (or voxels) and the number or dimension of channels, respectively.

[0044] For example, M and C can be 200 and 100 respectively. and These can be 100 and 1000 respectively. The above values ​​are merely examples and do not limit this disclosure. These values ​​may be used as examples in the following description.

[0045] In step S102, pixels in the image corresponding to the plurality of voxels are determined, wherein the coordinates of the pixels corresponding to each of the plurality of voxels in the coordinate system of the camera correspond to the coordinates of each voxel in the coordinate system of the lidar.

[0046] As an example, the coordinates of a voxel can refer to the coordinates of the center point of the voxel.

[0047] As an example, a transformation matrix can be obtained from the camera's intrinsic and extrinsic parameter matrices to transform coordinates from the LiDAR's coordinate system to the camera's coordinate system. For each voxel in the lidar's coordinate system, the 3D coordinates are... The coordinates corresponding to the coordinates of each voxel in the camera's coordinate system can be obtained through the following matrix operations. : .

[0048] In step S103, features of pixels in the image corresponding to the plurality of voxels are determined based on the feature map.

[0049] As an example, the coordinates of a pixel in a feature map can be determined based on the pixel's coordinates, where the value of the feature or element at that coordinate in the feature map is the feature of the pixel.

[0050] As an example, when the camera features are pyramid features, if only single-level features in the pyramid features are used, the features of each voxel correspond to the features in the single-level feature map. If multi-level features in the pyramid features are selected, the features of each voxel correspond to the features in the multi-level feature map.

[0051] For example, if two-level features from the pyramid feature set are used, then the feature of each voxel corresponds to one feature in each of the two feature maps, meaning that the feature of each voxel can correspond to two features. Those skilled in the art should understand that these two features indicate the features of pixels at the same coordinates in the image.

[0052] Figure 2 shows an example flowchart of a target detection method according to an embodiment of the present disclosure.

[0053] Referring to Figure 2, Figure 2 illustrates a feature extraction module 210, a mapping module 220, a splicing module 230, an initial fusion module 240, a deep fusion module 250, a filtering module 260, and an object detection module 270. The operations performed by each of these modules will be described later. Those skilled in the art will understand that Figure 2 may also include other modules, and at least one of the modules included in Figure 2 may be combined or separated.

[0054] As an example, the feature acquisition module 210 can acquire the features of the image and the features of the voxels. The mapping module 220 can determine the pixels in the image corresponding to multiple voxels, and determine the features of the pixels in the image corresponding to multiple voxels based on the feature map.

[0055] In step S104, the features of the pixels corresponding to the plurality of voxels are fused with the features of the plurality of voxels to obtain fused features.

[0056] As an example, the step of fusing the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels includes: concatenating the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain concatenated features; and fusing the concatenated features to obtain the fused features.

[0057] Referring to Figure 2, the stitching module 230 can stitch together the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels.

[0058] As an example, the stitching step includes: stitching the features of the pixels corresponding to the multiple voxels together with the features of the multiple voxels along the dimension of the feature channels of the multiple voxels.

[0059] As described above, since the LiDAR features and the matched image features are aligned in terms of feature count, they can be directly stitched along the channel dimension, keeping the total number of features unchanged.

[0060] For example, if the number of features (or elements) and the dimension or number of channels (of a certain level of feature map) in a certain level feature map are 200 and 100 respectively, and the number of features and the number or dimension of channels of voxels are 100 and 1000 respectively, then when only a certain level feature map (e.g., the highest level feature map) is used, the concatenated features can be represented as: .

[0061] As another example, if two-level feature maps are used (e.g., a single-level feature map) and secondary feature maps Then, the concatenated features can be represented as: If a three-level feature map is used, the concatenated features can be represented as: .

[0062] As another example, the features of the pixels corresponding to the plurality of voxels can be concatenated with the features of the plurality of voxels along the direction of the number of features.

[0063] As an example, since the dimension of the feature map is inconsistent with the dimension of the voxel features, the dimension of the feature map can be changed to be consistent with the dimension of the voxel features by dimensionality increase. Then, the features of the pixels corresponding to the multiple voxels after dimensionality increase are concatenated with the features of the multiple voxels along the direction of the number of features.

[0064] As an example, you can either set the feature of the added dimension of a pixel to 0 or use the feature of the original dimension of the pixel as the feature of the added dimension of the pixel.

[0065] For example, when only a single-level feature map is used (e.g., the highest-level feature map), the concatenated features can be represented as: When using two-level feature maps (e.g., a single-level feature map) and secondary feature maps When splicing, the features can be represented as: When using three-level feature maps, the concatenated features can be represented as: .

[0066] As an example, the fusion step includes: inputting the concatenated features into a convolutional layer to obtain initial fused features; and fusing the initial fused features based on an attention mechanism to obtain deep fused features.

[0067] As an example, the convolution operation can be performed on the stitched features based on the initial fusion module 240 in Figure 2 to obtain the initial fused features.

[0068] As an example, convolutional layers can be used to simply fuse the concatenated features to reduce the number of channels, thereby reducing the amount of subsequent computation.

[0069] As an example, the features of the pixels corresponding to the multiple voxels are concatenated with the features of the multiple voxels, and a convolution operation is performed on the concatenated features to obtain the initial fused features. , can be represented as: ,in, and These represent the splicing and convolution operations, respectively. Characteristics representing multiple voxels This represents the features of a pixel corresponding to multiple voxels.

[0070] As an example, a fusion network consisting of at least one sparse Transformer block (e.g., Dynamic Sparse Voxel Transformer (DSVT) and Flatformer, etc.) can be used to perform deep fusion on the initial fusion features to obtain fused features. For example, a stack of sparse transformer blocks can be used to perform deep fusion.

[0071] As an example, the features of the initial fusion can be deeply fused based on the deep fusion module 250 in Figure 2. As an example, the deep fusion module can be a sparse transformer based on DSVT blocks.

[0072] The Sparse Transformer adapts the attention mechanism to sparse voxel features. Its core idea is to ignore empty voxel positions, thus adapting the transformer to sparse data types. By inputting initial fused features into the Sparse Transformer, the powerful attention mechanism of the Transformer can be used to deeply fuse them to obtain deeply fused features. .

[0073] Since the specific computation process of sparse Transformer is existing technology, it will not be elaborated here.

[0074] In step S105, target detection is performed based on the fused features.

[0075] As an example, the fused features can be fed into a target detection module (e.g., a Transformer Decoder) to identify targets in the surrounding environment.

[0076] As an example, the steps of target detection based on the fused features include: filtering out foreground features from the fused features, and identifying targets in the surrounding environment based on the foreground features.

[0077] As an example, the steps for object detection based on the fused features include: determining the confidence level of each feature in the fused features; selecting the top K features with high confidence levels as foreground features; and performing object detection based on the foreground features.

[0078] As an example, foreground features can be filtered out based on the filtering module 260 in Figure 2.

[0079] As an example, features that have been discarded from the background (i.e., foreground features) can be input into the target detection module to detect targets in the surrounding environment.

[0080] As an example, object detection based on foreground features can be performed using the Transformer Decoder 270 in Figure 2.

[0081] Since the sparsity of point clouds and the number of voxels vary in different autonomous driving scenarios, the number of features in the fused features depends on the complexity of the scenario. The fused features are then input into the Transformer Decoder for computation, and the amount of computation varies depending on the scenario.

[0082] For example, in a Transformer Decoder, the main computational cost is the cross-attention between randomly generated queries (Q) and keys (i.e., the fused features). Since the order of magnitude of Q is typically... left and right The order of magnitude of the features is Therefore, the computational load of the object detection module mainly depends on the number of fused features. For example, in autonomous driving scenarios, due to the diversity of scenarios, the number of fused features varies widely (e.g., from 10,000 to 60,000). Therefore, directly inputting the fused features into the detection module not only results in varying computational loads depending on the scenario, but also increases detection latency when the number of features is large.

[0083] Since the background features in the fused features are not closely related to the target in the surrounding scene, while the target is mainly related to the foreground features, foreground features can be selected from the fused features, background features that are not closely related to the target (e.g., roads, sidewalks, etc.) can be discarded, while foreground features (e.g., vehicles, pedestrians, etc.) can be retained. The selected foreground features can then be input into the target detection module, thereby reducing the computational load of the target detection module.

[0084] Furthermore, a preset number of foreground features can be selected and input into the object detection module to perform object detection. This effectively reduces the uncertainty of computational load caused by the uneven distribution of data volume (the number of fused features). As an example, when training the selection module 260, for the coordinates (x, y, z) of each feature, the corresponding label can be generated using the bounding box in the labeled data based on the following equation. :

[0085] in, To provide ground truth labels for the detection boxes, and to ensure that more features associated with the target boxes can be selected, the detection boxes can be enlarged (e.g., enlarged by 1.5 times), so that features around the original target boxes are also considered foreground features.

[0086] The filtering module 260 is used to generate the confidence S of the feature that is the foreground feature in the magnified target box (i.e., the confidence is the confidence that the indicator feature is the foreground feature), and the focal loss function is used to supervise the training of the filtering module.

[0087] As an example, the screening step may include: determining the confidence level of each feature in the fused features as a foreground feature; and selecting K features with higher confidence levels as the foreground features.

[0088] As an example, a trained filtering module 260 can be used to determine the confidence level of each feature in the fused features as a foreground feature.

[0089] Based on the confidence level of each feature, the TopK method can be used to select the top K features (i.e., the K selected features with high confidence levels) as foreground features, and the remaining features are regarded as background features and discarded. Then, the selected K features are input into the object detection module (e.g., Transformer Decoder).

[0090] Compared to using all fused features, the proposed method reduces the computational complexity of the cross-attention operation in the object detection module to a fixed value. For example, when the voxel size is set to 0.1m... 0.1m At a depth of 0.075m, the average number of features is approximately 30,000. When we select a K value of 10,000, the computational cost of cross-attention can be reduced to approximately 1 / 9 of the original. Furthermore, since background features (mostly sidewalks on both sides of the road, the road itself, etc.) are discarded, the technical solution of this application will not result in a significant decrease in accuracy.

[0091] As an example, the setup of widely used 3D detectors (e.g., CMT, Transfusion, Centerformer) can be followed, and a DETR-like Transformer Decoder can be used as the object detection module. The Transformer Decoder mainly consists of several stacked Decoder layers composed of self-attention, cross-attention, and FFN. Since the Transformer Decoder is described in detail in related technologies, it will not be elaborated here.

[0092] According to embodiments of this disclosure, semantically rich image features and sparse radar features can be effectively fused without using BEV viewpoint transformation. In other words, the technique of this disclosure does not result in geometric loss (Z-axis information loss) or semantic loss.

[0093] According to embodiments of this disclosure, sparse LiDAR features are mapped to image features, effectively filtering image features associated with LiDAR based on feature dimension alignment. By fusing only the mapped image features with LiDAR features, the computational load can be reduced compared to using all image features in related technologies. Furthermore, by pre-filtering out background features that are not closely related to the target detection task, the computational load and memory usage in subsequent detection can be effectively reduced.

[0094] The target detection method according to an embodiment of the present disclosure has been described above with reference to Figures 1 and 2. The target detection apparatus according to an embodiment of the present disclosure will now be described with reference to Figure 3.

[0095] Figure 3 is a block diagram illustrating the structure of a target detection apparatus according to an embodiment of the present disclosure.

[0096] Referring to Figure 3, the target detection device 300 may include: a feature acquisition unit 310, a mapping unit 320, a feature determination unit 330, a fusion unit 340, and a target detection unit 350. Those skilled in the art will understand that the target detection device 300 may include other components, and at least one of the included components may be split or combined.

[0097] As an example, the feature acquisition unit 310 may be configured to acquire a feature map of an image based on an image captured by a camera and acquire features of multiple voxels based on point cloud data corresponding to the image acquired by a lidar.

[0098] As an example, the mapping unit 320 may be configured to determine pixels in the image corresponding to the plurality of voxels, wherein the coordinates of the pixels corresponding to each of the plurality of voxels in the coordinate system of the camera correspond to the coordinates of each voxel in the coordinate system of the lidar.

[0099] As an example, the feature determination unit 330 may be configured to determine the features of pixels in the image corresponding to the plurality of voxels based on the feature map.

[0100] As an example, the fusion unit 340 may be configured to fuse the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain fused features.

[0101] As an example, the target detection unit 350 can be configured to perform target detection based on the fused features.

[0102] As an example, the fusion unit 340 may be configured to: concatenate the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain concatenated features; and fuse the concatenated features to obtain the fused features.

[0103] As an example, the fusion unit 340 may be configured to: stitch together the features of the pixels corresponding to the multiple voxels with the features of the multiple voxels along the dimension of the channels of the features of the multiple voxels.

[0104] As an example, the fusion unit 340 may be configured to: input the concatenated features into a convolutional layer to obtain initial fused features; and fuse the initial fused features based on an attention mechanism to obtain the fused features.

[0105] As an example, the fusion unit 340 may be configured to input the initial fusion features into a deep fusion module consisting of at least one sparse transformer block to obtain the fused features.

[0106] As an example, the target detection unit 350 may be configured to: determine the confidence level of each feature in the fused features; select the top K features with high confidence levels as foreground features; and perform target detection based on the foreground features.

[0107] As an example, the feature map of the image is a pyramid feature map of the image.

[0108] According to embodiments of this disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor causes the at least one processor to perform a target detection method according to embodiments of this disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0109] According to embodiments of this disclosure, a computer program product may also be provided, wherein instructions in the computer program product are executable by a processor of a computer device to perform the target detection method described herein.

[0110] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

Claims

1. A target detection method, comprising: The feature map of the image is obtained based on the image captured by the camera, and the features of multiple voxels are obtained based on the point cloud data corresponding to the image obtained by the lidar. The pixels in the image corresponding to the plurality of voxels are determined, wherein the coordinates of the pixels corresponding to each voxel in the coordinate system of the camera correspond to the coordinates of each voxel in the coordinate system of the lidar; the features of the pixels corresponding to the plurality of voxels in the image are determined based on the feature map; the features of the pixels corresponding to the plurality of voxels are fused with the features of the plurality of voxels to obtain fused features; And target detection is performed based on the fused features.

2. The method as described in claim 1, wherein, The step of fusing the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels includes: concatenating the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain concatenated features; and fusing the concatenated features to obtain the fused features.

3. The method as described in claim 2, wherein the splicing step comprises: The features of the pixels corresponding to the multiple voxels are concatenated with the features of the multiple voxels along the dimension of the feature channels of the multiple voxels.

4. The method of claim 2, wherein the fusion step comprises: The concatenated features are input into a convolutional layer to obtain initial fused features; And the initial fusion features are fused based on an attention mechanism to obtain the fused features.

5. The method of claim 4, wherein, The step of fusing the initial fusion features based on the attention mechanism to obtain the fused features includes: inputting the initial fusion features into a deep fusion module composed of at least one sparse transform block to obtain the fused features.

6. The method of claim 1, wherein, The steps for object detection based on the fused features include: determining the confidence level of each feature in the fused features; selecting the top K features with high confidence levels as foreground features; and performing object detection based on the foreground features.

7. The method of claim 1, wherein, The feature map of the image is the pyramid feature map of the image.

8. A target detection device, comprising: The feature acquisition unit is configured to acquire feature maps of images captured by a camera and acquire features of multiple voxels based on point cloud data corresponding to the images acquired by a lidar. A mapping unit is configured to determine pixels in the image corresponding to the plurality of voxels, wherein the coordinates of the pixels corresponding to each voxel in the coordinate system of the camera correspond to the coordinates of each voxel in the coordinate system of the lidar; a feature determination unit is configured to determine features of the pixels corresponding to the plurality of voxels in the image based on the feature map; a fusion unit is configured to fuse the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain fused features; and a target detection unit is configured to perform target detection based on the fused features.

9. The target detection device as described in claim 8, wherein, The fusion unit is configured to: concatenate the features of the pixels corresponding to the plurality of voxels with the features of the plurality of voxels to obtain the concatenated features; And the spliced ​​features are fused to obtain the fused features.

10. A computer-readable storage medium for storing instructions, wherein, When the instruction is executed by at least one processor, the at least one processor performs the target detection method as described in any one of claims 1-7.

Images