A target detection method, computer program product and electronic device

Abstract

The application provides a target detection method, a computer program product and an electronic device, and is applied to the field of computer vision. The target detection method comprises the following steps: acquiring point cloud features and image features corresponding to the point cloud features; converting the point cloud features into a voxel space to obtain first voxel features in the voxel space, and converting the image features into the voxel space to obtain second voxel features in the voxel space; wherein a first coordinate system of the voxel space is different from a second coordinate system corresponding to a first acquisition device of point cloud data corresponding to the point cloud features and a third coordinate system corresponding to a second acquisition device of image data corresponding to the image features; fusing the first voxel features and the second voxel features to obtain to-be-detected features, and detecting a to-be-detected target based on the to-be-detected features.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more specifically, to a target detection method, a computer program product, and an electronic device. Background Technology

[0002] In existing technologies, multimodal unified representations for 3D object detection typically involve modal alignment at the input level or modal unification at the feature level. Modal alignment at the input level involves mapping point clouds or images to one modality as network input; for example, projecting image pixels into space using predicted depth, or projecting point clouds onto an image plane as an image input to the network for processing. Modal unification at the feature level generally involves projecting image features into space using predicted depth distribution and flattening the height layer to form a two-dimensional bird's-eye view feature.

[0003] However, in existing technologies, the image and point cloud are typically aligned to one modality directly at the input. For methods that perform modality alignment at the input level, the accuracy of object detection is low due to issues such as low depth prediction accuracy and the loss of inherent 3D spatial information when point clouds are directly projected onto the image. Similarly, methods that perform modality unification at the feature level suffer from inaccurate depth prediction, resulting in multiple heights of objects at the same location, leading to semantic ambiguity and also causing low object detection accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a target detection method, computer program product, and electronic device to solve the technical problem of low accuracy in target detection in the prior art.

[0005] In a first aspect, embodiments of this application provide a target detection method, comprising: acquiring point cloud features and corresponding image features; converting the point cloud features into a voxel space to obtain a first voxel feature in the voxel space; and converting the image features into the voxel space to obtain a second voxel feature in the voxel space; wherein the first coordinate system of the voxel space is different from the second coordinate system of the first acquisition device that acquires the point cloud data corresponding to the point cloud features and the third coordinate system of the second acquisition device that acquires the image data corresponding to the image features; fusing the first voxel feature and the second voxel feature to obtain a feature to be detected, and detecting the target to be detected based on the feature to be detected. In the above scheme, based on the image modality and the point cloud modality, a modality-independent voxel space is provided, which is different from the coordinate systems of the image modality and the point cloud modality. By converting the acquired point cloud features and image features into the aforementioned voxel space, the point cloud modalities and image modalities can be uniformly represented in the aforementioned voxel space. Therefore, this facilitates subsequent feature interaction and improves the accuracy of target detection.

[0006] In an optional implementation, the step of converting the image features to the voxel space to obtain a second voxel feature in the voxel space includes: determining the depth distribution corresponding to each pixel based on the image features; projecting the voxel point in the voxel space onto the two-dimensional image coordinate system corresponding to the image features to obtain the planar coordinate value and depth value of the voxel point in the two-dimensional image coordinate system; and obtaining the second voxel feature of the pixel point in the voxel space based on the planar coordinate value, the depth value, and the depth distribution of the pixel point corresponding to the voxel point. In the above scheme, since the image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to the voxel space. Through the above view transformation process, a feature representation of the image features in the three-dimensional voxel space can be established, thereby enabling point cloud modalities and image modalities to be uniformly represented in the above voxel space, thus improving the accuracy of target detection.

[0007] In an optional implementation, obtaining the second voxel feature of the pixel in the voxel space based on the planar coordinates, the depth value, and the depth distribution of the pixels corresponding to the voxel point includes: determining the feature value corresponding to the pixel in the image feature based on the planar coordinates, and determining the probability value corresponding to the pixel in the depth distribution based on the planar coordinates and the depth value; multiplying the feature value and the probability value to obtain the second voxel feature of the pixel in the voxel space. In the above scheme, since the image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to voxel space. Through the above view transformation process, a feature representation of the image features in three-dimensional voxel space can be established, thereby enabling a unified representation of point cloud modalities and image modalities in the voxel space, thus improving the accuracy of target detection.

[0008] In an optional implementation, the step of fusing the first voxel features and the second voxel features to obtain the feature to be detected includes: extracting features from the first voxel features using a first three-dimensional convolutional neural network to obtain a third voxel feature; and extracting features from the second voxel features using a second three-dimensional convolutional neural network to obtain a fourth voxel feature; and fusing the third voxel feature and the fourth voxel feature to obtain the feature to be detected. In the above scheme, after converting the point cloud features and image features to voxel space, two three-dimensional convolutional neural networks can be used to extract features from the obtained first voxel features and second voxel features respectively, so as to enhance the feature representation of the point cloud modality and the image modality in voxel space, thereby improving the accuracy of target detection.

[0009] In an optional implementation, during the feature extraction process of the first voxel features and the second voxel features, neighborhood-wide feature interactions are performed on the first voxel features and the second voxel features. In the above scheme, during the feature extraction process of the first voxel features and the second voxel features, neighborhood-wide feature interactions can also be performed on the first voxel features and the second voxel features to further enhance the feature representation of point cloud modalities and image modalities in voxel space, thereby improving the accuracy of target detection.

[0010] In an optional implementation, the method further includes: performing knowledge distillation on the fourth voxel features using the third voxel features. In the above scheme, after enhancing the feature representations of the point cloud modality and the image modality in voxel space, during the training of the convolutional neural network, the features corresponding to the point cloud modality can be used to perform knowledge distillation on the features corresponding to the image modality, thereby enabling the image modality to obtain the structured information contained in the point cloud modality. Therefore, the feature representation of the image modality in voxel space can be further enhanced to improve the accuracy of target detection.

[0011] In an optional implementation, detecting the target based on the features to be detected includes: detecting the target using a feature transformer-based decoder and the features to be detected. In the above scheme, after obtaining a unified multimodal voxel space, feature sampling can be performed in the voxel space based on the feature transformer-based decoder, thereby better utilizing multimodal features and achieving more efficient and accurate target detection.

[0012] Secondly, embodiments of this application provide a target detection device, comprising: an acquisition module for acquiring point cloud features and corresponding image features; a conversion module for converting the point cloud features to a voxel space to obtain a first voxel feature in the voxel space, and converting the image features to the voxel space to obtain a second voxel feature in the voxel space; wherein the first coordinate system of the voxel space is different from the second coordinate system of the first acquisition device that acquires the point cloud data corresponding to the point cloud features and the third coordinate system of the second acquisition device that acquires the image data corresponding to the image features; and a detection module for fusing the first voxel feature and the second voxel feature to obtain a feature to be detected, and detecting the target to be detected based on the feature to be detected. In the above scheme, based on the image modality and the point cloud modality, a modality-independent voxel space is provided, which is different from the coordinate systems of the image modality and the point cloud modality. By converting the acquired point cloud features and image features into the aforementioned voxel space, the point cloud modalities and image modalities can be uniformly represented in the aforementioned voxel space. Therefore, this facilitates subsequent feature interaction and improves the accuracy of target detection.

[0013] In an optional implementation, the conversion module is specifically used to: determine the depth distribution corresponding to each pixel based on the image features; project the voxel in the voxel space onto the two-dimensional image coordinate system corresponding to the image features to obtain the planar coordinates and depth values ​​of the voxel in the two-dimensional image coordinate system; and obtain the second voxel feature of the pixel in the voxel space based on the planar coordinates, the depth values, and the depth distribution of the pixels corresponding to the voxel. In the above scheme, since the image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to voxel space. Through the above view transformation process, a feature representation of the image features in three-dimensional voxel space can be established, thereby enabling a unified representation of point cloud modalities and image modalities in the voxel space, thus improving the accuracy of target detection.

[0014] In an optional implementation, the conversion module is further configured to: determine the feature value corresponding to the pixel in the image features based on the planar coordinate values, and determine the probability value corresponding to the pixel in the depth distribution based on the planar coordinate values ​​and the depth values; multiply the feature value by the probability value to obtain the second voxel feature of the pixel in the voxel space. In the above scheme, since the image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to voxel space. Through the above view transformation process, a feature representation of the image features in three-dimensional voxel space can be established, thereby enabling a unified representation of point cloud modalities and image modalities in the voxel space, thus improving the accuracy of target detection.

[0015] In an optional implementation, the detection module is specifically used to: extract features from the first voxel features using a first three-dimensional convolutional neural network to obtain a third voxel feature; and extract features from the second voxel features using a second three-dimensional convolutional neural network to obtain a fourth voxel feature; and fuse the third voxel feature and the fourth voxel feature to obtain the feature to be detected. In the above scheme, after converting the point cloud features and image features to voxel space, two three-dimensional convolutional neural networks can be used to extract features from the obtained first voxel features and second voxel features respectively, so as to enhance the feature representation of the point cloud modality and image modality in voxel space, thereby improving the accuracy of target detection.

[0016] In an optional implementation, during the feature extraction process of the first voxel features and the second voxel features, neighborhood-wide feature interactions are performed on the first voxel features and the second voxel features. In the above scheme, during the feature extraction process of the first voxel features and the second voxel features, neighborhood-wide feature interactions can also be performed on the first voxel features and the second voxel features to further enhance the feature representation of point cloud modalities and image modalities in voxel space, thereby improving the accuracy of target detection.

[0017] In an optional embodiment, the target detection device further includes a distillation module, used to perform knowledge distillation on the fourth voxel features using the third voxel features. In the above scheme, after enhancing the feature representations of the point cloud modality and the image modality in voxel space, during the training of the convolutional neural network, the features corresponding to the point cloud modality can be used to perform knowledge distillation on the features corresponding to the image modality, thereby enabling the image modality to obtain the structured information contained in the point cloud modality. Therefore, the feature representation of the image modality in voxel space can be further enhanced to improve the accuracy of target detection.

[0018] In an optional implementation, the detection module is specifically used to: detect the target using a feature transformer-based decoder and the features to be detected. In the above scheme, after obtaining a unified multimodal voxel space, feature sampling can be performed in the voxel space based on the feature transformer-based decoder, thereby better utilizing multimodal features and achieving more efficient and accurate target detection.

[0019] Thirdly, embodiments of this application provide a computer program product, including computer program instructions, which are read and executed by a processor to perform the target detection method as described in the first aspect.

[0020] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus; the processor and the memory communicate with each other via the bus; the memory stores computer program instructions executable by the processor, and the processor can execute the target detection method as described in the first aspect by calling the computer program instructions.

[0021] Fifthly, embodiments of this application provide a computer-readable storage medium that stores computer program instructions, which, when executed by a computer, cause the computer to perform the target detection method as described in the first aspect.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, embodiments of this application are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart of a target detection method provided in an embodiment of this application;

[0025] Figure 2 A structural block diagram of a target detection device provided in an embodiment of this application;

[0026] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0028] Please refer to Figure 1 , Figure 1 A flowchart of a target detection method provided in this application embodiment, the target detection method may include the following steps:

[0029] Step S101: Obtain point cloud features and their corresponding image features.

[0030] Step S102: Convert the point cloud features to a voxel space to obtain the first voxel features in the voxel space, and convert the image features to a voxel space to obtain the second voxel features in the voxel space.

[0031] Step S103: The first voxel feature and the second voxel feature are fused to obtain the feature to be detected, and the target to be detected is detected based on the feature to be detected.

[0032] Specifically, in step S101, the point cloud features are the feature data corresponding to the point cloud data under the point cloud modality. As one implementation, the first acquisition device can acquire point cloud data of the target to be detected, and then extract features based on the point cloud data to obtain the aforementioned point cloud data.

[0033] In this application, the specific implementation methods for the first acquisition device and the target to be detected are not specifically limited, and those skilled in the art can make appropriate selections according to the actual situation. For example, the first acquisition device can be a LiDAR, a 3D scanner, etc. As for the target to be detected, depending on the application scenario of the target detection method provided in this application, the target to be detected can be a person, a stationary object, a moving object, etc.

[0034] Furthermore, this application does not impose specific limitations on the specific implementation methods for feature extraction based on point cloud data, and those skilled in the art can make appropriate choices according to the actual situation. For example, point cloud data can be input into a sparse convolutional neural network and the output can be point cloud features corresponding to the point cloud data; or, point feature histograms (PFHs) can be used to extract features from the point cloud data to obtain the corresponding point cloud features, etc.

[0035] Similar to the point cloud data described above, in step S101, the image features are the feature data corresponding to the image data under the image modality. As one implementation, the second acquisition device can acquire image data of the target to be detected, and then extract features based on the image data to obtain the aforementioned image features.

[0036] In this application, the specific implementation of the second acquisition device is not limited. Those skilled in the art can make appropriate selections according to the actual situation. For example, the second acquisition device can be a camera, video camera, etc.

[0037] Furthermore, this application does not impose specific limitations on the specific implementation methods for feature extraction based on image data, and those skilled in the art can make appropriate choices according to the actual situation. For example, image data can be input into a convolutional neural network (CNN) and the output can be image features corresponding to the image data; or, the image data can be extracted using a histogram of oriented gradients (HOG) to obtain the corresponding image features, etc.

[0038] Based on the above embodiments, there are multiple ways to obtain point cloud features and their corresponding image features in step S101. For example, point cloud data and image data can be collected, and then features can be extracted to obtain point cloud features and image features; or, point cloud features and image features sent by an external device can be received; or, point cloud features and image features can be read from a cloud server, etc.

[0039] It is understood that, as one implementation method, the method of acquiring point cloud features can be the same as the method of acquiring image features, for example: receiving point cloud features and image features sent by the same or different external devices; as another implementation method, the method of acquiring point cloud features can also be the same as the method of acquiring image features, for example: receiving point cloud features sent by external devices and reading image features from a cloud server.

[0040] It should be noted that in the target detection method provided in this application embodiment, the point cloud features and image features in step S101 should be feature data obtained by collecting data for the same target to be detected, and the point cloud data and image data collected for the target to be detected should be at the same time or within the same time period.

[0041] In step S102 above, the voxel space is a three-dimensional space that is independent of both the point cloud mode and the image mode; in other words, the first coordinate system corresponding to the voxel space is different from the second coordinate system corresponding to the first acquisition device and the third coordinate system corresponding to the second acquisition device.

[0042] Therefore, converting point cloud features to a voxel space is equivalent to converting feature data in the second coordinate system to the first coordinate system; converting image features to a voxel space is equivalent to converting feature data in the third coordinate system to the first coordinate system.

[0043] Since the second coordinate system is typically a three-dimensional coordinate system, transforming feature data from the second coordinate system to the first coordinate system is equivalent to transforming data from one three-dimensional coordinate system to another. In this embodiment, the specific implementation method for transforming point cloud features is not limited; those skilled in the art can make appropriate selections by referring to existing technologies.

[0044] In contrast, since the third coordinate system is typically a two-dimensional coordinate system, transforming feature data from the third coordinate system to the first coordinate system is equivalent to transforming data from a two-dimensional coordinate system to a three-dimensional coordinate system. In this embodiment, as one approach, the image features can be first transformed by view transformation, and then the image features can be transformed to the first coordinate system; as another approach, the image features can be directly transformed using existing technology.

[0045] After step S102, the first voxel features and the second voxel features in the voxel space can be obtained. It can be seen that the feature data in the point cloud mode and the image features in the image mode have a unified representation in the voxel space, which facilitates feature interaction between the first voxel features and the second voxel features.

[0046] In this embodiment, the first voxel features and the second voxel features described above can be fused to obtain the fused features to be detected. This embodiment does not specifically limit the implementation method of the feature fusion; for example, the first voxel features and the second voxel features can be directly added together; or, the first voxel features and the second voxel features can be weighted and added together; or, the first voxel features and the second voxel features can be stacked together and fused using a convolutional neural network, etc.

[0047] It should be noted that when fusing based on the first voxel features and the second voxel features, one or more features can be selected as needed. For example, if no image features are currently available, fusion can be performed based solely on the first voxel features; or, if no point cloud features are currently available, fusion can be performed based solely on the second voxel features; or, fusion can be performed simultaneously based on both the first and second voxel features.

[0048] In step S103, after fusing the features to be detected in the voxel space, the target body can be detected based on these features. This application does not specifically limit the specific implementation method of the detection; those skilled in the art can make appropriate choices based on the actual situation. For example, the features to be detected can be detected based on the decoder of a feature transformer; or, the features to be detected can be input into a convolutional neural network for detection, etc.

[0049] In the above scheme, based on the image modality and point cloud modality, a modality-independent voxel space is provided. This voxel space has a different coordinate system than both the image modality and the point cloud modality. By transforming the acquired point cloud features and image features into the aforementioned voxel space, the point cloud modality and the image modality can be uniformly represented in the voxel space. Therefore, this facilitates subsequent feature interaction and improves the accuracy of target detection.

[0050] Furthermore, based on the above embodiments, this application provides a detailed description of an implementation method for converting image features in step S102. Step S102 specifically may include the following:

[0051] Step 1) Determine the depth distribution corresponding to each pixel based on image features.

[0052] Step 2): For a voxel point in voxel space, project the voxel point onto the two-dimensional image coordinate system corresponding to the image feature to obtain the planar coordinate value and depth value of the voxel point in the two-dimensional image coordinate system.

[0053] Step 3) Obtain the second voxel feature of the pixel in voxel space based on the planar coordinate value, depth value, and depth distribution of the pixel corresponding to the voxel point.

[0054] Specifically, the number of channels for the image features extracted by the convolutional neural network is C. Therefore, the image features can be considered as W×H×C data.

[0055] In step 1) above, the depth distribution corresponding to each pixel can be determined based on the image features described above. As one implementation, a 1×1 convolutional layer can be used to process the image features to obtain W×H×D data, which includes the depth distribution corresponding to each pixel in the image data.

[0056] Then, for each voxel point (x, y, z) in the voxel space, the voxel point in the first coordinate system can be projected onto the two-dimensional image coordinate system corresponding to the image feature. As one implementation, the projection matrix corresponding to the second acquisition device can be used to project the pixel point in the three-dimensional coordinate system onto the two-dimensional coordinate system, thereby obtaining the planar coordinates (u, v) of the voxel point in the third coordinate system and the corresponding depth value d, where the depth value d represents the distance of the pixel point from the plane of the two-dimensional image coordinate system corresponding to the image feature.

[0057] Finally, based on the depth distribution obtained in step 1) and the planar coordinates and depth values ​​in step 2), the image features in the two-dimensional coordinate system can be transformed into the three-dimensional coordinate system to obtain the second voxel features of the pixel in voxel space. The specific implementation of step 3) will be described in detail in subsequent embodiments and will not be presented here.

[0058] In the above scheme, since image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to voxel space. Through this view transformation process, a feature representation of the image features in three-dimensional voxel space can be established, enabling point cloud modalities and image modalities to be uniformly represented in the voxel space, thereby improving the accuracy of target detection.

[0059] Furthermore, based on the above embodiments, the step of obtaining the second voxel feature of a pixel in voxel space based on planar coordinate values, depth values, and the depth distribution of pixels corresponding to voxel points can specifically include the following:

[0060] Step 1) Determine the feature value corresponding to the pixel in the image feature based on the planar coordinate value, and determine the probability value corresponding to the pixel in the depth distribution based on the planar coordinate value and the depth value.

[0061] Step 2) Multiply the feature value by the probability value to obtain the second voxel feature of the pixel in voxel space.

[0062] Specifically, based on the planar coordinates (u, v) of a voxel, sampling can be performed in the image features; that is, the image feature value corresponding to the location of the planar coordinates (u, v) under W×H×C is extracted as the feature value corresponding to the pixel in the image features. Similarly, based on the planar coordinates (u, v) of a uniform voxel and the depth value d, sampling can be performed in the depth distribution data; that is, the probability value corresponding to the location of the lower coordinates (u, v, d) under W×H×D is extracted as the probability value corresponding to the pixel in the depth distribution.

[0063] Then, by multiplying the above feature value by the probability value, we can obtain the second voxel feature of the pixel in voxel space.

[0064] In the above scheme, since image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to voxel space. Through this view transformation process, a feature representation of the image features in three-dimensional voxel space can be established, enabling point cloud modalities and image modalities to be uniformly represented in the voxel space, thereby improving the accuracy of target detection.

[0065] Furthermore, based on the above embodiments, the step of fusing the first voxel feature and the second voxel feature in step S103 may specifically include the following:

[0066] Step 1) Use the first three-dimensional convolutional neural network to extract features from the first voxel features to obtain the third voxel features, and use the second three-dimensional convolutional neural network to extract features from the second voxel features to obtain the fourth voxel features.

[0067] Step 2) Fuse the third voxel feature with the fourth voxel feature to obtain the feature to be detected.

[0068] Specifically, before fusing the features from the point cloud modality with the features from the image modality, feature enhancement processing can be performed on the first voxel features and the second voxel features. As one implementation method, these features can be input into a three-dimensional convolutional neural network for sequential feature extraction to achieve feature enhancement.

[0069] For the first voxel feature, the first voxel feature can be input into the first three-dimensional convolutional neural network to obtain the output third voxel feature; similarly, for the second voxel feature, the second voxel feature can be input into the second three-dimensional convolutional neural network to obtain the output fourth voxel feature.

[0070] In this application, the specific structures of the first and second three-dimensional convolutional neural networks are not specifically limited, and those skilled in the art can make appropriate selections by referring to existing technologies. Furthermore, as one implementation, the structures of the first and second three-dimensional convolutional neural networks can be the same; as another implementation, the structures of the first and second three-dimensional convolutional neural networks can also be different; those skilled in the art can make appropriate adjustments according to the actual situation.

[0071] After obtaining the third and fourth voxel features, the third and fourth voxel features can be fused to obtain the feature to be detected. The specific implementation method for fusing the third and fourth voxel features is similar to the specific implementation method for fusing the first and second voxel features in the above embodiments, and will not be repeated here.

[0072] In the above scheme, after converting point cloud features and image features to voxel space, two three-dimensional convolutional neural networks can be used to extract features from the obtained first voxel features and second voxel features respectively, so as to enhance the feature representation of point cloud modalities and image modalities in voxel space, thereby improving the accuracy of target detection.

[0073] Furthermore, based on the above embodiments, during the feature extraction process of the first voxel features and the second voxel features, feature interaction within the neighborhood range is performed on the first voxel features and the second voxel features.

[0074] Specifically, during the feature extraction process of the first voxel features and the second voxel features, feature interaction within the neighborhood range of the first voxel features and the second voxel features can be performed to further enhance the features.

[0075] In this application, the specific implementation method for performing feature interaction within a neighborhood range is not limited, and those skilled in the art can make appropriate selections by referring to the prior art.

[0076] In the above scheme, during the feature extraction process of the first voxel features and the second voxel features, feature interaction within the neighborhood range of the first voxel features and the second voxel features can be performed to further enhance the feature representation of point cloud modalities and image modalities in voxel space, thereby improving the accuracy of target detection.

[0077] Furthermore, based on the above embodiments, in order to enhance the features, the embodiments of this application can also employ a cross-modal knowledge distillation method. In this embodiment, the target detection method provided by the embodiments of this application can further include the following steps:

[0078] Knowledge distillation is performed on the features of the fourth voxel using the features of the third voxel.

[0079] Specifically, when training a convolutional neural network, features in the point cloud modality (i.e., third voxel features) can be used to perform knowledge distillation on features in the image modality (i.e., fourth voxel features), thereby enabling the image to obtain the structured information contained in the point cloud.

[0080] As one implementation method, knowledge distillation of features in the point cloud modality to features in the image modality can be achieved by using features in the point cloud modality as follows: during training, the feature distance at the same location in the point cloud modality and the image modality is calculated, and then constrained by the mean squared error loss, thereby reducing the distance between features in different modalities in the feature space and promoting cross-modal knowledge transfer.

[0081] It should be noted that, in this embodiment of the application, in order to improve computational efficiency, only the feature distances of possible object locations can be calculated; of course, in order to improve the accuracy of the calculation, the feature distances of all locations can also be calculated, and this embodiment of the application does not make a specific limitation on this.

[0082] In the above scheme, after enhancing the feature representations of point cloud modalities and image modalities in voxel space, during the training of the convolutional neural network, the features corresponding to the point cloud modalities can be used to perform knowledge distillation on the features corresponding to the image modalities. This allows the image modalities to acquire the structured information contained in the point cloud modalities. Therefore, the feature representations of image modalities in voxel space can be further enhanced to improve the accuracy of object detection.

[0083] Furthermore, based on the above embodiments, this application provides a detailed description of an implementation method for detecting the feature to be detected in step S103. Step S103 may specifically include the following:

[0084] The target to be detected is detected using a decoder based on a feature transformer and the features to be detected.

[0085] Specifically, after obtaining the features to be detected in voxel space, a Transformer-based decoder can be used to obtain the potential object location and predict the target detection result at that location.

[0086] In one implementation, firstly, the target query feature and its corresponding position can be initialized; then, the target feature to be detected at the corresponding position in the voxel space can be sampled as the feature value based on the initialized query feature position; next, based on the obtained query feature and the corresponding feature value, multiple modules including self-attention, cross-region attention and feedforward network can be used to perform feature interaction and prediction, and finally, the target detection result corresponding to each query feature is predicted as the final prediction result.

[0087] In the above scheme, after obtaining a unified multimodal voxel space, feature sampling can be performed in the voxel space based on the decoder of the feature transformer, thereby making better use of multimodal features and performing more efficient and accurate target detection.

[0088] Please refer to Figure 2 , Figure 2 This application provides a structural block diagram of a target detection device 200, which may include: an acquisition module 201 for acquiring point cloud features and corresponding image features; a conversion module 202 for converting the point cloud features to a voxel space to obtain a first voxel feature in the voxel space, and converting the image features to the voxel space to obtain a second voxel feature in the voxel space; wherein the first coordinate system of the voxel space is different from the second coordinate system of the first acquisition device that acquires the point cloud data corresponding to the point cloud features and the third coordinate system of the second acquisition device that acquires the image data corresponding to the image features; and a detection module 203 for fusing the first voxel feature and the second voxel feature to obtain a feature to be detected, and detecting the target to be detected based on the feature to be detected.

[0089] In this embodiment, based on the image modality and point cloud modality, a modality-independent voxel space is provided. This voxel space has a different coordinate system than both the image modality and the point cloud modality. By transforming the acquired point cloud features and image features into the aforementioned voxel space, the point cloud modality and the image modality can be uniformly represented in the voxel space. Therefore, this facilitates subsequent feature interaction and improves the accuracy of target detection.

[0090] Furthermore, the conversion module 202 is specifically used for: determining the depth distribution corresponding to each pixel point based on the image features; for a voxel point in the voxel space, projecting the voxel point onto the two-dimensional image coordinate system corresponding to the image features to obtain the planar coordinate value and depth value of the voxel point in the two-dimensional image coordinate system; and obtaining the second voxel feature of the pixel point in the voxel space based on the planar coordinate value, the depth value, and the depth distribution of the pixel point corresponding to the voxel point.

[0091] In this embodiment, since image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to voxel space. Through the above view transformation process, a feature representation of the image features in three-dimensional voxel space can be established, thereby enabling point cloud modalities and image modalities to be uniformly represented in the voxel space, thus improving the accuracy of target detection.

[0092] Furthermore, the conversion module 202 is also configured to: determine the feature value corresponding to the pixel in the image feature based on the planar coordinate value, and determine the probability value corresponding to the pixel in the depth distribution based on the planar coordinate value and the depth value; multiply the feature value and the probability value to obtain the second voxel feature of the pixel in the voxel space.

[0093] In this embodiment, since image features do not contain structural information in three-dimensional space, a view transformation is required to convert the image features to voxel space. Through the above view transformation process, a feature representation of the image features in three-dimensional voxel space can be established, thereby enabling point cloud modalities and image modalities to be uniformly represented in the voxel space, thus improving the accuracy of target detection.

[0094] Furthermore, the detection module 203 is specifically used to: extract features from the first voxel features using a first three-dimensional convolutional neural network to obtain a third voxel feature, and extract features from the second voxel features using a second three-dimensional convolutional neural network to obtain a fourth voxel feature; and fuse the third voxel feature and the fourth voxel feature to obtain the feature to be detected.

[0095] In this embodiment, after converting point cloud features and image features to voxel space, two three-dimensional convolutional neural networks can be used to extract features from the obtained first voxel features and second voxel features respectively, so as to enhance the feature representation of point cloud modalities and image modalities in voxel space, thereby improving the accuracy of target detection.

[0096] Furthermore, during the feature extraction process of the first voxel feature and the second voxel feature, feature interaction within the neighborhood range is performed on the first voxel feature and the second voxel feature.

[0097] In the embodiments of this application, during the feature extraction process of the first voxel features and the second voxel features, feature interaction within the neighborhood range of the first voxel features and the second voxel features can be performed to further enhance the feature representation of point cloud modalities and image modalities in voxel space, thereby improving the accuracy of target detection.

[0098] Furthermore, the target detection device 200 also includes a distillation module for performing knowledge distillation on the fourth voxel feature using the third voxel feature.

[0099] In this embodiment, after enhancing the feature representations of the point cloud modality and the image modality in voxel space, during the training of the convolutional neural network, the features corresponding to the point cloud modality can be used to perform knowledge distillation on the features corresponding to the image modality, thereby enabling the image modality to obtain the structured information contained in the point cloud modality. Therefore, the feature representation of the image modality in voxel space can be further enhanced to improve the accuracy of target detection.

[0100] Furthermore, the detection module 203 is specifically used to: detect the target to be detected using a feature transformer-based decoder and the feature to be detected.

[0101] In this embodiment, after obtaining a unified multimodal voxel space, feature sampling can be performed in the voxel space based on the decoder of the feature transformer, thereby making better use of multimodal features and performing more efficient and accurate object detection.

[0102] Please refer to Figure 3 , Figure 3 This application provides a structural block diagram of an electronic device 300, which includes at least one processor 301, at least one communication interface 302, at least one memory 303, and at least one communication bus 304. The communication bus 304 enables direct communication between these components, the communication interface 302 facilitates signaling or data communication with other node devices, and the memory 303 stores machine-readable instructions executable by the processor 301. When the electronic device 300 is running, the processor 301 communicates with the memory 303 via the communication bus 304, and the target detection method described above is executed when the machine-readable instructions are invoked by the processor 301.

[0103] For example, the processor 301 in this embodiment of the application can read a computer program from the memory 303 via the communication bus 304 and execute the computer program to implement the following method: Step S101: Obtain point cloud features and their corresponding image features. Step S102: Convert the point cloud features to a voxel space to obtain the first voxel feature in the voxel space, and convert the image features to a voxel space to obtain the second voxel feature in the voxel space. Step S103: Fuse the first voxel feature and the second voxel feature to obtain the feature to be detected, and detect the target to be detected based on the feature to be detected.

[0104] The processor 301 may include one or more, and may be an integrated circuit chip with signal processing capabilities. The processor 301 may be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it may also be a special-purpose processor, including a Neural-network Processing Unit (NPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 301, some may be general-purpose processors, and others may be special-purpose processors.

[0105] The memory 303 includes one or more, which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0106] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof. In the embodiments of this application, electronic device 300 can be, but is not limited to, physical devices such as desktop computers, laptops, smartphones, smart wearable devices, and in-vehicle devices, or virtual devices such as virtual machines. Furthermore, electronic device 300 is not necessarily a single device; it can be a combination of multiple devices, such as a server cluster, etc.

[0107] This application also provides a computer program product, including a computer program stored on a computer-readable storage medium. The computer program includes computer program instructions. When the computer program instructions are executed by a computer, the computer can perform the steps of the target detection method in the above embodiments, such as: acquiring point cloud features and corresponding image features; converting the point cloud features to a voxel space to obtain a first voxel feature in the voxel space; and converting the image features to the voxel space to obtain a second voxel feature in the voxel space; wherein the first coordinate system of the voxel space is different from the second coordinate system corresponding to the first acquisition device that acquires the point cloud data corresponding to the point cloud features and the third coordinate system corresponding to the second acquisition device that acquires the image data corresponding to the image features; fusing the first voxel feature and the second voxel feature to obtain a feature to be detected, and detecting the target to be detected based on the feature to be detected.

[0108] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0109] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0111] It should be noted that if the function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0113] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target detection method, characterized in that, include: Obtain point cloud features and their corresponding image features; The point cloud features are transformed into a voxel space to obtain a first voxel feature in the voxel space, and the image features are transformed into the voxel space to obtain a second voxel feature in the voxel space; wherein, the first coordinate system of the voxel space is different from the second coordinate system of the first acquisition device that acquires the point cloud data corresponding to the point cloud features and the third coordinate system of the second acquisition device that acquires the image data corresponding to the image features, and the voxel space is an independent three-dimensional space that is unrelated to the second coordinate system and the third coordinate system; The first voxel feature and the second voxel feature are fused to obtain the feature to be detected, and the target to be detected is detected based on the feature to be detected. The step of converting the image features into the voxel space to obtain the second voxel features in the voxel space includes: Determine the depth distribution corresponding to each pixel based on the image features; For a voxel point in the voxel space, the voxel point is projected into the two-dimensional image coordinate system corresponding to the image feature to obtain the planar coordinate value and depth value of the voxel point in the two-dimensional image coordinate system. The second voxel feature of the pixel in the voxel space is obtained based on the plane coordinate value, the depth value, and the depth distribution of the pixel corresponding to the voxel point.

2. The target detection method according to claim 1, characterized in that, The step of obtaining the second voxel feature of the pixel in the voxel space based on the planar coordinate value, the depth value, and the depth distribution of the pixels corresponding to the voxel point includes: The feature value corresponding to the pixel in the image feature is determined based on the planar coordinate value, and the probability value corresponding to the pixel in the depth distribution is determined based on the planar coordinate value and the depth value. Multiplying the feature value by the probability value yields the second voxel feature of the pixel in the voxel space.

3. The target detection method according to claim 1, characterized in that, The step of fusing the first voxel features and the second voxel features to obtain the features to be detected includes: The first voxel feature is extracted using a first three-dimensional convolutional neural network to obtain the third voxel feature, and the second voxel feature is extracted using a second three-dimensional convolutional neural network to obtain the fourth voxel feature. The third voxel feature is fused with the fourth voxel feature to obtain the feature to be detected.

4. The target detection method according to claim 3, characterized in that, During the feature extraction process of the first voxel feature and the second voxel feature, a neighborhood-range feature interaction is performed on the first voxel feature and the second voxel feature.

5. The target detection method according to claim 3 or 4, characterized in that, The method further includes: Knowledge distillation is performed on the fourth voxel feature using the third voxel feature.

6. The target detection method according to any one of claims 1-4, characterized in that, The detection of the target based on the features to be detected includes: The target to be detected is detected using a feature transformer-based decoder and the features to be detected.

7. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.

8. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions that can be executed by the processor, and the processor can invoke the computer program instructions to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-6.

Images