Target detection methods, apparatus, computer equipment and storage media

Abstract

This application relates to a target detection method, apparatus, computer device, storage medium, and computer program product. The method includes: acquiring a 3D scene point cloud and a query language; determining candidate detection objects from the 3D scene point cloud and acquiring 2D images of the candidate detection objects; performing feature encoding on the 2D images, 3D scene point cloud, and query language respectively to obtain image-encoded features corresponding to the 2D images, first visual-encoded features corresponding to the 3D scene point cloud, and language-encoded features corresponding to the query language; performing feature fusion processing on the first visual-encoded features based on the image-encoded features and language-encoded features to obtain visual fusion-encoded features; decoding the visual fusion-encoded features to obtain second visual features; and determining the target detection object from the candidate detection objects based on the second visual features and the query language. This method can accurately detect targets.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a target detection method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of science and technology, object detection is being used in more and more fields. It is used to find all targets (objects) of interest in an image, so as to perform further analysis and processing on the targets.

[0003] Traditional techniques typically employ 3D (three-dimensional) vision grounding techniques for object detection. These techniques aim to locate object regions within a 3D point cloud scene based on a provided free-language description. Specifically, a set of 3D object proposals is first generated. Then, each proposal is fused with a language query to predict a similarity score between the 3D object proposal and the language query. Object detection is then performed on the 3D object proposals based on this similarity score. Furthermore, during similarity score prediction, the paired object-query score is maximized, while the score of unpaired objects is minimized.

[0004] However, traditional object detection methods still suffer from inaccuracies. Summary of the Invention

[0005] Therefore, it is necessary to provide an accurate target detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] Firstly, this application provides a target detection method, the method comprising:

[0007] Obtain the 3D scene point cloud and the language to be queried;

[0008] Candidate detection objects are determined from the three-dimensional scene point cloud, and two-dimensional images of the candidate detection objects are obtained;

[0009] The two-dimensional image, the three-dimensional scene point cloud, and the language to be queried are respectively feature encoded to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the language to be queried.

[0010] Based on the image coding features and the language coding features, the first visual coding features are subjected to feature fusion processing to obtain visual fusion coding features;

[0011] The visual fusion encoded features are decoded to obtain the second visual features;

[0012] Based on the second visual feature and the query language, the target detection object is determined from the candidate detection objects.

[0013] In one embodiment, prior to determining the candidate detection object from the 3D scene point cloud, the method further includes:

[0014] Retrieve the object-relational attribute language from the language to be queried;

[0015] Based on the object relational attribute language, data augmentation is performed on the 3D scene point cloud, and the 3D scene point cloud is updated.

[0016] In one embodiment, determining candidate detection objects from the 3D scene point cloud and acquiring 2D images of the candidate detection objects includes:

[0017] Clustering processing is performed on the point cloud of the 3D scene;

[0018] Candidate detection objects are identified from the clustered 3D scene point cloud, and 2D images of the candidate detection objects are obtained.

[0019] In one embodiment, the step of performing feature encoding on the two-dimensional image, the three-dimensional scene point cloud, and the query language respectively to obtain image encoding features corresponding to the two-dimensional image, first visual encoding features corresponding to the three-dimensional scene point cloud, and language encoding features corresponding to the query language includes:

[0020] The number of images in the two-dimensional image, the number of visual objects in the three-dimensional scene point cloud, and the number of words in the language to be queried are obtained.

[0021] Based on the number of images and the feature dimension of the two-dimensional image, feature encoding is performed on the two-dimensional image to obtain the image encoding feature corresponding to the two-dimensional image;

[0022] Based on the number of visual objects and the visual feature dimension of the three-dimensional scene point cloud, feature encoding is performed on the three-dimensional scene point cloud to obtain the first visual encoding feature corresponding to the three-dimensional scene point cloud.

[0023] Based on the number of words in the language and the linguistic feature dimension of the language to be queried, feature encoding is performed on the language to be queried to obtain the linguistic encoding features corresponding to the language to be queried.

[0024] In one embodiment, the step of performing multimodal feature fusion processing on the first visual coding feature based on the image coding feature and the language coding feature to obtain the visual fusion coding feature includes:

[0025] The first visual coding feature and the language coding feature are subjected to cross-attention processing to obtain the visual-language joint coding feature;

[0026] Cross-attention processing is performed on the visual language joint coding features and the image coding features to obtain visual fusion coding features.

[0027] In one embodiment, the step of decoding the visual fusion coding feature to obtain the second visual feature includes: performing multimodal feature coding fusion on the image coding feature, the first visual coding feature and the language coding feature to obtain the image fusion coding feature and the language fusion coding feature;

[0028] From the visual fusion coding features, determine a second candidate detection object whose correlation with the language coding features exceeds a preset threshold;

[0029] The second visual fusion coding features of the second candidate detection object are refined, and the refined second visual fusion coding features are cross-attention processed with the language fusion coding features to obtain the third visual coding features;

[0030] The third visual coding feature and the image fusion coding feature are subjected to cross-attention processing to obtain the fourth visual coding feature;

[0031] Cross-attention processing is performed on the fourth visual encoding feature and the visual fusion encoding feature to obtain the second visual feature of the second candidate detection object.

[0032] In one embodiment, determining the target detection object from the candidate detection objects based on the second visual feature and the query language includes:

[0033] Based on the second visual feature, the predicted three-dimensional parameters corresponding to the second candidate detection object are obtained, and the actual three-dimensional parameters corresponding to the second candidate detection object are obtained.

[0034] Based on the predicted 3D parameters and the actual 3D parameters, the 3D parameter loss value is obtained;

[0035] Based on the three-dimensional parameter loss value, the second visual feature is iteratively processed to obtain the third visual feature when the three-dimensional parameter loss value converges, and the language feature corresponding to the query language is obtained.

[0036] Based on the correlation between the third visual feature and the language feature, the target detection object is determined from the second candidate detection objects.

[0037] Secondly, this application provides a target detection device, the device comprising:

[0038] The data acquisition module is used to acquire the point cloud of the 3D scene and the language to be queried;

[0039] A two-dimensional analysis module is used to determine candidate detection objects from the three-dimensional scene point cloud and acquire two-dimensional images of the candidate detection objects;

[0040] The feature encoding module is used to perform feature encoding on the two-dimensional image, the three-dimensional scene point cloud, and the query language respectively, to obtain the image encoding feature corresponding to the two-dimensional image, the first visual encoding feature corresponding to the three-dimensional scene point cloud, and the language encoding feature corresponding to the query language;

[0041] The feature fusion module is used to perform feature fusion processing on the first visual coding feature based on the image coding feature and the language coding feature to obtain the visual fusion coding feature;

[0042] The feature decoding module is used to decode the visual fusion encoded features to obtain the second visual features;

[0043] The target detection module is used to determine the target detection object from the candidate detection objects based on the second visual features and the query language.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0045] Obtain the 3D scene point cloud and the language to be queried;

[0046] Candidate detection objects are determined from the three-dimensional scene point cloud, and two-dimensional images of the candidate detection objects are obtained;

[0047] The two-dimensional image, the three-dimensional scene point cloud, and the language to be queried are respectively feature encoded to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the language to be queried.

[0048] Based on the image coding features and the language coding features, the first visual coding features are subjected to feature fusion processing to obtain visual fusion coding features;

[0049] The visual fusion encoded features are decoded to obtain the second visual features;

[0050] Based on the second visual feature and the query language, the target detection object is determined from the candidate detection objects.

[0051] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0052] Obtain the 3D scene point cloud and the language to be queried;

[0053] Candidate detection objects are determined from the three-dimensional scene point cloud, and two-dimensional images of the candidate detection objects are obtained;

[0054] The two-dimensional image, the three-dimensional scene point cloud, and the language to be queried are respectively feature encoded to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the language to be queried.

[0055] Based on the image coding features and the language coding features, the first visual coding features are subjected to feature fusion processing to obtain visual fusion coding features;

[0056] The visual fusion encoded features are decoded to obtain the second visual features;

[0057] Based on the second visual feature and the query language, the target detection object is determined from the candidate detection objects.

[0058] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0059] Obtain the 3D scene point cloud and the language to be queried;

[0060] Candidate detection objects are determined from the three-dimensional scene point cloud, and two-dimensional images of the candidate detection objects are obtained;

[0061] The two-dimensional image, the three-dimensional scene point cloud, and the language to be queried are respectively feature encoded to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the language to be queried.

[0062] Based on the image coding features and the language coding features, the first visual coding features are subjected to feature fusion processing to obtain visual fusion coding features;

[0063] The visual fusion encoded features are decoded to obtain the second visual features;

[0064] Based on the second visual feature and the query language, the target detection object is determined from the candidate detection objects.

[0065] The aforementioned target detection method, apparatus, computer equipment, storage medium, and computer program product acquire a 3D scene point cloud and a query language; determine candidate detection objects from the 3D scene point cloud and acquire 2D images of the candidate detection objects; encode features for the 2D images, 3D scene point cloud, and query language respectively, obtaining image-encoded features corresponding to the 2D images, first visual-encoded features corresponding to the 3D scene point cloud, and language-encoded features corresponding to the query language; perform feature fusion processing on the visual-encoded features based on the image-encoded features and language-encoded features to obtain visual fusion-encoded features; decode the visual fusion-encoded features to obtain second visual-encoded features; and determine the target detection object from the candidate detection objects based on the second visual-encoded features and the query language. Throughout the process, by encoding, fusing, and decoding features from three dimensions—2D images, 3D vision, and language—2D image features can achieve better cross-modal feature fusion with language features. Therefore, by adding 2D image features mapped from the 3D scene point cloud as input to the 3D scene point cloud and the query language as input, the differences in cross-modal feature fusion between 3D vision and language can be better compensated for, resulting in more accurate target detection. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a diagram illustrating the application environment of a target detection method in one embodiment;

[0068] Figure 2 This is a flowchart illustrating a target detection method in one embodiment;

[0069] Figure 3 This is a flowchart illustrating the target detection method in another embodiment;

[0070] Figure 4 This is a structural block diagram of a target detection device in one embodiment;

[0071] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] The target detection method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. The user operates on terminal 102, interacting with it. Terminal 102 receives the user-uploaded 3D scene point cloud and query language, and generates a target detection request based on the 3D scene point cloud and query language, sending the target detection request to server 104. Server 104 extracts the 3D scene point cloud and query language from the target detection request, determines candidate detection objects from the 3D scene point cloud, and obtains 2D images of the candidate detection objects. It performs feature encoding on the 2D images, 3D scene point cloud, and query language respectively, obtaining image encoding features corresponding to the 2D images, first visual encoding features corresponding to the 3D scene point cloud, and language encoding features corresponding to the query language. Based on the image encoding features and language encoding features, it performs feature fusion processing on the first visual encoding features to obtain visual fusion encoding features. It decodes the visual fusion encoding features to obtain second visual features. Based on the second visual features and the query language, it determines the target detection object from the candidate detection objects. Further, server 104 sends the target detection object to terminal 102, which then displays it to the user.

[0074] Furthermore, the target detection method provided in this application embodiment can also be applied independently in the application environment of a terminal. Specifically, when the target detection method is applied independently to a terminal, the user operates on the terminal to achieve interaction with the terminal. The terminal receives the 3D scene point cloud and the language to be queried uploaded by the user, and performs a target detection task corresponding to the language to be queried in the 3D scene point cloud based on the 3D scene point cloud and the language to be queried.

[0075] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0076] In one exemplary embodiment, such as Figure 2As shown, a target detection method is provided, which is applied to... Figure 1 Taking server 104 as an example for explanation. Wherein:

[0077] S100: Obtain the 3D scene point cloud and the language to be queried.

[0078] Point clouds are massive collections of points representing the spatial distribution and surface characteristics of a target within the same spatial reference frame. Essentially, after acquiring the spatial coordinates of each sampled point on an object's surface, the resulting set of points is called a "point cloud." A 3D scene model is a concrete representation of the real world digitized in a computer, and point clouds can serve as a common 3D data format for 3D scenes. Compared to 2D images, 3D scene point clouds are sparse, noisy, and contain limited semantic information. The query language can be a description of an object, including but not limited to information such as the object's location and color.

[0079] Specifically, the user interacts with the terminal, which receives the user-uploaded 3D scene point cloud and query language. Based on these, the terminal generates an object detection request and sends it to the server. The server extracts the 3D scene point cloud and query language from the request and performs object detection operations on the 3D scene point cloud based on the query language. For example, if the 3D scene point cloud is of a living room, the query language could be something like "the red chair to the right of the sofa."

[0080] S200 identifies candidate detection objects from the 3D scene point cloud and acquires 2D images of the candidate detection objects.

[0081] Specifically, a 3D scene typically contains many objects. Therefore, candidate detection objects can be determined from the 3D point cloud corresponding to the 3D scene, i.e., the objects distributed in the 3D scene are used as candidate detection objects. Furthermore, since this application needs to introduce 2D images to compensate for the differences caused by the cross-modal fusion of 3D point clouds and language, the candidate detection objects also need to be processed to obtain 2D images of the candidate detection objects.

[0082] Furthermore, this application introduces a SIGGRAPH module, and the two-dimensional image of the candidate detection object is acquired through the virtual camera of the SIGGRAPH module. Specifically, for a given scene S and candidate detection object X... i The SIGGRAPH module can generate candidate detection objects X. i Two-dimensional image.

[0083] S300 performs feature encoding on the two-dimensional image, the three-dimensional scene point cloud, and the query language, respectively, to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the query language.

[0084] Specifically, the 3D scene point cloud is encoded, which essentially involves encoding the 3D scene point clouds corresponding to each candidate detection object. In addition, the 2D images of the candidate detection objects and the query language are also encoded.

[0085] Furthermore, a pre-trained feature extractor can be used to encode features of the 2D image of the candidate detection object, the 3D scene point cloud of the candidate detection object, and the query language. Typically, the PointNet++ backbone feature extractor is used to encode the 3D scene point cloud of the candidate detection object, obtaining the first visual encoding features corresponding to the 3D scene point cloud. The CLIP (Contrastive Language-Image Pre-Training) model is then used to encode features of the 2D image of the candidate detection object and the query language, obtaining the image encoding features corresponding to the 2D image and the language encoding features corresponding to the query language.

[0086] Furthermore, both the PointNet++ backbone feature extractor and the pre-trained CLIP model can be pre-trained. The training data can be from the current scene or from other scenes. During training, linear layers are designed for both the PointNet++ backbone feature extractor and the pre-trained CLIP model to calculate the classification loss, thereby constraining the 3D and 2D features obtained from the models and training feature extractors that conform to the 3D and 2D data features obtained from the projection.

[0087] S400, based on image coding features and language coding features, performs feature fusion processing on the first visual coding features to obtain visual fusion coding features.

[0088] Specifically, based on two-dimensional image coding features, it assists in the fusion of language coding features and first visual coding features. That is, it fuses image coding features, language coding features, and first visual coding features to obtain visually fused coding features. Further, the fusion processing between features is performed through N... E This is achieved through the interaction of attention layers. In practical applications, N is typically set to... E =3.

[0089] S500 decodes the visual fusion coding features to obtain the second visual features.

[0090] Specifically, the visual fusion encoded features are decoded to obtain the second visual features, which are visual features obtained by fusing the two-dimensional image with the query language. Furthermore, the decoding of the visual fusion encoded features is also achieved through an attention mechanism.

[0091] S600 determines the target detection object from candidate detection objects based on second visual features and the query language.

[0092] Specifically, the second visual feature at this time incorporates the features of the two-dimensional image and the query language. Based on the second visual feature and the query language, candidate detection objects related to the query language can be identified in the second visual feature, which can more accurately identify the target detection object than simply using the three-dimensional scene point cloud method.

[0093] In the aforementioned object detection method, a 3D scene point cloud and the query language are acquired. Candidate detection objects are determined from the 3D scene point cloud, and 2D images of these candidate objects are acquired. Feature encoding is performed on the 2D images, 3D scene point cloud, and query language to obtain image-encoded features corresponding to the 2D images, first visual encoding features corresponding to the 3D scene point cloud, and language encoding features corresponding to the query language. Based on the image-encoded features and language encoding features, feature fusion processing is performed on the visual encoding features to obtain visual fusion encoding features. The visual fusion encoding features are then decoded to obtain second visual encoding features. Based on the second visual encoding features and the query language, the target detection object is determined from the candidate detection objects. Throughout this process, by encoding, fusing, and decoding features from three dimensions—2D images, 3D vision, and language—the 2D image features can achieve better cross-modal feature fusion with language features. Therefore, by adding 2D image features mapped from the 3D scene point cloud as input to the 3D scene point cloud and the query language, the differences in cross-modal feature fusion between 3D vision and language can be better compensated for, resulting in more accurate object detection.

[0094] In an exemplary embodiment, a method for acquiring a two-dimensional image of a candidate detection object using a virtual camera includes: placing the virtual camera in front of the candidate detection object and then projecting the candidate detection object onto a two-dimensional plane to generate a corresponding two-dimensional image.

[0095] Furthermore, the virtual camera's position should be carefully set to generate an appropriate image representation of the 2D image. After excluding the upper and lower faces from the candidate detection objects, the centers of four faces are obtained. The distances between the centers of these four faces, Oli, and the scene center, Os, are calculated. The four faces can be represented by L, where L ranges from 0 to 3 and is an integer. The face L with the shortest distance to the scene center Os is the most prominent, ensuring that the candidate detection object is viewed from within the scene. At this point, the virtual camera is located within the scene at a distance d from the center of the face L with the shortest distance to the scene center Os. f The location is at a vertical distance d from the scene floor. up The location of the virtual camera. Therefore, the location of the virtual camera can be interpreted as Camp = [d]. f x, d f y, d up ], and the z-axis is aligned with the principal axis of the virtual camera, d f The orthogonal directions (x, y) are aligned with the corresponding axes of the image plane.

[0096] In one exemplary embodiment, before determining candidate detection objects from the 3D scene point cloud, the method further includes:

[0097] Retrieve the object-relational attribute language from the query language; based on the object-relational attribute language, perform data augmentation on the 3D scene point cloud and update the 3D scene point cloud.

[0098] Among them, object relation attribute language refers to the language that describes the relationship between objects in a 3D scene point cloud in the language to be queried, including position relation attribute language and color relation attribute language, etc.

[0099] Specifically, the object relational attribute language in the query language is obtained. For example, when the query language is "the red chair to the right of the sofa", the object relational attribute language in the query language is "to the right of the sofa" and "red". Based on the object relational attribute language in the query language, the point cloud data in the original 3D scene point cloud can be enhanced. Enhancement methods include: rotation, flipping, adding noise, translation, scaling, and color transformation.

[0100] Furthermore, when enhancing point cloud data using object-relational attribute language, the specific enhancement methods employed include:

[0101] 1. Rotation and Flipping: If around the Z-axis, generate a rotation angle theta_z, with a value range of [0, 360), plus a small random offset;

[0102] 2. Add noise: Generate a random number matrix of size (len(pc), 3), with values ​​in the range [0, 1), multiplied by 5e-3, and add noise to the first three columns (coordinate information) of the point cloud;

[0103] 3. Translation operation: Generate a shift vector of size (3,) with values ​​in the range [0, 1) minus 0.5, and add the shift vector shift to the first three columns (coordinate information) of the point cloud;

[0104] 4. Scaling operation: Generate a scaling factor (scale), with a value range of [0.98, 1.02), and multiply the first three columns (coordinate information) of the point cloud by the scaling factor (scale);

[0105] 5. Color transformation: Multiply the color value color by a random number matrix in the range [0.98, 1.02), with the same size as color, and subtract the average RGB value self.mean_rgb from the color value color.

[0106] In this embodiment, by performing data augmentation on the 3D scene point cloud, the robustness of the point cloud data can be improved, making the 3D scene point cloud data more accurate, and thus enabling more accurate identification of candidate detection objects from the 3D scene point cloud.

[0107] In an exemplary embodiment, determining candidate detection objects from a 3D scene point cloud and acquiring 2D images of the candidate detection objects includes:

[0108] Clustering is performed on the 3D scene point cloud; candidate detection objects are determined from the clustered 3D scene point cloud, and 2D images of the candidate detection objects are obtained.

[0109] Specifically, a 3D scene contains several objects, and the corresponding 3D scene point cloud also contains several discrete points corresponding to these objects. Therefore, clustering can be performed on these discrete points in the 3D scene point cloud, grouping the point clouds corresponding to individual objects together and separating the point clouds corresponding to different objects, thus obtaining point clouds that represent individual objects in the 3D scene point cloud. At this point, candidate detection objects can be efficiently identified from the clustered 3D scene point cloud, and 2D images of these candidate detection objects can be obtained.

[0110] Furthermore, the clustering process can be performed by an object detector, which here refers to an object detector in the 3D scene. The object detector identifies candidate detection objects from the 3D scene point cloud. This object detector is pre-trained based on scene data and supports outputting candidate detection objects corresponding to the 3D scene point cloud when given input.

[0111] In this embodiment, by performing clustering processing on the 3D scene point cloud, candidate detection objects can be determined more efficiently and accurately from the clustered 3D scene point cloud, avoiding errors in the candidate detection objects determined from the 3D scene point cloud due to human judgment mistakes.

[0112] In an exemplary embodiment, feature encoding is performed on the two-dimensional image, the three-dimensional scene point cloud, and the query language, respectively, to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the query language, including:

[0113] The process involves obtaining the number of images in a 2D image, the number of visual objects in a 3D scene point cloud, and the number of words in the query language. Based on the number of images and the feature dimensions of the 2D image, feature encoding is performed on the 2D image to obtain the corresponding image encoding features. Based on the number of visual objects and the visual feature dimensions of the 3D scene point cloud, feature encoding is performed on the 3D scene point cloud to obtain the corresponding first visual encoding features. Based on the number of words in the language and the language feature dimensions of the query language, feature encoding is performed on the query language to obtain the corresponding language encoding features.

[0114] Specifically, when encoding the 3D scene point cloud of candidate detection objects, the PointNet++ backbone feature extractor can be used. This adds the 3D scene point cloud features of the candidate detection objects to the learnable 3D position encoding in the PointNet++ backbone feature extractor, and uses a small MLP (Multilayer Perceptron) to transmit the encoded coordinates of the point cloud. Further, the first visual encoded features corresponding to the 3D scene point cloud... , where n v This represents the number of visual objects in the 3D scene point cloud, i.e., the number of candidate detection objects in the 3D scene point cloud; c v This represents the visual feature dimension of the 3D scene point cloud; the number of candidate detection objects in the 3D scene point cloud is obtained; based on the number of candidate detection objects and the visual feature dimension of the 3D scene point cloud of the candidate detection objects, feature encoding is performed on the 3D scene point cloud of the candidate detection objects to obtain the first visual encoding feature corresponding to the 3D scene point cloud of the candidate detection objects. That is, the obtained first visual encoding feature needs to be set based on the number of visual objects in the 3D scene point cloud and the visual feature dimension of the 3D scene point cloud.

[0115] When encoding features of the two-dimensional image of the candidate detection object and the query language, the CLIP model can be used. Image encoding features I∈R ni ×ciWhere ni is the number of images in the two-dimensional image and ci is the feature dimension of the two-dimensional image. That is, the number of images in the two-dimensional image is obtained, which is also the number of images of the candidate detection objects. Based on the number of images and the feature dimension of the two-dimensional image, the two-dimensional image is feature-encoded to obtain the image coding features corresponding to the two-dimensional image. That is, the obtained image coding features need to be set based on the number of images in the two-dimensional image and the feature dimension of the two-dimensional image.

[0116] In addition, language coding features , where n l c represents the number of words in the language to be queried. l The language feature dimension of the query language is the number of images obtained in the two-dimensional image, which is also the number of candidate detection objects. Based on the number of language words and the language feature dimension of the query language, the query language is feature-encoded to obtain the language encoding features corresponding to the query language. That is, the obtained language encoding features need to be set based on the number of language words and the language feature dimension of the query language.

[0117] In this embodiment, by acquiring the number of images in a two-dimensional image, the number of visual objects in a three-dimensional scene point cloud, and the number of words in the language to be queried, and by combining the feature dimensions of the two-dimensional image, the visual feature dimensions of the three-dimensional scene point cloud, and the language feature dimensions of the language to be queried, the two-dimensional image, the three-dimensional scene point cloud, and the three-dimensional scene point cloud can be accurately encoded, so that the obtained encoded features are within a specific range.

[0118] In an exemplary embodiment, based on image coding features and language coding features, a multimodal feature fusion process is performed on the first visual coding features to obtain visual fusion coding features, including:

[0119] Cross-attention processing is applied to the first visual coding features and the language coding features to obtain the visual-language joint coding features; cross-attention processing is applied to the visual-language joint coding features and the image coding features to obtain the visual fusion coding features.

[0120] Specifically, in multimodal fusion coding, the first visual coding features, language coding features, and image coding features are combined through N... EThe cross-attention layers influence each other. In each layer, the first visual encoding feature undergoes cross-attention processing with the language encoding feature and is updated using standard key-value attention to obtain the joint visual-language encoding feature. Further, a two-dimensional image encoding feature is introduced, and the resulting joint visual-language encoding feature is then subjected to standard cross-attention processing with the image encoding feature to obtain the final visual fusion encoding feature, which combines the language encoding feature and the two-dimensional image encoding feature. Furthermore, the updated visual fusion encoding feature, image encoding feature, and language encoding feature in each layer serve as input to the next layer. All features output from the last layer are then subjected to feature projection, which projects the fused features of the three modalities onto the same space for alignment loss calculation, thus narrowing the distance between similar features across the multimodalities. In other words, each language feature is composed of many words, each image feature is composed of multiple object images, and each visual feature is composed of multiple object features. To find the one-to-one correspondence between them, the predicted value of the cross-attention layer is compared with the true value ground (gt). Correct predictions are positive samples, and incorrect predictions are negative samples.

[0121] The expression for the loss is: ,and In the expression, D W P represents the Euclidean distance between two sample features, Y represents the feature dimension, M represents the label indicating whether the two samples match, and N represents the number of samples.

[0122] In this embodiment, by introducing two-dimensional image coding features, the difference between the fusion of the first visual coding features and the language coding features can be reduced, enabling the learning of a more accurate language-visual correspondence, thereby achieving more accurate and fine-grained target detection.

[0123] In one exemplary embodiment, such as Figure 3 As shown, S500 includes:

[0124] S510 performs multimodal feature encoding fusion on image coding features, first visual coding features, and language coding features to obtain image fusion coding features and language fusion coding features.

[0125] Specifically, in the cross-attention layer, when performing multimodal feature fusion processing on the first visual coding features based on image coding features and language coding features, multimodal feature coding fusion is also performed on the image coding features and language coding features simultaneously, resulting in image fused coding features and language fused coding features. Here, the language fused coding features refer to the fused coding features obtained after cross-attention processing with the first visual coding features, and the image fused coding features refer to the fused coding features obtained after cross-attention processing with the joint visual-language coding features.

[0126] S520: Identify a second candidate detection object from the visual fusion coding features whose correlation with the language coding features exceeds a preset threshold.

[0127] Specifically, non-parametric queries are used for visual fusion encoded features, meaning the visual fusion encoded features themselves are used as queries. However, not all visual fusion encoded features can be used as queries. Therefore, it is necessary to select visual fusion encoded features whose relevance to language encoded features exceeds a preset threshold for subsequent processing. One method for selecting visual fusion encoded features with relevance exceeding the preset threshold is through confidence score detection. Secondary candidate detection objects are then identified corresponding to visual fusion encoded features whose relevance to language encoded features exceeds the preset threshold. In other words, when fusing multimodal features, the correlation between each modality feature can be learned. The resulting fused human features can perform a preliminary screening of a large number of input visual features, allowing for decoding of the preliminarily selected objects.

[0128] S530, refine the second visual fusion coding features of the second candidate detection object, and perform cross-attention processing on the refined second visual fusion coding features and language fusion coding features to obtain the third visual coding features.

[0129] Specifically, the second visual fusion encoded features of the second candidate detection object are fed into the MLP model to obtain the query vector corresponding to the second visual fusion encoded features. The query vector is then decoded into fusion encoded features using multiple decoding layers in a residual manner. In each decoding layer, a certain attention processing operation is performed for decoding. First, the second visual fusion encoded features of the second candidate detection object are refined through query self-attention, achieving feature refinement within the feature modality. Second, after refining the second visual fusion encoded features, cross-attention processing is performed between the refined second visual fusion encoded features and the language fusion encoded features to obtain third visual encoded features, thereby adding constraints on the language features.

[0130] S540 performs cross-attention processing on the third visual coding features and the image fusion coding features to obtain the fourth visual coding features.

[0131] S550 performs cross-attention processing on the fourth visual coding feature and the visual fusion coding feature to obtain the second visual feature of the second candidate detection object.

[0132] Specifically, cross-attention processing is performed on the third visual coding feature and the image fusion coding feature to obtain the fourth visual coding feature. Furthermore, since the fourth visual coding feature at this point is the visual coding feature corresponding to the second subsequent detection object after filtering, it is a local feature of the 3D scene. Therefore, cross-attention processing is also performed on the fourth visual coding feature and the visual fusion coding feature of the entire scene point cloud to give each query global attention, thus obtaining the second visual feature of the second candidate detection object.

[0133] In this embodiment, by using an attention mechanism to decode the visual fusion coding features, the decoded second visual features can be constrained by language features and image features, thereby obtaining more accurate second visual features.

[0134] In an exemplary embodiment, determining the target detection object from the candidate detection objects based on the second visual features and the query language includes:

[0135] Based on the second visual features, the predicted 3D parameters corresponding to the second candidate detection object are obtained, and the actual 3D parameters corresponding to the second candidate detection object are also obtained. Based on the predicted 3D parameters and the actual 3D parameters, the 3D parameter loss value is obtained. Based on the 3D parameter loss value, the second visual features are iteratively processed to obtain the third visual features when the 3D parameter loss value converges, and the language features corresponding to the query language are obtained. Based on the correlation between the third visual features and the language features, the target detection object is determined from the second candidate detection objects.

[0136] Specifically, loss calculations are performed on the obtained features in each layer of the decoding layer. This involves designing linear layers to project the parameters of the second visual features obtained in each layer onto these linear layers, yielding the predicted 3D parameters corresponding to the second candidate detection object. These linear layers act as transformation layers, converting the original feature dimensions to the dimensions of specified features, thus enabling feature loss calculations with the actual 3D parameters of the same dimension. The specified feature dimensions in this application include, but are not limited to, the center coordinates, height, and width of the detection object, and may also include parameters such as category. The process involves obtaining the actual 3D parameters corresponding to the second candidate detection object; calculating the alignment loss between the predicted and actual 3D parameters to obtain the 3D parameter loss value; iterating through each layer of the decoding layer based on the 3D parameter loss value to obtain the third visual feature when the 3D parameter loss value converges. Since each layer can employ cross-attention to constrain language and image, the fourth visual feature most relevant to the language features can be determined from the final output third visual features, and the detection object corresponding to the fourth visual feature is taken as the target detection object, i.e., the target detection object is determined from the second candidate detection objects.

[0137] Furthermore, additional linear layers with branches similar to those in 3D constraints can be designed to apply alignment loss constraints to the center coordinates of the detected object in the 2D image, the position and orientation of the virtual camera, the category prediction of the 2D image, and the final category and position of the detected object.

[0138] In this embodiment, based on the second visual features, the predicted three-dimensional parameters corresponding to the second candidate detection object are obtained. By calculating the loss between the predicted and actual three-dimensional parameters, the loss value can be iteratively processed, thereby updating the predicted three-dimensional parameters based on the loss value. This updates the visual features corresponding to the predicted three-dimensional parameters, resulting in more accurate third visual features. Then, the feature with the highest correlation to the language features among the third visual features is queried, and the candidate detection object corresponding to this feature is used as the target detection object. In other words, each decoding layer has a predicted value that is ultimately compared with the actual value for loss calculation. Through multi-layer iteration, each layer has constraints, ensuring that the query most relevant to the language features achieves the highest correlation, thus enabling more accurate target detection.

[0139] In one embodiment, a two-dimensional image is generated as auxiliary features by taking 3D scene point cloud data and the corresponding language description of the target object in the 3D scene as input. This image is then fed into multimodal feature encoding, fusion, and decoding. The final target features are obtained by using the feature alignment loss between the multimodals generated during the constraint period, thereby determining the target detection object corresponding to the target features and locating the target object. Further, the object detection method includes:

[0140] 1. Obtain input data. Obtain the 3D scene point cloud and the query language, where the query language includes relational attribute words of objects in the 3D scene.

[0141] 2. Data Preprocessing. The 3D scene point cloud is augmented using relational attribute terms from the query language, and then updated.

[0142] 3. Object detection in 3D scene point cloud. This involves detecting objects composed of several points in the updated 3D scene point cloud, and using each object as a candidate for detection.

[0143] 4. Two-dimensional image acquisition. A virtual camera is used to project the candidate detection objects in the 3D scene point cloud onto a 2D plane to obtain a 2D image of the candidate detection objects.

[0144] 5. Feature Encoding. The PointNet++ backbone is used to encode the features of the 3D scene point clouds corresponding to candidate detection objects. The encoded 3D scene point cloud features, i.e., the first visual encoded features, need to be determined based on the number of candidate detection objects in the 3D scene point cloud and the dimensionality of the 3D scene point cloud features. Secondly, the CLIP model is used to encode the features of the 2D images corresponding to the candidate detection objects. The image encoded features need to be determined based on the number of 2D images corresponding to the candidate detection objects and the dimensionality of the 2D image features. Finally, the CLIP model is used to encode the features of the query language corresponding to the candidate detection objects. The language encoded features need to be determined based on the number of candidate detection objects in the 3D scene point cloud and the dimensionality of the 3D scene point cloud features.

[0145] 6. Multimodal Coding Feature Fusion. Image coding features, language coding features, and first-person visual coding features are fused through multiple cross-attention layers. In each cross-attention layer, image coding features, language coding features, and first-person visual coding features are fused, and the fused and updated coding features corresponding to the three features in each layer serve as the three inputs to the next cross-attention layer. Finally, the three fused features output from the last layer are projected onto the same space for alignment loss calculation, reducing the distance between the common features of the multimodal features.

[0146] 7. Multimodal Fusion Feature Decoding. Based on the visual fusion coding features obtained after loss alignment in the last layer of feature fusion, the correlation between the visual fusion coding features and the language coding features can be determined for preliminary screening of the visual fusion coding features, obtaining the top k visual fusion coding features with the highest correlation. These top k visual fusion coding features are used as the second visual fusion coding features, and their corresponding detection objects are used as the second candidate detection objects. The preliminary screening can be done by setting a threshold. The second visual fusion coding features are input into the decoding layer. First, the second visual fusion coding features are refined. Then, the refined second visual fusion coding features are cross-attention processed with the language fusion coding features to add constraints on language features, resulting in the third visual coding features. The third visual coding features are then cross-attention processed with the image fusion coding features to add constraints on image features, resulting in the fourth visual coding features. Finally, the fourth visual coding features are cross-attention processed with the visual fusion coding features of the entire scene to add global constraints, resulting in the second visual features of the second candidate detection objects.

[0147] 8. Prediction. The second visual feature is input into the prediction head. Based on the second visual feature, the predicted 3D parameters corresponding to the second candidate detection object are obtained, as well as the actual 3D parameters corresponding to the second candidate detection object. Based on the predicted and actual 3D parameters, the 3D parameter loss value is obtained. According to the 3D parameter loss value, the second visual feature is iteratively processed to obtain the third visual feature when the 3D parameter loss value converges, and the language features corresponding to the query language are obtained. The feature with the highest correlation between the third visual feature and the language feature is taken as the visual feature of the target detection object. Based on the visual feature of the target detection object, the target detection object is determined from the second candidate detection objects.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0149] Based on the same inventive concept, this application also provides a target detection apparatus for implementing the target detection method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more target detection method apparatus embodiments provided below can be found in the limitations of the target detection method described above, and will not be repeated here.

[0150] In one exemplary embodiment, such as Figure 4 As shown, a target detection device is provided, comprising: a data acquisition module 100, a two-dimensional analysis module 200, a feature encoding module 300, a feature fusion module 400, a feature decoding module 500, and a target detection module 600, wherein:

[0151] Data acquisition module 100 is used to acquire 3D scene point cloud and the language to be queried;

[0152] The two-dimensional analysis module 200 is used to determine candidate detection objects from the three-dimensional scene point cloud and acquire two-dimensional images of the candidate detection objects;

[0153] The feature encoding module 300 is used to perform feature encoding on the two-dimensional image, the three-dimensional scene point cloud, and the language to be queried, respectively, to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the language to be queried.

[0154] The feature fusion module 400 is used to perform feature fusion processing on the first visual coding features based on image coding features and language coding features to obtain visual fusion coding features;

[0155] The feature decoding module 500 is used to decode the visual fusion encoded features to obtain the second visual features;

[0156] The target detection module 600 is used to determine the target detection object from the candidate detection objects based on the second visual features and the query language.

[0157] In one embodiment, the system further includes a point cloud enhancement module, which is used to obtain the object relational attribute language in the query language; and to perform data enhancement on the 3D scene point cloud based on the object relational attribute language, thereby updating the 3D scene point cloud.

[0158] In one embodiment, the two-dimensional analysis module 200 is further used to perform clustering processing on the three-dimensional scene point cloud; determine candidate detection objects from the clustered three-dimensional scene point cloud, and obtain two-dimensional images of the candidate detection objects.

[0159] In one embodiment, the feature encoding module 300 is further configured to acquire the number of images in the two-dimensional image, the number of visual objects in the three-dimensional scene point cloud, and the number of words in the language to be queried; based on the number of images and the feature dimension of the two-dimensional image, feature encoding is performed on the two-dimensional image to obtain the image encoding feature corresponding to the two-dimensional image; based on the number of visual objects and the visual feature dimension of the three-dimensional scene point cloud, feature encoding is performed on the three-dimensional scene point cloud to obtain the first visual encoding feature corresponding to the three-dimensional scene point cloud; based on the number of words in the language and the language feature dimension of the language to be queried, feature encoding is performed on the language to be queried to obtain the language encoding feature corresponding to the language to be queried.

[0160] In one embodiment, the feature fusion module 400 is further configured to perform cross-attention processing on the first visual coding features and the language coding features to obtain visual-language joint coding features; and to perform cross-attention processing on the visual-language joint coding features and the image coding features to obtain visual fusion coding features.

[0161] In one embodiment, the feature decoding module 500 is further configured to perform multimodal feature encoding fusion on image encoding features, first visual encoding features, and language encoding features to obtain image fusion encoding features and language fusion encoding features; determine a second candidate detection object from the visual fusion encoding features whose correlation with the language encoding features exceeds a preset threshold; refine the second visual fusion encoding features of the second candidate detection object, and perform cross-attention processing on the refined second visual fusion encoding features and language fusion encoding features to obtain a third visual encoding feature; perform cross-attention processing on the third visual encoding features and image fusion encoding features to obtain a fourth visual encoding feature; and perform cross-attention processing on the fourth visual encoding features and visual fusion encoding features to obtain the second visual feature of the second candidate detection object.

[0162] In one embodiment, the target detection module 600 is further configured to: obtain the predicted three-dimensional parameters corresponding to the second candidate detection object based on the second visual features; obtain the actual three-dimensional parameters corresponding to the second candidate detection object; obtain a three-dimensional parameter loss value based on the predicted three-dimensional parameters and the actual three-dimensional parameters; perform iterative processing on the second visual features according to the three-dimensional parameter loss value to obtain the third visual features when the three-dimensional parameter loss value converges, and obtain the language features corresponding to the query language; and determine the target detection object from the second candidate detection objects based on the correlation between the third visual features and the language features.

[0163] Each module in the aforementioned target detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0164] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as 3D scene point clouds and query languages. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a target detection method.

[0165] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0166] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0167] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0168] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0170] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A target detection method, characterized in that, The method includes: Obtain the 3D scene point cloud and the language to be queried; Candidate detection objects are determined from the three-dimensional scene point cloud, and two-dimensional images of the candidate detection objects are obtained; The two-dimensional image, the three-dimensional scene point cloud, and the language to be queried are respectively feature encoded to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the language to be queried. Based on the image coding features and the language coding features, the first visual coding features are subjected to feature fusion processing to obtain visual fusion coding features; The visual fusion encoded features are decoded to obtain the second visual features; Based on the second visual feature and the query language, the target detection object is determined from the candidate detection objects; The step of performing multimodal feature fusion processing on the first visual coding features based on the image coding features and the language coding features to obtain visual fusion coding features includes: The first visual coding feature and the language coding feature are subjected to cross-attention processing to obtain the visual-language joint coding feature; The visual language joint coding features and the image coding features are subjected to cross-attention processing to obtain visual fusion coding features; The decoding process of the visual fusion encoded features to obtain the second visual feature includes: Multimodal feature encoding fusion is performed on the image encoding features, the first visual encoding features, and the language encoding features to obtain image fusion encoding features and language fusion encoding features; From the visual fusion coding features, determine a second candidate detection object whose correlation with the language coding features exceeds a preset threshold; The second visual fusion coding features of the second candidate detection object are refined, and the refined second visual fusion coding features are cross-attention processed with the language fusion coding features to obtain the third visual coding features; The third visual coding feature and the image fusion coding feature are subjected to cross-attention processing to obtain the fourth visual coding feature; Cross-attention processing is performed on the fourth visual encoding feature and the visual fusion encoding feature to obtain the second visual feature of the second candidate detection object.

2. The method according to claim 1, characterized in that, Before determining the candidate detection objects from the 3D scene point cloud, the method further includes: Retrieve the object-relational attribute language from the language to be queried; Based on the object relational attribute language, data augmentation is performed on the 3D scene point cloud, and the 3D scene point cloud is updated.

3. The method according to claim 1, characterized in that, The step of determining candidate detection objects from the 3D scene point cloud and acquiring 2D images of the candidate detection objects includes: Clustering processing is performed on the point cloud of the 3D scene; Candidate detection objects are identified from the clustered 3D scene point cloud, and 2D images of the candidate detection objects are obtained.

4. The method according to claim 1, characterized in that, The step of performing feature encoding on the two-dimensional image, the three-dimensional scene point cloud, and the query language respectively to obtain the image encoding features corresponding to the two-dimensional image, the first visual encoding features corresponding to the three-dimensional scene point cloud, and the language encoding features corresponding to the query language includes: The number of images in the two-dimensional image, the number of visual objects in the three-dimensional scene point cloud, and the number of words in the language to be queried are obtained. Based on the number of images and the feature dimension of the two-dimensional image, feature encoding is performed on the two-dimensional image to obtain the image encoding feature corresponding to the two-dimensional image; Based on the number of visual objects and the visual feature dimension of the three-dimensional scene point cloud, feature encoding is performed on the three-dimensional scene point cloud to obtain the first visual encoding feature corresponding to the three-dimensional scene point cloud. Based on the number of words in the language and the linguistic feature dimension of the language to be queried, feature encoding is performed on the language to be queried to obtain the linguistic encoding features corresponding to the language to be queried.

5. The method according to claim 1, characterized in that, The step of determining the target detection object from the candidate detection objects based on the second visual feature and the query language includes: Based on the second visual feature, the predicted three-dimensional parameters corresponding to the second candidate detection object are obtained, and the actual three-dimensional parameters corresponding to the second candidate detection object are obtained. Based on the predicted three-dimensional parameters and the actual three-dimensional parameters, the three-dimensional parameter loss value is obtained; Based on the three-dimensional parameter loss value, the second visual feature is iteratively processed to obtain the third visual feature when the three-dimensional parameter loss value converges, and the language feature corresponding to the query language is obtained. Based on the correlation between the third visual feature and the language feature, the target detection object is determined from the second candidate detection objects.

6. A target detection device, characterized in that, The device includes: The data acquisition module is used to acquire the point cloud of the 3D scene and the language to be queried; A two-dimensional analysis module is used to determine candidate detection objects from the three-dimensional scene point cloud and acquire two-dimensional images of the candidate detection objects; The feature encoding module is used to perform feature encoding on the two-dimensional image, the three-dimensional scene point cloud, and the query language respectively, to obtain the image encoding feature corresponding to the two-dimensional image, the first visual encoding feature corresponding to the three-dimensional scene point cloud, and the language encoding feature corresponding to the query language; The feature fusion module is used to perform feature fusion processing on the first visual coding feature based on the image coding feature and the language coding feature to obtain the visual fusion coding feature; The feature decoding module is used to decode the visual fusion encoded features to obtain the second visual features; The target detection module is used to determine the target detection object from the candidate detection objects based on the second visual features and the query language; The feature fusion module is further configured to perform cross-attention processing on the first visual coding feature and the language coding feature to obtain a visual-language joint coding feature; and to perform cross-attention processing on the visual-language joint coding feature and the image coding feature to obtain a visual fusion coding feature. The feature decoding module is further configured to perform multimodal feature encoding fusion on the image encoding features, the first visual encoding features, and the language encoding features to obtain image fusion encoding features and language fusion encoding features; determine a second candidate detection object from the visual fusion encoding features whose correlation with the language encoding features exceeds a preset threshold; refine the second visual fusion encoding features of the second candidate detection object, and perform cross-attention processing on the refined second visual fusion encoding features and the language fusion encoding features to obtain a third visual encoding feature; perform cross-attention processing on the third visual encoding features and the image fusion encoding features to obtain a fourth visual encoding feature; and perform cross-attention processing on the fourth visual encoding features and the visual fusion encoding features to obtain the second visual features of the second candidate detection object.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images