A 3D object detection method for autonomous driving

Abstract

The present invention relates to the field of autonomous driving technology, and discloses a method for 3D target detection in autonomous driving. The method comprises: determining an image to be detected and preset text query features; inputting the image to be detected and the preset text query features into a 3D perception model, and realizing 3D target detection and 3D scene description based on the 3D perception model; wherein the 3D perception model is based on 3D coordinate encoding features and a large language model, and the large language model is instance-level fused and aligned with the image to be detected, and the matching relationship between the 3D target in the image to be detected and each vector of the preset text query feature is determined, and a description of the spatial relationship between each 3D target and adjacent 3D targets is generated. The present invention can realize the perception of any known or unknown 3D object in a 3D scene; it improves the understanding of complex driving scenes by the on-board autonomous driving system, and makes the autonomous driving system's perception of objects in the surrounding world closer to human thinking logic.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving 3D target detection method. Background Art

[0002] In the current field of autonomous driving, vehicles rely on various on-board sensors as input, such as cameras, lidar, and millimeter-wave radar. Real-time perception of the surrounding environment and prediction of the actions of various actors are crucial to the entire system. According to the source of input data, it can be divided into 3D scene perception based on lidar, visual perception solutions based on BEV (Bird's Eye View) representation, and multimodal fusion perception solutions.

[0003] The existing method PETR (Position Embedding Transformation for Multi-View 3D Object Detection) extracts multi-view features from the vehicle camera image through a 2D image feature extraction network (swin-transformer) Figure 2 D features, and then embed the 3D position encoding into the multi-view features to generate 3D position-aware input features. The detection part uses a transformer decoder similar to DETR3D to iteratively optimize the query feature query and input features to obtain the target feature, and finally realizes end-to-end 3D object detection through the prediction network.

[0004] However, the above methods can only detect the marked obstacle categories, and have no detection capability for unmarked objects. In real scenarios, it is impossible to exhaust all the marked categories, so the mainstream perception methods can only detect certain fixed categories of objects, and have weak perception and processing capabilities for unknown scenarios. Conventional methods can use different types of perception algorithms, and for different driving scenarios (such as highways: highway barriers, toll booths, and urban areas), multiple perception modules run in time or in parallel, but this will result in cumbersome processing procedures.

[0005] In summary, mainstream perception algorithms are limited by the manual labeling of training data and are difficult to meet the needs of open-set perception in the real world, that is, existing models are difficult to fully identify object categories outside the training set. Therefore, when self-driving cars face complex scenes on urban roads, such as the sudden appearance of animals, they may react in a manner that is not in line with human common sense, affecting vehicle driving safety.

[0006] Therefore, designing a solution that can perceive and detect arbitrary 3D objects has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a 3D target detection method for autonomous driving in view of the problems existing in the prior art.

[0008] An embodiment of the present invention provides a method for detecting 3D objects in autonomous driving, comprising: determining an image to be detected and preset text query features; the image to be detected is a real-time image acquired by multiple onboard cameras during the autonomous driving process of the vehicle; the image to be detected and the preset text query features are input into a trained 3D perception model, and 3D object detection and 3D scene description are implemented based on the 3D perception model; wherein the 3D perception model is based on 3D coordinate encoding features and a large language model, and instance-level fusion and alignment of the large language model with the image to be detected is performed, and a matching relationship between the 3D object in the image to be detected and each vector of the preset text query features is determined, and a spatial relationship description of each 3D object and adjacent 3D objects is generated.

[0009] The beneficial effects of the present invention are as follows: the 3D perception model is based on a large language model and a multimodal (i.e., image data and text data) algorithm model. Through feature alignment of language-visual images, the relevant knowledge in the large language model is embedded into the visual BEV features, and the 3D coordinate encoding features are introduced in the feature alignment process, thereby determining the matching relationship between the 3D target in the image and each vector of the preset text query feature, generating a spatial relationship description for each 3D target and adjacent 3D targets, and exploring the intrinsic connection between each 3D target in the image, thereby realizing the perception of any known or unknown 3D objects in the 3D scene; improving the on-board autonomous driving system's understanding of complex driving scenarios, and making the autonomous driving system's perception of objects in the surrounding world closer to human thinking logic.

[0010] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of a method for detecting 3D objects in an autonomous driving system according to an exemplary embodiment of the present application is shown;

[0012] Figure 2 A flowchart of a method for pre-building and training a 3D perception model is shown as an exemplary embodiment of the present application;

[0013] Figure 3 for Figure 2 A flowchart of step S3 in the illustrated embodiment in an exemplary embodiment;

[0014] Figure 4 for Figure 2 A flowchart of step S4 in the illustrated embodiment in an exemplary embodiment;

[0015] Figure 5 for Figure 2A flowchart of step S5 in the illustrated embodiment in an exemplary embodiment;

[0016] Figure 6 for Figure 2 A flowchart of step S6 in the illustrated embodiment in an exemplary embodiment;

[0017] Figure 7 Schematic diagram of the process of aligning features of different modalities;

[0018] Figure 8 Schematic diagram for initializing the interaction between image query features and text query features;

[0019] Fig. 9 Schematic diagram of image-text feature interaction for coordinate embedding;

[0020] Fig.10 Schematic diagram of the relative orientation relationship between 3D objects predicted by image query features. DETAILED DESCRIPTION

[0021] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0022] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0023] The solution provided in the embodiments of the present invention can be executed by any electronic device, for example, a terminal device, or jointly executed by a terminal device and a server. The server can be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc., and is not limited here.

[0024] Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of the present application showing a method for detecting 3D objects in an autonomous driving system. Figure 1 As shown, in an exemplary embodiment, the method for 3D target detection based on autonomous driving may include steps S11 to S12, which are described in detail as follows:

[0025] S11, determining an image to be detected and a preset text query feature; the image to be detected is a real-time image acquired by multiple cameras on the vehicle during the automatic driving process of the vehicle;

[0026] S12, inputting the image to be detected and the preset text query features into the trained 3D perception model, and realizing 3D object detection and 3D scene description based on the 3D perception model;

[0027] Among them, the 3D perception model is based on 3D coordinate encoding features and a large language model. The large language model is instance-level fused and aligned with the image to be detected, and the matching relationship between the 3D target in the image to be detected and the vectors of the preset text query features is determined to generate a description of the spatial relationship between each 3D target and adjacent 3D targets.

[0028] In an embodiment of the present invention, the 3D perception model is based on a large language model and a multimodal algorithm model. Through feature alignment of language and visual images, the relevant knowledge in the large language model is embedded into the visual BEV features, and the 3D coordinate encoding features are introduced in the feature alignment process, thereby determining the matching relationship between the 3D target in the image and each vector of the preset text query feature, generating a spatial relationship description for each 3D target and adjacent 3D targets, and exploring the intrinsic connection between each 3D target in the image, thereby realizing the perception of any known or unknown 3D objects in the 3D scene; improving the on-board autonomous driving system's understanding of complex driving scenarios, and making the autonomous driving system's perception of objects in the surrounding world closer to human thinking logic.

[0029] See also Figure 2 , Figure 2 For execution Figure 1 Before the embodiment shown, step S10 is also included, which is a step of pre-building and training a 3D perception model. Figure 2 As shown, step S10 may specifically include steps S1 to S6. The 3D perception model is constructed and trained through the above steps, which are described in detail as follows:

[0030] S1, obtaining multiple frames of original images sampled by multiple cameras on the vehicle at the same time, and generating corresponding image text descriptions based on the target annotation information of each frame of the original image.

[0031] Specifically, take the same frame image of multiple cameras on the vehicle V is the camera number. Based on the annotation information (such as car, pedestrian, traffic sign, truck, ambulance), a text description is generated for each frame of the image to obtain P text : If the vehicle ahead is turning right, and there is a pedestrian on the right, the pedestrian is next to the traffic light.

[0032] Assume that the object categories to be detected are: car, truck, trailer, bus, bicycle, motorcycle, pedstrain. In the model training phase, it is generated based on the manually labeled information. For example, there is a pedestrian in front of the vehicle and a truck in front of the right. Then the text description is: there is a person in front, a truck in front of the right, etc. In the model reasoning phase: if you want to find obstacles in front of the vehicle in the image, the text description is: find the vehicle, pedestrian, truck, etc. in front.

[0033] S2, obtaining the 2D image features, 3D coordinate encoding features, text features generated by the image text description based on the large language model, and text position encoding features of each frame of the original image.

[0034] Get image set The 2D image features, 3D coordinate encoding features and image text description P of each frame image text Text features and text position encoding features based on the large language model LLM.

[0035] S2.1, the embodiment of the present invention uses a feature extraction network F based on the Trasnformer architecture net (such as Swin), after the network downsamples by d times, the image features are obtained As shown below:

[0036]

[0037] In the formula, V is the number of cameras, for example, if 6 cameras are used, then V=6; C represents the number of channels of the image data feature, for example, C=128, which means that the image data feature has 128 channels.

[0038] S2.2, according to the image resolution H, W, d, generate the corresponding homogenized image 3D coordinates:

[0039]

[0040] Where D represents the depth interval, X coord ,Y coord is the center coordinate of the image coordinate system (u, v) corresponding to the downsampling d times. H and W are evenly divided into grids with a spacing of d. The mathematical formula is as follows:

[0041]

[0042] Then the depth interval is divided along the camera line of sight [D near ,D far ]The depth coordinate Z obtained coord ,,The specific method can be uniform partition or linear growth algorithm. near and D far They respectively represent the closest distance and the farthest distance of an object from the center of the camera that can be sensed by the method provided by the embodiment of the present invention.

[0043] For example, the embodiment of the present invention selects the [0, 40m] interval and divides it into 80 depth interval coordinates [0.5, 1.0, 1.5...] at intervals of 0.5m.

[0044] S2.3, input homogenized image 3D coordinates Coords img , the camera's internal and external parameter matrix, Coords img Convert to the vehicle coordinate system IMU, and after feature embedding, obtain the 3D coordinate encoding feature image_pos_embedding.

[0045] S2.3.1, input the homogenized image 3D coordinates obtained above, based on the camera extrinsic matrix and the internal parameter matrix The image 3D position coordinates Coords img Convert to the vehicle coordinate system to obtain the homogenized 3D coordinates of each camera image and the vehicle normalized coordinates Coords ego , as shown below:

[0046]

[0047] S2.3.2, Aligned ego-vehicle normalized coordinates Coords ego Take the first three dimensions (x, y, z) to get the spatial coordinates Then adjust the spatial coordinates to the 3D coordinate dimension Get Coords' ego .

[0048] S2.3.3, Coords' ego Obtain 3D coordinate encoding features through the coordinate embedding layer pos_enbedding (convolutional network) As shown below:

[0049] coords pos_embedding =pos_enbedding(Coords pos )

[0050] Where D is the number of depth partition intervals, d depth is the 3D coordinate feature dimension, which is set to 256 in the embodiment of the present invention.

[0051] S2.3.4, for I t The image grid center coordinates Use sine and cosine encoding respectively. The specific formula is as follows:

[0052]

[0053]

[0054] Where pos is the coordinate value of the center of the image grid, d f is the coordinate embedding dimension. In this example, d is selected f =64, get the image 2D coordinate encoding feature

[0055] S2.3.5, Fusion of 3D coordinate encoding features coords pos_embedding and 2D coordinate encoding features, directly concatenate the two feature dimensions, and then pass through the convolutional network Pos 3d Fusion obtains the final 3D coordinate encoding features As shown below:

[0056] image_pos_embedding = Pos 3d ((coords pos_embedding ;coords pos_encoding ))

[0057] Among them, d pos =256, indicating the final 3D coordinate encoding feature dimension.

[0058] That is to say, concatenation is performed on the feature channel dimension C to obtain The characteristics of Pos 3d To fuse the convolutional network, (d depth +2×d f ) is reduced to d pos =256.

[0059] S2.4, using the large language model LLM to describe the input image text P text Encode. In this example, the large language model Bert is selected for encoding to obtain text features. Where L is the length of the text token, 784 is the dimension of the text word vector, and Bert can be replaced by a similar large language model such as GPT. text The dimension is reduced to 256 by linear transformation. where d text =256.

[0060] S3, based on the 2D image features and text features, image text query features are initialized and matched to obtain mutually matching image query features and text query features.

[0061] Conventional open set detection algorithms only initialize image query features and ignore the relationship between text features of each instance. The embodiment of the present invention explicitly matches image query features and text query features. After matching the relationship, the scene text description of each 3D object is associated with the 3D detection result, and the alignment of text features and image features of unlabeled categories is optimized.

[0062] S4, performing image and text feature fusion based on 2D image features, 3D coordinate coding features, text features and text position coding features to obtain fused image features and fused text features.

[0063] The embodiment of the present invention improves the text-image feature enhancement layer of the model proposed by Grounding DINO, and adds a text coordinate embedding and image 3D coordinate embedding feature fusion layer to meet the needs of 3D perception.

[0064] S5, performing 3D object detection and 3D scene description based on image query features, text query features, fused image features and fused text features, and obtaining 3D object detection results and 3D scene description features.

[0065] S6, determining the training loss of the 3D perception model according to the 3D object detection result and the 3D scene description feature, and obtaining the 3D perception model based on the training loss.

[0066] Existing solutions directly predict 3D objects in autonomous driving scenarios from image features. However, they face the problem of open set detection and are unable to identify unlabeled object categories. The embodiments of the present invention use a scene description branch to align the features contained in the large language model (LLM) with the image features, thereby achieving the purpose of zero-shot learning in 3D space.

[0067] In addition, the embodiments of the present invention can effectively fuse and align text descriptions and 3D detections at the instance level by initializing and pairing query feature queries, and then explicitly establish the relative positions of different objects, thereby enhancing the feature interaction between objects. Finally, the overall understanding of the entire scene is improved through text description supervision.

[0068] See also Figure 3 , Figure 3 for Figure 2 Step S3 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 3 As shown, step S3 may specifically include steps S3.1 to S3.4. The query feature initialization and matching optimization are performed through the above steps to obtain mutually matching image query features and text query features, which are described in detail as follows:

[0069] S3.1, obtain the initial value of the image query feature and the initial value of the text query feature by initializing the embedding operation.

[0070] Specifically, the maximum number of detected objects in each frame image is predefined as N, and the corresponding number of texts is N (N = 200, dim q = 256), and the initial value q of the image query feature is obtained by initializing the embedding operation nn.Emedding image and the initial value q of the text query feature text , as shown below:

[0071] q image =nn.Emedding(N,dim q )

[0072] q text =nn.Emedding(N,dim q )

[0073] S3.2, optimize the initial values ​​of the image query features and the initial values ​​of the text query features through the attention mechanism to obtain optimized image query features and text query features.

[0074] In an embodiment of the present invention, a deformable attention mechanism is performed on 2D image features, and a self-attention mechanism is performed on text features to obtain optimized image features and optimized text features; a cross-attention mechanism is performed on the optimized image features and image query feature initial values ​​to obtain optimized image query features; a cross-attention mechanism is performed on the optimized text features and text query feature initial values ​​to obtain optimized text query features.

[0075] Specifically, input image query feature initial value q image and the text query feature value initial q text , respectively, and 2D image features and text features F text Interact to obtain the query feature matching relationship and update the query feature. The interaction process is as follows Figure 8 As shown, the specific implementation process is as follows:

[0076] S3.2.1, Input 2D Image Features After the deformable attention network layer (DN Self-Attention), the updated image features are obtained Text feature F text Through the self-attention network layer (Self-Attention), the updated text feature F' is obtained text .

[0077] S3.2.2, Image query feature initial value q image and the updated image features As K and V, through cross-attention, we get the optimized image query feature q' image ; Similarly, the initial value of the text query feature q text Same as the updated text feature F' text Perform cross attention to obtain the optimized text query feature q' text The specific feature fusion process is as follows: Figure 7 As shown, the distribution of text features and image features gradually converges. In this figure, Source and Target represent the source domain and target domain respectively. For the biased dataset (left), the classification learned in the source domain may not be well transferred to the target domain. By minimizing the classification error and maximizing domain confusion, a representation with discriminativeness and domain invariance can be learned.

[0078] The specific formula for achieving feature fusion through cross-attention is:

[0079] q=CrossAtten(Q,K,V)

[0080] In the above formula, Q is the initial value of the query feature (q image / q text ).

[0081] S3.3, determining the query feature similarity of the optimized image query feature and the text query feature based on the feature similarity method.

[0082] In the embodiment of the present invention, the product of the transposition of the optimized image query feature and the optimized text feature is determined to obtain a query feature similarity matrix; and a bidirectional maximum matching algorithm is used to obtain the best matching pair of each optimized image query feature and optimized text query feature.

[0083] Specifically, input the optimized image query feature q' image and the optimized text query feature q' text , after the query feature initial fusion module Query Init Fusion performs feature fusion, the matching relationship between the query features is obtained. The specific method is to calculate the optimized image query feature q' image and the optimized text query feature q' text The feature similarity matrix Use the bidirectional maximum matching algorithm to obtain each optimized image query feature q' image and the optimized text query feature q' text The best matching pair Pair <Index image ,Index text >, after matching the relationship, the text query features and image query features corresponding to each object are associated, and the text features are migrated to the image feature field through S3.4 to optimize the image query features of unlabeled categories and assist in the recognition of unlabeled category objects (i.e., objects without manually labeled categories).

[0084] S3.4, based on the query feature similarity, the optimized image query features and text query features are updated to obtain image query features and text query features that match each other.

[0085] Specifically, an embodiment of the present invention performs matrix multiplication on the query feature similarity and the optimized text query feature, and uses the sum of the first product result and the optimized image query feature as the updated image query feature; performs matrix multiplication on the transpose of the query feature similarity and the optimized image query feature, and uses the sum of the second product result and the optimized text query feature as the updated text query feature; wherein the query feature similarity is obtained by performing matrix multiplication on the transpose of the optimized image query feature and the optimized text query feature.

[0086] The specific formula is as follows:

[0087]

[0088] In the formula, q' i ' mage represents the updated image query features, M sim Represents the query feature similarity matrix, q' textRepresents the optimized text query feature, q' image Represents the optimized image query features; q' t ' ext represents the updated text query features, represents the transpose of the query feature similarity matrix;

[0089]

[0090] q' image represents the optimized image query feature, q' text T represents the transpose of the optimized text query feature, N represents the number of targets in each frame image, and the number of texts is the same as the number of targets.

[0091] The embodiment of the present invention updates the image query features and the text query features to aggregate the features, concentrates the text features and the image features of the object on one query as much as possible, and makes the text query features and the image query features close in the feature space, thereby realizing multimodal feature alignment (text and image) and optimizing the image query features of unlabeled categories.

[0092] See also Figure 4 , Figure 4 for Figure 2 Step S4 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 4 As shown, step S4 may specifically include steps S4.1 to S4.2. The image and text features are fused through the above steps to obtain fused image features and fused text features, which are described in detail as follows:

[0093] S4.1, embed 3D coordinate encoding features in 2D image features, and embed text position encoding features in text features.

[0094] Specifically, the input 2D image features and text features F text , the two features are fused through the feature enhancement module to obtain updated text features and image features The embodiment of the present invention improves the text-image feature enhancement layer of the model proposed by GroundingDINO, and adds a text coordinate embedding and image 3D coordinate embedding feature fusion layer to meet the needs of 3D perception.

[0095] Get the image text description P text Corresponding text position encoding features Where L is the length of the text token. A text token refers to a meaningful language element in a text after the text is segmented.

[0096] Input 2D image features and text features F text The embodiment of the present invention is based on the GroundingDINO feature enhancement module FeatureEnhancer, and performs 2D image feature Introducing 3D coordinate encoding features, for text features F text Introducing text position encoding features.

[0097] S4.2, feature fusion is performed on the 2D image features and text features after position calibration to obtain fused image features and fused text features.

[0098] Specifically, the intermediate text features of the current feature fusion are obtained based on the sum of the text features and the text position coding features obtained in the previous feature fusion; the sum of the image features and the 3D coordinate coding features obtained in the previous feature fusion is determined, and the sum result is converted into a one-dimensional array to obtain the intermediate image features of the current feature fusion; feature enhancement processing is performed on the intermediate text features of the current feature fusion and the intermediate image features of the current feature fusion to obtain fused image features and fused text features.

[0099] In the embodiment of the present invention, the 2D image features and text features after position calibration are fused, and after n layers of interaction, a fused text feature is obtained. and fusion image features The specific formula is as follows:

[0100]

[0101]

[0102]

[0103] In the formula, represents the intermediate text features obtained from the nth interaction, represents the text features of the n-1th layer, text_pos_embedding represents the text position encoding features, L represents the number of text semantic elements, d pos represents the 3D coordinate encoding feature dimension, Represents the intermediate image features obtained from the nth interaction, and Flatten represents conversion to a one-dimensional array. represents the image features obtained by the n-1th interaction, image_pos_embedding represents the 3D coordinate encoding features, V represents the number of on-board surround view cameras; H, W, d represents the image resolution of 2D image features, W represents the width, H represents the height, and d represents the number of times the image is downsampled after passing through the feature extraction network; represents the fusion text features, Indicates the fused image features, and FeatureEnhancer indicates feature enhancement processing.

[0104] like Fig. 9 As shown, the embodiment of the present invention converts text features Add the text position encoding feature text_pos_embedding, generate features through the self-attention module (Self-Attention), as k and v, image features After adding the 3D coordinate encoding feature image_pos_embedding and performing the Flatten operation, it is input into the deformable self-attention module (DN Self-Attention) as Q; after passing through the feature enhancement module composed of two cross-attention modules (text-to-image and image-to-text), it passes through the feedforward neural network (FFN) respectively to generate the fused text features after multimodal fusion update and fusion image features

[0105] The above steps introduce 3D coordinate encoding features and text position encoding features, realize the alignment of text features and 3D features, and transfer the prior knowledge in the large language model LLM to the 3D features, so that it contains the ability to perceive unlabeled objects.

[0106] See also Figure 5 , Figure 5 for Figure 2 Step S5 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 5 As shown, step S4 may specifically include steps S5.1 to S5.6. The above steps are used to perform 3D object detection and 3D scene description to obtain 3D object detection results and 3D scene description features, which are described in detail as follows:

[0107] It should be noted that the step of performing 3D object detection may include S5.1 to S5.2 to obtain the 3D object detection result; and the step of performing 3D scene description to obtain 3D scene description features may include S5.3 to S5.6.

[0108] When performing 3D object detection, input fusion image features and image query feature q″ image , after the image decoder Image Decoder , and get the processed 3D detection results

[0109] S5.1, performing image decoding processing on the image query features and the fused image features to obtain the 3D object detection result.

[0110] Specifically, based on the DETR decoder, the coordinate feature query_pos of the image query feature is replaced by the 3D coordinate feature after flattening. As Q, As K, V, the decoder of the transformer architecture after L layers L Decode and iteratively optimize 3D image query features The formula is as follows:

[0111]

[0112] Same operation as PETR detection head, input image decoding features Finally, it is obtained through the 3D prediction branch cls_reg_branch (regression and classification convolutional neural network) Corresponding 3D object detection results Right now: Where N represents the number of predicted 3D detection boxes.

[0113] S5.2, perform binary matching and re-ranking of 3D target detection results based on target annotation information.

[0114] The 3D target detection results are binary matched and reordered based on the target annotation information, specifically including: performing non-maximum suppression (NMS) processing on the 3D target detection results, retaining a preset number of 3D target detection results; performing binary matching on the retained 3D target detection results and the target annotation information to obtain a corresponding relationship; and reordering the 3D target detection results based on the arrangement order of the target annotation information in the original image.

[0115] That is to say, the embodiments of the present invention After non-maximum suppression NMS processing, M detection results are retained Then with 3D labeled objects (i.e. target annotation information, manually annotated 3D box), perform binary matching to obtain the corresponding relationship, and then The order of the objects in the Rearrange to get

[0116] Detection results retained by non-maximum suppression NMS and manual annotation boxes For example, the number of the former box is M1, and the data of the latter box is M2. It needs to be matched by two parts to calculate A box in the specific and Which box in has the same category attributes and the highest spatial IoU (overlap).

[0117] The embodiment of the present invention filters out redundant detection frames through non-maximum suppression (NMS) processing. If two detection frames have a high degree of overlap, they can be regarded as one object to avoid interference with subsequent orientation prediction and reduce subsequent calculation complexity.

[0118] When describing a 3D scene, input fusion text features and text query feature q″ text , get the instance-level scene description feature F scene The specific steps are as follows:

[0119] S5.3, determining the nearest neighbor relationship of a preset number of objects of the nearest neighbors of each 3D object detection result in the reordered 3D object detection results.

[0120] Specifically, input the reordered 3D target detection results Get the indexes of the K nearest objects to each 3D object. The specific operation is: The jth detected object in the image is taken as the nearest neighbor of the jth detected object. The L2 distance is used to measure the distance between the jth detected object and the center of other detection boxes, and the nearest K objects are selected to obtain the nearest neighbor relationship of the jth detected object. j = {I j1 ,I j2 …I jk}, if the number of nearest objects is less than K, the index is supplemented with -1.

[0121] S5.4, update the text query feature vector corresponding to each 3D target detection result according to the text query feature, query feature matching relationship and nearest neighbor relationship.

[0122] In an embodiment of the present invention, a vector of a text query feature corresponding to a current 3D target detection result is obtained by querying a query feature matching relationship; a nearest neighbor relationship index is queried based on the vector of the text query feature to obtain vectors of multiple nearest text query features; the text similarity between the vector of the text query feature and the vector of each nearest text query feature is determined; a weighted sum is performed on the vector of the text query feature, the vectors of multiple nearest text query features, and the text similarity to obtain a text query feature vector corresponding to the current 3D target detection result.

[0123] Specifically, for the 3D target detection results For the jth object in the image, first find the corresponding text feature query vector by querying the feature matching relationship (i.e., the pair relationship between the image query feature and the text query feature matching). Then query the nearest neighbor relationship Near j The corresponding text query feature vector:

[0124]

[0125] Get the text features corresponding to the jth object through the similarity matrix and nearest neighbor text features The text similarity of And adopt the weighted summation method to obtain the updated text feature query vector corresponding to the jth object The formula is as follows:

[0126]

[0127] In the formula, Represents the text query feature vector corresponding to the current 3D object detection result, A vector representing the features of the text query, A vector representing the nearest text query features, Indicates text similarity.

[0128] S5.5, feature concatenate the text query feature vectors corresponding to all 3D object detection results to obtain instance-level scene text query features.

[0129] Specifically, directly convert the M 3D target detection results Corresponding text query features Perform feature splicing to obtain instance-level scene text query features The formula is as follows:

[0130]

[0131] S5.6, based on the instance-level scene text query features and the fused text features, cross-attention decoding is performed based on the instance description generator to obtain the instance-level 3D scene description features.

[0132] Specifically, input the instance description generator CaptionHead to query the feature with instance-level scene text As Q, text features As K, V. After cross-attention decoding, the instance-level scene text description feature F is obtained scene The formula is as follows:

[0133]

[0134] The embodiment of the present invention introduces 3D position coding features and adds instance-level 3D scene understanding to explicitly establish the intrinsic relationship between objects in the scene, thereby achieving the purpose of completing 3D perception of any scene with text prompts.

[0135] The text features after input optimization in the embodiment of the present invention As K, V, text query feature q″ text As Q, input the instance description generator CaptionHead based on CaptionFormer to generate instance-level scene description features F scene The conventional method is to randomly initialize the text query feature vector The example description generator decodes the text and generates a general description of the entire scene. The embodiment of the present invention uses a text query feature vector that matches the 3D detected object to generate a description of the spatial relationship between each instance and adjacent objects, explicitly establishes the spatial relationship between each 3D object, achieves alignment of large language model features and fine-grained instance 3D image features, improves the perception of unlabeled 3D objects based on text prompts, and can achieve the ability to understand the scene instance level.

[0136] See also Figure 6 , Figure 6 for Figure 2 Step S6 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 6 As shown, step S6 may specifically include steps S6.1 to S6.4. The training loss of the 3D perception model is determined through the above steps, which are described in detail as follows:

[0137] S6.1, determining the scene description feature loss based on the divergence loss according to the 3D scene description features and the image text description.

[0138] In the embodiment of the present invention, a large speech description model is used to perform feature encoding on the image text description to obtain the true value of the 3D scene description feature; the divergence loss is used as the loss metric function to determine the feature distribution difference between the 3D scene description feature and the true value of the 3D scene description feature to obtain the scene description feature loss. The specific implementation process is as follows:

[0139] S6.1.1, using the large language model BERT to encode the real scene description text H text Perform feature encoding to obtain the true value of 3D scene description features

[0140]

[0141] S6.1.2, using KL divergence loss KLD Loss As a loss metric function, the ground-truth 3D scene description feature And the instance-level scene text description feature F predicted by the model scene The difference in feature distribution between them is used to obtain the scene description feature loss L gen .

[0142]

[0143] The embodiment of the present invention describes the text H according to the real scene text , calculate H text The loss of the generated text features and the instance-level scene description features predicted by the model supervises the local scene perception ability based on instances in model training.

[0144] S6.2, according to the 3D target detection result and the target annotation information, determine the relative orientation loss based on the relative orientation probability.

[0145] S6.2.1, determining the true relative positions of the target objects corresponding to the target annotation information according to the target annotation information.

[0146] Enter 3D annotation information Output the one-hot encoding of the relative orientation of each object. The specific steps are: for any two 3D objects, first calculate the center of the first object obj1 (x1, y1, z1) and the center of the second object obj2 (x2, y2, z2) parallel to the ground vector Then calculate the angle difference θ between the vector and the positive direction of the X axis of the first object. Then determine the relative position Rel of any object ORI-raw (For the specific process of determining the relative orientation, please refer to the specific process of determining the relative orientation in S6.2.2). Finally, the relative orientation matrix of the real object is obtained through one-hot encoding. And Mask ORI [obj1,obj2] is assigned a value of 1, which means that there is a relative position relationship between the two objects.

[0147] S6.2.2, determine the relative orientation probability matrix obtained by predicting the text query features corresponding to the 3D object detection results.

[0148] In an embodiment of the present invention, a self-attention mechanism and feature fusion processing are performed on the text query features corresponding to the 3D target detection results to obtain relative orientation text query features; and the relative orientation of the spatial object corresponding to each feature vector of the relative orientation text query feature is predicted through a classification network.

[0149] The specific process of determining relative orientation is as follows:

[0150] 1) Input text query feature q″ text , after self-attention operation (SelfAtten), we get q″ text The interaction features q″ between each query vector in text-inter , then q″ text and q″ text-inter Perform fusion and dimension adjustment to obtain updated text query features As shown below:

[0151] q″ text-inter =SelfAtten(q″) text )

[0152]

[0153] 2) The relative orientation prediction is abstracted as a classification problem. The 360-degree space is evenly divided into 8 intervals (corresponding to east by north [0, 45°], north by east [45° 90°]…), and the two categories of the diagonal are set as (class i ,class 8-i ). Enter the relative position query feature Predict text query features through the classification network Rel_Predictor The relative position of each feature vector to the space object Rel ORI The formula is as follows:

[0154]

[0155] In the embodiment of the present invention, the relative orientation probability matrix Rel predicted by the input text query feature ORI , get the corresponding 3D detection object The relative position probability matrix By matching the relationship Pair, we get The text query vector index I corresponding to any j-th object in j , then Assign value to Rel ORI [I j ]. Finally, the relative orientation probability matrix corresponding to the 3D object is obtained like Fig.10 shown. Fig.10 This is a relative orientation relationship diagram between 3D objects predicted by image query features. A value of 1 indicates that each object corresponding to the query feature exists, and a value of 0 indicates that it does not exist. The relative orientation can be determined only when all objects exist. is a subvector of the image query feature vector.

[0156] S6.2.3, determining an orientation loss between the true relative orientation and the relative orientation probability matrix based on the true relative orientation, the relative orientation probability matrix, and the preset 3D object mask.

[0157] In an embodiment of the present invention, a preset 3D object mask is multiplied by a relative orientation probability matrix to obtain a denoised relative orientation probability matrix; and a cross entropy is used to calculate an orientation loss between the denoised relative orientation probability matrix and a true relative orientation.

[0158] Specifically, the input relative position probability matrix Rel ORI , the relative orientation probability matrix corresponding to the 3D object 3D Object Mask ORI , and finally get the relative orientation loss. This step uses the 3D object mask Mask ORI and Multiply to avoid interference from noise detection values, and then use cross entropy CrossEntropy to calculate and The loss between CE , as shown below:

[0159]

[0160] S6.2.4, regularize the orientation loss to obtain the relative orientation loss.

[0161] Considering the symmetry of the text position relationship, if object A is in front of the left of object B, then object B is behind the right of object A. Therefore, The probability should be close to Rel ORI , calculate the symmetric loss L of the two rg As a regularization term. As follows:

[0162] Combining the above two losses, we get the final relative orientation loss L ORI :

[0163] L ORI =L CE +L rg .

[0164] In the embodiment of the present invention, the 3D annotation information is input Initialize 3D object mask Mask ORI is a zero matrix, and the relative orientation probability matrix Rel predicted by the text query feature is calculated. ORI , the prediction loss of the relative position of the real object, and the learning of the model's ability to describe the scene.

[0165] S6.3, according to the 3D target detection result and the target annotation information, determine the 3D detection loss based on the PETR method.

[0166] The embodiment of the present invention adopts the PETR method, and the detection loss is divided into the classification loss L cls and 3D position regression loss L reg After weighted summation of the two parts, we get the 3D detection loss L det for:

[0167] L det =λ cls L cls +λ reg L reg

[0168] In the formula, λ cls and λ reg They are respectively the classification loss L cls and 3D position regression loss L reg The loss weight.

[0169] In the embodiment of the present invention, the 3D annotation information is input and 3D predicted objects The error is obtained by 3D detection loss L det , supervise the correctness of 3D perception results.

[0170] S6.4, perform weighted summation of scene description feature loss, relative orientation loss and 3D detection loss to obtain training loss, and obtain a 3D perception model based on the training loss.

[0171] Combining all the above losses, we get the final model framework training loss L all , the specific calculation method is as follows:

[0172] L all =β1L gen +β2L ori +β3L det .

[0173] Where β1, β2 and β3 are loss weights, L gen is the scene description feature loss, L ori is the relative position loss and L det is the 3D detection loss.

[0174] During the 3D perception model training process of the embodiment of the present invention, the images acquired by the surround-view camera and the 3D object annotation information (category, xyz spatial position, wl h box size) in the scene are input to train the deep learning model; during the model inference process, the surround-view camera images and the corresponding prompt information (such as a pedestrian crossing the road ahead) are input, and the 3D box information corresponding to the person is output.

[0175] In an embodiment of the present invention, the 3D perception model is based on a large language model and a multimodal algorithm model. Through feature alignment of language and visual images, the relevant knowledge in the large language model is embedded into the visual BEV features, and the 3D coordinate encoding features are introduced in the feature alignment process, thereby determining the matching relationship between the 3D target in the image and each vector of the preset text query feature, generating a spatial relationship description for each 3D target and adjacent 3D targets, and exploring the intrinsic connection between each 3D target in the image, thereby realizing the perception of any known or unknown 3D objects in the 3D scene; improving the on-board autonomous driving system's understanding of complex driving scenarios, and making the autonomous driving system's perception of objects in the surrounding world closer to human thinking logic.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D target detection method for autonomous driving, characterized in that: include: Determine an image to be detected and preset text query features; the image to be detected is a real-time image acquired by multiple onboard cameras during the vehicle's automatic driving process; Inputting the image to be detected and the preset text query feature into a trained 3D perception model, and realizing 3D object detection and 3D scene description based on the 3D perception model; The 3D perception model is based on 3D coordinate encoding features and a large language model, and performs instance-level fusion and alignment of the large language model features with the features of the image to be detected, and determines the matching relationship between the 3D target in the image to be detected and each vector of the preset text query feature, and generates a description of the spatial relationship between each 3D target and adjacent 3D targets; The method also includes: pre-building and training the 3D perception model, specifically including: Acquire multiple frames of original images sampled by multiple cameras on the vehicle at the same time, and generate corresponding image text descriptions based on the target annotation information of each frame of the original image; Obtaining 2D image features, 3D coordinate coding features, text features generated based on a large language model and text position coding features of the image text description of each frame of the original image; Based on the 2D image features and text features, image and text query features are initialized and matched to obtain mutually matching image query features and text query features; Perform image and text feature fusion based on the 2D image features, 3D coordinate coding features, text features and text position coding features to obtain fused image features and fused text features; Based on the image query features, text query features, fused image features and fused text features, 3D object detection and 3D scene description are performed to obtain 3D object detection results and 3D scene description features; The performing of 3D object detection and 3D scene description based on the image query feature, the text query feature, the fused image feature and the fused text feature to obtain the 3D object detection result and the 3D scene description feature includes: Performing image decoding processing on the image query feature and the fused image feature to obtain a 3D object detection result; Performing binary matching and reordering on the 3D target detection results based on the target annotation information; Determine the nearest neighbor relationship of a preset number of objects of the nearest neighbors of each 3D object detection result in the reordered 3D object detection results; Update the text query feature vector corresponding to each of the 3D object detection results according to the text query feature, the query feature matching relationship, and the nearest neighbor relationship; Concatenate the text query feature vectors corresponding to all 3D object detection results to obtain instance-level scene text query features; According to the instance-level scene text query feature and the fused text feature, a cross-attention decoding is performed based on the instance description generator to obtain an instance-level 3D scene description feature; The training loss of the 3D perception model is determined according to the 3D object detection result and the 3D scene description feature, and the 3D perception model is obtained based on the training loss.

2. The method according to claim 1, characterized in that The initialization and matching optimization of image and text query features based on the 2D image features and text features to obtain mutually matching image query features and text query features include: Obtaining the initial value of the image query feature and the initial value of the text query feature by initializing the embedding operation; Optimizing the initial value of the image query feature and the initial value of the text query feature through an attention mechanism to obtain optimized image query features and text query features; Determine the query feature similarity of the optimized image query feature and the text query feature based on the feature similarity method; The optimized image query features and text query features are updated based on the query feature similarity to obtain image query features and text query features that match each other.

3. The method according to claim 2, characterized in that The step of updating the optimized image query features and text query features based on the query feature similarity to obtain mutually matching image query features and text query features includes: Performing matrix multiplication on the query feature similarity and the optimized text query feature, and taking the sum of the first product result obtained and the optimized image query feature as the updated image query feature; Performing matrix multiplication on the transposed query feature similarity and the optimized image query feature, and taking the sum of the obtained second product result and the optimized text query feature as the updated text query feature; The query feature similarity is obtained by performing matrix multiplication of the transposed optimized image query feature and the optimized text query feature.

4. The method according to claim 1, characterized in that The image and text feature fusion is performed based on the 2D image feature, the 3D coordinate coding feature, the text feature and the text position coding feature to obtain the fused image feature and the fused text feature, including: Embedding the 3D coordinate encoding feature in the 2D image feature, and embedding the text position encoding feature in the text feature; The 2D image features and the text features after position calibration are fused to obtain fused image features and fused text features.

5. The method according to claim 4, characterized in that The step of fusing the 2D image features and the text features after position calibration to obtain fused image features and fused text features includes: The intermediate text features of the current feature fusion are obtained by summing the text features obtained in the previous feature fusion and the text position encoding features; Determine the sum of the image features obtained by the previous feature fusion and the 3D coordinate encoding features, and convert the sum result into a one-dimensional array to obtain the intermediate image features of the current feature fusion; Feature enhancement processing is performed on the intermediate text feature fused by the current sub-feature and the intermediate image feature fused by the current sub-feature to obtain a fused image feature and a fused text feature.

6. The method according to claim 1, characterized in that The updating of the text query feature vector corresponding to each of the 3D object detection results according to the text query feature, the query feature matching relationship, and the nearest neighbor relationship index includes: By querying the query feature matching relationship, a vector of the text query feature corresponding to the current 3D object detection result is obtained; Querying the nearest neighbor relationship index according to the vector of the text query feature to obtain vectors of multiple nearest neighbor text query features; Determining text similarity between the vector of the text query feature and the vector of each nearest text query feature; A weighted sum is performed on the vector of the text query feature, the vectors of multiple nearest text query features, and the text similarity to obtain a text query feature vector corresponding to the current 3D object detection result.

7. The method according to claim 1, characterized in that The determining the training loss of the 3D perception model according to the 3D object detection result and the 3D scene description feature, and obtaining the 3D perception model based on the training loss, comprises: Determining a scene description feature loss based on a divergence loss according to the 3D scene description feature and the image text description; Determining a relative orientation loss based on a relative orientation probability according to the 3D target detection result and the target annotation information; Determine the 3D detection loss based on the PETR method according to the 3D target detection result and the target annotation information; The scene description feature loss, relative orientation loss and 3D detection loss are weightedly summed to obtain the training loss, and the 3D perception model is obtained based on the training loss.

8. The method according to claim 7, characterized in that The determining the relative orientation loss based on the relative orientation probability according to the 3D target detection result and the target annotation information includes: Determine the real relative positions between the target objects corresponding to the target annotation information according to the target annotation information; Determine a relative orientation probability matrix obtained by predicting text query features corresponding to the 3D object detection result; Determine, according to the true relative orientation, the relative orientation probability matrix and a preset 3D object mask, an orientation loss between the true relative orientation and the relative orientation probability matrix; Regularization is performed on the orientation loss to obtain the relative orientation loss.

Images