A 3D dense annotation, electronic device and storage medium for autonomous driving scenes

Abstract

The present invention provides a 3D dense annotation method for an autonomous driving scene, which can be applied to the fields of big data and artificial intelligence technology. The method includes: extracting bird's-eye view features based on multi-view images and radar point cloud data; identifying and locating 3D objects in the scene by a detection head to ensure accurate capture of object features; using Relation Q-former technology to interact BEV features and 3D object features to obtain in-depth relationships between each object and the scene context, thereby enhancing a comprehensive understanding of the scene; based on adapter technology, avoiding retraining of existing large models, converting the above-obtained features into prompt word prompt input of a language model to generate a 3D object description. After the training is completed, the 3D dense annotation model can automatically locate and describe 3D objects according to a given scene picture. The present invention significantly improves the efficiency and accuracy of 3D object annotation in outdoor scenes of autonomous driving by combining multi-source data fusion, contextual relationship analysis and adapter technology.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, big data and artificial intelligence technology, and in particular to 3D dense annotation, electronic devices and storage media of autonomous driving scenes. Background Art

[0002] ‌3D dense annotation methods mainly include point cloud annotation and 3D cube annotation. Point cloud annotation involves classifying dense point cloud data collected by sensors such as lidar and annotating them with different attributes; 3D cube annotation is a annotation method based on two-dimensional plane images. By framing the edges of three-dimensional objects, the vanishing point is obtained and the relative distance between objects is measured. In addition, there is a 2D / 3D fusion annotation method, which simultaneously annotates image data collected by 2D and 3D sensors and establishes associations, which can more comprehensively describe the position and size of objects in the plane and three dimensions, further improving the accuracy and practicality of data annotation.

[0003] With the development of 3D dense annotation methods, 3D scene understanding based on natural language processing has shown its application potential in many fields, including cross-modal retrieval, robot navigation, and autonomous driving. However, existing technologies mainly focus on 3D annotation tasks for indoor scenes, and do not fully consider the significant differences between indoor and outdoor scenes, such as object states, point cloud density, camera perspectives, and regional characteristics of the scene. These differences lead to performance deficiencies in existing methods in outdoor scene applications such as autonomous driving. Summary of the invention

[0004] In view of the above problems, the present invention provides 3D dense annotation, electronic equipment and storage medium for autonomous driving scenes.

[0005] According to a first aspect of the present invention, a 3D dense annotation method for an autonomous driving scene is provided, comprising:

[0006] The 3D dense annotation model is used to process the 2D multi-view image samples to obtain multi-view image feature samples, and the 3D laser point cloud data samples are analyzed to obtain point cloud data feature samples. The multi-view image feature samples and the point cloud data feature samples are convolutionally fused to obtain multi-modal bird's-eye view feature samples.

[0007] Use the 3D dense annotation model to perform 3D object recognition and positioning on the multimodal bird's-eye view feature samples to obtain the 3D object set in the autonomous driving scene;

[0008] Based on information retrieval and learning representation mechanisms, the interactive relationships in the 3D object set are analyzed using a 3D dense annotation model to obtain the relationship between the 3D objects and the context of the autonomous driving scene;

[0009] The 3D dense annotation model is used to perform feature adaptation on the relationship between 3D objects and the context of the autonomous driving scene to obtain prompt word samples, which are then processed using a pre-trained large language model to generate a predicted description of the 3D object.

[0010] The prediction of 3D objects and the true value description of 3D objects are processed using a predefined loss function to obtain a loss value, and the 3D dense annotation model is iteratively trained based on the loss value until the preset conditions are met to obtain a trained 3D dense annotation model;

[0011] The trained 3D dense annotation model is deployed to the target autonomous driving scene, and the trained 3D dense annotation model is used to densely annotate 3D objects in the target autonomous driving scene.

[0012] According to an embodiment of the present invention, the 3D dense annotation model is used to perform image processing on 2D multi-view image samples to obtain multi-view image feature samples, including:

[0013] Based on the spatiotemporal attention mechanism, the multimodal bird's-eye view sample features obtained by training at the previous moment and the learnable bird's-eye view query vector at the current moment are used to obtain a learnable bird's-eye view query vector that integrates historical information.

[0014] Multi-view visual features are extracted from 2D multi-view image samples, and based on the spatial cross-attention mechanism, the spatiotemporal bird's-eye view encoder of the 3D dense annotation model is used to process the multi-view visual features and the learnable bird's-eye view query vector that integrates historical information to obtain the multi-view image feature samples mapped to the bird's-eye view space at the current moment.

[0015] According to an embodiment of the present invention, the above-mentioned point cloud analysis of the 3D laser point cloud data sample to obtain the point cloud data feature sample includes:

[0016] Voxel features are extracted from 3D laser point cloud data samples, and the voxel features are flattened in the height direction using a 3D dense annotation model and then aligned with multi-view image feature samples to obtain point cloud data feature samples.

[0017] According to an embodiment of the present invention, the 3D dense annotation model is used to perform 3D object recognition and positioning on the multimodal bird's-eye view feature samples, and the 3D object set in the autonomous driving scene is obtained, including:

[0018] Based on the predefined number of queries and predefined feature dimensions, the 3D object detection network of the 3D dense annotation model is used to perform 3D object detection on the multimodal bird's-eye view feature samples, and the 3D objects in the autonomous driving scene are identified and located based on the detection results to obtain a 3D object set.

[0019] According to an embodiment of the present invention, the above-mentioned information retrieval and learning representation mechanism is based on the use of a 3D dense annotation model to analyze the interactive relationship in the 3D object set, and the relationship between the 3D object and the autonomous driving scene context includes:

[0020] The 3D objects in the 3D object set are encoded using the multi-layer perceptron in the 3D dense annotation model to obtain a 3D object set aligned with the multimodal bird's-eye view feature samples;

[0021] The query-based information retrieval and learning representation network of the 3D dense annotation model are used to analyze the interaction between 3D objects in the aligned 3D object set and the interaction between 3D objects and the autonomous driving scene to obtain the relationship between 3D objects and the context of the autonomous driving scene.

[0022] According to an embodiment of the present invention, the above-mentioned 3D dense annotation model is used to perform feature adaptation on the relationship between the 3D object and the context of the autonomous driving scene, and the prompt word samples obtained include:

[0023] The multi-layer perceptron-based adapter of the 3D dense annotation model is used to perform feature transformation on the relationship between 3D objects and the context of the autonomous driving scene, and obtain prompt word samples that are aligned with the feature space of the pre-trained large language model.

[0024] According to an embodiment of the present invention, the above-mentioned processing of the prompt word sample by using the pre-trained large language model to generate a predicted description of the 3D object includes:

[0025] A pre-trained large language model is used to process prompt word samples and pre-defined prompt texts to generate a predicted description of the 3D object, wherein the predicted description of the 3D object includes a plurality of words for describing the 3D object.

[0026] According to an embodiment of the present invention, the predefined loss function is obtained by performing a weighted operation on an L1 regression loss function based on a 3D target box and a language annotation loss function based on a predefined weighting coefficient.

[0027] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0028] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0029] The 3D dense annotation of the above-mentioned autonomous driving scene provided by the present invention creates a bird's-eye view (BEV) from 3D point clouds and 2D multi-view images and obtains the relationship between each object and the scene context by identifying different 3D objects in the autonomous driving scene. The dense annotation of objects is achieved through a pre-trained large language model, which improves the accuracy of 3D object positioning and description in outdoor scenes; at the same time, it avoids retraining of the large language model and effectively utilizes the common sense reasoning ability of the large language model pre-trained under large-scale data. The above method provided by the present invention significantly improves the efficiency and practicality of the annotation process, and provides a new technical path for the understanding and application of outdoor 3D scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0031] Figure 1 is a flowchart of a 3D dense annotation method for an autonomous driving scene according to an embodiment of the present invention;

[0032] Figure 2 is an architectural diagram of a 3D dense annotation model according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of a 3D densely labeled model trained in an outdoor real scene according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of a training process of an outdoor 3D dense labeling method according to another embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of the structure of a 3D dense annotation device for an autonomous driving scene according to an embodiment of the present invention;

[0036] Figure 6 A block diagram of an electronic device suitable for implementing a 3D dense annotation method for an autonomous driving scenario according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0037] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0038] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0039] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0040] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0041] 3D dense annotation methods mainly include point cloud annotation and 3D cube annotation. Among them, the point cloud data obtained by point cloud annotation usually represents the three-dimensional spatial information of objects or scenes. By annotating different attributes, such as color, material, etc., it helps to train image recognition models and is often used in fields such as autonomous driving. Although 3D cube annotation is based on a two-dimensional plane, it can provide the position and size information of objects in three-dimensional space, which is of great significance for enhancing the visual and radar perception capabilities of autonomous driving models. In addition, in the fields of autonomous driving, robot navigation, etc., 3D dense annotation methods have broad application prospects.

[0042] In view of the technical problems existing in the prior art, the present invention first defines the outdoor 3D dense labeling task of autonomous driving, using the point cloud of the lidar and the panoramic camera image as input to detect and describe all objects in the scene. For this task, the present invention proposes an innovative 3D dense labeling method for autonomous driving scenes, which creates a bird's-eye view (BEV) from 3D point clouds and 2D multi-view images and identifies different objects through the detection head, uses the Relation Q-former to obtain the relationship between each object and the scene context, and realizes the dense labeling of objects through the large language model (LLM), thereby improving the accuracy of 3D object positioning and description in outdoor scenes. At the same time, the adapter is used to avoid retraining of the large language model, and effectively utilizes the common sense reasoning ability of the large language model pre-trained under large-scale data. The present invention significantly improves the efficiency and practicality of the labeling process, and provides a new technical path for outdoor 3D scene understanding and application.

[0043] The 3D dense labeling method for the autonomous driving scene provided by the present invention is further described in detail below with reference to specific embodiments and drawings.

[0044] Figure 1 3D dense labeling method for autonomous driving scenes according to an embodiment of the present invention.

[0045] like Figure 1 As shown, the above-mentioned 3D dense labeling method for the autonomous driving scene includes operations S110 to S160.

[0046] In operation S110, the 2D multi-view image samples are processed using a 3D dense annotation model to obtain multi-view image feature samples, point cloud analysis is performed on the 3D laser point cloud data samples to obtain point cloud data feature samples, and convolution fusion is performed on the multi-view image feature samples and the point cloud data feature samples to obtain multimodal bird's-eye view feature samples.

[0047] In operation S120, a 3D dense annotation model is used to perform 3D object recognition and positioning on the multimodal bird's-eye view feature samples to obtain a set of 3D objects in the autonomous driving scene.

[0048] The above-mentioned 3D object set includes different objects in the autonomous driving scene, such as various vehicles, pedestrians, green belts, signboards, etc., and also includes the interaction relationship between the above-mentioned different types of 3D objects and the environment in which the different types of 3D objects are located.

[0049] In operation S130, based on information retrieval and learning representation mechanism, the interactive relationship in the 3D object set is analyzed using a 3D dense annotation model to obtain the relationship between the 3D object and the autonomous driving scene context.

[0050] In operation S140, a 3D dense annotation model is used to perform feature adaptation on the relationship between the 3D object and the context of the autonomous driving scene to obtain a prompt word sample, and the prompt word sample is processed using a pre-trained large language model to generate a predicted description of the 3D object.

[0051] In operation S150, the prediction of the 3D object and the true value description of the 3D object are processed using a predefined loss function to obtain a loss value, and the 3D dense annotation model is iteratively trained based on the loss value until a preset condition is met to obtain a trained 3D dense annotation model.

[0052] In operation S160, the trained 3D densely labeled model is deployed to the target autonomous driving scene, and the trained 3D densely labeled model is used to densely label 3D objects in the target autonomous driving scene.

[0053] In the above operation S160, the acquisition of the relevant object information in the target autonomous driving scene is authorized by the user, and with the user's permission, the relevant data of the 3D objects involved is processed, stored and transmitted. The relevant process strictly complies with the provisions of laws and regulations, and strict confidentiality measures are taken. At the same time, the present invention can set an interface for agreeing or rejecting data collection in operation S160 for the relevant user to choose.

[0054] The 3D dense annotation of the above-mentioned autonomous driving scene provided by the present invention creates a bird's-eye view (BEV) from 3D point clouds and 2D multi-view images and obtains the relationship between each object and the scene context by identifying different 3D objects in the autonomous driving scene. The dense annotation of objects is achieved through a pre-trained large language model, which improves the accuracy of 3D object positioning and description in outdoor scenes; at the same time, it avoids retraining of the large language model and effectively utilizes the common sense reasoning ability of the large language model pre-trained under large-scale data. The above method provided by the present invention significantly improves the efficiency and practicality of the annotation process, and provides a new technical path for the understanding and application of outdoor 3D scenes.

[0055] The following is a specific embodiment and combined with the attached Figure 2 and 3 The above method provided by the present invention is further described in detail.

[0056] Figure 2 is an architectural diagram of a 3D dense annotation model according to an embodiment of the present invention.

[0057] Figure 3 It is a schematic diagram of a 3D densely labeled model that is trained in an outdoor real scene according to an embodiment of the present invention.

[0058] According to an embodiment of the present invention, the above-mentioned use of a 3D dense annotation model to perform image processing on 2D multi-view image samples to obtain multi-view image feature samples includes: based on a spatiotemporal attention mechanism, using the multimodal bird's-eye view sample features obtained by training at the previous moment and a learnable bird's-eye view query vector at the current moment to obtain a learnable bird's-eye view query vector that integrates historical information; extracting multi-view visual features from 2D multi-view image samples, and based on a spatial cross-attention mechanism, using the spatiotemporal bird's-eye view encoder of a 3D dense annotation model to process the multi-view visual features and the learnable bird's-eye view query vector that integrates historical information to obtain a multi-view image feature sample mapped to a bird's-eye view space at the current moment.

[0059] Specifically, for multi-view images ,in is the number of images, and is the image size, such as Figure 2As shown, the spatiotemporal BEV encoder is used ( Figure 2 The BEV Encoder in the image maps image features to the BEV space and models dynamic information by fusing historical BEV features. First, the multi-angle visual features are extracted from the original image through the image framework, and the spatial cross-attention technique is used with the learnable BEV query.

[0060] Interactively obtain spatial information, as shown in formula (1):

[0061] (1),

[0062] Among them, Backbone Figure 2 The Image Backbone shown in .

[0063] In order to extract time dependency and dynamic features, if the BEV feature of the previous moment Exists, BEV query First, we interact with it to get, as shown in formula (2):

[0064] (2),

[0065] The result is Alternative input to the spatial criss-cross attention layer.

[0066] According to an embodiment of the present invention, the above-mentioned point cloud analysis of 3D laser point cloud data samples to obtain point cloud data feature samples includes: extracting voxel features from the 3D laser point cloud data samples, and using a 3D dense annotation model to flatten the voxel features in the height direction and then align them with the multi-view image feature samples to obtain point cloud data feature samples.

[0067] like Figure 2 As shown, for the lidar data (Right now Figure 2 In the LIDAR Backbone, the point cloud framework is used to extract voxel features and flattened in the height direction to align the extracted 2D image features. , using the convolutional model to fuse the features of two different modalities (i.e. Figure 2 Fusion Module in the final result is a unified bird's-eye view feature ,(Right now Figure 2 Fused BEV Feature).

[0068] According to an embodiment of the present invention, the above-mentioned use of a 3D dense annotation model to perform 3D object recognition and positioning on multimodal bird's-eye view feature samples to obtain a 3D object set in the autonomous driving scene includes: based on a predefined query number and a predefined feature dimension, using a 3D object detection network of a 3D dense annotation model to perform 3D object detection on multimodal bird's-eye view feature samples, and based on the detection results, identifying and positioning the 3D objects in the autonomous driving scene to obtain a 3D object set.

[0069] Specifically, Figure 2 The Proposal Module detection head shown in the figure adopts a query-based object detection method based on the above BEV features. As input, identify and locate different 3D objects, and get ,Right now Figure 2 Object Proposals shown, where Query quantity for predefined objects, is the feature dimension.

[0070] According to an embodiment of the present invention, the above-mentioned information retrieval and learning representation mechanism is based on, and the interaction relationship in the 3D object set is analyzed using a 3D dense annotation model to obtain the relationship between the 3D objects and the autonomous driving scene context, including: using the multi-layer perceptron in the 3D dense annotation model to encode the 3D objects in the 3D object set to obtain a 3D object set aligned with the multimodal bird's-eye view feature samples; using the information retrieval and learning representation network based on the query mechanism of the 3D dense annotation model to analyze the interaction relationship between the 3D objects in the aligned 3D object set and the interaction relationship between the 3D objects and the autonomous driving scene to obtain the relationship between the 3D objects and the autonomous driving scene context.

[0071] The above-mentioned multi-layer perceptron and Relation Q-former technology analyzes the interaction between different objects and between objects and the environment, and obtains the relationship between each object and the scene context.

[0072] Specifically, Figure 2 The Captioner shown uses a multi-layer perceptron to Encode it to match the BEV feature The dimensions of are aligned, and then the two features are connected together and input into the Relation Q-former composed of several self-attention mechanisms, so as to achieve full interaction between the features and obtain the relationship between the object and the scene context, as shown in formula (3):

[0073] (3),

[0074] According to an embodiment of the present invention, the above-mentioned use of a 3D dense annotation model to perform feature adaptation on the relationship between the 3D object and the autonomous driving scene context to obtain prompt word samples includes: using a multi-layer perceptron-based adapter of the 3D dense annotation model to perform feature conversion on the relationship between the 3D object and the autonomous driving scene context to obtain prompt word samples that are aligned with the feature space of a pre-trained large language model.

[0075] According to an embodiment of the present invention, the above-mentioned use of a pre-trained large language model to process prompt word samples to generate a predicted description of a 3D object includes: using a pre-trained large language model to process prompt word samples and predefined prompt text to generate a predicted description of the 3D object, wherein the predicted description of the 3D object includes multiple words for describing the 3D object.

[0076] According to an embodiment of the present invention, the predefined loss function is obtained by performing a weighted operation on an L1 regression loss function based on a 3D target box and a language annotation loss function based on a predefined weighting coefficient.

[0077] Specifically, the present invention adopts a pre-trained large language model LLaMA (such as Figure 2 As shown in Figure 2, as a language generator, in order to ensure that the input features are consistent with the dimensions of the hidden layer of the large language model, MLP conversion features are used. Dimension.

[0078] The method further uses adapter technology to align the object representation with the feature space of the pre-trained model to bridge the gap between different modalities. The features processed by the adapter are used as the prompt word input of the large language model. , generate the corresponding 3D object description. The generation process is expressed as formula (4)~(6):

[0079] (4),

[0080] (5),

[0081] (6),

[0082] in, Indicates the system text prompt. is the generated text description, including words.

[0083] In the training phase, the standard cross entropy loss is adopted, and the teacher-forcing method is used for model training. The text description loss expression is shown in formula (7):

[0084] (7),

[0085] in, is the true value label, To pre-train the large language model weights, Represents the probability of each word.

[0086] The overall loss is the weighted term of the 3D object box L1 regression loss and the above language annotation loss, as shown in formula (8):

[0087] (8),

[0088] in, and is a hyperparameter.

[0089] After the training of the above 3D dense labeling model is completed, the last updated network parameters are obtained as the training results, and the 3D dense labeling model is deployed to ensure the efficiency and stability of the 3D dense labeling model in practical applications.

[0090] like Figure 3 As shown in the figure, the deployed 3D dense annotation model is compared with other methods (i.e. Figure 3 The GT method shown in Figure 3) has achieved obvious advantages in the annotation process.

[0091] Figure 4 It is a schematic diagram of the training process of an outdoor 3D dense labeling method according to another embodiment of the present invention.

[0092] like Figure 4 As shown, step S101 uses multi-view camera images and lidar point cloud data to generate bird's-eye view features through advanced image processing and point cloud analysis technology. This step provides a basis for subsequent object detection and scene understanding; step S102, based on the generated bird's-eye view features, the detection head identifies and locates different 3D objects; step S103, using Relation Q-former technology analyzes the interaction between different objects and between objects and the environment, and obtains the relationship between each object and the scene context. This step enhances the in-depth understanding of the scene and provides key information for object labeling. Step S104: Based on the adapter, the features obtained in step S103 are processed and converted into prompt input suitable for the large language model to generate a dense description of the 3D object. This step avoids retraining the existing language model and utilizes its common sense reasoning ability obtained through pre-training on large-scale data sets. Step S105: Determine whether the preset training termination condition is met, that is, the upper limit of the number of rounds of traversing the training data set, otherwise repeat steps S101-S104. Step S106: Training ends, the last updated network parameters are obtained as the training results, and the model is deployed to ensure the efficiency and stability of the model in practical applications.

[0093] Figure 5 3D dense labeling device for autonomous driving scenes according to an embodiment of the present invention.

[0094] like Figure 5 As shown, the 3D dense labeling device 500 for the above-mentioned autonomous driving scene includes a multimodal feature fusion module 510, a 3D object acquisition module 520, a contextual relationship analysis module 530, a prediction description word generation module 540, a labeling model training module 550 and a labeling model deployment module 560.

[0095] The multimodal feature fusion module 510 is used to perform image processing on 2D multi-view image samples using a 3D dense annotation model to obtain multi-view image feature samples, perform point cloud analysis on 3D laser point cloud data samples to obtain point cloud data feature samples, and perform convolution fusion on the multi-view image feature samples and the point cloud data feature samples to obtain multimodal bird's-eye view feature samples.

[0096] The 3D object acquisition module 520 is used to use the 3D dense annotation model to perform 3D object recognition and positioning on the multimodal bird's-eye view feature samples to obtain a set of 3D objects in the autonomous driving scene.

[0097] The contextual relationship analysis module 530 is used to analyze the interaction relationship in the 3D object set based on information retrieval and learning representation mechanism using a 3D dense annotation model to obtain the relationship between the 3D object and the context of the autonomous driving scene.

[0098] The prediction description word generation module 540 is used to use the 3D dense annotation model to perform feature adaptation on the relationship between the 3D object and the context of the autonomous driving scene, obtain prompt word samples, and use the pre-trained large language model to process the prompt word samples to generate a prediction description of the 3D object.

[0099] The annotation model training module 550 is used to process the prediction of 3D objects and the true value description of 3D objects using a predefined loss function to obtain a loss value, and iteratively train the 3D dense annotation model based on the loss value until the preset conditions are met to obtain a trained 3D dense annotation model.

[0100] The labeling model deployment module 560 is used to deploy the trained 3D dense labeling model to the target autonomous driving scene, and use the trained 3D dense labeling model to densely label the 3D objects in the target autonomous driving scene.

[0101] According to an embodiment of the present invention, any multiple modules of the multimodal feature fusion module 510, the 3D object acquisition module 520, the contextual relationship analysis module 530, the prediction description word generation module 540, the annotation model training module 550, and the annotation model deployment module 560 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the multimodal feature fusion module 510, the 3D object acquisition module 520, the contextual relationship analysis module 530, the prediction description word generation module 540, the annotation model training module 550, and the annotation model deployment module 560 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in any appropriate combination of any of them. Alternatively, at least one of the multimodal feature fusion module 510, the 3D object acquisition module 520, the contextual relationship analysis module 530, the prediction description word generation module 540, the annotation model training module 550, and the annotation model deployment module 560 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0102] Figure 6 A block diagram of an electronic device suitable for implementing a 3D dense annotation method for an autonomous driving scenario according to an embodiment of the present invention is schematically shown.

[0103] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0104] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0105] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0106] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0107] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0110] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A 3D dense annotation method for autonomous driving scenes, characterized in that: The method comprises: Performing image processing on 2D multi-view image samples using a 3D dense annotation model to obtain multi-view image feature samples, performing point cloud analysis on 3D laser point cloud data samples to obtain point cloud data feature samples, and performing convolution fusion on the multi-view image feature samples and the point cloud data feature samples to obtain multimodal bird's-eye view feature samples; Using the 3D dense annotation model to perform 3D object recognition and positioning on the multimodal bird's-eye view feature samples, to obtain a set of 3D objects in the autonomous driving scene; Based on information retrieval and learning representation mechanism, the interactive relationship in the 3D object set is analyzed using the 3D dense annotation model to obtain the relationship between the 3D object and the autonomous driving scene context; Using the 3D dense annotation model to perform feature adaptation on the relationship between the 3D object and the autonomous driving scene context to obtain a prompt word sample, and using a pre-trained large language model to process the prompt word sample to generate a predicted description of the 3D object; Processing the prediction of the 3D object and the true value description of the 3D object by using a predefined loss function to obtain a loss value, and iteratively training the 3D dense annotation model based on the loss value until a preset condition is met, thereby obtaining a trained 3D dense annotation model; The trained 3D dense annotation model is deployed to the target autonomous driving scene, and the trained 3D dense annotation model is used to densely annotate the 3D objects in the target autonomous driving scene.

2. The method according to claim 1, characterized in that The 3D dense annotation model is used to process the 2D multi-view image samples, and the multi-view image feature samples obtained include: Based on the spatiotemporal attention mechanism, the multimodal bird's-eye view sample features obtained by training at the previous moment and the learnable bird's-eye view query vector at the current moment are used to obtain a learnable bird's-eye view query vector that integrates historical information. Multi-view visual features are extracted from the 2D multi-view image samples, and based on the spatial cross-attention mechanism, the spatiotemporal bird's-eye view encoder of the 3D dense annotation model is used to process the multi-view visual features and the learnable bird's-eye view query vector that integrates historical information to obtain the multi-view image feature samples mapped to the bird's-eye view space at the current moment.

3. The method according to claim 1, characterized in that Perform point cloud analysis on 3D laser point cloud data samples, and obtain point cloud data feature samples including: Voxel features are extracted from the 3D laser point cloud data samples, and the voxel features are flattened in the height direction using the 3D dense annotation model and then aligned with the multi-view image feature samples to obtain point cloud data feature samples.

4. The method according to claim 1, characterized in that: The 3D dense annotation model is used to perform 3D object recognition and positioning on the multimodal bird's-eye view feature samples, and a set of 3D objects in the autonomous driving scene is obtained, including: Based on a predefined number of queries and a predefined feature dimension, a 3D object detection network of the 3D dense annotation model is used to perform 3D object detection on the multimodal bird's-eye view feature samples, and based on the detection results, 3D objects in the autonomous driving scene are identified and located to obtain the 3D object set.

5. The method according to claim 1, characterized in that Based on information retrieval and learning representation mechanism, the interactive relationship in the 3D object set is analyzed using the 3D dense annotation model to obtain the relationship between the 3D object and the autonomous driving scene context, including: Encoding the 3D objects in the 3D object set by using a multi-layer perceptron in the 3D dense annotation model to obtain a 3D object set aligned with the multimodal bird's-eye view feature samples; The information retrieval and learning representation network based on the query mechanism of the 3D dense annotation model is used to analyze the interaction relationship between the 3D objects in the aligned 3D object set and the interaction relationship between the 3D objects and the autonomous driving scene to obtain the relationship between the 3D objects and the context of the autonomous driving scene.

6. The method according to claim 1, characterized in that The 3D dense annotation model is used to perform feature adaptation on the relationship between the 3D object and the autonomous driving scene context, and the prompt word samples obtained include: The relationship between the 3D object and the autonomous driving scene context is subjected to feature transformation by using a multi-layer perceptron-based adapter of the 3D dense annotation model to obtain prompt word samples aligned with the feature space of the pre-trained large language model.

7. The method according to claim 1, characterized in that Processing the prompt word sample using a pre-trained large language model to generate a predicted description of the 3D object includes: The prompt word sample and the predefined prompt text are processed by using the pre-trained large language model to generate a predicted description of the 3D object, wherein the predicted description of the 3D object includes a plurality of words for describing the 3D object.

8. The method according to claim 1, characterized in that: The predefined loss function is obtained by weighted operation of an L1 regression loss function based on a 3D target box and a language annotation loss function based on a predefined weighting coefficient.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images