Method, device, equipment and medium for updating and fusing multimodal data for autonomous driving

Abstract

The present application relates to a method, apparatus, device, and medium for updating and fusing multimodal data for autonomous driving. The method comprises: obtaining point cloud data and image data corresponding to a target object; inputting the image data corresponding to the target object into an AI large model and updating the image data category corresponding to the target object; performing three-dimensional detection on the point cloud data corresponding to the target object based on a transformer neural network model to determine the three-dimensional point cloud data corresponding to the target object; performing target detection on the updated image data corresponding to the target object based on a single-stage monocular three-dimensional object detection model to determine the three-dimensional image data corresponding to the target object; and performing data fusion on the three-dimensional point cloud data and three-dimensional image data corresponding to the target object based on a preset target detection multimodal fusion model to determine the corresponding shared three-dimensional environment representation to complete the update and fusion. The present application can avoid the problem of missed detection caused by a single modality and reduce computational complexity.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a method for updating and fusing multimodal data for autonomous driving, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Autonomous driving technology is a key development direction for future smart transportation, and autonomous driving data plays a crucial role in its evolution. This data, containing a vast amount of annotated information, provides essential material for the training, verification, and optimization of autonomous driving systems, helping them learn and adapt to complex driving scenarios and making efficient and safe driving decisions.

[0003] Data labeling is a crucial component of autonomous driving data and a key step in training autonomous driving algorithms. Traditional manual labeling is complex and time-consuming, and due to the uncertainty of the real world, data may contain noise and inconsistencies, which can easily lead to labeling errors. Furthermore, due to the complexity and difficulty of the data collection process, some data suffer from insufficient or insufficient sample diversity, making manual labeling incomplete. This results in limited data categories, low information richness, an imbalance in the types of identifiable and labeled objects, complex matching of information across different dimensions, and high computational overhead.

[0004] Adapting to the existing technology, manual labeling is complex and time-consuming, which is prone to labeling errors. Manual labeling cannot achieve comprehensive coverage of data, resulting in limited data categories of autonomous driving multimodal data, low information richness, imbalance in the types of identifiable and labeled objects, complex matching of information in different dimensions, and high computational consumption. The applicant has made corresponding explorations to solve these problems. Summary of the Invention

[0005] The purpose of this application is to solve the above problems and provide an autonomous driving multimodal data update fusion method, corresponding device, electronic device and computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A method for updating and fusing multimodal data for autonomous driving proposed to meet one of the purposes of this application includes:

[0008] Responding to multimodal data update fusion instructions, obtaining point cloud data and image data corresponding to target objects from multiple sensors of the autonomous driving vehicle;

[0009] Input the image data corresponding to the target object into a preset AI large model, and perform feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object;

[0010] Performing three-dimensional detection on the point cloud data corresponding to the target object based on a preset transformer neural network model to determine the three-dimensional point cloud data corresponding to the target object;

[0011] performing target detection on the updated image data corresponding to the target object based on a preset single-stage monocular three-dimensional object detection model to detect the three-dimensional position and shape information corresponding to the target object, thereby determining the three-dimensional image data corresponding to the target object;

[0012] Based on the preset target detection multimodal fusion model, the three-dimensional point cloud data and three-dimensional image data corresponding to the target object are fused to determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and three-dimensional image data to complete the update fusion of autonomous driving multimodal data.

[0013] Optionally, the step of performing three-dimensional detection on the point cloud data corresponding to the target object based on a preset transformer neural network model to determine the three-dimensional point cloud data corresponding to the target object includes:

[0014] Encode the point cloud data corresponding to the target object into bird's-eye view features based on a standard sparse voxelization network;

[0015] performing scale conversion on the bird's-eye view features based on a multi-scale center proposal network to determine the bird's-eye view features of different scales, and predicting the position of the center of the target object;

[0016] In the transformer and decoder of the transformer neural network model, self-attention layers and cross-attention layers are used to aggregate features from other centers and multi-scale feature maps;

[0017] Using a multi-frame mode, the bird's-eye view features of the previous frames are fused with the bird's-eye view features of the current frame in the central head and cross-attention layers. The spatial perception-based fusion module calculates the pixel-level attention weights according to the bird's-eye view features of the current frame, and adds time embedding to distinguish different frames. The three-dimensional point cloud data corresponding to the target object is determined based on the attention weights and the added time.

[0018] Optionally, the step of performing target detection on the updated image data corresponding to the target object based on a preset single-stage monocular three-dimensional object detection model to detect the three-dimensional position and shape information corresponding to the target object includes:

[0019] Inputting the updated image data corresponding to the target object into the convolutional neural network pre-trained in the single-stage monocular 3D object detection model to extract target object features and construct a multi-scale feature pyramid;

[0020] At each feature level of the feature pyramid, a shared detection head is used to predict the category, attribute, center, and 3D regression target of the object;

[0021] According to the two-dimensional size of the target object in the image, objects of different sizes are assigned to feature maps of different levels, and positive and negative samples are determined based on the distance between the center of the object and the feature point;

[0022] Summarize the prediction results of all levels of the feature pyramid and use the rotation object detection model to remove overlapping boxes;

[0023] According to the predicted 3D regression target and the camera intrinsic parameters, the 2.5D center is converted into coordinates in 3D space, and the 3D bounding box of the target object is calculated to determine the corresponding 3D position and shape information of the target object.

[0024] Optionally, the step of performing data fusion on the three-dimensional point cloud data and the three-dimensional image data corresponding to the target object based on a preset target detection multimodal fusion model to determine a shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and the three-dimensional image data includes:

[0025] Dividing the three-dimensional space of the target object into regular voxels, and converting points in the three-dimensional space into voxel coordinates based on a voxel coordinate conversion algorithm;

[0026] For each input 3D point cloud data and 3D image data corresponding to the target object, converting the points of the 3D point cloud data and 3D image data into corresponding voxel coordinates thereof, and fusing the converted voxel coordinates corresponding to the points of the point cloud data or image data with the voxel grid;

[0027] When fusing new points, the state of the voxel is updated according to the distance and normal vector information of the point to perform data fusion of the three-dimensional point cloud data and the three-dimensional image data corresponding to the target object, and determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and the three-dimensional image data.

[0028] Optionally, the voxel coordinate conversion algorithm is:

[0029]

[0030]

[0031]

[0032] Where x represents the horizontal index, y represents the vertical index, and z represents the vertical index. (x′, y′, z′) represents a point in three-dimensional space. Δx, Δy, and Δz are the sizes of the voxel grid, and floor is the floor function.

[0033] Optionally, the step of inputting the image data corresponding to the target object into a preset AI large model, performing feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object includes:

[0034] In response to the image data update instruction, generating a category label based on the image data corresponding to the target object and the image data update instruction based on a preset AI large model;

[0035] The image data corresponding to the target object is subjected to feature extraction processing based on an image-text retrieval method, an image code is generated according to the input text and instructions, and the category label is matched with the image data corresponding to the target object to automatically update the category of the image data corresponding to the target object.

[0036] Optionally, after the step of inputting the image data corresponding to the target object into a preset AI large model and performing feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object, the method further includes:

[0037] Determine the two-dimensional bounding box of the image data corresponding to each target object, estimate the size and orientation of the view frustum, and obtain the camera center and line of sight direction from the camera's intrinsic and extrinsic parameters, where the intrinsic parameters include one or more of the camera's focal length and optical center, and the extrinsic parameters include one or more of the camera's position and orientation.

[0038] Estimate the height of the viewing cone based on prior knowledge and the depth image, and estimate the width and depth of the viewing cone based on the width and height of the two-dimensional bounding box, combined with the intrinsic parameters of the camera and prior information;

[0039] Based on the estimated size, direction, height, width and depth of the viewing cone, the coordinates of the eight vertices of the viewing cone are calculated and determined. The eight vertices are respectively located at the top of the viewing cone, the four corner points of the bottom surface, and the far end and the near end of the viewing cone, so as to project the image data corresponding to the target object onto the three-dimensional point cloud data corresponding to the target object.

[0040] An autonomous driving multimodal data updating and fusion device provided for another purpose of the present application includes:

[0041] a data acquisition module configured to acquire point cloud data and image data corresponding to a target object from a plurality of sensors of the autonomous driving vehicle;

[0042] an image data updating module, configured to input the image data corresponding to the target object into a preset AI large model, and perform feature extraction processing on the image data corresponding to the target object based on an image text retrieval method to update the image data category corresponding to the target object;

[0043] a three-dimensional point cloud data determination module, configured to perform three-dimensional detection on the point cloud data corresponding to the target object based on a preset transformer neural network model, and determine the three-dimensional point cloud data corresponding to the target object;

[0044] a three-dimensional image data determination module configured to perform target detection on the updated image data corresponding to the target object based on a preset single-stage monocular three-dimensional object detection model, detect the three-dimensional position and shape information corresponding to the target object, and determine the three-dimensional image data corresponding to the target object;

[0045] The multimodal data fusion module is configured to perform data fusion on the three-dimensional point cloud data and three-dimensional image data corresponding to the target object based on a preset target detection multimodal fusion model, determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and three-dimensional image data, and complete the update fusion of autonomous driving multimodal data.

[0046] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the autonomous driving multimodal data update fusion method described in the present application.

[0047] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the autonomous driving multimodal data update fusion method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0048] Compared with the existing technology, this application addresses the problems of manual labeling in the existing technology, which are complex and time-consuming, prone to labeling errors, and unable to achieve comprehensive data coverage. This leads to limited data categories, low information richness, imbalanced types of identifiable and labeled objects, complex matching of information in different dimensions, and high computational consumption. This application provides, but is not limited to, the following beneficial effects:

[0049] First, this application uses AI big model technology to update data categories, dynamically adding category information that is missing from the original data. This expands the original 23 categories of objects that can be identified and labeled to 5,000 categories, greatly expanding the data categories and enriching the information of multimodal data fusion. At the same time, it improves labeling efficiency, eliminating the need for additional manual labeling, and the original system framework does not need to be modified due to target category restrictions.

[0050] Secondly, this application implements 3D detection of point cloud data and image data based on the transformer neural network model and the single-stage monocular 3D object detection model, respectively, and then matches and fuses the detected target object data to achieve fusion annotation of multimodal data. Multimodal data can improve the accuracy of annotation content and avoid the problem of missed detection caused by a single modality. At the same time, fusion in the same dimension can reduce computational complexity and improve annotation efficiency.

[0051] Third, based on the F-PointNet algorithm, this application projects the target object in the two-dimensional image data output by the AI ​​large model into the corresponding point cloud data, thereby obtaining data such as the depth and orientation of the target object, enriching the annotation content, avoiding recognition errors caused by a single modality, improving the accuracy of the annotation content, and effectively realizing the combination of prior knowledge in the large model and multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0053] Figure 1 Schematic diagram of the process of updating and fusing multimodal data for autonomous driving in an embodiment of the present application;

[0054] Figure 2 This is a principle block diagram of the method for updating and fusing multimodal data for autonomous driving in an embodiment of the present application;

[0055] Figure 3 This is a rendering of the 2D image data information output based on the AI ​​large model in the embodiment of this application;

[0056] Figure 4 This is an effect diagram of the annotation of three-dimensional information of an image based on the F-Poinnet algorithm in an embodiment of the present application;

[0057] Figure 5 This is a schematic diagram of a process for determining three-dimensional point cloud data corresponding to a target object in an embodiment of the present application;

[0058] Figure 6 Schematic diagram of the process of detecting the three-dimensional position and shape information corresponding to the target object in an embodiment of the present application;

[0059] Figure 7 Schematic diagram of a process for fusing three-dimensional point cloud data and three-dimensional image data corresponding to a target object in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of a process for updating the image data category corresponding to a target object in an embodiment of the present application;

[0061] Figure 9 Schematic diagram of a process for projecting image data corresponding to a target object onto three-dimensional point cloud data corresponding to the target object in an embodiment of the present application;

[0062] Figure 10 This is a principle block diagram of the autonomous driving multimodal data updating and fusion device in an embodiment of the present application;

[0063] Figure 11 Schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0064] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0065] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0066] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0067] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0068] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0069] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0070] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0071] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0072] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0073] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0074] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0075] Based on the above example scenarios, please refer to Figure 1 as well as Figure 2 In one embodiment, the autonomous driving multimodal data update and fusion method of the present application includes:

[0076] Step S10: Responding to the multimodal data update fusion instruction, obtaining point cloud data and image data corresponding to the target object from multiple sensors of the autonomous driving vehicle;

[0077] The terminal device can obtain point cloud data and image data corresponding to the target object from multiple sensors of the autonomous driving vehicle. The sensors include one or any multiple of lidar or cameras. The point cloud data corresponding to the target object can be obtained from the lidar in the autonomous driving vehicle, and the image data corresponding to the target object can be obtained from the camera in the autonomous driving vehicle.

[0078] Step S20: input the image data corresponding to the target object into a preset AI large model, and perform feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object;

[0079] The image data corresponding to the target object is input into a preset AI big model, and feature extraction processing is performed on the image data corresponding to the target object based on the image text retrieval method, and the target object is framed in the image data corresponding to the target object. The AI ​​big model can be a Transformer model, and the working process of the AI ​​big model is as follows:

[0080] Transformer, as a unified computing engine, uses an encoder-decoder architecture to encode and decode different inputs and outputs. The basic formula of Transformer is as follows:

[0081]

[0082]

[0083] in, represents the hidden state of the i-th position in the i-th layer of the encoding layer, represents the hidden state of the i-th position of the decoder layer, FFN represents the feedforward neural network, SelfAttention represents the self-attention mechanism, and CrossAttention represents the cross-attention mechanism.

[0084] A unified vocabulary is used to represent different data types, including text, images, and locations. Text is segmented and embedded using byte pair encoding (BPE), images are convolved and sliced ​​using the ResNet module, and then quantized using VQGAN's discrete encoding, and locations are discretized using coordinate values.

[0085] Different instructions are used to represent different tasks. An instruction is a piece of text that tells the model what to do. Instructions can be concatenated with input and output into a sequence and separated by special symbols.

[0086] Multi-task pre-training is performed on multiple datasets, including cross-modal and unimodal data. Pre-training tasks include visual localization, visual description, image-text matching, image question answering, object detection, image filling, text filling, etc. The pre-training goal is to minimize the cross-entropy loss. The formula for cross-entropy loss is as follows:

[0087]

[0088] Among them, P θ represents the conditional probability distribution of the model parameter θ, y i Represents the encoding of the i-th position of the output sequence.

[0089] Fine-tuning or zero-shot inference is performed on downstream tasks to generate corresponding outputs based on different task instructions. For example, in image classification tasks, the AI ​​large model generates category labels based on the input image and instructions. In text-to-image generation tasks, the AI ​​large model generates image encoding based on the input text and instructions to update the image data category corresponding to the target object.

[0090] In some embodiments, after the target object is framed in the image data corresponding to the target object, the framed target object is projected back to the point cloud data corresponding to the target object based on the F-PointNet algorithm to further obtain target object information, wherein the target object information includes one or any combination of information such as orientation and depth. The image data corresponding to the target object is projected onto the three-dimensional point cloud data corresponding to the target object based on the F-PointNets model. F-PointNets is an algorithm for 3D object detection, which extracts a frustum from a 2D data image and performs 3D target detection. The following are the basic steps of the F-PointNets algorithm to obtain a frustum by projecting the two-dimensional image data corresponding to the target object onto the three-dimensional point cloud data corresponding to the target object:

[0091] a. Get parameters from 2D bounding boxes: In 2D object detection, we obtain a 2D bounding box for each target object. A 2D bounding box is usually represented by four vertex coordinates, namely the coordinates of the upper left corner and the lower right corner: (x_min, y_min) and (x_max, y_max);

[0092] b. Estimate the size and direction of the view frustum: Obtain the camera center (cx, cy, cz) and line of sight direction (l, m, n) from the camera's intrinsic and extrinsic parameters. The intrinsic parameters include information such as the camera's focal length and optical center, while the extrinsic parameters include the camera's position and orientation.

[0093] c. Estimate the height of the frustum using prior knowledge and the depth image. Use the width and height of the 2D bounding box, combined with the camera's intrinsic parameters and prior information, to estimate the width and depth of the frustum.

[0094] d. Using the estimated size, orientation, height, width, and depth of the viewing cone, calculate the coordinates of the eight vertices of the viewing cone. These eight vertices are located at the top of the viewing cone, the four corners of the bottom surface, and the far and near ends of the viewing cone.

[0095] See also Figure 3 as well as Figure 4 After the above steps, the 2D image data is effectively projected onto the 3D point cloud data, and the 2D image data corresponding to the target object is projected onto the 3D point cloud data corresponding to the target object, so as to perform data fusion of the updated 2D image data corresponding to the target object and the 3D point cloud data corresponding to the target object. The AI ​​large model is combined with the F-Poinnet algorithm to complete the annotation of the 3D information of the image. Because the current AI large model outputs 2D image data, the AI ​​large model combined with the F-Poinnet algorithm can directly complete the annotation of the 3D information of the image.

[0096] Step S30: performing three-dimensional detection on the point cloud data corresponding to the target object based on a preset transformer neural network model to determine the three-dimensional point cloud data corresponding to the target object;

[0097] The transformer neural network model can be a CenterFormer model, which is a center-based transformer neural network model that can be used for 3D object detection. Based on the preset CenterFormer model, three-dimensional detection is performed on the point cloud data corresponding to the target object, and the point cloud data corresponding to the target object is encoded into bird's-eye view features based on a standard sparse voxelization network; the bird's-eye view features are scale-converted based on a multi-scale center proposal network to determine the bird's-eye view features of different scales and predict the position of the center of the target object; in the transformer and decoder of the transformer neural network model, self-attention layers and cross-attention layers are used to aggregate features from other centers and multi-scale feature maps; in a multi-frame mode, the bird's-eye view features of the previous frames are fused with the bird's-eye view features of the current frame in the center head and cross-attention layers; a spatially aware fusion module is used to calculate pixel-level attention weights based on the bird's-eye view features of the current frame, and time embedding is added to distinguish different frames. The three-dimensional point cloud data corresponding to the target object is determined based on the attention weights and the added time.

[0098] Step S40: performing target detection on the updated image data corresponding to the target object based on a preset single-stage monocular 3D object detection model to detect the 3D position and shape information corresponding to the target object, thereby determining the 3D image data corresponding to the target object;

[0099] The single-stage monocular 3D object detection model may be an FCOS3D model, which is a center-based single-stage monocular 3D object detection network that adds a 3D regression branch and a center definition.

[0100] Specifically, the updated image data corresponding to the target object is input into the convolutional neural network pre-trained in the FCOS3D model to extract the target object features and construct a multi-scale feature pyramid; on each feature layer of the feature pyramid, a shared detection head is used to predict the object category, attribute, center and three-dimensional regression target; according to the two-dimensional size of the target object on the image, objects of different sizes are assigned to feature maps of different levels, and positive and negative samples are determined according to the distance between the object center and the feature point; the prediction results of the feature pyramid for all levels are summarized, and the rotating target detection model is used to remove overlapping frames; according to the predicted three-dimensional regression target and the camera intrinsic parameters, the 2.5D center is converted into coordinates in three-dimensional space, and the three-dimensional bounding box of the target object is calculated to determine the corresponding three-dimensional position and shape information of the target object.

[0101] In some embodiments, the rotation target detection model may be an NMS target detection model or the like.

[0102] Step S50: Based on the preset target detection multimodal fusion model, the three-dimensional point cloud data and the three-dimensional image data corresponding to the target object are fused to determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and the three-dimensional image data to complete the update fusion of the autonomous driving multimodal data.

[0103] The target detection multimodal fusion model can be a Voxel-based Fusion algorithm model, etc. The Voxel-based Fusion algorithm model is taken as an example, which does not constitute a limitation to the present application. The Voxel-based Fusion algorithm model is an algorithm commonly used in 3D reconstruction and visual SLAM. It is mainly used to fuse point clouds or depth images from multiple sensors into a shared 3D environment representation. Based on the preset Voxel-based Fusion algorithm model, the 3D point cloud data and 3D image data corresponding to the target object are subjected to data fusion, and the shared 3D environment representation corresponding to the 3D point cloud data and 3D image data is determined, which can improve the accuracy and robustness of target object perception detection.

[0104] Specifically, the three-dimensional space of the target object is divided into regular voxels, and the points in the three-dimensional space are converted into voxel coordinates based on the voxel coordinate conversion algorithm; for each input three-dimensional point cloud data and three-dimensional image data corresponding to the target object, the points of the three-dimensional point cloud data and three-dimensional image data are converted into their corresponding voxel coordinates, and the voxel coordinates corresponding to the points of the converted point cloud data or image data are fused with the voxel grid; when fusing new points, the state of the voxels is updated according to the distance and normal vector information of the points to perform data fusion of the three-dimensional point cloud data and three-dimensional image data corresponding to the target object, and determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and three-dimensional image data to complete the update and fusion of multimodal data for autonomous driving.

[0105] As can be seen from the above embodiments, compared with the prior art, this application aims to adapt to the problems of the prior art, such as the complexity and time-consuming manual labeling, the easy occurrence of labeling errors, the inability of manual labeling to achieve comprehensive coverage of the data, the limited data categories of autonomous driving multimodal data, low information richness, imbalanced types of identifiable labeled objects, complex matching of information in different dimensions, and high computational consumption. This application includes but is not limited to the following:

[0106] Beneficial effects:

[0107] First, this application uses AI big model technology to update data categories, dynamically adding category information that is missing from the original data. This expands the original 23 categories of objects that can be identified and labeled to 5,000 categories, greatly expanding the data categories and enriching the information of multimodal data fusion. At the same time, it improves labeling efficiency, eliminating the need for additional manual labeling, and the original system framework does not need to be modified due to target category restrictions.

[0108] Secondly, this application implements 3D detection of point cloud data and image data based on the transformer neural network model and the single-stage monocular 3D object detection model, respectively, and then matches and fuses the detected target object data to achieve fusion annotation of multimodal data. Multimodal data can improve the accuracy of annotation content and avoid the problem of missed detection caused by a single modality. At the same time, fusion in the same dimension can reduce computational complexity and improve annotation efficiency.

[0109] Third, based on the F-PointNet algorithm, this application projects the target object in the two-dimensional image data output by the AI ​​large model into the corresponding point cloud data, thereby obtaining data such as the depth and orientation of the target object, enriching the annotation content, avoiding recognition errors caused by a single modality, improving the accuracy of the annotation content, and effectively realizing the combination of prior knowledge in the large model and multimodal data.

[0110] Based on any embodiment of this application, please refer to Figure 5 The step of performing three-dimensional detection on the point cloud data corresponding to the target object based on a preset transformer neural network model and determining the three-dimensional point cloud data corresponding to the target object includes:

[0111] Step S301: Encode the point cloud data corresponding to the target object into bird's-eye view features based on a standard sparse voxel network;

[0112] Step S303: performing scale conversion on the bird's-eye view feature based on a multi-scale center proposal network to determine the bird's-eye view features of different scales, and predicting the position of the center of the target object;

[0113] Step S305: In the transformer and decoder of the transformer neural network model, self-attention layers and cross-attention layers are used to aggregate features from other centers and multi-scale feature maps;

[0114] Step S307: Using a multi-frame mode, the bird's-eye view features of the previous frames are fused with the bird's-eye view features of the current frame in the central head and cross-attention layers. The spatial perception-based fusion module calculates the pixel-level attention weight according to the bird's-eye view features of the current frame, and adds time embedding to distinguish different frames. The three-dimensional point cloud data corresponding to the target object is determined according to the attention weight and the added time.

[0115] Specifically, the raw point cloud data is encoded into a bird's-eye view (BEV) feature representation through a standard sparse voxelization network; a multi-scale center proposal network (CPN) is used to convert the BEV features to different scales and predict the location of the object center; the features of the object center are extracted as query embeddings for the transformer decoder; in the transformer and decoder, self-attention layers and cross-attention layers are used to aggregate features from other centers and multi-scale feature maps; the cross-attention layer uses a small window or a deformable region to limit the scope of attention learning, thereby reducing computational complexity. The output object representation is then regressed to other object attributes to generate the final object prediction; using a multi-frame mode, the BEV features of the previous frames are fused with the BEV features of the current frame in the center head and cross-attention layers; a spatially aware fusion module is used to calculate pixel-level attention weights based on the BEV features of the current frame, and a temporal embedding is added to distinguish different frames. The 3D point cloud data corresponding to the target object is determined based on the attention weights and the added time.

[0116] Based on any embodiment of this application, please refer to Figure 6 The step of performing target detection on the updated image data corresponding to the target object based on a preset single-stage monocular three-dimensional object detection model to detect the three-dimensional position and shape information corresponding to the target object includes:

[0117] Step S401: inputting the updated image data corresponding to the target object into the convolutional neural network pre-trained in the single-stage monocular 3D object detection model to extract target object features and construct a multi-scale feature pyramid;

[0118] Step S403: On each feature layer of the feature pyramid, a shared detection head is used to predict the category, attribute, center, and three-dimensional regression target of the object;

[0119] Step S405: assign objects of different sizes to feature maps of different levels according to the two-dimensional size of the target object in the image, and determine positive and negative samples based on the distance between the object center and the feature point;

[0120] Step S407: Summarize the prediction results of all levels of the feature pyramid and use the rotation object detection model to remove overlapping frames;

[0121] Step S409: Convert the 2.5D center into coordinates in three-dimensional space based on the predicted three-dimensional regression target and the camera intrinsic parameters, calculate the three-dimensional bounding box of the target object, and determine the corresponding three-dimensional position and shape information of the target object.

[0122] Specifically, the updated image data corresponding to the target object is subjected to a pre-trained ResNet101 convolutional neural network to extract features, and a multi-scale feature pyramid is constructed using the FPN model; at each feature layer, a shared detection head is used to predict the object's category, attribute, center, and 3D regression target, and the 3D regression target includes the 2D offset from the feature point to the object center, the depth of the object center, the 3D size of the object, the orientation angle, and the orientation category of the object; objects of different sizes are assigned to feature maps of different levels according to the 2D size of the object on the image, and positive and negative samples are determined according to the distance between the object center and the feature point. A 2D Gaussian distribution can be used to define the center and use it to filter low-quality predictions; the prediction results of all levels are aggregated, and the NMS rotation target detection model is used to remove overlaps; then, based on the predicted 3D regression target and the camera intrinsic parameters, the 2.5D center is converted to coordinates in 3D space, and the 3D bounding box of the object is calculated to determine the corresponding three-dimensional position and shape information of the target object.

[0123] Based on any embodiment of this application, please refer to Figure 7 The step of fusing the three-dimensional point cloud data and the three-dimensional image data corresponding to the target object based on a preset target detection multimodal fusion model to determine a shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and the three-dimensional image data includes:

[0124] Step 501: Divide the three-dimensional space of the target object into regular voxels, and convert points in the three-dimensional space into voxel coordinates based on a voxel coordinate conversion algorithm;

[0125] Step 503: for each input 3D point cloud data and 3D image data corresponding to the target object, convert the points of the 3D point cloud data and 3D image data into their corresponding voxel coordinates, and fuse the converted voxel coordinates corresponding to the points of the point cloud data or image data with the voxel grid;

[0126] Step S505: When fusing new points, the voxel state is updated according to the distance and normal vector information of the points to perform data fusion of the three-dimensional point cloud data and the three-dimensional image data corresponding to the target object, and determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and the three-dimensional image data.

[0127] Specifically, the three-dimensional space of the target object is divided into regular voxels, and the points in the three-dimensional space are converted into voxel coordinates based on the voxel coordinate conversion algorithm; the three-dimensional space of the target object is divided into a regular cubic grid, namely voxels (voxel). Assuming that the size of the voxel grid is Δx, Δy and Δz, then the entire three-dimensional space can be represented as a set of voxels, and the voxel coordinates representing the three-dimensional space are (x, y, z), where x represents the horizontal index, y represents the vertical index, and z represents the vertical index. Each voxel represents a three-dimensional space unit of a fixed size. The point (x′, y′, z′) in the three-dimensional space can be converted into voxel coordinates (x, y, z) by the voxel coordinate conversion algorithm, and the formula is as follows:

[0128]

[0129]

[0130]

[0131] Among them, floor represents the rounding down function

[0132] For each input 3D point cloud data and 3D image data corresponding to the target object, the points need to be converted into voxel coordinates so that they can be fused in the voxel grid. Assuming that an input point cloud or depth image contains n points, the coordinates of the i-th point are (x_i′, y_i′, z_i′), then its corresponding voxel index in the voxel grid is (x_i, y_i, z_i)

[0133] For each input data source, the transformed point is fused with the voxel grid using TSDF. When a new point (x_i′, y_i′, z_i′) is fused, the voxel state is updated based on the distance and normal vector information of the point. The specific formula is as follows:

[0134]

[0135] Where weight is the previous weight, indicating the number of previous fusions; distance is the distance from the point (x_i′, y_i′, z_i′) to the voxel (x_i, y_i, z_i) surface

[0136] During the fusion process, optimization techniques can be employed to adjust the state of the voxel grid to minimize inconsistencies between input data or to optimize reconstruction quality.

[0137] As can be seen from the above embodiments, the present application realizes 3D detection of point cloud data and image data respectively based on the transformer neural network model and the single-stage monocular 3D object detection model, and then matches and fuses the target object data obtained by detection, thereby realizing the fusion annotation of multimodal data. Multimodal data can improve the accuracy of the annotation content and avoid the problem of missed detection caused by a single modality. At the same time, fusion in the same dimension can reduce the complexity of calculation and improve the annotation efficiency.

[0138] Based on any embodiment of this application, please refer to Figure 8 The steps of inputting the image data corresponding to the target object into a preset AI large model and performing feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object include:

[0139] Step 201: In response to an image data update instruction, a category label is generated based on the image data corresponding to the target object and the image data update instruction based on a preset AI large model;

[0140] Step 203: Perform feature extraction processing on the image data corresponding to the target object based on the image text retrieval method, generate image coding according to the input text and instructions, match the category label with the image data corresponding to the target object, and automatically update the category of the image data corresponding to the target object.

[0141] Specifically, the image data corresponding to the target object is input into a preset AI large model. Based on the preset AI large model, a category label is generated according to the image data corresponding to the target object and the image data update instruction. The image data corresponding to the target object is subjected to feature extraction processing based on the image text retrieval method. The AI ​​large model may be a Transformer model. The working process of the Transformer model is as follows:

[0142] The Transformer model, as a unified computing engine, uses an encoder-decoder architecture to encode and decode different inputs and outputs. The basic formula of the Transformer is as follows:

[0143]

[0144]

[0145] in, represents the hidden state of the i-th position in the i-th layer of the encoding layer, represents the hidden state of the i-th position of the decoder layer, FFN represents the feedforward neural network, SelfAttention represents the self-attention mechanism, and CrossAttention represents the cross-attention mechanism.

[0146] A unified vocabulary is used to represent different data types, including text, images, locations, etc. Text is segmented and embedded using byte pair encoding (BPE), images are convolved and sliced ​​using the ResNet module, and then quantized using the discrete encoding of VQGAN, and locations are discretized using coordinate values. Different instructions are used to represent different tasks. An instruction is a piece of text that tells the model what to do. Instructions can be concatenated with input and output into a sequence and separated by special symbols; multi-task pre-training is performed on multiple datasets, including cross-modal and unimodal data. Pre-training tasks include visual positioning, visual description, image-text matching, image question answering, target detection, image filling, text filling, etc. The pre-training goal is to minimize the cross-entropy loss. The formula for cross-entropy loss is as follows:

[0147]

[0148] Among them, P θ represents the conditional probability distribution of the model parameter θ, y i Represents the encoding of the i-th position in the output sequence. Fine-tuning or zero-shot inference is performed on downstream tasks, generating corresponding outputs based on different task instructions. Feature extraction is performed on the image data corresponding to the target object based on the image text retrieval method. Image encoding is generated based on the input text and instructions. The category label is matched with the image data corresponding to the target object to automatically update the category of the image data corresponding to the target object.

[0149] It can be seen from the above embodiments that the present application realizes the update of data categories based on AI big model technology, can dynamically add category information that is lacking in the original data, and expands the original 23 categories of targets that can be identified and labeled to 5,000 categories, greatly expanding the data categories and enriching the information of multimodal data fusion. At the same time, it improves the labeling efficiency, and no additional manual labeling is required. The original system framework does not need to be modified due to the limitation of target categories.

[0150] Based on any embodiment of this application, please refer to Figure 9 , inputting the image data corresponding to the target object into a preset AI large model, performing feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object, including:

[0151] Step 100: Determine the two-dimensional bounding box of the image data corresponding to each target object, estimate the size and direction of the view frustum, and obtain the camera center and line of sight direction from the camera intrinsic parameters and extrinsic parameters, wherein the intrinsic parameters include one or more of the camera focal length and optical center, and the extrinsic parameters include one or more of the camera position and orientation.

[0152] Step 200: Estimate the height of the viewing cone based on prior knowledge and the depth image, and estimate the width and depth of the viewing cone based on the width and height of the two-dimensional bounding box, combined with the intrinsic parameters of the camera and prior information;

[0153] Step 300: Based on the estimated size, direction, height, width, and depth of the viewing cone, the coordinates of the eight vertices of the viewing cone are calculated and determined. The eight vertices are located at the top of the viewing cone, the four corner points of the bottom surface, and the far end and the near end of the viewing cone, so as to project the image data corresponding to the target object onto the three-dimensional point cloud data corresponding to the target object.

[0154] Specifically, the image data corresponding to the target object can be projected onto the corresponding 3D point cloud data based on the F-PointNets model. F-PointNets is an algorithm for 3D object detection that extracts the frustum from a 2D image and performs 3D object detection. The following are the basic steps of the F-PointNets algorithm to project the frustum from 2D to 3D:

[0155] a. Get parameters from 2D bounding boxes: In 2D object detection, we obtain a 2D bounding box for each target object. A 2D bounding box is usually represented by four vertex coordinates, namely the coordinates of the upper left corner and the lower right corner: (x_min, y_min) and (x_max, y_max);

[0156] b. Estimate the size and direction of the view frustum: Obtain the camera center (cx, cy, cz) and line of sight direction (l, m, n) from the camera's intrinsic and extrinsic parameters. The intrinsic parameters include information such as the camera's focal length and optical center, while the extrinsic parameters include the camera's position and orientation.

[0157] c. Estimate the height of the frustum using prior knowledge and the depth image. Use the width and height of the 2D bounding box, combined with the camera's intrinsic parameters and prior information, to estimate the width and depth of the frustum.

[0158] d. Using the estimated size, orientation, height, width, and depth of the viewing cone, calculate the coordinates of the eight vertices of the viewing cone. These eight vertices are located at the top of the viewing cone, the four corners of the bottom surface, and the far and near ends of the viewing cone.

[0159] Through the above steps, the 2D image data is effectively projected onto the 3D point cloud data, and the image data corresponding to the target object is projected onto the 3D point cloud data corresponding to the target object.

[0160] It can be seen from the above embodiments that this application is based on the F-PointNet algorithm, which projects the target object in the two-dimensional image data output by the AI ​​large model into the corresponding point cloud data, thereby obtaining data such as the depth and orientation of the target object, enriching the annotation content, avoiding recognition errors caused by a single modality, improving the accuracy of the annotation content, and effectively realizing the combination of prior knowledge in the large model and multimodal data.

[0161] See also Figure 10 An autonomous driving multimodal data update and fusion device provided to meet one of the purposes of this application includes a data acquisition module 1100, an image data update module 1200, a three-dimensional point cloud data determination module 1300, a three-dimensional image data determination module 1400 and a multimodal data fusion module 1500. Among them, the data acquisition module 1100 is configured to acquire point cloud data and image data corresponding to a target object from multiple sensors of the autonomous driving vehicle; the image data update module 1200 is configured to input the image data corresponding to the target object into a preset AI large model and perform feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the image data category corresponding to the target object; the three-dimensional point cloud data determination module 1300 is configured to perform three-dimensional detection on the point cloud data corresponding to the target object based on a preset transformer neural network model to determine the three-dimensional point cloud data corresponding to the target object; the three-dimensional image data determination module 1400 is configured to perform target detection on the updated image data corresponding to the target object based on a preset single-stage monocular three-dimensional object detection model to detect the three-dimensional position and shape information corresponding to the target object to determine the three-dimensional image data corresponding to the target object; the multimodal data fusion module 1500 is configured to perform data fusion on the three-dimensional point cloud data and three-dimensional image data corresponding to the target object based on a preset target detection multimodal fusion model to determine a shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and the three-dimensional image data to complete the update and fusion of the autonomous driving multimodal data.

[0162] Based on any embodiment of this application, please refer to Figure 11 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 11As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a method for updating and fusion of multimodal data for autonomous driving. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may execute the method for updating and fusion of multimodal data for autonomous driving of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0163] In this embodiment, the processor is used to execute Figure 10 The memory stores the program code and various data required to execute the specific functions of each module and its submodule in the autonomous driving multimodal data update and fusion device. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the autonomous driving multimodal data update and fusion device of this application. The server can call the server's program code and data to execute the functions of all submodules.

[0164] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the autonomous driving multimodal data update fusion method described in any embodiment of the present application.

[0165] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the autonomous driving multimodal data update and fusion method described in any embodiment of the present application.

[0166] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0167] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0168] In summary, multimodal data can improve the accuracy of annotation content and avoid missed detection problems caused by a single modality. At the same time, fusion in the same dimension can reduce the complexity of calculation and improve annotation efficiency.

Claims

1. A method for updating and fusing multimodal data for autonomous driving, characterized in that: include: Responding to multimodal data update fusion instructions, obtaining point cloud data and image data corresponding to target objects from multiple sensors of the autonomous driving vehicle; Input the image data corresponding to the target object into a preset AI large model, and perform feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object; The point cloud data corresponding to the target object is encoded into bird's-eye view features based on a standard sparse voxelization network; the bird's-eye view features are scaled based on a multi-scale center proposal network to determine the bird's-eye view features of different scales and predict the position of the center of the target object; in the transformer and decoder of the transformer neural network model, self-attention layers and cross-attention layers are used to aggregate features from other centers and multi-scale feature maps; in a multi-frame mode, the bird's-eye view features of the previous frames are fused with the bird's-eye view features of the current frame in the center head and cross-attention layers, and a spatially aware fusion module is used to calculate pixel-level attention weights based on the bird's-eye view features of the current frame, and time embedding is added to distinguish different frames. The three-dimensional point cloud data corresponding to the target object is determined based on the attention weights and the added time; The updated image data corresponding to the target object is input into a convolutional neural network pre-trained in a single-stage monocular 3D object detection model to extract target object features and construct a multi-scale feature pyramid; on each feature layer of the feature pyramid, a shared detection head is used to predict the category, attribute, center and 3D regression target of the object; objects of different sizes are assigned to feature maps of different levels according to the two-dimensional size of the target object in the image, and positive and negative samples are determined according to the distance between the center of the object and the feature point; the prediction results of all levels of the feature pyramid are summarized, and the rotating target detection model is used to remove overlapping frames; based on the predicted 3D regression target and the camera intrinsic parameters, the 2.5D center is converted into coordinates in 3D space, and the 3D bounding box of the target object is calculated, and the corresponding 3D position and shape information of the target object are determined to determine the corresponding 3D image data of the target object; The three-dimensional space of the target object is divided into regular voxels, and the points in the three-dimensional space are converted into voxel coordinates based on the voxel coordinate conversion algorithm; for each input three-dimensional point cloud data and three-dimensional image data corresponding to the target object, the points of the three-dimensional point cloud data and three-dimensional image data are converted into their corresponding voxel coordinates, and the voxel coordinates corresponding to the points of the converted point cloud data or image data are fused with the voxel grid; when fusing new points, the state of the voxels is updated according to the distance and normal vector information of the points to perform data fusion of the three-dimensional point cloud data and three-dimensional image data corresponding to the target object, and determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and three-dimensional image data to complete the update and fusion of multimodal data for autonomous driving.

2. The method for updating and fusing multimodal data for autonomous driving according to claim 1, characterized in that: The voxel coordinate conversion algorithm is: ； Where x represents the horizontal index, y represents the vertical index, and z represents the vertical index. (x', y', z') represents a point in three-dimensional space. Δx, Δy, and Δz are the sizes of the voxel grid, and floor is the floor function.

3. The method for updating and fusing multimodal data for autonomous driving according to claim 1, characterized in that: The steps of inputting the image data corresponding to the target object into a preset AI large model, performing feature extraction processing on the image data corresponding to the target object based on an image-text retrieval method to update the category of the image data corresponding to the target object include: In response to the image data update instruction, generating a category label based on the image data corresponding to the target object and the image data update instruction based on a preset AI large model; The image data corresponding to the target object is subjected to feature extraction processing based on an image-text retrieval method, an image code is generated according to the input text and instructions, and the category label is matched with the image data corresponding to the target object to automatically update the category of the image data corresponding to the target object.

4. The method for updating and fusing multimodal data for autonomous driving according to any one of claims 1 to 3, characterized in that: After inputting the image data corresponding to the target object into a preset AI large model and performing feature extraction processing on the image data corresponding to the target object based on an image text retrieval method to update the category of the image data corresponding to the target object, the method includes: Determine the two-dimensional bounding box of the image data corresponding to each target object, estimate the size and orientation of the view frustum, and obtain the camera center and line of sight direction from the camera's intrinsic and extrinsic parameters, where the intrinsic parameters include one or more of the camera's focal length and optical center, and the extrinsic parameters include one or more of the camera's position and orientation. Estimate the height of the viewing cone based on prior knowledge and the depth image, and estimate the width and depth of the viewing cone based on the width and height of the two-dimensional bounding box, combined with the intrinsic parameters of the camera and the prior information; Based on the estimated size, direction, height, width and depth of the viewing cone, the coordinates of the eight vertices of the viewing cone are calculated and determined. The eight vertices are respectively located at the top of the viewing cone, the four corner points of the bottom surface, and the far end and the near end of the viewing cone, so as to project the image data corresponding to the target object onto the three-dimensional point cloud data corresponding to the target object.

5. An automatic driving multimodal data updating and fusion device, characterized in that: include: a data acquisition module configured to respond to a multimodal data update fusion instruction and acquire point cloud data and image data corresponding to a target object from multiple sensors of the autonomous driving vehicle; an image data updating module, configured to input the image data corresponding to the target object into a preset AI large model, and perform feature extraction processing on the image data corresponding to the target object based on an image text retrieval method to update the image data category corresponding to the target object; A three-dimensional point cloud data determination module is configured to encode point cloud data corresponding to a target object into bird's-eye view features based on a standard sparse voxelization network; scale the bird's-eye view features based on a multi-scale center proposal network to determine the bird's-eye view features of different scales, and predict the position of the center of the target object; in the transformer and decoder of the transformer neural network model, self-attention layers and cross-attention layers are used to aggregate features from other centers and multi-scale feature maps; in a multi-frame mode, the bird's-eye view features of the previous frames are fused with the bird's-eye view features of the current frame in the center head and cross-attention layers; a spatially aware fusion module is used to calculate pixel-level attention weights based on the bird's-eye view features of the current frame, and time embedding is added to distinguish different frames. The three-dimensional point cloud data corresponding to the target object is determined based on the attention weights and the added time; A three-dimensional image data determination module is configured to input the updated image data corresponding to the target object into a convolutional neural network pre-trained in a single-stage monocular three-dimensional object detection model to extract target object features and construct a multi-scale feature pyramid; at each feature layer of the feature pyramid, a shared detection head is used to predict the object's category, attribute, center, and three-dimensional regression target; objects of different sizes are assigned to feature maps of different levels according to the two-dimensional size of the target object on the image, and positive and negative samples are determined according to the distance between the object center and the feature point; the prediction results of all levels of the feature pyramid are summarized, and overlapping frames are removed using a rotation target detection model; based on the predicted three-dimensional regression target and camera intrinsic parameters, the 2.5D center is converted into coordinates in three-dimensional space, and the three-dimensional bounding box of the target object is calculated, and the three-dimensional position and shape information corresponding to the target object are determined to determine the three-dimensional image data corresponding to the target object; The multimodal data fusion module is configured to divide the three-dimensional space of the target object into regular voxels, and convert the points in the three-dimensional space into voxel coordinates based on the voxel coordinate conversion algorithm; for each input three-dimensional point cloud data and three-dimensional image data corresponding to the target object, convert the points of the three-dimensional point cloud data and three-dimensional image data into their corresponding voxel coordinates, and fuse the voxel coordinates corresponding to the points of the converted point cloud data or image data with the voxel grid; when fusing new points, update the state of the voxel according to the distance and normal vector information of the point to perform data fusion of the three-dimensional point cloud data and three-dimensional image data corresponding to the target object, determine the shared three-dimensional environment representation corresponding to the three-dimensional point cloud data and three-dimensional image data, and complete the update and fusion of multimodal data for autonomous driving.

6. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 4 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images