Autonomous driving data fusion method, device, equipment and medium

Abstract

This application relates to a method, apparatus, device, and medium for autonomous driving data fusion. The method includes: acquiring two-dimensional image data, determining a cone based on target detection results and semantic segmentation results of a target object in the two-dimensional image data, performing target detection on lidar point cloud data within the cone, and determining three-dimensional point cloud data of the target object; fusing the target detection results and semantic segmentation results of the target object with the three-dimensional point cloud data corresponding to the target object based on an attention query matching mechanism to determine multimodal fusion data corresponding to the autonomous driving vehicle; inputting the two-dimensional image data into a trained multimodal large model to determine traffic scene logical description data; and transmitting the multimodal fusion data of the autonomous driving vehicle and the traffic scene logical description data to a preset database. This application can accurately and clearly determine the structure and layout of the traffic scene, achieving a more comprehensive scene understanding.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to an autonomous driving data fusion method, corresponding device, electronic device and computer-readable storage medium. Background Art

[0002] In recent years, due to the continuous increase in the number of car users, problems such as congestion and safety accidents faced by road traffic have become increasingly serious. Autonomous driving technology, with the collaboration of vehicle networking technology and artificial intelligence driving, can reasonably plan travel routes and time, thereby greatly improving travel efficiency and reducing energy consumption to a certain extent.

[0003] Currently, the processing of multimodal data for autonomous driving is limited by small models and lacks keyword query library updates and optimization. Small models require the annotation of large amounts of business data sets at the data level, which is tedious and time-consuming. Repeated development is required at the model level, resulting in low reuse rates and accuracy. In addition, the R&D cycle for business support is long and the versatility is low. This results in a relatively simple and inaccurate understanding and logical description of traffic scenarios. At the same time, the matching and fusion between 2D and 3D multimodal data is not accurate and precise.

[0004] To sum up, the existing technology requires labeling a large number of business data sets, which is tedious and time-consuming. The model level needs to be repeatedly developed, with low reuse rate and low precision. In addition, the R&D cycle in business support is long and the versatility is low, resulting in low overall accuracy in understanding and logical description of traffic scenes. At the same time, the matching and fusion between 2D and 3D multimodal data are not accurate and accurate. This application makes corresponding explorations to solve these problems. Summary of the Invention

[0005] The purpose of this application is to solve the above problems and provide an autonomous driving data fusion method, corresponding device, electronic device and computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] An autonomous driving data fusion method proposed to meet one of the purposes of this application includes the following steps:

[0008] In response to an autonomous driving data fusion event, obtain two-dimensional image data and lidar point cloud data from the autonomous driving vehicle, perform target detection and semantic segmentation on the two-dimensional image data, and determine a target detection result and a semantic segmentation result corresponding to the target object;

[0009] Determine a cone based on the target detection result and semantic segmentation result corresponding to the target object, perform target detection on the lidar point cloud data within the cone, and determine the three-dimensional point cloud data corresponding to the target object;

[0010] Based on the attention query matching mechanism, the target detection result and the semantic segmentation result corresponding to the target object are fused with the three-dimensional point cloud data corresponding to the target object to determine the multimodal fusion data corresponding to the autonomous driving vehicle;

[0011] Input the two-dimensional image data into the trained multimodal large model, update the keyword query library corresponding to the target object, and perform reasoning on the two-dimensional image data based on the keywords corresponding to the target object to determine the traffic scene logical description data corresponding to the autonomous driving vehicle;

[0012] The multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle are transmitted to a preset database to determine the autonomous driving data corresponding to the autonomous driving vehicle.

[0013] Optionally, after the step of obtaining the two-dimensional image data and the lidar point cloud data from the autonomous driving vehicle, the following steps are included:

[0014] In response to an image preprocessing instruction, preprocessing the two-dimensional image data;

[0015] The two-dimensional image data is resized, denoised, and enhanced.

[0016] Optionally, the step of fusing the target detection result and semantic segmentation result corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism includes the following steps:

[0017] Inputting the two-dimensional image data into a fully convolutional neural network to convert the two-dimensional image data into its corresponding feature vector;

[0018] Determine a query vector, a key vector, and a value vector corresponding to the two-dimensional image data according to the feature vector;

[0019] An attention score result and a weighted sum result are calculated based on the query vector, key vector, and value vector.

[0020] Optionally, the step of fusing the target detection result and semantic segmentation result corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism includes the following steps:

[0021] Performing target detection on the three-dimensional point cloud data based on a preset 3D target detection model to determine the position of the target object in the three-dimensional point cloud data;

[0022] Calculating an attention score between a current position of the target object in the two-dimensional image data and other positions of the target object in the three-dimensional point cloud data;

[0023] The attention scores are normalized so that the sum of the scores of all positions is equal to 1.

[0024] Optionally, the step of performing target detection and semantic segmentation on the two-dimensional image data to determine a target detection result and a semantic segmentation result corresponding to the target object includes:

[0025] Inputting multiple frames of the two-dimensional image data into a feature extraction network of a pre-trained target convolutional neural network model, and determining a feature map output by the feature extraction network;

[0026] Inputting the feature map into the pyramid structure of the target convolutional neural network model, and determining a pyramid feature map output by the pyramid structure;

[0027] The target is detected according to the pyramid feature map to obtain the multiple target detection results, and the pyramid feature map is input into the decoder of the target convolutional neural network model to determine the multiple semantic segmentation results output by the decoder, wherein the multiple semantic segmentation results are multiple segmentation mask maps.

[0028] Optionally, the step of performing target detection and semantic segmentation on the two-dimensional image data to determine a target detection result and a semantic segmentation result corresponding to the target object includes the following steps:

[0029] Fusing dynamic target objects and static target objects in the target detection results respectively to determine the target detection results, wherein the dynamic target objects include one or more of pedestrians, motor vehicles, or non-motor vehicles, and the static target objects include one or more of traffic signs or ground signs;

[0030] Accumulating the confidences of the segmentation categories corresponding to the pixel points at each position in the multiple segmentation mask images to obtain a total confidence value of one or more segmentation categories corresponding to each position;

[0031] Determine the target segmentation category of each position as the segmentation category corresponding to the maximum value of the total confidence value, and obtain the semantic segmentation result.

[0032] Optionally, after the step of transmitting the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle to a preset database, the following steps are included:

[0033] Responding to the vehicle's autonomous driving instructions, it obtains the corresponding multimodal fusion data of the autonomous driving vehicle and the traffic scene logic description data from the preset database;

[0034] Fitting the lane lines and lane surfaces corresponding to the autonomous driving vehicle according to the multimodal fusion data, and determining the fitted lane lines and lane surfaces;

[0035] The fitted lane lines, lane surfaces and traffic scene logic description data are integrated to determine the autonomous driving data corresponding to the autonomous driving vehicle, so as to drive the autonomous driving vehicle to perform autonomous driving.

[0036] An autonomous driving data fusion device provided for another purpose of the present application includes:

[0037] an image data processing module configured to respond to an autonomous driving data fusion event, obtain two-dimensional image data and lidar point cloud data from the autonomous driving vehicle, perform target detection and semantic segmentation on the two-dimensional image data, and determine a target detection result and a semantic segmentation result corresponding to the target object;

[0038] a point cloud data processing module configured to determine a cone based on a target detection result and a semantic segmentation result corresponding to the target object, perform target detection on the lidar point cloud data within the cone, and determine three-dimensional point cloud data corresponding to the target object;

[0039] a multimodal fusion data determination module configured to fuse the target detection results and semantic segmentation results corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on an attention query matching mechanism to determine multimodal fusion data corresponding to the autonomous driving vehicle;

[0040] a traffic scene data determination module configured to input the two-dimensional image data into a trained multimodal large model, update a keyword query library corresponding to the target object, and perform inference on the two-dimensional image data based on the keywords corresponding to the target object to determine traffic scene logical description data corresponding to the autonomous driving vehicle;

[0041] The autonomous driving data determination module is configured to transmit the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle to a preset database to determine the autonomous driving data corresponding to the autonomous driving vehicle.

[0042] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the autonomous driving data fusion method described in the present application.

[0043] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the autonomous driving data fusion method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0044] Compared with the existing technology, this application addresses the problems in the existing technology such as the need to label a large number of business data sets, which is cumbersome and time-consuming, the need for repeated development at the model level, low reuse rate, low accuracy, long R&D cycle in business support, low versatility, resulting in low overall accuracy and relatively simple understanding of traffic scenes and logical descriptions, as well as low precision and accuracy in matching and fusion of 2D and 3D multimodal data. This application proposes an autonomous driving data fusion method, including but not limited to the following beneficial effects:

[0045] First, the Transformer attention query matching mechanism is used to match and fuse 2D image data with 3D point cloud data. This allows for rapid matching and fusion of 2D and 3D point cloud data within the cone, generating richer, more detailed, and diverse large-scale datasets while ensuring accurate data processing.

[0046] Secondly, based on AI big model technology, it can realize the recognition, understanding and logical description of traffic scenes. Based on the constantly updated keyword query library, it can realize the identification and extraction of objects in traffic scenes, effectively capture semantic associations, and generate labels. The adaptability makes the model more flexible and applicable to different scenarios. It can accurately and clearly determine the structure and layout of traffic scenes and achieve a more comprehensive scene understanding.

[0047] Furthermore, this application greatly enriches the traffic understanding scenarios of autonomous driving by improving the precision and accuracy of multimodal data fusion of autonomous driving vehicles, thereby significantly improving the safety of autonomous driving, enabling further development of autonomous driving technology, and facilitating the further promotion and application of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 This is an exemplary architecture used in the autonomous driving data fusion method of this application;

[0050] Figure 2 This is a flow chart of the autonomous driving data fusion method in an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of a process for preprocessing two-dimensional image data in an embodiment of the present application;

[0052] Figure 4 Schematic diagram of the process of fusing target detection results and semantic segmentation results corresponding to the target object with three-dimensional point cloud data based on the attention query matching mechanism in an embodiment of the present application;

[0053] Figure 5 This is a flow chart of normalizing attention scores in an embodiment of the present application;

[0054] Figure 6 Schematic diagram of the process of determining target detection results and semantic segmentation results corresponding to the target object in an embodiment of the present application;

[0055] Figure 7 Schematic diagram of a process for fusing dynamic target objects and static target objects in target detection results to determine target detection results and semantic segmentation results in an embodiment of the present application;

[0056] Figure 8 This is a schematic diagram of the process of driving an autonomous driving vehicle to perform autonomous driving in an embodiment of the present application;

[0057] Figure 9 This is a principle block diagram of the autonomous driving data fusion device in an embodiment of the present application;

[0058] Figure 10 Schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0060] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0062] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0063] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0064] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0065] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0066] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0067] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0068] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0069] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0070] Based on the above example scenarios, please refer to Figure 1 as well as Figure 2 In one embodiment, the autonomous driving data fusion method of the present application includes the following steps:

[0071] Step S10: Responding to the autonomous driving data fusion event, obtaining two-dimensional image data and lidar point cloud data from the autonomous driving vehicle, performing target detection and semantic segmentation on the two-dimensional image data, and determining a target detection result and a semantic segmentation result corresponding to the target object;

[0072] The terminal device in the autonomous driving vehicle can respond to the autonomous driving data fusion event and obtain the two-dimensional image data and lidar point cloud data in the autonomous driving vehicle. The two-dimensional image data can be obtained from the camera or camera in the autonomous driving vehicle, and the lidar point cloud data can be obtained from the lidar in the autonomous driving vehicle. After obtaining the two-dimensional image data and lidar point cloud data in the autonomous driving vehicle, the two-dimensional image data can be subjected to target detection and semantic segmentation based on a preset target detection model to determine the target detection result and semantic segmentation result corresponding to the target object.

[0073] In some embodiments, multimodal data collected by the autonomous driving vehicle in the perception device or road-side perception device is obtained, and the multimodal data includes one or more of the following: lidar point cloud data, camera image data, or vehicle-mounted IMU data. The collected data is imported and preprocessed, including size unification, denoising, image enhancement, etc. of the image or video data. The selected pictures are imported into the code environment, and the pictures are read one by one through the index, and necessary preprocessing operations are performed. The preprocessing includes image resizing and image normalization to ensure that the image data meets the input (224*224) requirements of the model to ensure the quality and consistency of the data. Target detection of two-dimensional image data can be performed based on the Swin-Transformer module in the Transformer model. Semantic segmentation of two-dimensional image data can be performed based on the Cross-viewTransformers module in the Transformer model to determine the target detection results and semantic segmentation results corresponding to the target object, and obtain the target detection box or detection results such as the target category, quantity, size, etc. corresponding to the target object.

[0074] In some embodiments, multiple frames of the two-dimensional image data can be input into a feature extraction network of a pre-trained target convolutional neural network model to determine a feature map output by the feature extraction network; the feature map can be input into a pyramid structure of the target convolutional neural network model to determine a pyramid feature map output by the pyramid structure; the target is detected according to the pyramid feature map to obtain the multiple target detection results, and the pyramid feature map is input into a decoder of the target convolutional neural network model to determine multiple semantic segmentation results output by the decoder, wherein the multiple semantic segmentation results are multiple segmentation mask maps.

[0075] In some embodiments, dynamic target objects and static target objects in the target detection results are fused separately to determine the target detection results, wherein the dynamic target objects include one or any multiple of pedestrians, motor vehicles or non-motor vehicles, and the static target objects include one or any multiple of traffic signs or ground signs; the confidence of the segmentation category corresponding to the pixel point at each position in the multiple segmentation mask images is accumulated to obtain a total confidence value of one or more segmentation categories corresponding to each position; the target segmentation category of each position is determined to be the segmentation category corresponding to the maximum value in the total confidence value to obtain the semantic segmentation result.

[0076] Step S20: determining a cone based on the target detection result and the semantic segmentation result corresponding to the target object, performing target detection on the lidar point cloud data within the cone, and determining three-dimensional point cloud data corresponding to the target object;

[0077] A cone is determined based on the target detection result and semantic segmentation result corresponding to the target object, cone projection is performed on the lidar point cloud data, target detection is performed on the lidar point cloud data within the cone, and three-dimensional point cloud data corresponding to the target object is determined.

[0078] Specifically, after determining the target detection result and semantic segmentation result corresponding to the target object, the internal and external parameters of the two-dimensional image data and the lidar point cloud information are calibrated, and the target detection model is used to detect the bounding box of the target object in the two-dimensional image data. With the camera as the origin, a cone is projected along the direction of the bounding box. The lidar point cloud data is subjected to target detection within the cone to determine the three-dimensional point cloud data corresponding to the target object. Since the detected three-dimensional point cloud data is not necessarily in the exact center of the image, the axis of the generated cone does not necessarily coincide with the coordinate axis of the camera. In order to make the network have better rotation invariance, a rotation is performed so that the Z axis of the camera coincides with the axis of the cone.

[0079] Step S30: fusing the target detection result and semantic segmentation result corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism to determine multimodal fusion data corresponding to the autonomous driving vehicle;

[0080] After determining the three-dimensional point cloud data corresponding to the target object, the target detection result and semantic segmentation result corresponding to the target object in the two-dimensional image data are fused with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism. Based on the target detection result and semantic segmentation result corresponding to the target object in the two-dimensional image data, feature mapping, and the three-dimensional point cloud data corresponding to the target object, based on the Transformer attention query matching mechanism, through the Attention idea, the target detection result and semantic segmentation result corresponding to the target object in the two-dimensional image data are first input into the fully convolutional neural network FCN, and converted into its corresponding implicit representation or feature vector to obtain the query (Query), key (Key i ) and value (Value) are:

[0081] Query = W_q*X

[0082] Key i =W_k*X

[0083] Value=W_v*X

[0084] Where X represents the representation matrix of the input sequence, with each row corresponding to the feature vector of a position; W_q, W_k, and W_v represent learnable weight matrices, which are used to linearly transform the input sequence to obtain the corresponding query, key, and value; Q, K, and V represent the query vector, key vector, and value vector, respectively, which are used to calculate the attention score and weighted sum, that is:

[0085] Dot product: Similarity(Query,Key i )=Query·Key i

[0086]

[0087] MLP network: Similarity(Query,Key i )=MLP(Query,Key i )

[0088] In some embodiments, effective object detection and positioning can be performed on point cloud data based on the VoxelNet algorithm. The attention score between the current position of the target object in the two-dimensional image data and other positions of the target object in the three-dimensional point cloud data is calculated. The attention score is normalized so that the sum of the scores of all positions is equal to 1 to ensure the distribution of attention. The attention scoring mechanism and normalized attention matching are as follows:

[0089]

[0090]

[0091] Among them, L x Indicates the length of the input sentence.

[0092] The implicit representation of each position in the input sequence is weighted and summed according to its corresponding attention weight to obtain a weighted context vector. The output of the Transformer model reflects the part of the input sequence that should be paid attention to at the current position and determines the multimodal fusion data corresponding to the autonomous driving vehicle.

[0093]

[0094]

[0095] Step S40: Input the two-dimensional image data into the trained multimodal large model, update the keyword query library corresponding to the target object, and perform reasoning on the two-dimensional image data based on the keywords corresponding to the target object to determine the traffic scene logical description data corresponding to the autonomous driving vehicle;

[0096] Specifically, the multimodal large model includes the LLAVA large model, etc., establishes a keyword query library, continuously updates the vocabulary of the vocabulary library, and inputs image features into the LLAVA large model by setting query terms. The LLAVA large model searches for data in the keyword query library according to the input query terms. The pytorch function library is used to load and read the pre-trained LLAVA large model. After the LLAVA large model is loaded, the model is initialized by passing a set of prompts. The prompts represent a set of instructions for what information to query for the input image. According to the loaded model, the image data is understood and reasoned. By calling the model function and taking the pre-processed two-dimensional image data as input, the LLAVA large model will return the understanding result of the image, thereby capturing the dynamic changes and related information in the traffic scene, realizing the understanding of the scene, generating a logical description of the traffic scene, and determining the traffic scene logical description data corresponding to the autonomous driving vehicle.

[0097] In some embodiments, it is assumed that the loaded LLAVA large model is After initialization through prompts, it is F φ , the input image is recorded as X I , for X I Perform feature extraction:

[0098] X V =g(X I )

[0099] X V ∈RB×C×H×W

[0100]

[0101] Among them, B is the batch, which is set to 14; C is the number of channels, which is 3; H and W are the image height and width, both are 224, g(·) is the feature extraction, X V is the feature vector, as the input sequence.

[0102] Step S50: Transmit the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle to a preset database to determine the autonomous driving data corresponding to the autonomous driving vehicle.

[0103] After determining the traffic scene logic description data corresponding to the autonomous driving vehicle, the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle are transmitted to a preset database to determine the autonomous driving data corresponding to the autonomous driving vehicle.

[0104] In some embodiments, after determining the autonomous driving data corresponding to the autonomous driving vehicle, the terminal device of the autonomous driving vehicle can respond to the vehicle autonomous driving instructions, obtain the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle from a preset database; fit the lane lines and lane surfaces corresponding to the autonomous driving vehicle according to the multimodal fusion data, and determine the fitted lane lines and lane surfaces; fuse the fitted lane lines, lane surfaces and traffic scene logic description data to determine the autonomous driving data corresponding to the autonomous driving vehicle, so as to drive the autonomous driving vehicle to perform autonomous driving.

[0105] As can be seen from the above embodiments, compared with the prior art, the present application addresses the problems in the prior art such as the need to label a large number of business data sets, which is cumbersome and time-consuming, the need for repeated development at the model level, low reuse rate, low precision, long R&D cycle in business support, low versatility, resulting in low overall accuracy and relatively simple understanding of traffic scenes and logical descriptions, as well as low precision and accuracy in matching and fusion of 2D and 3D multimodal data. The present application proposes an autonomous driving data fusion method, including but not limited to the following beneficial effects:

[0106] First, the Transformer attention query matching mechanism is used to match and fuse 2D image data with 3D point cloud data. This allows for rapid matching and fusion of 2D and 3D point cloud data within the cone, generating richer, more detailed, and diverse large-scale datasets while ensuring accurate data processing.

[0107] Secondly, based on AI big model technology, it can realize the recognition, understanding and logical description of traffic scenes. Based on the constantly updated keyword query library, it can realize the identification and extraction of objects in traffic scenes, effectively capture semantic associations, and generate labels. The adaptability makes the model more flexible and applicable to different scenarios, which helps the structure and layout of traffic scenes and achieves a more comprehensive scene understanding.

[0108] Furthermore, this application greatly enriches the traffic understanding scenarios of autonomous driving by improving the precision and accuracy of multimodal data fusion of autonomous driving vehicles, thereby significantly improving the safety of autonomous driving, enabling further development of autonomous driving technology, and facilitating the further promotion and application of autonomous driving vehicles.

[0109] Based on any embodiment of this application, please refer to Figure 3 After obtaining the two-dimensional image data and lidar point cloud data from the autonomous driving vehicle, the following steps are included:

[0110] Step 100: responding to an image preprocessing instruction, preprocessing the two-dimensional image data;

[0111] Step 200: perform size normalization, denoising, and image enhancement on the two-dimensional image data.

[0112] Specifically, the system acquires multimodal data collected by the autonomous vehicle's sensing devices or roadside sensing devices. The multimodal data may include one or more of the following: lidar point cloud data, camera image data, or vehicle-mounted IMU data. The collected data may be imported and preprocessed, including resizing, denoising, and image enhancement of the image or video data. Selected images are imported into the code environment, and the images are read one by one through indexing, followed by necessary preprocessing operations. The preprocessing includes image resizing and image normalization to ensure that the image data complies with the model's input (224*224) requirements, thereby ensuring data quality and consistency.

[0113] It can be seen from the above embodiments that by performing size unification, denoising and image enhancement on the two-dimensional image data, the quality and consistency of the data can be guaranteed, and the target detection results and semantic segmentation results of the two-dimensional image data can be made more accurate, which is conducive to the fusion of multimodal data of autonomous driving vehicles.

[0114] Based on any embodiment of this application, please refer to Figure 4 The step of fusing the target detection result and the semantic segmentation result corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism includes the following steps:

[0115] Step S301: input the two-dimensional image data into a fully convolutional neural network, and convert the two-dimensional image data into its corresponding feature vector;

[0116] Step S303: determining a query vector, a key vector, and a value vector corresponding to the two-dimensional image data according to the feature vector;

[0117] Step S305: Calculate and determine the attention score result and the weighted sum result based on the query vector, key vector, and value vector.

[0118] Specifically, the target detection results and semantic segmentation results corresponding to the target objects in the two-dimensional image data are first input into the fully convolutional neural network FCN, and converted into their corresponding implicit representations or feature vectors to obtain the query (Query), key (Key i ) and value (Value) are:

[0119] Query = W_q*X

[0120] Key i =W_k*X

[0121] Value=W_v*X

[0122] Where X represents the representation matrix of the input sequence, with each row corresponding to the feature vector of a position; W_q, W_k, and W_v represent learnable weight matrices, which are used to linearly transform the input sequence to obtain the corresponding query, key, and value; Q, K, and V represent the query vector, key vector, and value vector, respectively, which are used to calculate the attention score and weighted summation result, that is:

[0123] Dot product: Similarity(Query,Key i )=Query·Key i

[0124] Cosine similarity:

[0125] MLP network: Similarity(Query,Key i )=MLP(Query,Key i )

[0126] Based on any embodiment of this application, please refer to Figure 5 The step of fusing the target detection result and the semantic segmentation result corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism includes the following steps:

[0127] Step S3001: performing target detection on the three-dimensional point cloud data based on a preset 3D target detection model to determine the position of the target object in the three-dimensional point cloud data;

[0128] Step S3003: Calculate an attention score between the current position of the target object in the two-dimensional image data and other positions of the target object in the three-dimensional point cloud data;

[0129] Step S3003: normalize the attention scores so that the sum of the scores of all positions is equal to 1.

[0130] Specifically, the VoxelNet algorithm can be used to effectively detect and locate objects in point cloud data. The attention score between the current position of the target object in the two-dimensional image data and other positions of the target object in the three-dimensional point cloud data is calculated. The attention score is normalized so that the sum of the scores of all positions is equal to 1 to ensure the distribution of attention. The attention scoring mechanism and normalized attention matching are as follows:

[0131]

[0132]

[0133] Among them, L x Indicates the length of the input sentence.

[0134] The implicit representation of each position in the input sequence is weighted and summed according to its corresponding attention weight to obtain a weighted context vector. The output of the Transformer model reflects the part of the input sequence that should be paid attention to at the current position and determines the multimodal fusion data corresponding to the autonomous driving vehicle.

[0135]

[0136]

[0137] Based on any embodiment of this application, please refer to Figure 6 The steps of performing target detection and semantic segmentation on the two-dimensional image data and determining target detection results and semantic segmentation results corresponding to the target object include:

[0138] Step S101: inputting multiple frames of the two-dimensional image data into a feature extraction network of a pre-trained target convolutional neural network model, and determining a feature map output by the feature extraction network;

[0139] Step S103: inputting the feature map into the pyramid structure of the target convolutional neural network model, and determining a pyramid feature map output by the pyramid structure;

[0140] Step S105: Detect the target according to the pyramid feature map to obtain the multiple target detection results, and at the same time input the pyramid feature map into the decoder of the target convolutional neural network model to determine the multiple semantic segmentation results output by the decoder, wherein the multiple semantic segmentation results are multiple segmentation mask maps.

[0141] Based on any embodiment of this application, please refer to Figure 7 , performing target detection and semantic segmentation on the two-dimensional image data, and determining target detection results and semantic segmentation results corresponding to the target object, comprising the following steps:

[0142] Step S1001: Fusing dynamic target objects and static target objects in the target detection results to determine the target detection results, wherein the dynamic target objects include one or more of pedestrians, motor vehicles, or non-motor vehicles, and the static target objects include one or more of traffic signs or ground signs;

[0143] Step S1003: Accumulate the confidence of the segmentation category corresponding to the pixel point at each position in the multiple segmentation mask images to obtain a total confidence value of one or more segmentation categories corresponding to each position;

[0144] Step S1005 : Determine that the target segmentation category of each position is the segmentation category corresponding to the maximum value of the total confidence value, and obtain the semantic segmentation result.

[0145] It can be seen from the above embodiments that based on the multi-task convolutional neural network, one branch detects motor vehicles, non-motor vehicles, pedestrians, road signs, and traffic signs, and the other branch performs lane line and road surface segmentation. Then, the target detection results and semantic segmentation results corresponding to the target objects in the two-dimensional image data are fused, including the fusion of dynamic and static target detection results and the fusion of semantic segmentation results. The traffic scene understanding results determined are richer. It not only simply identifies the lane lines in the road, but can also identify lane surface areas and distinguish different lanes. It combines motor vehicles, non-motor vehicles, and pedestrians to identify lane directions and distinguish non-motor vehicle lanes from motor vehicle lanes, which can provide accurate and rich traffic scene understanding information for autonomous driving vehicles.

[0146] Based on any embodiment of this application, please refer to Figure 8 After the step of transmitting the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle to a preset database, the following steps are included:

[0147] Step S501: Responding to the vehicle autonomous driving instruction, obtaining multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle from a preset database;

[0148] Step S503: fitting the lane lines and lane surfaces corresponding to the autonomous driving vehicle based on the multimodal fusion data to determine the fitted lane lines and lane surfaces;

[0149] Step S505: Fusing the fitted lane lines, lane surfaces, and traffic scene logic description data to determine the autonomous driving data corresponding to the autonomous driving vehicle, so as to drive the autonomous driving vehicle to perform autonomous driving.

[0150] Lane line fitting is performed based on the multimodal fusion data corresponding to the fused autonomous driving vehicle to obtain fitted lane lines and lane surfaces; traffic scene understanding is performed based on the fitted lane lines and lane surfaces and the traffic scene logic description data, and the autonomous driving data corresponding to the autonomous driving vehicle is determined to drive the autonomous driving vehicle to perform autonomous driving.

[0151] It can be seen from the above embodiments that this application greatly enriches the traffic understanding scenarios of autonomous driving by improving the precision and accuracy of multimodal data fusion of autonomous driving vehicles, thereby significantly improving the safety of autonomous driving, enabling the further development of autonomous driving technology, and facilitating the further promotion and application of autonomous driving vehicles.

[0152] See also Figure 9, an autonomous driving data fusion device provided to meet one of the purposes of the present application includes an image data processing module 1100, a point cloud data processing module 1200, a multimodal fusion data determination module 1300, a traffic scene data determination module 1400 and an autonomous driving data determination module 1500. Among them, the image data processing module 1100 is configured to respond to an autonomous driving data fusion event, obtain two-dimensional image data and lidar point cloud data in the autonomous driving vehicle, perform target detection and semantic segmentation on the two-dimensional image data, and determine the target detection result and semantic segmentation result corresponding to the target object; the point cloud data processing module 1200 is configured to determine a cone based on the target detection result and semantic segmentation result corresponding to the target object, perform target detection on the lidar point cloud data within the cone, and determine the three-dimensional point cloud data corresponding to the target object; the multimodal fusion data determination module 1300 is configured to perform target detection on the target object based on the attention query matching mechanism. The semantic segmentation result is fused with the three-dimensional point cloud data corresponding to the target object to determine the multimodal fusion data corresponding to the autonomous driving vehicle; the traffic scene data determination module 1400 is configured to input the two-dimensional image data into the trained multimodal large model, update the keyword query library corresponding to the target object, and infer the two-dimensional image data according to the keywords corresponding to the target object to determine the traffic scene logical description data corresponding to the autonomous driving vehicle; the autonomous driving data determination module 1500 is configured to transmit the multimodal fusion data and traffic scene logical description data corresponding to the autonomous driving vehicle to a preset database to determine the autonomous driving data corresponding to the autonomous driving vehicle.

[0153] Based on any embodiment of this application, please refer to Figure 10 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 10 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence, and when the computer-readable instructions are executed by the processor, the processor may implement an autonomous driving data fusion method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the autonomous driving data fusion method of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that, Figure 10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0154] In this embodiment, the processor is used to execute Figure 9 The memory stores the program code and various data required to execute the specific functions of each module and its submodule in the autonomous driving data fusion device. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the autonomous driving data fusion device of this application, and the server can call the server's program code and data to execute the functions of all submodules.

[0155] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the autonomous driving data fusion method described in any embodiment of the present application.

[0156] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the autonomous driving data fusion method described in any embodiment of the present application.

[0157] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0158] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0159] To sum up, this application greatly enriches the traffic understanding scenarios of autonomous driving by improving the precision and accuracy of multimodal data fusion of autonomous driving vehicles, thereby significantly improving the safety of autonomous driving, enabling further development of autonomous driving technology, and facilitating the further promotion and application of autonomous driving vehicles.

Claims

1. A method for fusion of autonomous driving data, characterized in that: The steps include: In response to an autonomous driving data fusion event, two-dimensional image data and lidar point cloud data from the autonomous driving vehicle are obtained, target detection and semantic segmentation are performed on the two-dimensional image data, and target detection results and semantic segmentation results corresponding to the target object are determined, which includes: Inputting multiple frames of the two-dimensional image data into a feature extraction network of a pre-trained target convolutional neural network model to determine a feature map output by the feature extraction network; inputting the feature map into a pyramid structure of the target convolutional neural network model to determine a pyramid feature map output by the pyramid structure; detecting a target based on the pyramid feature map to obtain multiple target detection results, and simultaneously inputting the pyramid feature map into a decoder of the target convolutional neural network model to determine multiple semantic segmentation results output by the decoder, wherein the multiple semantic segmentation results are multiple segmentation mask maps; The dynamic target objects and the static target objects in the target detection results are respectively fused to determine the target detection results, wherein the dynamic target objects include one or more of pedestrians, motor vehicles or non-motor vehicles, and the static target objects include one or more of traffic signs or ground signs; the confidence of the segmentation category corresponding to the pixel point at each position in the multiple segmentation mask images is accumulated to obtain a total confidence value of one or more segmentation categories corresponding to each position; the target segmentation category of each position is determined to be the segmentation category corresponding to the maximum value in the total confidence value, to obtain the semantic segmentation result; Determine a cone based on the target detection result and semantic segmentation result corresponding to the target object, perform target detection on the lidar point cloud data within the cone, and determine the three-dimensional point cloud data corresponding to the target object; Based on the attention query matching mechanism, the target detection result and the semantic segmentation result corresponding to the target object are fused with the three-dimensional point cloud data corresponding to the target object to determine the multimodal fusion data corresponding to the autonomous driving vehicle; Input the two-dimensional image data into the trained multimodal large model, update the keyword query library corresponding to the target object, and perform reasoning on the two-dimensional image data based on the keywords corresponding to the target object to determine the traffic scene logical description data corresponding to the autonomous driving vehicle; The multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle are transmitted to a preset database to determine the autonomous driving data corresponding to the autonomous driving vehicle.

2. The autonomous driving data fusion method according to claim 1, characterized in that: After obtaining the two-dimensional image data and lidar point cloud data from the autonomous driving vehicle, the following steps are included: In response to an image preprocessing instruction, preprocessing the two-dimensional image data; The two-dimensional image data is resized, denoised, and enhanced.

3. The autonomous driving data fusion method according to claim 1, characterized in that: The step of fusing the target detection result and the semantic segmentation result corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism includes the following steps: Inputting the two-dimensional image data into a fully convolutional neural network to convert the two-dimensional image data into its corresponding feature vector; Determine a query vector, a key vector, and a value vector corresponding to the two-dimensional image data according to the feature vector; An attention score result and a weighted sum result are calculated based on the query vector, key vector, and value vector.

4. The autonomous driving data fusion method according to claim 3, characterized in that: The step of fusing the target detection result and the semantic segmentation result corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on the attention query matching mechanism includes the following steps: Performing target detection on the three-dimensional point cloud data based on a preset 3D target detection model to determine the position of the target object in the three-dimensional point cloud data; Calculating an attention score between a current position of the target object in the two-dimensional image data and other positions of the target object in the three-dimensional point cloud data; The attention scores are normalized so that the sum of the scores of all positions is equal to 1.

5. The autonomous driving data fusion method according to any one of claims 1 to 4, characterized in that: After the step of transmitting the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle to a preset database, the following steps are included: Responding to the vehicle's autonomous driving instructions, it obtains the corresponding multimodal fusion data of the autonomous driving vehicle and the traffic scene logic description data from the preset database; Fitting the lane lines and lane surfaces corresponding to the autonomous driving vehicle according to the multimodal fusion data, and determining the fitted lane lines and lane surfaces; The fitted lane lines, lane surfaces and traffic scene logic description data are integrated to determine the autonomous driving data corresponding to the autonomous driving vehicle, so as to drive the autonomous driving vehicle to perform autonomous driving.

6. An autonomous driving data fusion device, characterized in that: include: An image data processing module is configured to respond to an autonomous driving data fusion event, obtain two-dimensional image data and lidar point cloud data from the autonomous driving vehicle, perform target detection and semantic segmentation on the two-dimensional image data, and determine target detection results and semantic segmentation results corresponding to the target object, including: Inputting multiple frames of the two-dimensional image data into a feature extraction network of a pre-trained target convolutional neural network model to determine a feature map output by the feature extraction network; inputting the feature map into a pyramid structure of the target convolutional neural network model to determine a pyramid feature map output by the pyramid structure; detecting a target based on the pyramid feature map to obtain multiple target detection results, and simultaneously inputting the pyramid feature map into a decoder of the target convolutional neural network model to determine multiple semantic segmentation results output by the decoder, wherein the multiple semantic segmentation results are multiple segmentation mask maps; The dynamic target objects and the static target objects in the target detection results are respectively fused to determine the target detection results, wherein the dynamic target objects include one or more of pedestrians, motor vehicles or non-motor vehicles, and the static target objects include one or more of traffic signs or ground signs; the confidence of the segmentation category corresponding to the pixel point at each position in the multiple segmentation mask images is accumulated to obtain a total confidence value of one or more segmentation categories corresponding to each position; the target segmentation category of each position is determined to be the segmentation category corresponding to the maximum value in the total confidence value, to obtain the semantic segmentation result; a point cloud data processing module configured to determine a cone based on a target detection result and a semantic segmentation result corresponding to the target object, perform target detection on the lidar point cloud data within the cone, and determine three-dimensional point cloud data corresponding to the target object; a multimodal fusion data determination module configured to fuse the target detection results and semantic segmentation results corresponding to the target object with the three-dimensional point cloud data corresponding to the target object based on an attention query matching mechanism to determine multimodal fusion data corresponding to the autonomous driving vehicle; a traffic scene data determination module configured to input the two-dimensional image data into a trained multimodal large model, update a keyword query library corresponding to the target object, and perform inference on the two-dimensional image data based on the keywords corresponding to the target object to determine traffic scene logical description data corresponding to the autonomous driving vehicle; The autonomous driving data determination module is configured to transmit the multimodal fusion data and traffic scene logic description data corresponding to the autonomous driving vehicle to a preset database to determine the autonomous driving data corresponding to the autonomous driving vehicle.

7. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 5 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images