Automatic driving scene representation vector generation method and device, model training method and computer readable storage medium

Abstract

The invention discloses an automatic driving scene representation vector generation method and device, a model training method and a computer readable storage medium, and the method comprises the steps: obtaining scene data of an automatic driving vehicle, and the scene data comprises image information, point cloud information and state information; processing the image information, the point cloud information and the state information to obtain a fusion feature vector, and inputting the fusion feature vector into a trained scene representation learning model to generate a representation vector of an automatic driving scene; wherein the scene representation learning model completes training in a multi-task learning mode, training tasks comprise a scene similarity task and a scene classification task, the training data set comprises a plurality of pieces of training data, and a single piece of training data is composed of labeled similar scene pairs. According to the method, the automatic driving scene is accurately understood, and the scene representation vector is better applied to scene mining applications such as scene classification, scene retrieval and extreme scene discovery.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving, and particularly to a method, device, and computer-readable storage medium for representing and learning an autonomous driving scene. Background Art

[0002] In the current field of autonomous driving technology, how to better understand the content of autonomous driving scenes for applications in scene mining applications such as scene classification, scene retrieval, and discovery of extreme scenes has become a key issue. The main technical problems are as follows:

[0003] 1. The content included in autonomous driving scenes is complex, including images and point cloud information captured by sensors, driving behavior information such as the throttle opening and steering wheel angle of the vehicle itself, and information such as the type and size of obstacles calculated by perception algorithms. How to extract important information related to it from complex scene information according to the needs of scene mining applications has become an important technical problem in this field;

[0004] 2. Understanding autonomous driving scenes itself is a multi-modal problem, and how to effectively fuse multi-modal information such as images and point clouds is another important technical problem;

[0005] 3. Understanding autonomous driving scenes itself is still a time series problem, and how to understand dynamic behaviors such as entry and exit of autonomous driving scenes from a time series containing multi-frame information is another important technical problem.

[0006] The current mainstream technical solution is to first assign multiple labels to a section of autonomous driving scene in a rule-based manner. The scope of the labels includes weather type, road type, driving behavior, etc., and then apply them to scene mining applications such as scene classification, scene retrieval, and discovery of extreme scenes based on label combinations. However, this label-based scene mining technical solution has the following disadvantages: corresponding operators need to be developed for each label, and it is cumbersome and time-consuming to complete the operator development for all labels; the operator strategies for each label are mostly rule-based, and there are problems of low recall rate and low accuracy for a single label; when applied to scene classification, scene retrieval, and discovery of extreme scene applications based on label combinations, there are problems of low recall rate and low accuracy. Summary of the Invention

[0007] The technical problem to be solved by the embodiments of the present invention is to provide a method, device, and computer-readable storage medium for representing and learning an autonomous driving scene to accurately understand an autonomous driving scene.

[0008] To solve the above technical problem, the present invention provides a method for generating a representation vector of an autonomous driving scene, including the following steps:

[0009] Obtain the scenario data of an autonomous driving vehicle, where the scenario data includes image information, point cloud information, and status information;

[0010] Process the image information, point cloud information, and status information to obtain a fused feature vector, and input it into a pre-trained scenario representation learning model to generate a representation vector of the autonomous driving scenario; among them, the scenario representation learning model is trained through a multi-task learning method, and the training tasks include a scenario similarity task and a scenario classification task. The training dataset includes multiple training data, and a single training data consists of labeled similar scenario pairs.

[0011] As can be seen from the above steps, the embodiment of the present invention realizes simple and efficient multi-modal information fusion. The feature vectors of images, point clouds, etc. are concatenated to obtain a fused feature vector, which is used as the input of a pre-trained scenario representation learning model; and the scenario representation learning model is trained through a multi-task learning method, which can make the model have stronger generalization ability, be able to better understand the similarity and category of the autonomous driving scenario, and adapt to a variety of different scenarios and tasks.

[0012] Preferably, the image information comes from the obstacle image information captured by a camera sensor, including a time series of multiple frames of images with the same structure, and each frame of image is a three-channel structure of red, green, and blue; the point cloud information comes from the obstacle spatial information captured by a lidar sensor, including a time series of multiple frames of point clouds with the same structure, and each frame of point cloud is a three-dimensional coordinate structure; the status information comes from the position and speed information calculated by the autonomous driving algorithm, including a time series of multiple frames of status vectors with the same structure.

[0013] Preferably, the processing of the image information, point cloud information, and status information to obtain a fused feature vector specifically includes:

[0014] Use the pre-trained scenario representation learning model to extract features from the image information and point cloud information respectively to obtain an image feature vector and a point cloud feature vector, and concatenate the image feature vector, the point cloud feature vector, and the status vector to obtain the fused feature vector.

[0015] Preferably, the scenario similarity task is the main task in multi-task learning, which is used to learn the similarity score of two given scenarios. The more similar the two scenarios are, the higher the similarity score of the two scenarios. The training samples of the scenario similarity task consist of similar scenario sample pairs and dissimilar scenario sample pairs; the scenario classification task is the secondary task in multi-task learning, which is used to learn the category of a given scenario. The training samples of the scenario classification task consist of scenario-scenario classification pairs.

[0016] Preferably, the annotation method for the similar scene pairs is a combination of automated annotation and manual annotation: The automated annotation method scores the similarity between the current scene and each candidate scene through a scene similarity large model, and filters out the scenes that meet the threshold and are the most similar as the similar scene pairs of the current scene according to the similarity score threshold; Manual annotation then manually reviews the similar scene pairs generated by the automated annotation method, and deletes the scene pairs that do not meet the manual review criteria from the training set.

[0017] The present invention also provides an automatic driving scene representation vector generation device, including:

[0018] An acquisition module, configured to acquire the scene data of an autonomous vehicle, where the scene data includes image information, point cloud information, and status information;

[0019] A scene representation learning model, configured to process the image information, point cloud information, and status information to obtain a fused feature vector, and generate a representation vector of the automatic driving scene according to the fused feature vector; Among them, the scene representation learning model is trained through a multi-task learning method, the training tasks include a scene similarity task and a scene classification task, and the training data set includes multiple pieces of training data, and a single piece of training data consists of labeled similar scene pairs.

[0020] The present invention also provides an automatic driving scene representation vector generation device, including:

[0021] One or more processors;

[0022] A memory;

[0023] One or more applications, where the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the automatic driving scene representation vector generation method described above.

[0024] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program; Among them, the computer program controls the device where the computer-readable storage medium is located to execute the automatic driving scene representation vector generation method when running.

[0025] The present invention also provides a model training method for training an automatic driving scene representation learning model, including the following steps:

[0026] Collect the scene data of an autonomous vehicle, where the scene data includes image information, point cloud information, and status information;

[0027] Build a training dataset for learning the representation of autonomous driving scenarios, where each piece of training data consists of labeled similar scenario pairs;

[0028] Construct a scenario representation learning model, where the input of the scenario representation learning model is a fused feature vector obtained by processing the image information, point cloud information, and state information, and the output is a representation vector of the autonomous driving scenario;

[0029] Train the constructed scenario representation learning model through multi-task learning, combining the scenario similarity task and the scenario classification task.

[0030] As can be seen from the above steps, the embodiment of the present invention realizes simple and efficient multi-modal information fusion. The image feature vector, point cloud feature vector, and state vector are concatenated to obtain a fused feature vector, which is used as the input of the trained scenario representation learning model; using labeled similar scenario pairs as the training dataset can effectively utilize the data and simplify the data annotation process at the same time; and training the scenario representation learning model through multi-task learning can make the model have stronger generalization ability, be able to better understand the similarity and category of autonomous driving scenarios, adapt to a variety of different scenarios and tasks, and provide strong support for the development and practical application of autonomous driving systems.

[0031] Preferably, the scenario data of the autonomous vehicle includes multiple pieces of driving scenario data corresponding to multiple vehicles respectively, and the single driving scenario data of the multiple pieces of driving scenario data indicates the scenario information when the corresponding vehicle is driving; wherein, the point cloud information comes from the obstacle space information captured by a lidar sensor, and the point cloud information of a single driving scenario is a point cloud time series containing multiple frames with the same structure, and each frame of point cloud is a three-dimensional coordinate structure; the image information comes from the obstacle image information captured by a camera sensor, and the image information of a single driving scenario is an image time series containing multiple frames with the same structure, and each frame of image is a three-channel structure of red, green, and blue; the state information comes from the position and speed information calculated by an autonomous driving algorithm and includes a state vector sequence containing multiple frames with the same structure.

[0032] Preferably, the processing of the image information, point cloud information, and state information to obtain a fused feature vector specifically includes:

[0033] Use the trained scenario representation learning model to extract features from the image information and point cloud information respectively to obtain an image feature vector and a point cloud feature vector, and concatenate the image feature vector, point cloud feature vector, and state vector to obtain the fused feature vector.

[0034] Preferably, in the training dataset, the annotation method for similar scene pairs is a combination of automated annotation and manual annotation: The automated annotation method scores the similarity between the current scene and each candidate scene through a scene similarity large model, and filters out the scenes that meet the threshold and are the most similar as the similar scene pairs of the current scene according to the similarity score threshold; Manual annotation then manually reviews the similar scene pairs generated by the automated annotation method, and deletes the scene pairs that do not meet the manual review criteria from the training set.

[0035] Preferably, the scene representation learning model includes a feature extraction layer, a feature fusion layer, a transformer layer, and a learning task layer. The feature extraction layer includes an image feature extraction layer and a point cloud feature extraction layer. The image feature extraction layer performs the same image feature extraction operation on each frame of image of each scene. Its input is a three-channel structure of red, green, and blue, and the output is an image feature vector; The point cloud feature extraction network performs the same point cloud feature extraction operation on each frame of point cloud of each scene. Its input is a three-dimensional coordinate structure, and the output is a point cloud feature vector; The feature fusion layer concatenates the image feature vector, point cloud feature vector, and state vector extracted by the feature extraction layer into a fused feature vector.

[0036] Preferably, the transformer layer takes the fused feature vector concatenated by the image feature vector, point cloud feature vector, and state vector as input, processes the order and position information in the time series data through position encoding, and assigns weights to the feature vectors at each time point through the self-attention mechanism to learn dynamic information from the time series scene.

[0037] Preferably, the learning task layer includes a scene similarity learning task layer and a scene classification task layer. The scene similarity learning task layer is a fully connected layer connected after the transformer layer, which is used to output the representation vector of the scene; The scene classification learning task layer is a fully connected layer plus a normalized exponential layer connected after the transformer layer, which is used to output the classification probability distribution of the scene.

[0038] Preferably, the scene similarity task is the main task in multi-task learning, which is used to learn the similarity score between two given scenes. The more similar the two scenes are, the higher the similarity score of the two scenes. The training samples of the scene similarity task consist of similar scene sample pairs and dissimilar scene sample pairs; The scene classification task is a secondary task in multi-task learning, which is used to learn the category of a given scene. The training samples of the scene classification task consist of scene-scene classification pairs.

[0039] The present invention also provides a model training device, including:

[0040] One or more processors;

[0041] Memory;

[0042] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the model training method described above.

[0043] The present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the model training method described above.

[0044] Implementing the present invention has the following beneficial effects: The embodiments of the present invention achieve simple and efficient multi-modal information fusion. Feature vectors such as images and point clouds are spliced to obtain a fused feature vector, which is used as the input of a trained scene representation learning model. Using labeled similar scene pairs as the training data set can effectively utilize the data and simplify the data annotation process. The scene representation learning model is trained through a multi-task learning method, which can make the model have stronger generalization ability, better understand the similarity and category of the autonomous driving scene, adapt to a variety of different scenes and tasks, and provide strong support for the development and practical application of the autonomous driving system. Description of the Drawings

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart of a method for learning the representation of an autonomous driving scene according to Embodiment 1 of the present invention.

[0047] Figure 2 It is a schematic flowchart of a model training method according to Embodiment 2 of the present invention.

[0048] Figure 3 It is a schematic diagram of the structure of an autonomous driving scene representation learning model in the embodiments of the present invention. Detailed Embodiments

[0049] The following descriptions of the embodiments are with reference to the drawings, which are used to illustrate specific embodiments in which the present invention can be implemented.

[0050] Please refer to Figure 1 As shown, Embodiment 1 of the present invention provides a method for generating an autonomous driving scene representation vector, including the following steps:

[0051] Obtain the scene data of an autonomous vehicle, where the scene data includes image information, point cloud information, and status information;

[0052] Process the image information, point cloud information, and status information to obtain a fused feature vector, and input it into a pre-trained scene representation learning model to generate a representation vector of the autonomous driving scene; among them, the scene representation learning model is trained through a multi-task learning method, and the training tasks include a scene similarity task and a scene classification task. The training dataset includes multiple pieces of training data, and a single piece of training data consists of labeled similar scene pairs.

[0053] From the above steps, it can be seen that the embodiment of the present invention realizes simple and efficient multi-modal information fusion. The feature vectors such as images and point clouds are concatenated to obtain a fused feature vector, which is used as the input of the pre-trained scene representation learning model; and the scene representation learning model is trained through a multi-task learning method, which can make the model have stronger generalization ability, be able to better understand the similarity and category of the autonomous driving scene, and adapt to a variety of different scenes and tasks.

[0054] Specifically, the scene data of the autonomous vehicle is used to indicate the scene information when the autonomous vehicle is driving. Among them, the image information comes from the obstacle image information captured by the camera sensor, including a time series of multiple frames of images with the same structure, and each frame of image is a three-channel RGB structure; the point cloud information comes from the obstacle space information captured by the lidar sensor, including a time series of multiple frames of point clouds with the same structure, and each frame of point cloud is a three-dimensional coordinate structure; the status information comes from the information such as position and speed calculated by the autonomous driving algorithm, including a time series of multiple frames of status vectors with the same structure.

[0055] The scene representation learning model includes a feature extraction layer, a feature fusion layer, a transformer layer, and a learning task layer. The input of this scene representation learning model is the aforementioned fused feature vector, and the output is the representation vector of the autonomous driving scene. The feature extraction layer includes an image feature extraction layer and a point cloud feature extraction layer. The image feature extraction layer performs the same image feature extraction operation on each frame of the image of the obtained scene. Its input is a three-channel RGB structure, and the output is an image feature vector; the point cloud feature extraction layer performs the same point cloud feature extraction operation on each frame of the point cloud of the obtained scene. Its input is a three-dimensional coordinate structure, and the output is a point cloud feature vector. The feature fusion layer concatenates the image feature vector, point cloud feature vector, and status vector extracted by the feature extraction layer for each frame into a fused feature vector.

[0056] The transformer layer receives the fused feature vector concatenated by the image feature vector, the point cloud feature vector, and the state vector as input. Through its internal self-attention mechanism, it can capture the internal correlations and dependencies among these feature vectors, enabling the scene representation learning model to better understand and utilize these multi-modal features and provide a more advanced and abstract feature representation for the subsequent learning task layer. In addition, due to the characteristics of the transformer structure, it can handle variable-length sequences, which is very beneficial for the processing of time series data in the autonomous driving scenario.

[0057] The learning task layer includes a scene similarity learning task layer and a scene classification task layer. The scene similarity learning task layer is a fully connected layer connected after the transformer layer, used to output the representation vector of the scene; the scene classification learning task layer is a fully connected layer plus a softmax layer connected after the transformer layer, used to output the classification probability distribution of the scene.

[0058] The training tasks of the scene representation learning model include a scene similarity task and a scene classification task. The scene similarity task is the main task in the multi-task learning, used to learn the similarity score of two given scenes. The more similar the two scenes are, the higher the similarity score of the two scenes. The scene similarity task uses a triplet loss function, and the corresponding training samples are composed of similar scene sample pairs and dissimilar scene sample pairs; the scene classification task is the secondary task in the multi-task learning, learning the category of the given scene, used to assist the scene similarity task learning. The scene classification task uses a cross-entropy loss function, and the corresponding training samples are composed of scene-scene classification pairs.

[0059] In the training dataset of the scene representation learning model, a single training data is composed of labeled similar scene pairs. The labeling method of the similar scene pairs is a combination of automatic labeling and manual labeling: the automatic labeling method scores the similarity between the current scene and each candidate scene through a scene similarity large model, and filters out the scene that meets the threshold and is the most similar as the similar scene pair of the current scene according to the similarity score threshold; the manual labeling then manually reviews the similar scene pairs generated by the automatic labeling method and deletes the scene pairs that do not meet the manual review criteria from the training set.

[0060] The representation vector of the autonomous driving scene generated by the embodiments of the present invention can be well applied to data mining applications such as scene classification, scene retrieval, and extreme scene discovery. Experiments show that the recall rate and accuracy in these three applications are effectively improved.

[0061] Please refer to Figure 2 As shown in the figure, Embodiment 2 of the present invention provides a model training method for training an autonomous driving scene representation learning model, including the following steps:

[0062] Collect the scenario data of an autonomous driving vehicle, where the scenario data includes image information, point cloud information, and status information;

[0063] Establish a training data set for autonomous driving scenario representation learning, where each piece of training data consists of labeled similar scenario pairs;

[0064] Construct a scenario representation learning model, where the input of the scenario representation learning model is a fused feature vector obtained by processing the image information, point cloud information, and status information, and the output is a representation vector of the autonomous driving scenario;

[0065] Train the constructed scenario representation learning model through a multi-task learning approach, combining the scenario similarity task and the scenario classification task.

[0066] As can be seen from the above steps, the embodiment of the present invention realizes simple and efficient multi-modal information fusion. The image feature vector, point cloud feature vector, and status vector are concatenated to obtain a fused feature vector, which is used as the input of the trained scenario representation learning model; using labeled similar scenario pairs as the training data set can effectively utilize the data and simplify the data annotation process at the same time; and training the scenario representation learning model through a multi-task learning approach can make the model have stronger generalization ability, be able to better understand the similarity and category of the autonomous driving scenario, adapt to a variety of different scenarios and tasks, and provide strong support for the development and practical application of the autonomous driving system.

[0067] Specifically, the scenario data of the collected autonomous driving vehicle includes multiple driving scenario data corresponding to multiple vehicles respectively, and the single driving scenario data of the multiple driving scenario data indicates the scenario information when the corresponding vehicle is driving. Among them, the point cloud information comes from the obstacle space information captured by the lidar sensor, and the point cloud information of a single driving scenario is a point cloud time series containing multiple frames with the same structure, and each frame of point cloud is a three-dimensional coordinate structure. The image information comes from the obstacle image information captured by the camera sensor; the image information of a single driving scenario is an image time series containing multiple frames with the same structure, and each frame of image is a three-channel structure of red, green, and blue. The status information comes from the information such as position and speed calculated by the autonomous driving algorithm, and the status data in the scenario data is a status vector sequence containing multiple frames with the same structure.

[0068] In the training dataset of the scene representation learning model, the annotation method for similar scene pairs is a combination of automated annotation and manual annotation: The automated annotation method scores the similarity between the current scene and each candidate scene through a large scene similarity model, and filters out the scenes that meet the threshold and are the most similar as the similar scene pairs of the current scene according to the similarity score threshold; Manual annotation then manually reviews the similar scene pairs generated by the automated annotation method, and deletes the scene pairs that do not meet the manual review criteria from the training set.

[0069] Please combine Figure 3 As shown, the scene representation learning model includes a feature extraction layer, a feature fusion layer, a transformer layer, and a learning task layer. The input of this scene representation learning model is the aforementioned fused feature vector, and the output is the representation vector of the autonomous driving scene. The feature extraction layer includes an image feature extraction layer and a point cloud feature extraction layer. The image feature extraction layer performs the same image feature extraction operation on each frame of image of each scene. Its input is a three-channel RGB structure, and the output is an image feature vector; The point cloud feature extraction network performs the same point cloud feature extraction operation on each frame of point cloud of each scene. Its input is a three-dimensional coordinate structure, and the output is a point cloud feature vector. The feature fusion layer concatenates each frame of image feature vector, point cloud feature vector, and state vector extracted by the feature extraction layer into a fused feature vector.

[0070] The transformer layer receives the fused feature vector, which is concatenated by the image feature vector, the point cloud feature vector, and the state vector, as input. Through its internal self-attention mechanism, it can capture the internal correlations and dependencies between these feature vectors, enabling the scene representation learning model to better understand and utilize these multimodal features and provide a more advanced and abstract feature representation for the subsequent learning task layer. In addition, due to the characteristics of the transformer structure, it can handle variable-length sequences, which is very beneficial for the processing of time series data in the autonomous driving scenario. The fused feature vector fuses the multimodal feature vectors at different time points to form a continuous time series feature sequence. The fused feature vector at each time point synthesizes the image, point cloud, and state information to represent the scene at that time point more comprehensively. The transformer layer uses positional encoding to ensure that the model can consider the order and position information in the time series data. Since the fused feature vector contains information from multiple modalities, positional encoding can help the model distinguish the features at different time points and capture the temporal dependencies between them. Through the self-attention mechanism, the transformer layer can capture more complex dynamic changes and internal correlations in the time series data. The fused feature vector at each time point may contain important information from different modalities, and the self-attention mechanism can adaptively assign different weights to each time point to focus on the features that are more critical for the current task, thus enabling the full learning and utilization of the dynamic information in the time series scene. This processing method helps the model better understand and grasp the evolution process of the scene, thereby improving the performance of subsequent tasks.

[0071] The learning task layer includes a scene similarity learning task layer and a scene classification task layer. The scene similarity learning task layer is a fully connected layer connected after the transformer layer, which is used to output the representation vector of the scene; the scene classification learning task layer is a fully connected layer plus a softmax layer connected after the transformer layer, which is used to output the classification probability distribution of the scene.

[0072] Through the above design of the transformer layer and the learning task layer, the scene representation learning model can output the representation vector of the scene and the classification probability distribution of the scene, making the results of the model have a certain interpretability.

[0073] In this embodiment, a scene representation learning model is trained and constructed through a multi-task learning method, combining a scene similarity task and a scene classification task. The scene similarity task is the main task in the multi-task learning, which is used to learn the similarity score of two given scenes. The more similar the two scenes are, the higher the similarity score of the two scenes. The scene similarity task uses a triplet loss function, and the corresponding training samples are composed of similar scene sample pairs and dissimilar scene sample pairs. The scene classification task is the secondary task in the multi-task learning, which learns the category of a given scene and is used to assist the scene similarity task in learning. The scene classification task uses a cross-entropy loss function, and the corresponding training samples are composed of scene-scene classification pairs.

[0074] Through the above steps, a high-quality autonomous driving scene representation model is trained. In actual application, the image information, point cloud information, and state information in the scene data of the autonomous driving vehicle are extracted for features and concatenated into a fused feature vector, which is input into the scene representation learning model, and then the representation vector of the autonomous driving scene can be generated and applied to data mining applications such as scene classification, scene retrieval, and extreme scene discovery.

[0075] It should be noted that the embodiment of the present invention is an end-to-end technology, covering the entire process from raw data input to final task output without additional intermediate processing or human intervention. For example, the embodiment of the present invention directly extracts features from the original image, point cloud, and state data and finally outputs the representation vector or classification result of the scene without cumbersome intermediate processing. The embodiment of the present invention integrates multiple modules, such as feature extraction, transformer layer, task learning layer, etc. These modules work together to jointly achieve end-to-end learning and inference. In the end-to-end framework, the entire system can be optimized and adjusted uniformly instead of optimizing each component separately, thereby improving the overall performance. In a complex scenario such as autonomous driving, the end-to-end technical solution helps to reduce error accumulation, improve system efficiency, and overall performance.

[0076] Corresponding to the autonomous driving scene representation vector generation method described in the first embodiment of the present invention, the third embodiment of the present invention further provides an autonomous driving scene representation vector generation device, including:

[0077] An acquisition module, configured to acquire the scene data of the autonomous driving vehicle, where the scene data includes image information, point cloud information, and state information;

[0078] A scene representation learning model is used to process the image information, point cloud information, and status information to obtain a fused feature vector, and generate a representation vector of the autonomous driving scene according to the fused feature vector. Among them, the scene representation learning model is trained through a multi-task learning method, and the training tasks include a scene similarity task and a scene classification task. The training dataset includes multiple pieces of training data, and a single piece of training data consists of labeled similar scene pairs.

[0079] Corresponding to the autonomous driving scene representation vector generation method described in the foregoing Embodiment 1 of the present invention, Embodiment 4 of the present invention further provides an autonomous driving scene representation vector generation device, including:

[0080] One or more processors;

[0081] A memory;

[0082] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the autonomous driving scene representation vector generation method described in Embodiment 1 of the present invention.

[0083] Corresponding to the autonomous driving scene representation vector generation method described in the foregoing Embodiment 1 of the present invention, Embodiment 5 of the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program. Among them, the computer program controls the device where the computer-readable storage medium is located to execute the autonomous driving scene representation vector generation method when running.

[0084] Corresponding to the model training method described in the foregoing Embodiment 2 of the present invention, Embodiment 6 of the present invention further provides a model training device, including:

[0085] One or more processors;

[0086] A memory;

[0087] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the model training method described in Embodiment 2 of the present invention.

[0088] Corresponding to the model training method described in the foregoing Embodiment 2 of the present invention, Embodiment 7 of the present invention further provides a computer-readable storage medium, and the computer-readable storage medium includes a stored computer program. Among them, the computer program controls the device where the computer-readable storage medium is located to execute the model training method described in Embodiment 2 of the present invention when running.

[0089] Preferably, the computer program may be divided into one or more modules / units (such as computer program 1, computer program 2, ……), and the one or more modules / units are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the device.

[0090] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor. The processor is the control center of the device and connects various parts of the device through various interfaces and circuits.

[0091] The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory may be a high-speed random access memory, or may also be a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., or the memory may also be other volatile solid-state storage devices.

[0092] It should be noted that the above device may include but is not limited to a processor and a memory, which can be understood by those skilled in the art.

[0093] For the working principles and processes of the above embodiments, please refer to the descriptions of Embodiment 1 and Embodiment 2 of the present invention respectively above, and details are not described herein again.

[0094] As can be seen from the above description, compared with the prior art, the beneficial effects of the present invention are as follows: The embodiments of the present invention achieve simple and efficient multi-modal information fusion, splice feature vectors such as images and point clouds to obtain a fused feature vector, and use it as the input of a trained scene representation learning model; use labeled similar scene pairs as the training data set, which can effectively utilize the data and simplify the data annotation process; while the scene representation learning model is trained through a multi-task learning method, which can make the model have stronger generalization ability, better understand the similarity and categories of the autonomous driving scene, adapt to a variety of different scenes and tasks, and provide strong support for the development and practical application of the autonomous driving system.

[0095] The above-disclosed are only the preferred embodiments of the present invention. Of course, the scope of the rights of the present invention cannot be limited thereby. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope covered by the present invention.

Claims

1. A method for generating an autonomous driving scenario representation vector, characterized in that, Including the following steps: Obtain the scenario data of the autonomous vehicle, where the scenario data includes image information, point cloud information, and status information; Process the image information, point cloud information, and status information to obtain a fused feature vector, and input it into a pre-trained scenario representation learning model to generate a representation vector of the autonomous driving scenario; among them, the scenario representation learning model is trained through a multi-task learning method, and the training tasks include a scenario similarity task and a scenario classification task. The training dataset includes multiple pieces of training data, and a single piece of training data consists of labeled similar scenario pairs.

2. The method according to claim 1, wherein The image information comes from the obstacle image information captured by the camera sensor, including a time series of multiple frames of images with the same structure, and each frame of image is a three-channel structure of red, green, and blue; the point cloud information comes from the obstacle space information captured by the lidar sensor, including a time series of multiple frames of point clouds with the same structure, and each frame of point cloud is a three-dimensional coordinate structure; the status information comes from the position and speed information calculated by the autonomous driving algorithm, including a time series of multiple frames of state vector sequences with the same structure.

3. The method according to claim 2, characterized in that, The processing of the image information, point cloud information, and status information to obtain a fused feature vector specifically includes: Use the pre-trained scenario representation learning model to extract features from the image information and point cloud information respectively to obtain an image feature vector and a point cloud feature vector, and splice the image feature vector, point cloud feature vector, and state vector to obtain the fused feature vector.

4. The method according to claim 1, characterized in that The scenario similarity task is the main task in multi-task learning, which is used to learn the similarity score of two given scenarios. The more similar the two scenarios are, the higher the similarity score of the two scenarios. The training samples of the scenario similarity task consist of similar scenario sample pairs and dissimilar scenario sample pairs; the scenario classification task is the secondary task in multi-task learning, which is used to learn the category of a given scenario. The training samples of the scenario classification task consist of scenario-scenario classification pairs.

5. The method according to claim 1, wherein The annotation method for the similar scenario pairs is a combination of automatic annotation and manual annotation: the automatic annotation method scores the similarity between the current scenario and each candidate scenario through a scenario similarity large model, and filters out the scenario that meets the threshold and is the most similar as the similar scenario pair of the current scenario according to the similarity score threshold; Manual annotation then manually reviews the similar scenario pairs generated by the automatic annotation method, and deletes the scenario pairs that do not meet the manual review criteria from the training set.

6. An automatic driving scenario representation vector generation device, characterized in that Including: An acquisition module for acquiring the scenario data of the autonomous vehicle, where the scenario data includes image information, point cloud information, and status information; A scenario representation learning model for processing the image information, point cloud information, and status information to obtain a fused feature vector, and generating a representation vector of the autonomous driving scenario according to the fused feature vector; among them, the scenario representation learning model is trained through a multi-task learning method, and the training tasks include a scenario similarity task and a scenario classification task. The training dataset includes multiple pieces of training data, and a single piece of training data consists of labeled similar scenario pairs.

7. An apparatus for generating an autonomous driving scenario representation vector, characterized in that, Including: One or more processors; A memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method for generating an autonomous driving scenario representation vector according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the method for generating an autonomous driving scenario representation vector according to any one of claims 1 to 5.

9. A model training method for training an autonomous driving scene representation learning model, characterized in that, Comprising the following steps: Collect the scenario data of the autonomous driving vehicle, where the scenario data includes image information, point cloud information, and status information; Establish a training data set for autonomous driving scenario representation learning, where each piece of training data consists of labeled similar scenario pairs; Construct a scenario representation learning model, where the input of the scenario representation learning model is a fused feature vector obtained by processing the image information, point cloud information, and status information, and the output is a representation vector of the autonomous driving scenario; Train the constructed scenario representation learning model through a multi-task learning method, combining a scenario similarity task and a scenario classification task.

10. The method according to claim 9, wherein The scenario data of the autonomous driving vehicle includes multiple driving scenario data corresponding to multiple vehicles respectively, and the single driving scenario data of the multiple driving scenario data indicates the scenario information of the corresponding vehicle during driving; wherein, the point cloud information comes from the obstacle space information captured by a lidar sensor, and the point cloud information of a single driving scenario is a point cloud time series containing multiple frames of the same structure, and each frame of point cloud is a three-dimensional coordinate structure; the image information comes from the obstacle image information captured by a camera sensor, and the image information of a single driving scenario is an image time series containing multiple frames of the same structure, and each frame of image is a three-channel structure of red, green, and blue; the status information comes from the position and speed information calculated by the autonomous driving algorithm, and includes a status vector sequence containing multiple frames of the same structure.

11. The method according to claim 10, wherein The processing of the image information, point cloud information, and status information to obtain a fused feature vector specifically includes: Use the trained scenario representation learning model to extract features from the image information and point cloud information respectively to obtain an image feature vector and a point cloud feature vector, and splice the image feature vector, the point cloud feature vector, and the status vector to obtain the fused feature vector.

12. The method according to claim 9, wherein In the training data set, the annotation method for similar scenario pairs is a combination of automatic annotation and manual annotation: the automatic annotation method scores the similarity between the current scenario and each candidate scenario through a scenario similarity large model, and filters out the scenarios that meet the threshold and are the most similar as the similar scenario pairs of the current scenario according to the similarity score threshold; Manual annotation then manually reviews the similar scenario pairs generated by the automatic annotation method, and deletes the scenario pairs that do not meet the manual review criteria from the training set.

13. The method according to claim 10, wherein The described scene representation learning model includes a feature extraction layer, a feature fusion layer, a Transformer layer, and a learning task layer. The feature extraction layer includes an image feature extraction layer and a point cloud feature extraction layer. The image feature extraction layer performs the same image feature extraction operation on each frame of the image in each scene. Its input is a three-channel RGB structure, and the output is an image feature vector. The point cloud feature extraction network performs the same point cloud feature extraction operation on each frame of the point cloud in each scene. Its input is a three-dimensional coordinate structure, and the output is a point cloud feature vector. The feature fusion layer concatenates the image feature vector, point cloud feature vector, and state vector extracted by the feature extraction layer into a fused feature vector.

14. The method according to claim 13, wherein, The described Transformer layer takes the fused feature vector concatenated by the image feature vector, point cloud feature vector, and state vector as input, processes the order and position information in the time series data through position encoding, and assigns weights to the feature vectors at each time point through the self-attention mechanism to learn dynamic information from the time series scene.

15. The method according to claim 13, characterized in that The described learning task layer includes a scene similarity learning task layer and a scene classification task layer. The scene similarity learning task layer is a fully connected layer connected after the Transformer layer, which is used to output the representation vector of the scene. The scene classification learning task layer is a fully connected layer plus a softmax layer connected after the Transformer layer, which is used to output the classification probability distribution of the scene.

16. The method according to claim 9, wherein The described scene similarity task is the main task in multi-task learning, which is used to learn the similarity score of two given scenes. The more similar the two scenes are, the higher the similarity score of the two scenes. The training samples of the scene similarity task consist of similar scene sample pairs and dissimilar scene sample pairs. The scene classification task is a secondary task in multi-task learning, which is used to learn the category of a given scene. The training samples of the scene classification task consist of scene-scene classification pairs.

17. A model training device, characterized in that, Comprising: One or more processors; A memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors. The one or more applications are configured to execute the model training method according to any one of claims 9 to 16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program. Wherein, the computer program controls the device where the computer-readable storage medium is located to execute the model training method according to any one of claims 9 to 16 when running.

Images