Auxiliary driving method and device, electronic equipment and storage medium

Abstract

The invention discloses an auxiliary driving method and device, electronic equipment and a storage medium, and relates to the technical field of intelligent driving, and the method comprises the steps: obtaining to-be-processed driving data containing a plurality of modes, and compared with a single mode, the method can more comprehensively describe a high-speed driving scene, and provides rich information for decision making; according to the method, multi-modal driving data to be processed are subjected to representation conversion to obtain a first feature vector, a second feature vector and a third feature vector, different feature vectors can be used for different analysis tasks, and the detailed division is beneficial to realizing multi-level decision making; the behavior, position and time sequence information of a target object are obtained through multiple tasks such as mask processing, object positioning and time sequence prediction, the analysis dimension is refined, and the defect that a general multi-mode large model lacks a special processing mechanism is overcome; finally, multi-dimensional information is integrated to output an auxiliary driving strategy, multiple factors are comprehensively considered, a more reasonable decision is made, and the problems that a general model is insufficient in information integration and difficult to adapt to complex tasks are solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to an assisted driving method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of intelligent driving technology, assisted driving systems have played an important role in improving driving safety and reducing traffic accidents. The role of assisted driving systems is particularly significant in complex scenarios such as highways.

[0003] Related technologies, based on large multimodal models, directly interpret scenes end-to-end, taking video as input and outputting driving recommendations directly through the model. However, in highway scenarios, complex tasks such as vehicle interactions and the timing of lane changes place higher demands on assisted driving systems. Existing general-purpose models are generally unable to meet these multi-layered decision-making requirements. Summary of the Invention

[0004] The present application provides an assisted driving method, device, electronic device and storage medium to at least solve the problem in the related art that the general multimodal large model cannot meet the multi-level decision-making requirements of complex tasks in highway scenarios.

[0005] The present application provides an assisted driving method, including: obtaining driving data to be processed, the driving data to be processed including data of multiple modalities; performing characterization conversion on the driving data to be processed to obtain a first eigenvector, a second eigenvector, and a third eigenvector; performing mask processing based on the first eigenvector to determine a target object corresponding to the first eigenvector, and performing action judgment on the target object to obtain behavior information of the target object; performing object positioning based on the second eigenvector to obtain position information of the target object; performing time series prediction based on the third eigenvector to obtain time series information of the target object; and integrating the behavior information, position information, and time series information to output an assisted driving strategy.

[0006] The present application also provides a driving assistance device, comprising:

[0007] A data acquisition module is used to acquire driving data to be processed, where the driving data to be processed includes data of multiple modalities;

[0008] a data representation conversion module, configured to perform representation conversion on the driving data to be processed to obtain a first eigenvector, a second eigenvector, and a third eigenvector;

[0009] The multi-task head module is configured to perform mask processing based on the first eigenvector to determine the target object corresponding to the first eigenvector, and perform action judgment on the target object to obtain the target object's behavior information; perform object positioning based on the second eigenvector to obtain the target object's position information; and perform time series prediction based on the third eigenvector to obtain the target object's time series information;

[0010] The information integration module is used to integrate behavior information, location information and timing information and output assisted driving strategies.

[0011] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned assisted driving methods when executing the computer program.

[0012] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned assisted driving methods are implemented.

[0013] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned assisted driving methods when executed by a processor.

[0014] This application acquires unprocessed driving data containing multiple modalities, which can more comprehensively describe the driving scene than a single modality and provide rich information for decision-making. The acquired unprocessed driving data undergoes characterization transformation to obtain first, second, and third eigenvectors. Different eigenvectors can be used for different analysis tasks, and this meticulous division facilitates multi-level decision-making. Masking based on the first eigenvector accurately identifies the target object, which is then used to determine its behavior and obtain its behavioral information. Object localization using the second eigenvector accurately determines the target object's position. Time series prediction based on the third eigenvector obtains the target object's timing information, facilitating early decision-making. Behavior, position, and timing information are integrated to output an assisted driving strategy. This multi-factor decision-making approach can meet the multi-level decision-making requirements of complex tasks in highway scenarios. Refining the analysis dimensions compensates for the lack of specialized processing mechanisms in general multimodal large models. Finally, integrating the multi-dimensional information to output an assisted driving strategy, comprehensively considering multiple factors, enables more rational decision-making, overcoming the limitations of general models in information integration and adaptability to complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flowchart of an assisted driving method provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of the architecture of a multi-task head multi-modal large model provided in an embodiment of the present application;

[0018] Figure 3 A data processing flow chart of a positioning task head provided in an embodiment of the present application;

[0019] Figure 4 A data processing flow chart of an action mask task head provided in an embodiment of the present application;

[0020] Figure 5 A data processing flow chart of a time series prediction task header provided in an embodiment of the present application;

[0021] Figure 6 A schematic diagram of the structure of a driving assistance device provided in an embodiment of the present application;

[0022] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0025] In order to more clearly illustrate the embodiments of the present application, the following briefly introduces the technical terms used in the embodiments:

[0026] Multi-task-heads refer to multiple specialized sub-modules connected to the end of the model body in machine learning models, especially multimodal large models or other deep learning architectures. Each sub-module corresponds to a specific task, aiming to enable the model to process and execute multiple different types of tasks simultaneously.

[0027] A multimodal big model is an artificial intelligence model that can process and fuse data from multiple different modalities, such as images, text, audio, video, etc., to achieve more powerful perception, understanding and generation capabilities.

[0028] A vision-language model (VLM) is an artificial intelligence model that fuses visual and language information. It aims to understand and generate content involving both images (vision) and text (language), processing and correlating information from different modalities to achieve more comprehensive and accurate perception and understanding. Such models are typically based on deep learning architectures and are trained on large amounts of image-text data to learn the mapping relationship between visual and language features. VLMs have a wide range of applications at the intersection of computer vision and natural language processing, such as scene understanding and command interaction in autonomous driving.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the assisted driving method depends, the specific application environment architecture or specific hardware architecture is described here.

[0031] The execution of assisted driving methods depends on a specific application environment and hardware architecture consisting of sensors, computing units, communication modules, etc.

[0032] The data collection layer deploys multiple types of sensors to collect multimodal data. For example, cameras capture road images, millimeter-wave radar and lidar obtain information such as the distance, speed, and direction between the vehicle and obstacles, the Global Positioning System (GPS) and inertial measurement unit (IMU) determine the vehicle's position, speed, and attitude, and high-precision maps provide static information such as road topology and speed limits.

[0033] Data Processing and Decision-Making Layer: The central computing unit (CCU), equipped with high-performance chips such as artificial intelligence (AI) chips and graphics processing units (GPUs), is responsible for running multi-task, multimodal, and large models. It performs operations such as representation conversion, mask processing, object positioning, and time series prediction on collected data, integrating the information and outputting assisted driving strategies. Edge computing devices can also be used for partial data preprocessing, reducing pressure on the CCU and improving response speed.

[0034] Execution and Communication Layer: The actuator receives decision instructions and controls the vehicle's steering, braking, acceleration, and other systems. Furthermore, the IoV communication module enables vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) information exchange, acquiring information such as the driving status of surrounding vehicles and traffic light changes to assist in optimizing decision-making.

[0035] An embodiment of the present application provides an assisted driving method, and the method is described in detail in conjunction with the execution process of the assisted driving method.

[0036] S101: Acquire driving data to be processed.

[0037] The driving data to be processed includes data in multiple modalities. This includes, but is not limited to, text data, image data, LiDAR data, and map data. The image data can be continuous frames of video captured by the vehicle's surround-view camera during high-speed driving.

[0038] For example, the text data may be a driver's voice command, in which the driver asks, "Can I change lanes to the left?", "Can I speed up?", "Which exit should I take to go to a certain place?", etc.

[0039] An embodiment of the present application provides a multi-task head multimodal large model, and steps S102 to S106 are executed by the multi-task head multimodal large model. The multi-task head multimodal large model includes: an encoder, a pre-trained language model and a multi-task head. Among them, the encoder is used to encode the multimodal driving data to be processed to obtain the feature vectors corresponding to the data of each modality; the pre-trained language model is used to perform preliminary mapping on the feature vectors output by the encoder to obtain intermediate feature vectors; the multi-task head is used to perform advanced mapping on the intermediate feature vectors to obtain the output results of the multi-task head, and the multi-task head includes an action mask task head, a positioning task head and a timing prediction task head. The multi-task head multimodal large model provided in the present application can be obtained by adding a multi-task head to the end of an open source pre-trained model. The open source pre-trained model includes but is not limited to Shusheng Wanxiang 2.5 (InternVL-2.5) and Tongyi Qianwen Visual Language Model (Tongyi Qianwen Qwen2-VL).

[0040] It should be emphasized that the multi-task heads run independently, handle different tasks, and process the features output by the model body in parallel, so that the multimodal large model can handle multiple different types of tasks at the same time, and finally output integrated information to provide comprehensive decision-making information for assisted driving.

[0041] This design approach improves the model's flexibility and versatility, enabling large, multi-task, multi-modal models to handle a wide range of tasks simultaneously. Because different task heads share the feature representations of the main model, they can interact with each other to a certain extent and learn collaboratively, helping to improve the model's performance across tasks.

[0042] S102 : Perform representation conversion on the driving data to be processed to obtain a first eigenvector, a second eigenvector, and a third eigenvector.

[0043] In some embodiments, the representation conversion of the processed driving data includes an encoding phase and a preliminary mapping phase. The encoding phase includes encoding the processed driving data to obtain embedded representations corresponding to the data of each modality. The preliminary mapping phase includes mapping the embedded representations corresponding to the data of each modality according to a pre-trained language model to obtain a first feature vector, a second feature vector, and a third feature vector. The pre-trained language model may be a language model of a VLM.

[0044] Optionally, the encoding phase includes encoding the text data contained in the processed driving data to obtain text embeddings (TextEmbeddings); and encoding the image data (Surround-ViewVideoInput) contained in the processed driving data to obtain image embeddings. The preliminary mapping phase includes mapping the text embeddings and image embeddings based on the pre-trained language model to obtain feature vectors (Latent Representations). Here, the distinguishing mark is the second feature vector.

[0045] For example, Figure 2 As shown in the figure, the text data (Text Input) contained in the processed driving data is encoded by the text encoder (VLM Text Encoder) to obtain a text embedding representation, and the image data contained in the processed driving data is encoded by the video encoder (VLM Video Encoder) to obtain an image embedding (marked as Surround-View Video Embeddings in the figure). Figure 3 As shown in the figure, the second feature vector (Text Interaction Features) is obtained by mapping the text embedding representation and the image embedding representation through the pre-trained language model (Language Model).

[0046] The above embodiment unifies various complex and diverse forms of raw data into representations in a vector space, reducing the complexity of data processing and improving the efficiency of subsequent calculations. By using a pre-trained language model to map the embedded representation, it is possible to further explore the deep semantic information in the data and convert low-level features into more semantically representative feature vectors. This helps the model better capture the semantic relationships and logical structures in the data and improves the model's ability to understand and analyze the data. Data of different modalities contain information from different aspects. By obtaining the embedded representation corresponding to the data of each modality in the encoding stage and then using the pre-trained language model for mapping in the preliminary mapping stage, these multimodal information can be fused together, allowing the model to comprehensively utilize information from multiple modalities to understand the data.

[0047] In some embodiments, before executing step S102 (performing characterization conversion on the driving data to be processed to obtain the first feature vector, the second feature vector and the third feature vector), the method further includes: encoding the image data contained in the driving data to be processed to obtain a bird's-eye view feature. Figure 3As shown, the image data (Surround-View Video Input) is encoded by a Bird-Eye-View Encoder to obtain Bird-Eye-View Features (BEV Features). The Bird-Eye-View Features are used to enhance the subsequent positioning task in step S104.

[0048] The above-described embodiment converts the scattered, localized image information contained in the image data into a feature representation with a global perspective, thereby comprehensively considering the overall layout and characteristics of the surrounding environment and reducing positioning errors. The bird's-eye view feature captures rich environmental details captured by the surround-view camera, such as the position, posture, and appearance of surrounding objects. Subsequent analysis and processing of the bird's-eye view feature enables better perception of changes in the surrounding environment.

[0049] In some embodiments, before executing step S102 (performing a representation transformation on the processed driving data to obtain a first eigenvector, a second eigenvector, and a third eigenvector), the method further includes: performing mask processing on the image data contained in the processed driving data to obtain mask data; performing object tracking on the mask data to obtain attribute information of the target object, the attribute information including object feature information and object identification.

[0050] Mask processing involves both noise addition and denoising. Noising introduces a certain amount of noise to simulate the various interference factors found in high-speed driving scenarios, allowing the model to focus more on the key features of target objects during training and data processing. This improves the detection capability of target objects and enhances the model's adaptability and robustness to real-world scenarios. Denoising removes irrelevant noise from the driving data to be processed, making the features of target objects more prominent and clear, facilitating accurate identification and location of target objects. In complex high-speed driving scenarios, mask processing can better highlight target objects such as vehicles and pedestrians, reduce background interference, and enhance detection accuracy.

[0051] Mask data is image data containing object masks, which is used to enhance the pre-trained language model's ability to understand target objects in high-speed driving scenarios. Optionally, a diffusion mask generation module can be used to mask the driving data. This diffusion mask generation module can be the open-source pre-trained model DiffuMask.

[0052] Optionally, the mask data is tracked by a tracking module. The tracking module is used to re-identify (Re-identification, ReID) and track target objects in a multi-frame image sequence. Tracking modules include but are not limited to: ByteTrack-V2 module and CenterTrack module. ReID is achieved by determining whether the objects in a multi-frame image sequence are the same target object, extracting the features of the target object, and comparing the feature similarity of the target objects in different frames. The tracking algorithm predicts the position of the target object in the subsequent frame based on information such as the position, appearance, and motion characteristics of the target object in the previous frame. The tracking algorithm includes but is not limited to end-to-end tracking algorithms based on correlation filtering, Kalman filtering, or deep learning.

[0053] The attribute information of the target object, including object characteristics and object identification. Object characteristics include pedestrians’ clothing and body shape, and vehicles’ color and shape.

[0054] For example, Figure 4 or Figure 5 As shown in the figure, the image data (marked as Surround-View Video Input in the figure) is masked by the diffusion module (Diffusion Module), including the noise addition operation (Noising) and the noise removal operation (De-Noising), to obtain the mask data (Surround-View Video Input With Mask). Figure 5 As shown in the figure, the tracking module performs ReID and tracking on the mask data to obtain the target object's attribute information, including object features and object ID. The tracking module can accurately track the target object's position and trajectory across multiple image frames and maintain consistent identification in high-speed driving scenarios.

[0055] The above-described embodiment refines and refines the outline of the target object through masking, thereby more accurately segmenting the target object from the background. This helps the tracking algorithm better track the target object's movement and avoids target loss or misidentification during the tracking process. The tracking module distinguishes and identifies different target objects, enabling the system to track multiple targets simultaneously and accurately record their motion trajectories.

[0056] In some embodiments, the encoding stage further includes encoding the mask data to obtain a mask embedding representation. The preliminary mapping stage further includes mapping the text embedding representation and the mask embedding representation according to a pre-trained language model to obtain a first feature vector.

[0057] For example, the mask data is encoded by a video encoder to obtain a masked surround-view video embedding representation (MaskedSurround-View Video Embeddings). Figure 4 As shown in FIG, the first feature vector is obtained by mapping the text embedding representation and the mask embedding representation through the pre-trained language model (LanguageModel).

[0058] The above embodiment encodes the mask data to obtain a mask embedding representation, and then combines it with the text embedding representation, so that the model can fuse data information of two different modalities, image and text. This multimodal fusion can make full use of the complementarity of data of different modalities to more comprehensively describe the characteristics of the object. By mapping the text embedding representation and the mask embedding representation through the pre-trained language model to obtain the first feature vector, it can use its pre-trained language knowledge and semantic understanding ability to deeply fuse and transform the embedding representations of the two modalities, which helps the model understand the complex semantic relationship between the object in the image and the text description, generate the essential features that accurately reflect the target object, and improve the performance of the model in the positioning task.

[0059] In some embodiments, the preliminary mapping stage further includes mapping the text embedding representation, the mask embedding representation and the attribute information according to the pre-trained language model to obtain a third feature vector. Figure 5 As shown in FIG, the third feature vector is obtained by mapping the text embedding representation, the mask embedding representation and the attribute information (such as Object ID) through the pre-trained language model (Language Model).

[0060] The above embodiment enables the model to deeply integrate text descriptions, image features, and specific attributes of objects, learn more complex and representative feature combinations, and generate a third feature vector that can better capture the intrinsic patterns and semantic relationships in the data. This enhanced feature representation helps improve the accuracy of the model in time series prediction tasks.

[0061] S103 : performing mask processing based on the first eigenvector to determine that the first eigenvector corresponds to a target object, and performing action judgment on the target object to obtain behavior information of the target object.

[0062] like Figure 2 As shown, the action mask task head (Mask Head) performs mask processing and action judgment tasks to obtain the target object and its behavior information (Mark Output) corresponding to the first feature vector. The target object may be an obstacle.

[0063] In some embodiments, a mask is applied to the first feature vector to determine that the first feature vector corresponds to a target object, and an action judgment is performed on the target object to obtain behavioral information of the target object. The first feature vector is obtained by mapping a text embedding representation and a mask embedding representation based on a pre-trained language model. Behavioral information includes the target object's direction of movement, turning angle, acceleration or deceleration, etc.

[0064] like Figure 4 As shown, the action masking task head includes a multi-layer perceptron (MLP). The MLP performs masking on the first eigenvector to determine whether the first eigenvector corresponds to the target object, and then performs action judgment on the target object to obtain the target object's behavior information (Mark Output). Among them, MLP is a feedforward neural network consisting of an input layer, one or more hidden layers, and an output layer, and the neurons in each layer are connected by weights. Its working principle is to pass the input first eigenvector through the input layer to the hidden layer. In the hidden layer, after weighted summation and nonlinear activation function processing, the first eigenvector is feature extracted and transformed, and then the processed result is passed to the output layer. The output layer then performs final prediction or classification operations based on the received information. MLP establishes a mapping relationship between different action features and behavior patterns by learning a large amount of sample data, thereby inferring the behavioral intention of the target object based on the current action features.

[0065] The above embodiment uses the motion masking task head to mask the first feature vector, highlighting the portion of the first feature vector relevant to the target object and suppressing irrelevant information, thereby accurately locating the target object. It also captures the target object's motion characteristics and accurately determines its behavior pattern. This addresses the inadequate object segmentation and behavior judgment capabilities of general-purpose large multimodal models in related technologies.

[0066] S104: Perform object positioning based on the second eigenvector to obtain position information of the target object.

[0067] like Figure 2 As shown in the figure, the Detection Head performs the object localization task and obtains the target object's location information (Detection Output) corresponding to the second feature vector. The second feature vector is obtained by mapping the text embedding representation and the image embedding representation based on the pre-trained language model. The localization task is used to locate the target object and output the coordinates of the target object's center point in each frame.

[0068] In some embodiments, based on encoding the image data to obtain the bird's-eye view features, the bird's-eye view features and the second feature vector are used to locate the object to obtain the position information of the target object. Figure 4 As shown in the figure, the localization task head includes a vision transformer and an MLP. The Vision Transformer and MLP are used to localize the bird's-eye view features (BEVFeatures) and the second feature vector (Text Interaction Features) to obtain the location information of the target object (Detection Output).

[0069] In the above embodiment, the bird's-eye view feature is combined with the second feature vector through the positioning task head, thereby realizing the fusion of multi-source data, making the positioning information more comprehensive, and different types of data complement each other, which can more accurately determine the position of the target object and reduce positioning errors, thereby solving the problem of insufficient positioning capabilities of general multimodal large models in related technologies.

[0070] S105 . Perform time series prediction based on the third eigenvector to obtain time series information of the target object.

[0071] like Figure 2 As shown, the temporal prediction task is performed by the temporal head to obtain the temporal information (Temporal Output) of the target object corresponding to the third feature vector. The third feature vector can be obtained by mapping the text embedding representation, the mask embedding representation, and the attribute information according to the pre-trained language model.

[0072] like Figure 5 As shown, the time series prediction task head includes an MLP. The MLP performs time series prediction on the third eigenvector to obtain the target object's time series information (temporal output), such as the target object's movement speed, acceleration changes, and behavior mode switching in different scenarios.

[0073] The above-described embodiment integrates multi-dimensional information through the time series prediction task head to model the time series changes of the target object. This more comprehensively considers the impact of various factors on the object's time series information and accurately captures the target object's changing trends over time, thereby improving the accuracy of the prediction. Using the third eigenvector containing multimodal information for time series prediction helps improve the model's generalization capabilities across different scenarios and data distributions. This addresses the problem of insufficient time series information processing capabilities of general-purpose multimodal large models in related technologies.

[0074] In the above steps S103 to S105, the corresponding feature vectors are advanced mapped by the multi-task head to obtain the behavior information, position information and timing information of the target object respectively, so that the model can be optimized for each task more finely. Compared with the single-task model, it can more comprehensively and accurately extract the features related to the behavior, position and timing of the target object. There is a synergistic effect between the multi-task heads. They can share the information in the feature vectors and make full use of the correlation between different tasks. By jointly optimizing multiple tasks, the prediction performance of the model for each task is improved. The multi-task heads can be executed in parallel during the calculation process, allowing the model to process large amounts of data more quickly, which is suitable for high-speed driving scenarios with high real-time requirements. By fusing these different types of features, the model can better cope with various complex scenarios and changes, and improve the generalization ability in high-speed driving scenarios.

[0075] S106: Integrate behavior information, location information, and timing information, and output an assisted driving strategy.

[0076] Assisted driving strategies include speed control, lane keeping, lane change decisions, following distance adjustment and other instructions.

[0077] Based on the pre-trained language model, the behavioral information, location information and timing information of the target object are integrated to determine the assisted driving strategy, thereby improving decision-making accuracy, enhancing environmental adaptability and optimizing the driving experience.

[0078] For example, Figure 2 As shown in the figure, assume that the target object's behavior information is the white car ahead changing lanes right. The target object's position information is the sequence of changes in the white car's center coordinates, such as {[411, 623], [507, 594], [592, 548], [677, 541]}, indicating that the white car moved from the center coordinates [411, 623] to [507, 594], then to [507, 594], then to [507, 594], and finally to [677, 541]. The target object's timing information is 2.8 seconds. The pre-trained language model integrates the outputs of the above multi-task heads to produce an assisted driving policy (Final Output), indicating that the white car ahead at [677, 541] completes its lane change to the right after 2.8 seconds, after which the vehicle can be controlled to change lanes to the left.

[0079] Optionally, the assisted driving strategy can be output through voice broadcast or text display.

[0080] The above-described embodiment uses a pre-trained language model to comprehensively integrate the target object's behavioral, location, and timing information, accurately grasping the surrounding environment and providing a comprehensive and accurate basis for the formulation of assisted driving strategies. The pre-trained language model's ability to integrate multi-source information enables it to effectively handle complex high-speed driving scenarios, rapidly adjusting assisted driving strategies based on various information about the target object to ensure safe driving.

[0081] It should be noted that the multi-task head multimodal large model provided in the embodiment of the present application adjusts and optimizes the internal parameters of the pre-trained language model and the multi-task head according to the loss function during the training process; and freezes the pre-trained weights of the diffusion mask generation module, tracking module and visual encoder.

[0082] The memory parameters of the pre-trained language model include, but are not limited to, word embeddings and neural network connection weights. The internal parameters of the multi-task head include the MLP weights. Pre-trained weights are model parameters obtained by pre-training on large-scale datasets. Freezing these pre-trained weights prevents them from being updated during model training to preserve the learned general features. This prevents overfitting to a specific dataset in new training tasks, which could lead to loss of previously good feature extraction capabilities.

[0083] This training approach combines the strengths of different model components, leveraging the general feature extraction capabilities of pre-trained models while adapting to specific tasks and datasets by opening up specific weights for training. This improves training efficiency while also achieving better task performance. The addition of multi-task heads avoids the tedious process of training a separate model for each task, saving computing resources and time.

[0084] In some embodiments, after outputting the assisted driving strategy, it also includes predicting the vehicle's driving trajectory, and then generating autonomous driving instructions based on the vehicle's driving trajectory and the assisted driving strategy, and then controlling the vehicle's driving components to execute the autonomous driving instructions to achieve autonomous driving in high-speed driving scenarios.

[0085] Optionally, the vehicle's driving trajectory is predicted by analyzing the relative position and motion relationship between the vehicle and the target object. Alternatively, based on the hidden Markov model, the vehicle's driving state is regarded as a hidden state, and the transition probability of the hidden state is inferred by observing the behavior information, position information, and timing information of the target object, thereby predicting the vehicle's driving trajectory. After the vehicle's driving trajectory is predicted, the assisted driving strategy output by the multi-task head multimodal large model is combined to generate autonomous driving instructions, which may include vehicle speed. The vehicle's driving components are further controlled to execute autonomous driving instructions, such as adjusting the vehicle's power output, steering wheel angle, and braking force, thereby achieving autonomous driving in high-speed driving scenarios.

[0086] The above-mentioned embodiments combine accurate vehicle trajectory prediction to precisely control the vehicle's speed, steering, braking and other operations, providing passengers with a more comfortable driving experience. Optionally, during vehicle driving, the actual driving status of the vehicle is monitored in real time and compared with the predicted trajectory and planned path. If it is found that the actual driving trajectory is inconsistent with expectations, for example, due to external interference or prediction errors causing the vehicle to deviate from the planned path, this information is passed to the trajectory prediction and decision planning module through a feedback mechanism. These modules then adjust and optimize the prediction and planning based on the feedback information, generate new control instructions, and return the vehicle to the correct driving trajectory. Through continuous feedback and optimization, a closed-loop control system is formed, enabling the vehicle to achieve autonomous driving in various complex environments.

[0087] In summary, the assisted driving method provided by the embodiments of the present application has the following beneficial effects compared to related technologies:

[0088] (1) Obtaining multimodal driving data to be processed. The fusion of multimodal data can more comprehensively describe the highway scene, overcoming the problem of insufficient information in single-modal data. Through multimodal data input, it can more accurately capture various information in the highway environment and provide a rich information basis for subsequent complex task decision-making.

[0089] (2) The multimodal driving data to be processed is transformed into the first, second, and third eigenvectors. Different eigenvectors can be used for different analysis tasks. This detailed division helps to achieve multi-level decision-making. For example, the first eigenvector can focus on the attribute and behavior analysis of the target object; the second eigenvector focuses on the spatial position of the target object; and the third eigenvector focuses on the time series changes of the target object. This helps the model process and analyze the data more efficiently, converting complex multimodal data into a form suitable for model processing, and improving the model's generalization ability and decision accuracy.

[0090] (3) Multiple tasks, including mask processing based on the first eigenvector, object positioning based on the second eigenvector, and time series prediction based on the third eigenvector. Mask processing can identify the target object and perform action judgment to obtain behavioral information; object positioning can obtain the position information of the target object; and time series prediction can obtain the time series information of the target object. These different tasks analyze and understand the target object from different angles, thereby obtaining more comprehensive and detailed information, meeting the multi-level decision-making needs of complex tasks in highway scenarios. In contrast, the general multimodal large model lacks a specialized processing mechanism for these specific tasks, making it difficult to simultaneously take into account multiple aspects such as behavior judgment, position positioning, and time series prediction, resulting in insufficient precision and accuracy in decision-making in highway scenarios.

[0091] (4) By integrating the target object's behavior, location, and timing information, and comprehensively considering the various states and changing trends of the target object, a more comprehensive and reasonable assisted driving strategy can be formulated. This decision-making method based on multi-dimensional information can better cope with the complex situations in highway scenarios. For example, according to the behavior, location, and future timing changes of the target vehicle, the vehicle's driving trajectory, speed adjustment, and lane change decisions can be reasonably planned. However, the general multimodal large model cannot effectively integrate these different levels of information, resulting in the output decision results not being well adapted to the complex task requirements in highway scenarios.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0093] The embodiment of the present application also provides a driving assistance device, such as Figure 6 As shown, the auxiliary driving device includes:

[0094] A data acquisition module 601 is used to acquire driving data to be processed, where the driving data to be processed includes data of multiple modalities;

[0095] A data representation conversion module 602 is configured to perform representation conversion on the driving data to be processed to obtain a first eigenvector, a second eigenvector, and a third eigenvector;

[0096] The multi-task head module 603 is configured to perform mask processing based on the first eigenvector to determine the target object corresponding to the first eigenvector, and perform action judgment on the target object to obtain behavior information of the target object; perform object positioning based on the second eigenvector to obtain position information of the target object; and perform time series prediction based on the third eigenvector to obtain time series information of the target object.

[0097] The information integration module 604 is used to integrate behavior information, location information and timing information and output an assisted driving strategy.

[0098] As an optional implementation provided in the embodiment of the present application, the data representation conversion module 602 includes:

[0099] The encoding submodule is used to encode the driving data to be processed and obtain the corresponding embedding representation of the data of each modality;

[0100] The mapping submodule is used to map the embedding representations corresponding to the data of each modality according to the pre-trained language model to obtain feature vectors.

[0101] As an optional implementation provided in an embodiment of the present application, the driving data to be processed includes text data and image data;

[0102] The device also includes a mask submodule, which is used to: perform mask processing on the image data to obtain mask data; and perform object tracking on the mask data to obtain attribute information of the target object, where the attribute information includes object feature information and object identification.

[0103] As an optional implementation provided in an embodiment of the present application, the encoding submodule is specifically used to: encode text data to obtain a text embedding representation; encode image data to obtain an image embedding representation; and encode mask data to obtain a mask embedding representation.

[0104] As an optional implementation provided in an embodiment of the present application, the mapping submodule is specifically used to: map the text embedding representation and the mask embedding representation according to the pre-trained language model to obtain a first feature vector; map the text embedding representation and the image embedding representation according to the pre-trained language model to obtain a second feature vector; map the text embedding representation, the mask embedding representation and the attribute information according to the pre-trained language model to obtain a third feature vector.

[0105] As an optional implementation provided in an embodiment of the present application, the device further includes a bird's-eye view encoder, configured to: encode the image data to obtain a bird's-eye view feature;

[0106] The multi-task head module is specifically used to locate the object based on the bird's-eye view feature and the second feature vector to obtain the position information of the target object.

[0107] As an optional implementation provided in an embodiment of the present application, the device also includes a control module for predicting the vehicle's driving trajectory; generating autonomous driving instructions based on the vehicle's driving trajectory and assisted driving strategy; and controlling the vehicle's driving components to execute the autonomous driving instructions.

[0108] For the description of the features in the embodiments corresponding to the assisted driving device, please refer to the relevant description of the embodiments corresponding to the assisted driving method, and will not be repeated here.

[0109] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned assisted driving method embodiments.

[0110] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned assisted driving method embodiments when run.

[0111] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0112] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned assisted driving method embodiments are implemented.

[0113] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned assisted driving method embodiments.

[0114] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] The above is a detailed introduction to the assisted driving method, device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A driving assistance method, characterized in that: include: Acquiring driving data to be processed, where the driving data to be processed includes data of multiple modalities; Performing characterization conversion on the driving data to be processed to obtain a first eigenvector, a second eigenvector, and a third eigenvector; performing mask processing based on the first feature vector to determine a target object corresponding to the first feature vector, and performing action judgment on the target object to obtain behavior information of the target object; Performing object positioning based on the second feature vector to obtain position information of the target object; Performing time series prediction based on the third eigenvector to obtain time series information of the target object; The behavior information, the location information, and the timing information are integrated to output an assisted driving strategy.

2. The method according to claim 1, characterized in that The characterization conversion of the driving data to be processed to obtain a first eigenvector, a second eigenvector, and a third eigenvector includes: Encoding the driving data to be processed to obtain an embedding representation corresponding to the data of each modality; Mapping the embedded representations corresponding to the data of each modality according to the pre-trained language model to obtain the first feature vector, the second feature vector, and the third feature vector.

3. The method according to claim 2, characterized in that The driving data to be processed includes text data and image data; After obtaining the driving data to be processed and before performing characterization conversion on the driving data to be processed to obtain the first eigenvector, the second eigenvector, and the third eigenvector, the method further includes: performing mask processing on the image data to obtain mask data; Object tracking is performed on the mask data to obtain attribute information of the target object, where the attribute information includes object feature information and object identification.

4. The method according to claim 3, characterized in that The encoding of the to-be-processed driving data to obtain an embedded representation corresponding to the data of each modality includes: Encoding the text data to obtain a text embedding representation; Encoding the image data to obtain an image embedding representation; The mask data is encoded to obtain a mask embedding representation.

5. The method according to claim 4, characterized in that Mapping the embedded representations corresponding to the data of each modality according to the pre-trained language model to obtain the first feature vector, the second feature vector, and the third feature vector includes: Mapping the text embedding representation and the mask embedding representation according to the pre-trained language model to obtain a first feature vector; Mapping the text embedding representation and the image embedding representation according to the pre-trained language model to obtain a second feature vector; The text embedding representation, the mask embedding representation, and the attribute information are mapped according to the pre-trained language model to obtain a third feature vector.

6. The method according to claim 5, characterized in that After obtaining the driving data to be processed and before performing characterization conversion on the driving data to be processed to obtain the first eigenvector, the second eigenvector, and the third eigenvector, the method further includes: encoding the image data to obtain bird's-eye view features; The performing object positioning based on the second feature vector to obtain the position information of the target object includes: performing object positioning based on the bird's-eye view feature and the second feature vector to obtain the position information of the target object.

7. The method according to claim 1, characterized in that After integrating the behavior information, the location information, and the timing information and outputting the assisted driving strategy, the method further includes: Predict vehicle trajectories; generating an automatic driving instruction based on the vehicle driving trajectory and the assisted driving strategy; Control the vehicle driving component to execute the automatic driving instructions.

8. A driving assistance device, characterized in that: include: A data acquisition module, configured to acquire driving data to be processed, wherein the driving data to be processed includes data of multiple modalities; a data representation conversion module, configured to perform representation conversion on the driving data to be processed to obtain a first eigenvector, a second eigenvector, and a third eigenvector; a multi-task head module, configured to perform mask processing based on the first feature vector to determine a target object corresponding to the first feature vector, and perform action judgment on the target object to obtain behavior information of the target object; Performing object positioning based on the second feature vector to obtain position information of the target object; Performing time series prediction based on the third eigenvector to obtain time series information of the target object; The information integration module is used to integrate the behavior information, the location information and the timing information to output an assisted driving strategy.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the assisted driving method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the assisted driving method according to any one of claims 1 to 7 are implemented.

Images