Method for controlling vehicle, method and device for training multimodal model, and vehicle

Abstract

The present application provides a method for controlling a vehicle, a method for training a multimodal model, an apparatus, and a vehicle, and relates to the field of artificial intelligence, specifically to technical fields such as vehicle control, deep learning, and autonomous driving. The method for controlling a vehicle includes: receiving multimodal perception data for vehicle driving; using a trained multimodal model to plan vehicle driving motions based on the multimodal perception data based on prompt words, thereby obtaining driving motion information for a specified future time period, wherein the driving motion information represents the vehicle's driving motion, wherein the prompt words are used to control the multimodal model to use the driving motion information as a target and perform semantic understanding of the multimodal perception data; and controlling vehicle driving based on the driving motion information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, specifically to technical fields such as vehicle control, deep learning, and autonomous driving, and more specifically to a method for controlling a vehicle, a method for training a multimodal model, an apparatus, a vehicle, equipment, a medium, and a program product. Background Art

[0002] With the rapid development of artificial intelligence technology, the vehicle's autonomous driving function can be realized based on artificial intelligence algorithms, so as to improve the vehicle's operating efficiency and travel efficiency through the vehicle's autonomous driving function in scenarios such as logistics transportation, mining operations, and public transportation.

[0003] During the process of implementing the inventive concept of the present invention, the inventors found that vehicles driving based on the automatic driving function have poor driving stability and are difficult to meet actual scenario requirements. Summary of the Invention

[0004] In view of the above problems, the present application provides a method for controlling a vehicle, a method for training a multimodal model, an apparatus, a vehicle, a device, a medium and a program product.

[0005] According to a first aspect of the present application, a method for controlling a vehicle is provided, comprising: receiving multimodal perception data for vehicle driving; based on prompt words, using a trained multimodal model to plan driving actions for the vehicle based on the multimodal perception data, and obtaining driving action information for a specified future time period, wherein the driving action information represents the driving action of the vehicle, wherein the prompt words are used to control the multimodal model to use the driving action information as a target and perform semantic understanding on the multimodal perception data; and controlling the driving of the vehicle based on the driving action information.

[0006] Another aspect of the present application also provides a method for training a multimodal model, including: receiving sample multimodal perception data and labeled driving action information for vehicle driving, the above-mentioned labeled driving action information represents the driving action of the above-mentioned vehicle in a sample specified time period; based on sample prompt words, using a multimodal model to plan the driving action of the above-mentioned vehicle based on the above-mentioned sample multimodal perception data, and obtain sample driving action information for the sample specified time period, wherein the above-mentioned sample prompt words are used to control the above-mentioned multimodal model to use the above-mentioned sample driving action information as a target, and perform semantic understanding on the above-mentioned sample multimodal perception data; training the above-mentioned multimodal model based on the above-mentioned sample driving action information and the above-mentioned labeled driving action information to obtain a trained multimodal model.

[0007] Another aspect of the present application also provides a device for controlling a vehicle, including: a perception data receiving module for receiving multimodal perception data for vehicle driving; a driving action information obtaining module for planning the driving action of the vehicle based on the multimodal perception data using a trained multimodal model based on prompt words, and obtaining driving action information for a specified time period in the future, wherein the driving action information represents the driving action of the vehicle, wherein the prompt words are used to control the multimodal model to use the driving action information as a target and perform semantic understanding on the multimodal perception data; and a control module for controlling the driving of the vehicle based on the driving action information.

[0008] Another aspect of the present application also provides a device for training a multimodal model, including: a sample receiving module for receiving sample multimodal perception data and labeled driving action information for vehicle driving, the above-mentioned labeled driving action information representing the driving action of the above-mentioned vehicle in the sample specified time period; a sample driving action information acquisition module for planning the driving action of the above-mentioned vehicle based on the above-mentioned sample multimodal perception data based on sample prompt words, and obtaining sample driving action information for the sample specified time period, wherein the above-mentioned sample prompt words are used to control the above-mentioned multimodal model to use the above-mentioned sample driving action information as a target and perform semantic understanding on the above-mentioned sample multimodal perception data; a training module for training the above-mentioned multimodal model based on the above-mentioned sample driving action information and the above-mentioned labeled driving action information to obtain a trained multimodal model.

[0009] Another aspect of the present application provides a vehicle, characterized in that it includes: a processor configured to execute the above-mentioned method of controlling the vehicle.

[0010] Another aspect of the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0011] Another aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0012] Another aspect of the present application further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the computer program or instructions are executed by a processor.

[0013] According to the embodiments of the present application, since prompt words can be used to control the trained multimodal model to use driving action information as a target to perform semantic understanding of multimodal perception data, the multimodal model can generate driving action information representing the vehicle's driving action in a specified future time period by more fully understanding the multimodal perception data used for vehicle driving under the control of the prompt words. In this way, the vehicle's driving can be controlled by more accurate driving action information to improve the stability of vehicle driving control. At the same time, by outputting driving action information from the multimodal model, the results output by the multimodal model can have a strong correlation with the driving action that controls the vehicle's driving, thereby improving the interpretability and comprehensibility of the multimodal model's output results. In addition, a multimodal model with strong prediction accuracy can be trained based on smaller training resources to reduce the difficulty of training the multimodal model, further improve the generalization ability of the multimodal model for controlling vehicles in diverse autonomous driving scenarios, and reduce the cost of model deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0015] Figure 1 A diagram showing an application scenario of the method and device for controlling a vehicle according to an embodiment of the present application is shown;

[0016] Figure 2 A flow chart showing a method for controlling a vehicle according to an embodiment of the present application is shown;

[0017] Figure 3A A schematic diagram showing the principle of a multimodal model according to an embodiment of the present application is shown;

[0018] Figure 3B A schematic diagram showing the principle of a decoder according to an embodiment of the present application is shown;

[0019] Figure 4 An application scenario diagram of a method for controlling a vehicle according to an embodiment of the present application is shown;

[0020] Figure 5 A flowchart of a method for training a multimodal model according to an embodiment of the present application is shown;

[0021] Figure 6 A structural block diagram of a vehicle control device according to an embodiment of the present application is shown;

[0022] Figure 7 A structural block diagram of an apparatus for training a multimodal model according to an embodiment of the present application is shown;

[0023] Figure 8A block diagram of an electronic device suitable for implementing a method for controlling a vehicle and a method for training a multimodal model according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the technical solution of this application, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] The inventors discovered that deep learning models with large parameters, such as multimodal large models (MLLMs), are used in scenarios like autonomous driving to plan driving trajectories and control vehicles. For example, large multimodal models like the Vision-Language-Action (VLA) model can incorporate natural language instructions expressed in speech and combine them with vehicle perception data for path planning. However, controlling vehicles based on large multimodal models can lead to problems such as poor vehicle stability and high model deployment costs.

[0030] Embodiments of the present application provide a method for controlling a vehicle, a method for training a multimodal model, an apparatus, a vehicle, a device, a medium, and a program product. The method for controlling a vehicle includes: receiving multimodal perception data for vehicle driving; using a trained multimodal model to plan vehicle driving motions based on the multimodal perception data based on a prompt word, thereby obtaining driving motion information for a specified future time period, wherein the driving motion information represents the vehicle's driving motion, wherein the prompt word is used to control the multimodal model to use the driving motion information as a target and perform semantic understanding on the multimodal perception data; and controlling vehicle driving based on the driving motion information.

[0031] According to the embodiments of the present application, since prompt words can be used to control the trained multimodal model to use driving action information as a target to perform semantic understanding of multimodal perception data, the multimodal model can generate driving action information representing the vehicle's driving action in the future period by more fully understanding the multimodal perception data used for vehicle driving under the control of the prompt words. In this way, the vehicle's driving can be controlled by more accurate driving action information to improve the stability of vehicle driving control. At the same time, by outputting driving action information from the multimodal model, the results output by the multimodal model can have a strong correlation with the driving action that controls the vehicle's driving, thereby improving the interpretability and comprehensibility of the multimodal model's output results. In addition, a multimodal model with strong prediction accuracy can be trained based on smaller training resources, thereby reducing the difficulty of training the multimodal model and further improving the generalization ability of the multimodal model for vehicle control in diverse autonomous driving scenarios, thereby reducing the cost of model deployment.

[0032] Figure 1 A diagram showing an application scenario of the method and device for controlling a vehicle according to an embodiment of the present application is shown.

[0033] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first vehicle 101, a second vehicle 102, a third vehicle 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first vehicle 101, the second vehicle 102, the third vehicle 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0034] The first vehicle 101, the second vehicle 102, and the third vehicle 103 can each interact with the server 105 via the network 104 to receive or send messages, etc. The first vehicle 101, the second vehicle 102, and the third vehicle 103 can also interact with each other via the network 104 to receive or send messages. The first vehicle 101, the second vehicle 102, and the third vehicle 103 can be unmanned vehicles, or can also include vehicles driven by a driver.

[0035] The first vehicle 101 , the second vehicle 102 , and the third vehicle 103 may be any type of vehicle, such as a mining car, a truck, a car, and the like.

[0036] Server 105 may be a server that provides various services, such as a server that provides background management for the movement and operation status of users utilizing first vehicle 101, second vehicle 102, and third vehicle 103 (for example only). The background management server may analyze and process data such as broadcast messages received from the vehicles, and provide feedback to any of the vehicles.

[0037] It should be noted that the vehicle control method provided in the embodiment of the present application can generally be performed by any one of the first vehicle 101, the second vehicle 102, and the third vehicle 103. The vehicle control device provided in the embodiment of the present application can also be provided in any one of the first vehicle 101, the second vehicle 102, and the third vehicle 103.

[0038] Alternatively, the method for controlling a vehicle provided in the embodiment of the present application may also generally be executed by the server 105. Accordingly, the apparatus for controlling a vehicle provided in the embodiment of the present application may generally be provided in the server 105. The method for controlling a vehicle provided in the embodiment of the present application may also be executed by a server or server cluster that is different from the server 105 and that is capable of communicating with the first vehicle 101, the second vehicle 102, the third vehicle 103 and / or the server 105. Accordingly, the apparatus for controlling a vehicle provided in the embodiment of the present application may also be provided in a server or server cluster that is different from the server 105 and that is capable of communicating with the first vehicle 101, the second vehicle 102, the third vehicle 103 and / or the server 105.

[0039] It should be understood that Figure 1The number of vehicles, networks and servers in the embodiment is only illustrative. Any number of vehicles, networks and servers may be provided as needed.

[0040] Figure 2 A flow chart of a method for controlling a vehicle according to an embodiment of the present application is shown.

[0041] like Figure 2 As shown, the method for controlling a vehicle in this embodiment includes operations S210 to S230.

[0042] In operation S210 , multimodal perception data for vehicle travel is received.

[0043] In operation S220 , based on the prompt word, a driving action plan for the vehicle is performed using the trained multimodal model based on the multimodal perception data to obtain driving action information for a future specified time period.

[0044] In operation S230, the vehicle is controlled to travel based on the travel motion information.

[0045] According to embodiments of the present application, multimodal perception data used for vehicle driving may include perception data related to the vehicle's driving environment, such as image data representing obstacle information such as the location and speed of obstacles around the vehicle, but is not limited thereto. It may also include data related to road information such as lane lines, or may also include perception data in any other modality, such as voice modality or text modality. The embodiments of the present application do not limit the specific data information type of the multimodal perception data.

[0046] In some embodiments, the multimodal perception data includes at least two of image modality data, text modality data, and map data.

[0047] For example, the image modality data may include image modality data representing the driving environment of the vehicle, such as roadside environment images and vehicle-mounted perspective images collected for the driving environment of the vehicle.

[0048] For example, text modal data may include voice data representing the driving intention of the vehicle, text data, and other data related to text semantics. For example, text modal data may include the driver's or other personnel's voice control message "arrive at unloading point A to unload the cargo."

[0049] Exemplarily, the map data is related to the driving environment of the vehicle. For example, the map data may be high-precision map data, standard map data, etc. related to the vehicle's driving area.

[0050] In some embodiments, the multimodal perception data may also include point cloud data representing the vehicle's driving environment, such as lidar point cloud data, millimeter-wave radar perception data, etc.

[0051] It should be noted that the embodiments of the present application do not limit the method of obtaining multimodal perception data. For example, it can be collected based on vehicle-mounted sensors such as vehicle-mounted cameras and vehicle-mounted lidars, or it can also be obtained by the vehicle from other terminals or servers through a communication module.

[0052] According to embodiments of the present application, driving action information represents a vehicle's driving action. A vehicle's driving action may be related to the vehicle's state of motion during driving. For example, driving action information may represent any type of driving action, such as the vehicle's speed, direction, braking action, acceleration action, or the like. Embodiments of the present application do not limit the specific type of driving action represented by the driving action information, as long as it can represent the vehicle's state of motion.

[0053] According to an embodiment of the present application, the multimodal model may be a multimodal large model (MLLMs) for processing multimodal data. The multimodal model may be used to process cross-modal multimodal perception data to more fully understand the perception data related to the vehicle's driving environment. The prompt words are used to control the multimodal model to use driving action information as a target for semantic understanding of the multimodal perception data, so that the multimodal model can be used to process the prompt words and multimodal perception data to control the multimodal model to use driving action information as the regression target of the model algorithm, so as to improve the interpretability and reliability of the driving action information output by the multimodal model as an output result, and then the vehicle can be controlled to drive stably according to the driving action information, so as to improve the vehicle's driving stability and safety, and improve the vehicle's driving efficiency. It should be noted that the multimodal model in the method for controlling a vehicle provided in the embodiment of the present disclosure may be a trained multimodal model, and the embodiment of the present disclosure will not be repeated here.

[0054] In some embodiments, the driving action information includes at least one of the following: driving speed data for a specified time; driving curvature data for a specified time; and driving heading angle data for a specified time.

[0055] According to an embodiment of the present application, the future specified time period includes one or more specified moments. For example, the future specified time period may be a time period of 3 seconds after the current moment, and the multiple specified moments may be the 1st second, 2nd second, and 3rd second after the current moment.

[0056] According to an embodiment of the present application, driving speed data may represent the speed value that a vehicle should have at a specified time, for example, the speed should be 3 m / s one second after the current time. Driving heading angle data may represent the heading angle that a vehicle should have at a specified time. Driving curvature data may represent the degree of curvature of a vehicle's driving motion at a specified time, for example, the driving curvature data may be represented as a driving curvature value of 1 rad / m one second after the current time.

[0057] It should be noted that the driving speed data, driving curvature data, and driving heading angle data in the driving action information may be driving action data in a preset coordinate system, such as a driving speed value and a driving heading angle value in a geodetic coordinate system. The embodiments of this application do not limit the specific configuration of the preset coordinate system, and the preset coordinate system may be selected based on actual needs.

[0058] In some embodiments, controlling vehicle driving based on driving action information may include: determining power control instructions and steering control instructions based on driving speed data and driving curvature data in the driving action information; and controlling the vehicle to drive in a specified future time period based on the power control instructions and steering control instructions.

[0059] According to an embodiment of the present application, the power control instruction may be an instruction for controlling the power output degree or the braking force control degree of the vehicle. For example, the power control instruction may include an accelerator opening instruction, a gear setting instruction, a braking degree control instruction, etc.

[0060] According to an embodiment of the present application, the steering control instruction may include an instruction for indicating the heading angle that the vehicle should have at a specified moment. For example, the steering control instruction may include a steering wheel rotation angle control instruction of the vehicle, etc.

[0061] In some embodiments, the future specified time period may include multiple specified moments, each of which corresponds to driving speed data and driving curvature data. The driving action information at multiple specified moments output by the multimodal model can be expressed as Among them, M ego is a set of driving action information at multiple specified moments, m t is the driving action information at the specified time t, and the driving action information m t Including driving speed data and driving curvature data. For example, driving action information can be m t =(k t , s t ), where s t represents the driving speed data at a specified time t, k t Indicates the travel curvature data at a specified time t.

[0062] It should be noted that the driving speed data and the driving curvature data can be represented based on data of any data type, including but not limited to numerical values, vectors, character strings, etc.

[0063] According to an embodiment of the present application, a multimodal model, under the control of prompt words, performs driving motion planning on multimodal perception data to output driving speed data and driving curvature data. The driving speed data and driving curvature data can be used to represent the degree of vehicle power control and vehicle steering control in the vehicle's driving motion, respectively. This allows power control instructions and steering control instructions to be generated directly based on the driving speed data and driving curvature data to control vehicle driving. This avoids the difficulty in understanding the control method for the vehicle's driving motion by generating power control signals or steering control signals based on trajectory points output by the model, and avoids the problem of reduced stability and reliability caused by controlling vehicle driving based on trajectory points, thereby improving vehicle driving efficiency and safety.

[0064] According to an embodiment of the present application, based on prompt words, using a trained multimodal model to plan vehicle driving actions based on multimodal perception data may include: using action embedding features of the action type representing driving action information as prompt words, using the multimodal model to perform feature fusion on the multimodal perception data to obtain intermediate fusion features related to the action type; and determining the driving action information for the action type based on the intermediate fusion features.

[0065] According to embodiments of the present application, action embedding features representing action types serve as prompts and can be used to control a multimodal model to target driving action information corresponding to the action type, thereby performing semantic understanding of multimodal perception data. For example, an action type could be a driving speed type or a driving curvature type, and multiple action embedding features could correspond to each of the driving speed and curvature types, respectively.

[0066] In this way, the multimodal model can fully integrate the multimodal perception data representing the vehicle driving environment or vehicle control intention under the control condition that the action driving information corresponding to the action type is used as the target, so that the intermediate fusion features can more accurately represent the driving action semantics related to the action type on the basis of more accurately integrating the semantic attributes of the multimodal perception data. Then, the action driving information for the action type can be determined based on the intermediate fusion features related to the action type, so that the action information corresponding to the action type can be accurately adapted to the driving environment semantics and text control semantics represented by the multimodal perception data, thereby improving the stability of vehicle control based on the driving action information.

[0067] In some embodiments, the decoder of the multimodal model can be used to process the intermediate fusion features corresponding to the multiple action types to obtain the driving action information corresponding to the multiple action types.

[0068] In some embodiments, based on a multimodal model, feature fusion is performed on multimodal perception data based on action embedded features as prompt words, and the intermediate fusion features corresponding to the action type after fusion are used to determine driving action information with numerical type. This can avoid the large amount of computational overhead generated by outputting the planned trajectory points through the multimodal model, thereby saving the computational overhead and control delay generated by executing and deploying the multimodal model to control vehicle driving, and realize the flexibility and adaptability of deploying a large multimodal model on the vehicle side to execute the method provided in the embodiment of the present application, and reduce the problems of poor driving stability and reduced driving reliability caused by control delay, thereby improving the efficiency and stability of vehicle control.

[0069] According to an embodiment of the present application, the prompt word includes action embedding features corresponding to multiple action types respectively, the multiple prompt words are arranged in sequence, and the arrangement positions of the multiple prompt words correspond to multiple specified moments in a future specified time period.

[0070] In some embodiments, the action embedding feature can be represented as an empty placeholder (token) vector to prompt the multimodal model the action type of the action embedding feature. Multiple action embedding features can be arranged in the following manner to represent multiple driving action information related to the action type at a specified time. For example, multiple action embedding features are based on the feature sequence E emb ={(e11,e21), (e12,e22),……(e1t,e2t)}, where e1t represents the action embedding feature corresponding to the driving rate type at the t-th specified moment, and e2t represents the action embedding feature corresponding to the driving curvature type at the t-th specified moment. (e1t,e2t) can represent prompt words used to prompt the multimodal model to predict the driving rate data and driving curvature data at the t-th specified moment.

[0071] In a comparative example of the present application, the input data of the multimodal model is multimodal perception data, and the output is the trajectory point at a specified time in a specified future period. The features corresponding to the multimodal perception data include visual perception features and vehicle control text features. The visual perception features and vehicle control text features are represented by X input1 =(V env ,L query ): where X input is the input feature of the multimodal model, V env Visual perception features (or image modality features), L query Control text features for text mode.

[0072] The multimodal model is fed with input X inputThe multimodal model obtains the vehicle trajectory by smoothing the trajectory points at T specified moments in the future specified period.

[0073] In one embodiment of the present application, the input data of the multimodal model is the action embedding feature sequence E obtained by arranging the multimodal perception data and multiple action embedding features in sequence. emb The input data of the multimodal model is X input2 =(V env ,L query ,E emb ), the multimodal model can perform parallel data processing under the prompt control of the action embedded feature sequence and output the driving action information at multiple specified moments.

[0074] According to an embodiment of the present application, multiple prompt words are arranged in sequence to prompt the multimodal model to generate driving action information for multiple specified moments. This allows the multimodal model to output multiple sequential driving action instructions for the vehicle in a specified future time period in parallel, thereby enabling the multimodal model to calculate multiple driving action instructions in parallel. This avoids the redundant computational overhead generated by iteratively generating the t-th trajectory point based on the t-1th trajectory point generated in the current round and the multimodal input data using the multimodal model. Furthermore, by generating driving action information for multiple specified moments in parallel, the time delay in generating control instructions for controlling vehicle driving is reduced, thereby reducing the computational overhead of the computing equipment used to control the vehicle while improving vehicle control efficiency, avoiding the impact of large control signal output delays on vehicle driving stability and reliability, and improving vehicle driving safety in scenarios such as autonomous driving.

[0075] According to an embodiment of the present application, using a multimodal model to perform feature fusion on multimodal perception data includes: determining the perceptual features that characterize the perceptual data of each modality in the multimodal perception data; based on multiple action embedding features arranged in sequence as prompt words, fusing multiple perceptual features according to the attention mechanism to obtain intermediate fusion features related to multiple specified moments in a specified future time period.

[0076] According to an embodiment of the present application, the perceptual features corresponding to the perceptual data of the modalities can be obtained by performing feature extraction on the perceptual data of at least two modalities, such as image modal data, text modal data, and map data. For example, the multimodal perceptual data can be feature embedded based on the feature embedding layer of the attention network algorithm to obtain the perceptual features of each modal perceptual data. However, it is not limited to this. The perceptual features corresponding to the perceptual data of the modalities can also be obtained by performing feature extraction on the perceptual data of at least two modalities based on other types of neural network algorithms such as convolutional neural network algorithms. The embodiments of the present application do not limit the specific algorithm type for determining perceptual features.

[0077] According to an embodiment of the present application, an encoder of a multimodal model can be used to perform attention fusion on multiple perceptual features and action embedding features corresponding to multiple specified moments, thereby obtaining intermediate fused features associated with each of the multiple specified moments. Multiple driving action information is determined based on the multiple intermediate fused features.

[0078] According to an embodiment of the present application, determining driving action information for an action type based on intermediate fusion features may include: fusing multiple intermediate fusion features based on an attention mechanism to obtain target fusion features related to each of the multiple specified moments; and performing driving action detection based on multiple target fusion features to obtain driving action information for the action type at multiple specified moments.

[0079] In some embodiments, multiple intermediate fusion features can be fused based on the attention mechanism based on the decoder of the trained multimodal model, so that the intermediate fusion features corresponding to multiple specified moments can fully learn the correlation and continuity between the driving action information of multiple specified moments in the future, and the target fusion features corresponding to multiple specified moments can represent the driving action under the condition of fully learning the feature semantics contained in the multimodal data, so as to improve the adaptation of the driving action information represented by the target fusion features to the driving conditions represented by multimodal perception data such as the driving environment and control text, and improve the degree of adaptation between the driving action information corresponding to the multiple specified moments and action types output subsequently, thereby improving the control accuracy and reliability of the vehicle through the driving action information.

[0080] In some implementations, by fusing multiple intermediate fusion features to determine target fusion features at multiple specified moments, and further determining driving action information of various action types at multiple specified moments based on the multiple target fusion features, it is possible to achieve parallel decoding processing of the encoded features of the feature semantics of the fused multimodal perception data. This allows the multimodal model to directly output driving action information at multiple specified moments, avoiding the model reasoning method in which the multimodal model needs to rely on at least the prediction results of the previous round as input each time it outputs a current prediction result, and avoids the model reasoning time of repeated feature decoding, thereby reducing the device calculation time and computing overhead of vehicles, servers and other devices that deploy multimodal models in the process of controlling vehicles, meeting the real-time requirements of vehicle control, and enabling the method provided in the embodiment of the present application to improve the flexibility of large model deployment by performing real-time control of vehicles in devices with limited computing power, such as vehicles.

[0081] Figure 3A A schematic diagram showing the principle of a multimodal model according to an embodiment of the present application is shown.

[0082] Figure 3BA schematic diagram showing the principle of a decoder according to an embodiment of the present application is shown.

[0083] like Figure 3A As shown, the trained multimodal model includes an encoder, a decoder, and a detection layer. The encoder and decoder can be constructed based on the Transformer network algorithm, and the detection layer can be constructed based on the multi-layer perceptron algorithm. The multiple perceptual features corresponding to the multimodal perception data are image modality perception features 310 and text modality perception features 320. The image modality perception features 310 can be determined by feature extraction of image modality data representing the driving environment, and the text modality perception features 320 can be determined by feature extraction of text modality data used to control the vehicle.

[0084] The prompt word may include multiple action embedding features 330 arranged in a preset order. Each block in the multiple action embedding features 330 can be understood as an action embedding feature corresponding to the action type. The image modality perception feature 310, the text modality perception feature 320 and the multiple action embedding features 330 are input into the encoder of the multimodal model. The multiple action embedding features 330 can be used as embedding vectors corresponding to the placeholders, so as to use the encoder to perform feature fusion on the image modality perception feature 310, the text modality perception feature 320 and the multiple action embedding features 330 based on the attention network algorithm to obtain multiple intermediate fusion features 340, wherein each block in the multiple intermediate fusion features 340 represents an intermediate fusion feature corresponding to a specified moment, and the arrangement order of the multiple intermediate fusion features 340 corresponds to the arrangement order of the multiple action embedding features 330.

[0085] Combine Figure 3A and Figure 3B As shown, intermediate fused features 340 corresponding to multiple specified moments are input into a decoder. The decoder then fuses these intermediate fused features using an attention mechanism to generate multiple target fused features 350 corresponding to the multiple specified moments. A detection layer constructed using a multi-layer perceptron algorithm processes these multiple target fused features 350 to perform a one-time detection and output of driving motion information at the multiple specified moments, generating driving motion information 360 corresponding to the multiple specified moments. This driving motion information may include driving motion data {m1, m2, …, mt} for t specified moments, where mt represents the driving speed data and driving curvature data at the tth specified moment.

[0086] In one embodiment, multiple target fusion features can be expressed as H=VLA(X input ), where VLA() represents the algorithm execution process of the encoder and decoder, X input represents the multimodal perception data and multiple action embedding features input into the multimodal model, and H represents multiple target fusion features.

[0087] In some embodiments, the action embedding feature is determined based on the following operations: determining multiple preset text tags for the multimodal model; performing data fusion on the text embedding features corresponding to each of the multiple preset text tags to obtain the action embedding feature for at least one action type.

[0088] According to an embodiment of the present application, the preset text tag can be understood as a placeholder (or token) for the preset text word, and the text embedding feature corresponding to the preset text tag can be stored in a preset text word feature library to facilitate feature embedding of the text words input into the multimodal model based on the preset text word feature library.

[0089] According to an embodiment of the present application, by performing data fusion on the text embedding features of preset text tags, the fused action embedding features can be made different from the text embedding features, thereby avoiding confusion between the action embedding features and the perceptual features of the multimodal perception data input by the multimodal model, which would cause interference in model understanding.

[0090] In some embodiments, by adding action embedding features for action types to a preset text word feature library to update the embedding feature set, the multimodal model can target driving action information of specific action types for semantic understanding, thereby improving the multimodal model's adaptability to semantic understanding of driving action information.

[0091] In some embodiments, data fusion can be performed on multiple text embedding features based on at least one of a mean algorithm, a standard deviation algorithm, and a convolution fusion algorithm. The embodiments of this application do not limit the specific type of algorithm used to perform data fusion, as long as the action embedding feature is different from the text embedding features in the preset text word feature library.

[0092] In one example, a mean vector of text embedding features in a preset text word feature library is calculated based on a mean algorithm, and the mean vector is determined as an action embedding feature related to at least one action type, and a mapping relationship is established between the action embedding feature and the action type, so as to adapt the action embedding feature to the input rule requirements of the multimodal model.

[0093] In some embodiments, the method for controlling a vehicle may further include: determining driving speed data and driving curvature data associated with each of multiple specified moments from driving action information; performing integration operations on the driving speed data and driving curvature data associated with each of the multiple specified moments to obtain driving heading angles for the multiple specified moments; performing trajectory point planning based on the driving heading angles and driving speed data for the multiple specified moments to obtain multiple planned trajectory points; and determining a planned driving trajectory based on the multiple planned trajectory points.

[0094] In one embodiment, after the multimodal model predicts the driving speed data and driving curvature data at t specified moments, the heading angle θ at the specified moment t is calculated by integrating the driving speed data and the driving curvature data based on the following formula (1): t .

[0095] ;Formula (1)

[0096] Where k(t) represents the driving curvature value for the driving curvature data at the specified time t, s(t) represents the driving speed value for the driving speed data at the specified time t, and θ t-1 Indicates the heading angle at the specified time t-1.

[0097] It should be noted that the driving speed data and driving curvature data of the vehicle at the starting designated time in the future designated period can be collected based on sensors, so as to calculate the heading angles at multiple subsequent designated time points using formula (1).

[0098] In one embodiment, trajectory point planning based on driving heading angles and driving speed data for multiple specified moments may include: for the driving heading angles and driving speed data at any specified moment, calculating the speed component based on the driving heading angles and driving speed data related to the specified moment to obtain the speed component at any specified moment; performing an integral calculation operation on the speed components for multiple specified moments to obtain the planned trajectory points for the specified moments.

[0099] For example, the velocity components of the vehicle in the x-axis direction and the y-axis direction in the preset coordinate system at a specified time t can be determined based on the following formulas (2) and (3).

[0100] ;Formula (2)

[0101] ;Formula (3)

[0102] in, and They represent the velocity components of the vehicle in the x-axis and y-axis directions at the specified time t, s t is the driving speed value at the specified time t.

[0103] In one example, the velocity components for multiple specified moments can be integrated based on the following formula (4) to obtain the trajectory point (x t ,y t )

[0104] ;Formula (4)

[0105] Among them, xt and y t They represent the x-coordinate and y-coordinate of the trajectory point in the preset coordinate system at the specified time t, respectively. τ is the integral variable. In formula (4), the preset trajectory point (x0, y0) can be used as the starting trajectory point to perform the integration operation shown in formula (4).

[0106] In one embodiment, the initial preset trajectory point (x0, y0) can also be used as the data input of the following formula (5), and the speed component v can be calculated by combining the driving speed data and driving curvature data at t specified moments. x and v y , and perform the following formula (5) cumulative trapezoidal algorithm operation to obtain the heading angle and trajectory points.

[0107] ;Formula (5)

[0108] in, is the time step.

[0109] For example, the initial preset trajectory point (x0, y0) = (0, 0), the initial heading angle θ0 = 0, and the time step s, t>i≥1, driving action information at multiple specified moments output by the multimodal model As shown in Table 1.

[0110] Table 1

[0111]

[0112] According to the prediction results of vehicle speed and curvature in the vehicle's driving action information, the corresponding vehicle trajectory point set within the final specified time period of 3s in the future is obtained as shown in Table 2

[0113] Table 2

[0114]

[0115] According to the embodiments of the present application, by planning the vehicle's driving trajectory based on driving action information, it is possible to avoid the large amount of computing power consumption generated by iteratively predicting multiple trajectory points by setting a trajectory point detection head for the multimodal model, reduce the computing power resource requirements of the multimodal model deployed on the vehicle side, and improve the timeliness and accuracy of vehicle control.

[0116] According to the method provided in the embodiments of the present application, a driving trajectory can be generated based on driving action information such as driving curvature data, driving speed data, etc. that are highly interpretable. It can provide a stable and highly interpretable method for driving trajectory planning and generation, so as to promote a new method for driving trajectory planning for vehicles.

[0117] It should be noted that the trajectory points, planned trajectory points, and vehicle trajectory points involved in the embodiments of the present application represent the same meaning, and the planned trajectory and driving trajectory represent the same or similar meanings. The embodiments of the present application will not be repeated here.

[0118] Figure 4 An application scenario diagram of a method for controlling a vehicle according to an embodiment of the present application is shown.

[0119] like Figure 4 As shown, in this application scenario, a trained multimodal model is deployed on the computing device of vehicle 401. While driving, vehicle 401 can process input multimodal perception data 410 through the multimodal model of the computing device and output prediction results 420 for multiple specified future time periods.

[0120] Among them, the multimodal perception data 410 includes a driving environment image, a driver's control text and a prompt word, and the prompt word may include "Please provide the driving speed and driving curvature of the vehicle in the next 3 seconds. The output mode is {(k1, s1), (k2, s2), (k3, s3), (k4, s4), (k5, s5), (k6, s6)}".

[0121] The prediction result 420 output by the multimodal model may include a text reply content "The driving speed and driving curvature in the next 3 seconds are predicted, and the output results are {(0.02, 0.01), (0.03, 0.2), (0.00, 0.0), (0.1, 0.01), (0.02, 0.3), (0.4, 0.01)}", where "{(0.02, 0.01), (0.03, 0.2), (0.00, 0.0), (0.1, 0.01), (0.02, 0.3), (0.4, 0.01)}" are the driving speed values ​​and driving curvature values ​​at 6 specified moments of 0.5 seconds each in the next 3 seconds. The driving action information in the prediction result 420 can be used to control the driving of the vehicle 401 to realize the automatic driving function of the vehicle 401.

[0122] The method for controlling a vehicle provided in the embodiment of the present application improves the interpretability of the model output results by determining the output results of the multimodal model as driving action information, so as to facilitate rapid connection between the vehicle control system and the model output results, and predicts the driving action information at multiple specified moments through parallel decoding to replace the iterative reasoning of driving trajectory points, thereby achieving one-time output of multiple driving actions, improving the reasoning efficiency of the multimodal model, and meeting the actual operational needs of the vehicle-side control system to control the vehicle driving in real time. The regression mechanism of the multimodal model is optimized by using action embedding features representing the action type as structured prompt words, so that the vehicle can be widely used in complex autonomous driving scenarios with extremely high real-time and safety requirements, and has high practical value and engineering promotion prospects.

[0123] An embodiment of the present application also provides a method for training a multimodal model. The trained multimodal model determined based on the method for training a multimodal model provided in the embodiment of the present application can be applied to the method for controlling a vehicle in the above embodiment.

[0124] Figure 5 A flowchart of a method for training a multimodal model according to an embodiment of the present application is shown.

[0125] like Figure 5 As shown, the method for training a multimodal model includes operations S510 to S530.

[0126] In operation S510 , sample multimodal perception data and label driving action information for vehicle driving are received.

[0127] In operation S520 , based on the sample prompt word, a multimodal model is used to plan a driving action of the vehicle based on the sample multimodal perception data to obtain sample driving action information for the sample specified time period.

[0128] In operation S530 , a multimodal model is trained based on the sample driving action information and the label driving action information to obtain a trained multimodal model.

[0129] In one example, the sample multimodal perception data includes sample data of multiple modalities such as images, speech, and text.

[0130] According to an embodiment of the present application, the tag driving action information represents the driving action of the vehicle in the sample specified time period, for example, it may be the actual driving action of the vehicle at multiple sample specified moments in the sample specified time period.

[0131] According to an embodiment of the present application, the sample prompt words are used to control the multimodal model to take the sample driving action information as the target and perform semantic understanding on the sample multimodal perception data.

[0132] It should be noted that the technical terms involved in the method for training a multimodal model provided in the embodiment of the present application, including but not limited to sample driving action information, sample multimodal perception data, etc., have the same or similar technical attributes as the corresponding technical terms involved in the method for controlling a vehicle provided in the embodiment of the present application, including but not limited to driving action information, multimodal perception data, etc., and the embodiments of the present application will not be repeated here.

[0133] In some embodiments, the sample prompt words include sample action embedding features corresponding to multiple sample action types, the multiple prompt words are arranged in sequence, and the arrangement positions of the multiple prompt words correspond to multiple specified moments in the sample specified time period.

[0134] In one embodiment, the sample prompt word can be expressed in the following form:

[0135] Please provide the vehicle's driving speed and curvature for the next three seconds. The output format is: [ ([MASK],[MASK]), ([MASK], [MASK]), ..., ([MASK], [MASK]) ] , where [MASK] represents the sample action embedding feature. ([MASK], [MASK]) indicates that the prediction target is six sets of predicted sample driving speed and curvature data within three seconds of the current time, with a sample time interval determined every 0.5 seconds. The labeled driving action information consists of six sets of actual driving speed and curvature values ​​for the vehicle at multiple specified times during the specified time period.

[0136] In some embodiments, the sample driving action information and the labeled driving action information are, respectively, first and second parameter values ​​corresponding to the sample action type. For example, the first and second parameter values ​​are, respectively, the predicted curvature value and labeled curvature value corresponding to the driving curvature type at a specified time t. In another example, the first and second parameter values ​​are, respectively, a first array including sample curvature values ​​and sample velocity values ​​corresponding to the specified time t, and a second array including labeled curvature values ​​and labeled velocity values ​​corresponding to the specified time t.

[0137] In some embodiments, training a multimodal model based on sample driving action information and labeled driving action information may include: processing a first parameter value and a second parameter value using a loss function to obtain a loss value; and training the multimodal model based on the loss value.

[0138] For example, the first parameter value and the second parameter value may be processed using a loss function based on the following formula (6).

[0139] ;Formula (6)

[0140] Among them, L pos represents the loss value, is the second parameter value of the tag driving action information, m tis the first parameter value of the sample driving action information, t is the serial number of the sample specified time, and T is the number of sample specified time. Among them, the first parameter value and the second parameter used to calculate the loss value are numerical types, for example, they can be rate values ​​and curvature values. Therefore, by using the loss function to process the difference between the prediction results and labels of numerical types, it is possible to avoid errors caused by loss calculations for other data types such as trajectory point coordinates, thereby optimizing the prediction accuracy of the multimodal model during training, avoiding errors caused by the multimodal model processing text-based embedded feature representations, and enabling the multimodal model to be deployed in computing devices such as vehicle terminals to achieve better vehicle control performance and engineering applicability.

[0141] In some embodiments, the multimodal model reasoning process can be performed based on an on-board computing platform. First, the trained multimodal model is converted into a format on the training platform, and then on the on-board computing platform, the multimodal model is converted into a format supported by the on-board computing platform. Model reasoning can convert driving action information into a driving trajectory according to actual needs and display it on the on-board end for the driver to browse, or directly use the driving speed value and driving curvature value in the driving action information to control vehicle driving.

[0142] Based on the above-mentioned method for controlling a vehicle and method for training a multimodal model, the present application also provides a device for controlling a vehicle and a device for training a multimodal model. Figure 6 and Figure 7 The device is described in detail.

[0143] Figure 6 A structural block diagram of a device for controlling a vehicle according to an embodiment of the present application is shown.

[0144] like Figure 6 As shown, the device 600 for controlling a vehicle in this embodiment includes a perception data receiving module 610 , a driving action information obtaining module 620 and a control module 630 .

[0145] The perception data receiving module 610 is used to receive multimodal perception data for vehicle driving.

[0146] The driving action information acquisition module 620 is used to plan the vehicle's driving actions based on the multimodal perception data using a trained multimodal model based on the prompt word, and obtain driving action information for a specified time period in the future. The driving action information represents the vehicle's driving action, wherein the prompt word is used to control the multimodal model to use the driving action information as a target and perform semantic understanding of the multimodal perception data.

[0147] The control module 630 is used to control the vehicle driving based on the driving action information.

[0148] According to an embodiment of the present application, the driving action information obtaining module includes: an intermediate fusion feature obtaining submodule and a driving action information determining submodule.

[0149] The intermediate fusion feature acquisition submodule is used to use the action embedding feature of the action type representing the driving action information as a prompt word, and use the multimodal model to perform feature fusion on the multimodal perception data to obtain the intermediate fusion features related to the action type. Among them, the prompt word is used to control the multimodal model to take the driving action information corresponding to the action type as the target, and perform semantic understanding of the multimodal perception data.

[0150] The driving action information determination submodule is used to determine the driving action information for the action type based on the intermediate fusion features.

[0151] According to an embodiment of the present application, the intermediate fusion feature obtaining submodule includes: a first determining unit and a first obtaining unit.

[0152] The first determining unit is configured to determine a perceptual feature representing each modal perceptual data in the multimodal perceptual data.

[0153] The first acquisition unit is used to fuse multiple perception features based on the attention mechanism based on multiple action embedding features arranged in sequence as prompt words, so as to obtain intermediate fusion features related to multiple specified moments in the future specified time period, wherein multiple driving action information is determined based on the multiple intermediate fusion features.

[0154] According to an embodiment of the present application, the driving action information determination submodule includes: a second obtaining unit and a third obtaining unit.

[0155] The second obtaining unit is used to fuse multiple intermediate fusion features based on the attention mechanism to obtain target fusion features related to multiple specified moments.

[0156] The third obtaining unit is configured to perform driving action detection based on multiple target fusion features to obtain driving action information for action types at multiple specified moments.

[0157] According to an embodiment of the present application, the action embedding feature is determined based on the following operations: determining multiple preset text tags for the multimodal model; performing data fusion on the text embedding features corresponding to each of the multiple preset text tags to obtain the action embedding feature for at least one action type.

[0158] According to an embodiment of the present application, the prompt word includes action embedding features corresponding to multiple action types respectively, the multiple prompt words are arranged in sequence, and the arrangement positions of the multiple prompt words correspond to multiple specified moments in a future specified time period.

[0159] According to an embodiment of the present application, the driving action information includes at least one of the following: driving speed data for a specified time; driving curvature data for a specified time; and driving heading angle data for a specified time.

[0160] According to an embodiment of the present application, the control module includes: a first determining submodule and a control submodule.

[0161] The first determination submodule is configured to determine a power control instruction and a steering control instruction based on the driving speed data and the driving curvature data in the action information.

[0162] The control submodule is used to control the vehicle to travel in a specified future time period based on the power control command and the steering control command.

[0163] According to an embodiment of the present application, the device for controlling a vehicle further includes: a first determination module, a first acquisition module, a trajectory point acquisition module, and a second determination module.

[0164] The first determining module is configured to determine driving speed data and driving curvature data respectively associated with a plurality of designated moments from the driving action information, wherein the future designated period includes the plurality of designated moments.

[0165] The first obtaining module is used to perform an integration operation on the driving speed data and the driving curvature data respectively related to a plurality of designated moments to obtain the driving heading angles for the plurality of designated moments.

[0166] The trajectory point acquisition module is used to plan trajectory points based on the driving heading angle and driving speed data for multiple specified moments to obtain multiple planned trajectory points.

[0167] The second determining module is used to determine the planned driving trajectory based on multiple planned trajectory points.

[0168] According to an embodiment of the present application, the trajectory point acquisition module includes: a velocity component acquisition submodule and a planned trajectory point acquisition submodule.

[0169] The speed component obtaining submodule is used to calculate the speed component according to the driving heading angle and driving speed data at any specified moment, so as to obtain the speed component at any specified moment.

[0170] The planned trajectory point acquisition submodule is used to perform an integral calculation operation on the velocity components for multiple specified moments to obtain the planned trajectory points for the specified moments.

[0171] According to an embodiment of the present application, the multimodal perception data includes at least two of the following: image modal data representing the driving environment of the vehicle; text modal data representing the driving intention of controlling the vehicle; and map data related to the driving environment.

[0172] Figure 7 A structural block diagram of an apparatus for training a multimodal model according to an embodiment of the present application is shown.

[0173] like Figure 7 As shown, the apparatus 700 for training a multimodal model in this embodiment includes a sample receiving module 710 , a sample driving action information obtaining module 720 and a training module 730 .

[0174] The sample receiving module 710 is used to receive sample multimodal perception data and labeled driving action information for vehicle driving, where the labeled driving action information represents the driving action of the vehicle in the sample specified time period.

[0175] The sample driving action information acquisition module 720 is used to plan the vehicle's driving action based on the sample prompt words using the multimodal model and the sample multimodal perception data, and obtain the sample driving action information for the sample specified time period, wherein the sample prompt words are used to control the multimodal model to use the sample driving action information as the target and perform semantic understanding on the sample multimodal perception data.

[0176] The training module 730 is used to train the multimodal model based on the sample driving action information and the labeled driving action information to obtain a trained multimodal model.

[0177] According to an embodiment of the present application, the sample driving action information and the label driving action information are respectively the first parameter value and the second parameter value corresponding to the sample action type; wherein the training module includes: a loss value determination submodule and a training submodule.

[0178] a loss value determination submodule, configured to process the first parameter value and the second parameter value using a loss function to obtain a loss value; and

[0179] The training submodule is used to train the multimodal model based on the loss value.

[0180] According to an embodiment of the present application, the sample prompt words include sample action embedding features corresponding to multiple sample action types, the multiple prompt words are arranged in sequence, and the arrangement positions of the multiple prompt words correspond to multiple specified moments in a future specified time period.

[0181] An embodiment of the present application also provides a vehicle, comprising: a processor configured to execute the method for controlling the vehicle provided in the embodiment of the present application.

[0182] Figure 8 A block diagram of an electronic device suitable for implementing a method for controlling a vehicle and a method for training a multimodal model according to an embodiment of the present application is shown.

[0183] like Figure 8As shown, an electronic device 800 according to an embodiment of the present application includes a processor 801, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0184] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0185] According to an embodiment of the present application, electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to bus 804. Electronic device 800 may also include one or more of the following components connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or modem. Communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 810 as needed, so that computer programs read from the removable media can be installed into storage section 808 as needed.

[0186] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0187] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0188] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided in the embodiments of the present application.

[0189] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the processor 801 executes the computer program. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0190] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0191] In such an embodiment, the computer program can be downloaded and installed from the network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0192] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0193] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0194] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0195] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A method for controlling a vehicle, characterized in that: The method comprises: receiving multimodal perception data for vehicle driving; Using multiple action embedding features corresponding to the driving speed type and the driving curvature type as prompt words, the trained multimodal model is used to perform feature fusion on the multimodal perception data to obtain intermediate fusion features related to multiple specified moments in the future specified time period; The multimodal model processes the intermediate fusion features to output the driving speed data and the driving curvature data at the plurality of designated moments, the prompt word being used to control the multimodal model to use the driving speed data and the driving curvature data as targets to perform semantic understanding on the multimodal perception data, the arrangement positions of the plurality of action embedding features corresponding to the plurality of designated moments, and the driving speed data and the driving curvature data being numerical data in a preset coordinate system; determining a power control command and a steering control command based on the driving speed data and the driving curvature data; and Based on the power control command and the steering control command, the vehicle is controlled to travel during the future specified period.

2. The method according to claim 1, characterized in that The utilizing the multimodal model to perform feature fusion on the multimodal perception data includes: Determining a perceptual feature representing perceptual data of each modality in the multimodal perceptual data; Based on the multiple action embedding features arranged in sequence as prompt words, the multiple perception features are fused according to the attention mechanism to obtain intermediate fusion features related to the multiple specified moments in the future specified time period.

3. The method according to claim 2, characterized in that The method of utilizing the multimodal model to process the intermediate fusion features to output the driving speed data and the driving curvature data at the plurality of designated moments includes: fusing the plurality of intermediate fusion features based on an attention mechanism to obtain a plurality of target fusion features that are respectively related to the specified moments; and Driving action detection is performed based on the multiple target fusion features to obtain the driving speed data and driving curvature data at the multiple specified moments.

4. The method according to claim 1, wherein The action embedding feature is determined based on the following operations: determining a plurality of preset text tags for the multimodal model; Data fusion is performed on the text embedding features corresponding to each of the plurality of preset text tags to obtain an action embedding feature for at least one action type.

5. The method according to claim 1, characterized in that The method further comprises: determining travel speed data and travel curvature data associated with each of a plurality of specified moments; performing an integration operation on the driving speed data and the driving curvature data respectively associated with the plurality of designated moments to obtain the driving heading angles for the plurality of designated moments; Performing trajectory point planning based on the driving heading angle and driving speed data for the plurality of designated moments to obtain a plurality of planned trajectory points; and The planned driving trajectory is determined based on a plurality of the planned trajectory points.

6. The method according to claim 5, characterized in that The performing trajectory point planning based on the driving heading angle and driving speed data for the plurality of designated moments includes: For the driving heading angle and driving speed data at any specified moment, calculating the speed component based on the driving heading angle and driving speed data related to the specified moment to obtain the speed component at the specified moment; An integral calculation operation is performed on the velocity components for the plurality of designated moments to obtain a planned trajectory point for the designated moment.

7. The method according to claim 1, characterized in that The multimodal sensing data includes at least two of the following: Image modality data representing the driving environment of the vehicle; Text modal data representing a driving intention to control the vehicle; Map data related to the driving environment.

8. A method for training a multimodal model, characterized in that The method comprises: Receiving sample multimodal perception data and labeled driving action information for vehicle driving, wherein the labeled driving action information represents the driving action of the vehicle in a sample specified time period; The sample action embedding feature representing the sample action type is used as the sample prompt word, and the multimodal model is used to perform feature fusion on the sample multimodal perception data to obtain the intermediate fusion features of the sample related to multiple specified moments in the specified time period of the sample; Outputting sample driving action information by processing the sample intermediate fusion features using the multimodal model, the sample driving action information including sample driving speed data and sample driving curvature data at the plurality of specified moments, the sample driving speed data and the sample driving curvature data being data of numerical type in a preset coordinate system, the sample action types including driving speed types and driving curvature types, the sample prompt words being used to target the sample driving speed data and sample driving curvature data corresponding to the sample action types, and performing semantic understanding on the sample multimodal perception data, wherein the arrangement positions of the plurality of sample action embedded features correspond to the plurality of specified moments; The multimodal model is trained based on the sample driving action information and the label driving action information to obtain a trained multimodal model.

9. The method according to claim 8, characterized in that The sample driving action information and the labeled driving action information are respectively a first parameter value and a second parameter value corresponding to the sample action type; wherein training the multimodal model based on the sample driving action information and the labeled driving action information includes: Processing the first parameter value and the second parameter value using a loss function to obtain a loss value; and The multimodal model is trained based on the loss value.

10. A device for controlling a vehicle, characterized in that: The device comprises: A perception data receiving module, configured to receive multimodal perception data for vehicle driving; a driving action information acquisition module, configured to use an action embedding feature representing an action type as a prompt word, utilize a trained multimodal model to plan the vehicle's driving action based on the multimodal perception data, and obtain driving action information of a numerical type in a preset coordinate system, wherein the driving action information is used for a future specified period of time and is related to the action type; and a control module, configured to control the vehicle to travel based on the travel action information; Wherein, the driving action information acquisition module is configured as follows: Using multiple action embedding features corresponding to the driving speed type and the driving curvature type as prompt words, the trained multimodal model is used to perform feature fusion on the multimodal perception data to obtain intermediate fusion features related to multiple specified moments in the future specified time period; The multimodal model processes the intermediate fusion features to output the driving speed data and the driving curvature data at the plurality of designated moments, the prompt word being used to control the multimodal model to use the driving speed data and the driving curvature data as targets to perform semantic understanding on the multimodal perception data, and the arrangement positions of the plurality of action embedding features corresponding to the plurality of designated moments; Wherein, the control module is configured as follows: determining a power control command and a steering control command based on the driving speed data and the driving curvature data; and Based on the power control command and the steering control command, the vehicle is controlled to travel during the future specified period.

11. A device for training a multimodal model, characterized in that: The device comprises: A sample receiving module, configured to receive sample multimodal perception data and labeled driving action information for vehicle driving, wherein the labeled driving action information represents the driving action of the vehicle in a sample specified time period; a sample driving action information acquisition module, configured to use a sample action embedded feature representing a sample action type as a sample prompt word, utilize a multimodal model to plan the vehicle's driving action based on the sample multimodal perception data, and obtain sample driving action information of a numerical type in a preset coordinate system, wherein the sample prompt word is used to target the driving action information corresponding to the action type, and to perform semantic understanding on the sample multimodal perception data, wherein the sample driving action information is used for a sample specified future time period and is related to the sample action type; a training module, configured to train the multimodal model based on the sample driving action information and the labeled driving action information to obtain a trained multimodal model; The sample driving action information acquisition module is configured as follows: The sample action embedding feature representing the sample action type is used as the sample prompt word, and the multimodal model is used to perform feature fusion on the sample multimodal perception data to obtain the intermediate fusion features of the sample related to multiple specified moments in the specified time period of the sample; The multimodal model is used to process the sample intermediate fusion features to output sample driving action information, wherein the sample driving action information includes sample driving speed data and sample driving curvature data at multiple specified moments, wherein the sample driving speed data and sample driving curvature data are data of numerical type in a preset coordinate system, and the sample action type includes a driving speed type and a driving curvature type. The sample prompt word is used to take the sample driving speed data and sample driving curvature data corresponding to the sample action type as targets, and perform semantic understanding on the sample multimodal perception data. The arrangement positions of the multiple sample action embedded features correspond to the multiple specified moments.

12. A vehicle, characterized in that: include: A processor configured to execute the method according to any one of claims 1 to 7.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 9.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

15. A computer program product, characterized in that A computer program is included which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images