Vehicle control method and vehicle

Abstract

The embodiment of the invention provides a vehicle control method and a vehicle. The method comprises the steps of obtaining target query information corresponding to a target vehicle driving track prediction task according to obtained multiple pieces of image feature information and query information corresponding to an obstacle track prediction task, a map element prediction task and the target vehicle driving track prediction task, and according to target query information corresponding to the target vehicle traveling track prediction task, obtaining multiple pieces of target vehicle traveling track information corresponding to the multiple pieces of target vehicle traveling track granularity, and according to the multiple pieces of target vehicle traveling track information corresponding to the multiple pieces of target vehicle traveling track granularity, controlling the target vehicle to travel. Thus, the query information corresponding to the target vehicle driving track prediction task is obtained according to the query information of the multiple tasks and the multiple pieces of image feature information, more comprehensive and accurate multiple pieces of target vehicle driving track information corresponding to the multiple pieces of target vehicle driving track granularity can be obtained, and vehicle auxiliary driving control can be better achieved.

Description

Technical Field

[0001] The present application relates to the field of vehicle assisted driving technology, and in particular to a vehicle control method and a vehicle. Background Art

[0002] With the continuous development of vehicles, assisted driving technology is becoming a key development direction in the current and future automotive fields. Assisted driving technology uses advanced perception and decision-making systems to perceive the surrounding environment, predict the vehicle's driving trajectory based on this information, and then control the vehicle's driving trajectory to achieve assisted driving. The goal is to reduce driver fatigue and enhance driving safety. Therefore, the development of more comprehensive and accurate trajectory prediction for better assisted driving control has become particularly important. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, the embodiments of the present application provide a vehicle control method and a vehicle, which can predict the driving trajectory more comprehensively and accurately to better perform vehicle assisted driving control, make vehicle assisted driving control safer, and better meet actual driving needs, thereby improving user experience.

[0004] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a vehicle control method, which includes: determining multiple image information corresponding to different perspectives corresponding to a target vehicle, performing image feature extraction on the multiple image information, and obtaining multiple image feature information; determining query information corresponding to a target task, the target tasks including an obstacle trajectory prediction task, a map element prediction task, and a target vehicle driving trajectory prediction task; obtaining target query information corresponding to the target vehicle driving trajectory prediction task based on multiple image feature information and query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task; obtaining multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities based on the target query information corresponding to the target vehicle driving trajectory prediction task; and controlling the driving of the target vehicle based on the multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities.

[0005] Using the above technical solution, target query information corresponding to the target vehicle trajectory prediction task is obtained based on multiple image feature information and query information corresponding to multiple tasks: obstacle trajectory prediction tasks, map element prediction tasks, and target vehicle trajectory prediction tasks. Based on the target query information corresponding to the target vehicle trajectory prediction task, multiple target vehicle trajectory information corresponding to multiple target vehicle trajectory granularities is obtained. The target vehicle is then controlled based on the multiple target vehicle trajectory information corresponding to the multiple target vehicle trajectory granularities. In this way, by fusing multiple image feature information with query information from multiple tasks, multiple target vehicle trajectory information corresponding to multiple target vehicle trajectory granularities is obtained, and the target vehicle is then controlled based on the multiple target vehicle trajectory information corresponding to the multiple target vehicle trajectory granularities. Multi-task prediction makes the predicted target vehicle trajectory information corresponding to each target vehicle trajectory granularity more comprehensive and accurate. Consequently, vehicle driving is controlled based on the multiple target vehicle trajectory information corresponding to the multiple target vehicle trajectory granularities, enabling more comprehensive and accurate vehicle assisted driving control. This means better assisted driving control, safer vehicle assisted driving control, and better meeting actual driving needs, improving user experience.

[0006] In a possible implementation of the first aspect above, the multiple target vehicle driving trajectory granularities include time trajectories corresponding to different time intervals, spatial trajectories corresponding to different spatial intervals, trajectories corresponding to different driving styles, and trajectories corresponding to different driving scenarios.

[0007] By adopting the above technical solution, the target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities is determined, and the vehicle driving is controlled according to the target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities, which can better and more comprehensively perform vehicle automatic driving control and improve user experience.

[0008] In a possible implementation of the first aspect above, based on the target query information corresponding to the target vehicle driving trajectory prediction task, multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities are obtained, including: dividing the target query information corresponding to the target vehicle driving trajectory prediction task into target vehicle driving trajectory granularities to obtain target query information corresponding to multiple target vehicle driving trajectory granularities, the target query information corresponding to each target vehicle driving trajectory granularity includes target query information corresponding to multiple driving states; fusing the target query information corresponding to the same driving state at each target vehicle driving trajectory granularity to obtain target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state; and predicting target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities for each driving state based on the target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state based on a multi-layer perceptron.

[0009] Using the above technical solution, the target query information corresponding to the target vehicle driving trajectory prediction task is divided into target vehicle driving trajectory granularity. The target query information corresponding to each target vehicle driving trajectory granularity is then divided into different driving states. The target query information of each target vehicle driving trajectory granularity under the same driving state is then fused to obtain target vehicle driving trajectory query information of different target vehicle driving trajectory granularity under different driving states, and target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularity under different driving states is obtained. In this way, vehicle driving is controlled based on target vehicle driving trajectory information of different target vehicle driving trajectory granularity under a certain driving state. This allows the target vehicle to be controlled based on target vehicle driving trajectory information of various driving requirements and driving scenarios, making vehicle assisted driving control more precise.

[0010] In a possible implementation of the first aspect above, controlling the driving of a target vehicle according to multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities includes: determining a target driving state from multiple driving states, and controlling different driving execution devices of the target vehicle according to the target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities of the target driving state to control the driving of the target vehicle.

[0011] By adopting the above technical solution, different execution devices of the target vehicle are controlled based on the target vehicle driving trajectory information of different target vehicle driving trajectory granularities under a certain driving state to realize assisted driving of the target vehicle. The target vehicle driving trajectory information of various driving needs and driving scenarios can be integrated to realize different control of different driving execution devices of the target vehicle, making the vehicle assisted driving control more comprehensive.

[0012] In a possible implementation of the first aspect above, the query information includes initial query information and historical query information, and the target query information corresponding to the target vehicle driving trajectory prediction task is obtained based on multiple image feature information and query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task, including: obtaining the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task based on the initial query information and historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task; obtaining the context query information corresponding to the target vehicle driving trajectory prediction task based on the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task based on the cross-attention mechanism; obtaining the target query information of the target vehicle driving trajectory prediction task based on the context query information of the target vehicle driving trajectory prediction task and multiple image feature information based on the deformable attention mechanism.

[0013] By adopting the above technical solution, the time series query information corresponding to the obstacle trajectory prediction task, map element prediction task and target vehicle driving trajectory prediction task is cross-learned based on the cross-attention mechanism to obtain more comprehensive and accurate context query information corresponding to the target vehicle driving trajectory prediction task.

[0014] In a possible implementation of the first aspect above, based on the initial query information and historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task, the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task is obtained, including: based on the cross-attention mechanism, based on the initial query information of the obstacle trajectory prediction task and the historical query information of the obstacle trajectory prediction task, the time series query information of the obstacle trajectory prediction task is obtained; based on the cross-attention mechanism, based on the initial query information of the map element prediction task and the historical query information of the map element prediction task, the time series query information of the map element prediction task is obtained; based on the cross-attention mechanism, based on the initial query information of the target vehicle driving trajectory prediction task and the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task, the time series query information of the target vehicle driving trajectory prediction task is obtained.

[0015] Using the above technical solution, the historical query information corresponding to the obstacle trajectory prediction task, map element prediction task, and target vehicle driving trajectory prediction task, as well as the initial query information of the target vehicle driving trajectory prediction task, are cross-learned to obtain more comprehensive and accurate time series query information for the target vehicle driving trajectory prediction task.

[0016] In a possible implementation of the first aspect above, the method further includes: obtaining context query information of the obstacle trajectory prediction task based on the time series query information of the obstacle trajectory prediction task based on the self-attention mechanism, and obtaining context query information of the map element prediction task based on the time series query information of the map element prediction task; obtaining target query information corresponding to the obstacle trajectory prediction task based on multiple image feature information and the context query information of the obstacle detection task based on the deformable attention mechanism, and obtaining target query information corresponding to the map element prediction task based on multiple image feature information and the context query information of the map element prediction task; predicting obstacle trajectory information based on the target query information corresponding to the obstacle trajectory prediction task based on a multi-layer perceptron, and predicting map element information based on the target query information corresponding to the map element prediction task; and controlling the driving of the target vehicle based on multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities, as well as the obstacle trajectory information and the map element information.

[0017] By adopting the above technical solution, the vehicle driving is controlled based on the predicted obstacle trajectory information and map element information, so that the information based on which the vehicle assisted driving control is based is more abundant and more comprehensive, and the vehicle assisted driving control is safer.

[0018] In a possible implementation of the first aspect above, the method further includes: using the obtained target query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task as the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task involved in the next control of the target vehicle's driving.

[0019] On the second aspect, the implementation method of the present application also discloses a vehicle control method, including determining multiple image information corresponding to different perspectives corresponding to the target vehicle, and determining query information corresponding to the target task, the target task including an obstacle trajectory prediction task, a map element prediction task and a target vehicle driving trajectory prediction task; inputting the multiple image information and the query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task into the target machine learning model, so that the target machine learning model performs image feature extraction on the multiple image information to obtain multiple image feature information, and obtains target query information corresponding to the target vehicle driving trajectory prediction task based on the multiple image feature information and the query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task; obtains multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities based on the target query information corresponding to the target vehicle driving trajectory prediction task; and controls the driving of the target vehicle according to the multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities.

[0020] In one implementation of the second aspect above, the query information includes initial query information and historical query information, and the target machine learning model obtains the target query information corresponding to the target vehicle driving trajectory prediction task in the following manner: the initial query information and historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task are input into the spatiotemporal interaction module of the target machine learning model to obtain the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task; the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task is input into the collaborative interaction module of the target machine learning model to obtain the context query information corresponding to the target vehicle driving trajectory prediction task; the multiple image feature information and the context query information corresponding to the target vehicle driving trajectory prediction task are input into the deformable attention module of the target machine learning model to obtain the target query information corresponding to the target vehicle driving trajectory prediction task.

[0021] In an implementation of the second aspect above, the method further includes: obtaining context query information of the obstacle trajectory prediction task based on the time series query information corresponding to the obstacle trajectory prediction task based on the collaborative interaction module, and obtaining context query information of the map element prediction task based on the time series query information of the map element prediction task; obtaining target query information corresponding to the obstacle detection task based on multiple image feature information and the context query information of the obstacle trajectory prediction task based on the deformable attention module, and obtaining target query information corresponding to the map element prediction task based on multiple image feature information and the context query information of the map element prediction task.

[0022] In an implementation of the second aspect above, based on the target query information corresponding to the target vehicle driving trajectory prediction task, multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities is obtained, including: inputting the target query information corresponding to the target vehicle driving trajectory prediction task into the multilayer perceptron of the target machine learning model, so that the multilayer perceptron predicts multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities based on the target query information of the target vehicle driving trajectory prediction task.

[0023] In an implementation of the second aspect above, the method further includes inputting target query information corresponding to the obstacle trajectory prediction task into a multilayer perceptron to obtain obstacle trajectory information, inputting target query information corresponding to the map element prediction task into the multilayer perceptron to obtain map element information, and controlling the driving of the target vehicle based on multiple target vehicle driving trajectory information corresponding to the granularity of the multiple target vehicles, as well as the obstacle trajectory information and the map element information.

[0024] By adopting the above technical solution, the target machine learning model takes multiple image information corresponding to different perspectives of the target vehicle and query information of the obstacle trajectory prediction task, map element prediction task, and target vehicle driving trajectory prediction task as input, and directly outputs the predicted obstacle trajectory information, map element information, and multiple target vehicle driving trajectory information corresponding to the granularity of multiple target vehicle driving trajectories. The end-to-end data processing process reduces the data format conversion processing steps and speeds up the generation efficiency of the prediction results.

[0025] On the third aspect, the implementation of the present application also discloses a vehicle control method, which is applied to a cloud server, and the cloud server is used to implement the vehicle control method provided by any implementation method of the first aspect above, or to implement the vehicle control method provided by the second aspect above.

[0026] In a fourth aspect, the implementation of the present application further discloses a vehicle for implementing the vehicle control method provided by any one of the implementations of the first aspect, or implementing the vehicle control method provided by the second aspect.

[0027] In the fifth aspect, the implementation method of the present application also discloses a computer-readable storage medium, which stores a computer program. The computer program can be executed by a vehicle to implement the vehicle automatic driving control method provided by any implementation method of the above-mentioned first aspect, or to implement the vehicle control method provided by the above-mentioned second aspect.

[0028] In the sixth aspect, the implementation method of the present application also discloses a computer program product, including a computer program. When the computer program is executed by a vehicle, it implements the vehicle automatic driving control method provided by any implementation method of the above-mentioned first aspect, or implements the vehicle control method provided by the above-mentioned second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings used in the description of the implementation methods.

[0030] Figure 1 A schematic flow chart of a vehicle control method provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a process for determining target query information corresponding to a target vehicle driving trajectory prediction task provided by an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of a process for determining the time series query information corresponding to each task provided by an embodiment of the present invention;

[0033] Figure 4A schematic diagram of a process for determining target vehicle driving trajectory information of each target vehicle driving trajectory granularity provided by an embodiment of the present invention;

[0034] Figure 5 A schematic diagram illustrating the principle of granularity division of target vehicle driving trajectory and driving state division provided by an embodiment of the present invention;

[0035] Figure 6 Another schematic flow chart of a vehicle control method provided by an embodiment of the present invention;

[0036] Figure 7 Schematic diagram of the principles of obstacle trajectory prediction, map element prediction, and target vehicle trajectory prediction based on a target machine learning model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] As mentioned above, it is particularly important to predict vehicle driving trajectories more comprehensively and accurately in order to better perform vehicle assisted driving control.

[0038] Currently, end-to-end assisted driving technology is gradually attracting attention. End-to-end assisted driving technology uses a single deep learning model to directly map the raw data collected by sensors (such as image information corresponding to different perspectives) into driving trajectories sampled at equal intervals, and then controls vehicle driving based on the driving trajectory.

[0039] End-to-end assisted driving technologies primarily include Vectorized Autonomous Driving (VAD) and Unified Autonomous Driving (Uniad). VAD focuses on improving dynamic perception and planning efficiency through efficient scene representation and vectorization technology, making it suitable for scenarios with high real-time requirements. UniAad integrates multiple tasks through a unified framework, strengthening inter-module collaboration and global decision-making capabilities, making it suitable for generalized applications in complex scenarios.

[0040] The input to VAD and Nniad is sensor data (such as image information), and the output is vehicle trajectory information sampled at equal intervals. This refers to the plotting of predicted path points at equal time intervals (for example, one path point every 5 seconds) to obtain a trajectory consisting of multiple path points. However, current assisted driving trajectory predictions are limited to single trajectory predictions. Learning vehicle trajectories based on deep learning models is difficult, resulting in poor learning outcomes and low predicted trajectory accuracy. Furthermore, vehicle control based on a single trajectory cannot meet diverse driving scenarios.

[0041] Based on this, the present application provides a vehicle control method that obtains target query information corresponding to a target vehicle driving trajectory prediction task by fusing query information and image feature information from multiple tasks. Furthermore, based on the target query information corresponding to the target vehicle driving trajectory prediction task, multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities is determined, so as to control the driving of the target vehicle based on the multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities. In this way, through multi-task prediction, the target vehicle driving trajectory information corresponding to each target vehicle driving trajectory granularity obtained can be made more comprehensive and accurate, and vehicle driving can be controlled based on the multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities, thereby enabling better and more comprehensive vehicle assisted driving control.

[0042] Next, the vehicle control method provided by the implementation of this application is described in detail.

[0043] like Figure 1 As shown, the vehicle control method provided by the implementation of this application specifically includes the following steps.

[0044] S100, determining a plurality of image information corresponding to different viewing angles corresponding to a target vehicle, performing image feature extraction on the plurality of image information, and obtaining a plurality of image feature information.

[0045] S200, determining query information corresponding to a target task, where the target task includes an obstacle trajectory prediction task, a map element prediction task, and a target vehicle driving trajectory prediction task.

[0046] S300 , obtaining target query information corresponding to the target vehicle driving trajectory prediction task based on multiple image feature information and query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task.

[0047] S400 , obtaining a plurality of target vehicle driving trajectory information corresponding to a plurality of target vehicle driving trajectory granularities according to target query information corresponding to a target vehicle driving trajectory prediction task.

[0048] S500 , controlling the target vehicle to travel according to multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities.

[0049] In the implementation of this application, target query information corresponding to the target vehicle trajectory prediction task is obtained based on multiple image feature information and query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle trajectory prediction task. Based on the target query information corresponding to the target vehicle trajectory prediction task, multiple target vehicle trajectory information corresponding to multiple target vehicle trajectory granularities is obtained. The target vehicle is then controlled based on the multiple target vehicle trajectory information corresponding to the multiple target vehicle trajectory granularities. In this way, by fusing the image feature information with the query information of each task, multiple target vehicle trajectory information corresponding to multiple target vehicle trajectory granularities is obtained, and vehicle driving is controlled based on the target vehicle trajectory information corresponding to different target vehicle trajectory granularities. Multi-task prediction makes the predicted target vehicle trajectory information corresponding to each target vehicle trajectory granularity more comprehensive and accurate. Vehicle driving is then controlled based on the multiple target vehicle trajectory information corresponding to multiple target vehicle trajectory granularities, enabling better and more comprehensive vehicle assisted driving control, better meeting actual driving needs, and improving the user experience.

[0050] In the implementation of this application, the target vehicle trajectory prediction is achieved based on a deep learning model (as an example of a target machine learning model). For example, the deep learning model is an end-to-end transformer model.

[0051] First, step S100 is executed to capture N perspectives of the target vehicle (i.e., multi-perspective images) based on a device such as a 360-degree surround view camera of the target vehicle, which are recorded as img_0-img_n (as an example of multiple image information corresponding to different perspectives).

[0052] Through the Convolutional Neural Network (CNN), the multi-view images of the surround view are feature extracted to obtain multiple image features img_feats, which are recorded as img_feats=CNN(img_0, ..., img_n) (as an example of multiple image feature information).

[0053] Among them, CNN networks such as resnet50 can be used to perform image feature extraction processing on multi-view images.

[0054] Furthermore, step S200 is executed to determine query information of the target task, wherein the target task includes the obstacle trajectory prediction task, the map element prediction task, and the ego vehicle driving trajectory prediction task (ie, the target vehicle driving trajectory prediction task).

[0055] The obstacle trajectory prediction task involves both obstacle detection and obstacle trajectory prediction. Obstacles can be pedestrians, vehicles, or roadblocks. If the obstacle is a pedestrian, the obstacle trajectory prediction task is a pedestrian trajectory prediction task. If the obstacle is a vehicle, the obstacle trajectory prediction task is a vehicle trajectory prediction task. If the obstacle is a roadblock, the obstacle trajectory prediction task is a roadblock location detection task.

[0056] Furthermore, the query information includes initial query information and historical query information.

[0057] Exemplarily, queries for multiple tasks are initialized (as an example of initial query information for each task). The queries for each task include an od query, denoted as q_od (as an example of initial query information corresponding to the obstacle trajectory prediction task), a map query, denoted as q_map (as an example of initial query information corresponding to the map element prediction task), and a plan query, denoted as q_plan (as an example of initial query information corresponding to the target vehicle driving trajectory prediction task).

[0058] Where q_od, q_map, and q_plan are real vectors of (N_od, 256), (N_map, 256), and (N_plan, 256), respectively. 256 is the vector dimension, but other values are possible. N_od, N_map, and N_plan are the number of vectors.

[0059] Furthermore, we obtain the od query of the history frame of each task, recorded as history_q_od (as the historical query information corresponding to the obstacle trajectory prediction task), the map query, recorded as history_q_map (as the historical query information corresponding to the map element prediction task), and the plan query, recorded as history_q_plan (as the historical query information corresponding to the target vehicle trajectory prediction task).

[0060] Furthermore, step S300 is executed to obtain target query information corresponding to the target vehicle trajectory prediction task based on the multiple image feature information and query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle trajectory prediction task. The target query information is obtained by fusing the multiple image feature information and query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle trajectory prediction task, and performing feature extraction.

[0061] In one implementation of this application, Figure 2 As shown, obtaining the target query information corresponding to the target vehicle driving trajectory prediction task includes the following steps.

[0062] S310 , obtaining time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task based on the initial query information and the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task.

[0063] For example, the cross attention mechanism is used to extract the temporal query information of each task and update itself.

[0064] In one implementation of this application, Figure 3 As shown, obtaining the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task includes the following steps.

[0065] S311 , based on the cross-attention mechanism, obtain the temporal query information corresponding to the obstacle trajectory prediction task according to the initial query information of the obstacle trajectory prediction task and the historical query information of the obstacle trajectory prediction task.

[0066] For example, crossattention is used to extract temporal information from q_od of the obstacle trajectory prediction task and history_q_od of the history frame, and the information is updated, which is recorded as q_od = cross_atten(q_od, history_q_od) (as an example of temporal query information for the obstacle trajectory prediction task).

[0067] S312 , obtaining temporal query information corresponding to the map element prediction task based on the initial query information of the map element prediction task and the historical query information of the map element prediction task based on the cross attention mechanism.

[0068] Exemplarily, crossattention is used to extract temporal information from q_map of the map element prediction task and history_q_map of the history frame, and the information is updated, which is recorded as q_map=cross_atten(q_map, history_q_map) (as an example of temporal query information for the map element prediction task).

[0069] S313, based on the cross-attention mechanism, obtains the time series query information corresponding to the target vehicle driving trajectory prediction task according to the initial query information of the target vehicle driving trajectory prediction task and the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task.

[0070] Exemplarily, cross attention is used to extract temporal information from the q_plan of the target vehicle driving trajectory prediction task and the history_q_od, history_q_map, and history_q_plan of the history frame, and update itself, which is recorded as q_plan=cross_atten(q_plan, [history_q_od, history_q_map, history_q_plan]) (as an example of temporal query information for the target vehicle driving trajectory prediction task).

[0071] It should be noted that using cross attention to extract temporal query information is a mature technique, widely used in methods such as VAD and UniAD. While q_od and q_map only need to perform cross attention with their own historical query information, q_plan needs to perform cross attention with the historical query information of each task.

[0072] S320, based on the cross-attention mechanism, obtains context query information of the target vehicle driving trajectory prediction task according to the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task.

[0073] Exemplarily, cross attention is used to extract context information from q_plan of the target vehicle trajectory prediction task, q_od of the obstacle trajectory prediction task, and q_map of the map element prediction task, and update themselves, which is recorded as q_plan=cross_atten(q_plan, [q_od, q_map]) (as an example of context query information for the target vehicle trajectory prediction task).

[0074] Furthermore, in another implementation of the present application, based on the self-attention mechanism, the context query information of the obstacle trajectory prediction task is obtained according to the temporal query information of the obstacle trajectory prediction task, and the context query information of the map element prediction task is obtained according to the temporal query information of the map element prediction task.

[0075] For example, the q_od of the obstacle trajectory prediction task is self-updated using self attention to obtain a new q_od for the obstacle trajectory prediction task, which is recorded as q_od = self_atten(q_od, q_od) (as an example of context query information for the obstacle trajectory prediction task).

[0076] In addition, the q_map of the map element prediction task is self-updated using self attention to obtain a new q_map for the map element prediction task, which is denoted as q_map=self_atten(q_map, q_map) (as an example of contextual query information for the map element prediction task).

[0077] It should be noted that the query that uses self attention and cross attention to extract contextual information is a mature technology and is widely used in methods such as VAD and UNIAD.

[0078] S330 , based on a deformable attention mechanism, obtains target query information corresponding to the target vehicle driving trajectory prediction task according to the context query information of the target vehicle driving trajectory prediction task and multiple image feature information.

[0079] Exemplarily, deformable attention is used to extract image information and context query information of q_plan and img_feats of the target vehicle driving trajectory prediction task, and then update itself to obtain the query information (plan query) of the final target vehicle driving trajectory prediction task of this frame, which is recorded as q_plan=deform_atten(q_plan,img_feats) (as an example of the target query information of the target vehicle driving trajectory prediction task).

[0080] Furthermore, in another implementation of the present application, based on the deformable attention mechanism, the target query information corresponding to the obstacle trajectory prediction task is obtained according to multiple image feature information and the context query information of the obstacle trajectory prediction task, and the target query information corresponding to the map element prediction task is obtained according to multiple image feature information and the context query information of the map element prediction task.

[0081] For example, deformable attention is used to extract image information and contextual query information of q_od and img_feats of the obstacle detection task, and the query information of q_od is updated to obtain the query information (od query) of the final obstacle trajectory prediction task of this frame, which is recorded as q_od = deform_atten(q_od, img_feats) (as an example of the target query information of the obstacle trajectory prediction task).

[0082] In addition, deformable attention is used to extract image information and context query information of q_map from q_map and img_feats of the map element prediction task, and the query information of q_map is updated. The final query information (map query) of the map element prediction task for this frame is obtained as q_map=deform_atten(q_map, img_feats) (as an example of image query information for the map element prediction task).

[0083] It should be noted that using deformable attention to extract query information from images is a mature technology and is widely used in methods such as sparse4d.

[0084] Furthermore, step S400 is executed to obtain a plurality of target vehicle driving trajectory information corresponding to a plurality of target vehicle driving trajectory granularities according to the target query information corresponding to the target vehicle driving trajectory prediction task.

[0085] Among them, in the implementation of the present application, the granularity of multiple target vehicle driving trajectories includes time trajectories corresponding to different time intervals, spatial trajectories corresponding to different spatial intervals, trajectories corresponding to different driving styles, and trajectories corresponding to different driving scenarios.

[0086] Exemplarily, the driving trajectory can obtain trajectory points according to time intervals, and multiple trajectory points form a driving trajectory information, wherein the time intervals can be equal intervals or unequal intervals. For example, an equal interval is 5 seconds. At 0 seconds, the vehicle is at point A. At 5 seconds, it is predicted that the vehicle will travel to point B. At 10 seconds, it is predicted that the vehicle will travel to point C. ..., and so on. The prediction is made according to one trajectory point every 5 seconds. Another example of an equal interval is 2 seconds. At 0 seconds, the vehicle is at point A. At 2 seconds, it is predicted that the vehicle will travel to point B. At 4 seconds, it is predicted that the vehicle will travel to point C. ..., and so on. The prediction is made according to one trajectory point every 2 seconds. For example, an unequal interval is 0 seconds. At 3 seconds, it is predicted that the vehicle will travel to point B. At 5 seconds, it is predicted that the vehicle will travel to point C. ..., and so on. The trajectory point prediction is made based on different time intervals.

[0087] Furthermore, the driving trajectory can be obtained by obtaining trajectory points based on distance intervals. Multiple trajectory points form a driving trajectory information. The spatial intervals can be equal or unequal. For example, if the vehicle starts at point A, it is predicted that the vehicle will be at point B after traveling 1 meter, at point C after traveling 2 meters, and so on. The prediction is made based on one trajectory point per meter. For unequal intervals, for example, if the vehicle starts at point A, it is predicted that the vehicle will be at point B after traveling 1 meter, at point C after traveling 2 meters, at point D after traveling 4 meters, and so on. The trajectory point prediction is made based on different distance intervals.

[0088] Furthermore, driving trajectories can be predicted based on driving style, such as aggressive or comfortable. An aggressive driving style indicates a driver's preference for speed. Therefore, trajectory point predictions can be made based on different driving speeds, generating driving trajectory information based on multiple trajectory points. For example, taking a speed of 80 km / h (i.e., an aggressive driving style), trajectory points corresponding to different time intervals or spatial intervals at that speed are predicted. For example, taking a speed of 50 km / h (i.e., a comfortable driving style), trajectory points corresponding to different time intervals or spatial intervals at that speed are predicted.

[0089] Driving trajectories can be predicted based on driving scenarios, such as reversing, underground parking, parallel parking, normal road driving, highway driving, and urban road driving. Based on these different driving scenarios, trajectory point predictions can be performed for each scenario, and driving trajectory information can be derived from multiple trajectory points. For example, trajectory points corresponding to different time intervals or spatial intervals can be predicted for each driving scenario.

[0090] Further, in the implementation of this application, see Figure 4 , according to the target query information corresponding to the target vehicle driving trajectory prediction task, a plurality of target vehicle driving trajectory information corresponding to a plurality of target vehicle driving trajectory granularities is obtained, including the following steps.

[0091] S410 , dividing the target query information corresponding to the target vehicle driving trajectory prediction task into target vehicle driving trajectory granularity to obtain target query information corresponding to multiple target vehicle driving trajectory granularities, where the target query information corresponding to each target vehicle driving trajectory granularity includes target query information corresponding to multiple driving states.

[0092] S420 , performing fusion processing on target query information corresponding to the same driving state at each target vehicle driving trajectory granularity to obtain target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state.

[0093] S430 , predicting target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities for each driving state based on the target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state based on a multi-layer perceptron.

[0094] For example, Figure 5As shown in the figure, the query corresponding to the target vehicle trajectory prediction task undergoes an align-fuse process, and then the multilayer perceptron (MLP) is used to predict the ego vehicle's trajectory information (also known as the target vehicle's trajectory information).

[0095] Specifically, the query corresponding to the target vehicle driving trajectory prediction task (i.e., target query information) is divided (expanded) into queries of different target vehicle driving trajectory granularities (as examples of target query information corresponding to each target vehicle driving trajectory granularity), namely multi-granularity driving trajectory query information (Multi-Granularity PlanningQuery), such as temporal trajectories corresponding to different time intervals (temporal), spatial trajectories corresponding to different spatial intervals (spatial), trajectories corresponding to different driving styles (driving-style), etc. Each granularity corresponds to queries of multiple modalities (i.e., driving states) (as examples of target query information corresponding to each driving state included in each target vehicle driving trajectory granularity), such as left turn, right turn, straight, U-turn, left lane change, right lane change, overtaking, and other driving states.

[0096] The target vehicle trajectory prediction task has N_plan as the target query information q_plan. This information is first divided (expanded) into multiple groups N_granu, called trajectory granularity. Each group contains M queries, called modes (i.e., driving states), where N_plan = N_granu * M.

[0097] Furthermore, each granularity corresponds to multiple modalities. For the same modality, it has its own true value at different driving trajectory granularities. For example, for a left-turn driving state, there are left-turn time trajectory, left-turn spatial trajectory, left-turn aggressive driving style trajectory, left-turn data driving style trajectory, etc.

[0098] like Figure 5The reference group shown is a query group of different modalities corresponding to a spatial trajectory granularity with a certain interval between trajectory points. The j-th query group is a query group of different modalities corresponding to a certain driving style (the i-th query group). That is, the temporal trajectory granularity includes trajectory granularities corresponding to different time intervals (e.g., multiple columns composed of boxes of different thicknesses corresponding to time in the figure), the spatial trajectory granularity includes trajectory granularities corresponding to different spatial intervals (e.g., multiple columns composed of boxes of different thicknesses corresponding to space in the figure), and the trajectory granularity corresponding to different driving styles (e.g., multiple columns composed of boxes of different thicknesses corresponding to driving styles in the figure). The trajectory granularity of each target vehicle includes target query information for each modality. Furthermore, the reference group is matched with the ground truth (GT).

[0099] The queries for each target vehicle trajectory granularity under the same modality are fused through addition (sum) to obtain the query corresponding to each target vehicle trajectory granularity of the modality, which is recorded as a fused plan query (i.e., query information, serving as an example of target vehicle trajectory query information corresponding to different target vehicle trajectory granularities). In this way, the same process is performed on each modality to obtain M fused plan queries.

[0100] That is, assuming q_plan_ij is the plan query for the target vehicle's trajectory granularity at mode j, then q_plan_fused_j is the plan query for different trajectory granularities under mode j. Then q_plan_fused_j = sum_i(q_plan_ij), and the output waypoints for each mode's trajectory granularity are output (outputs waypoints).

[0101] Taking the left-turn driving state as an example, the left-turn driving state corresponding to each target vehicle's driving trajectory granularity is fused to obtain the query information (fusedquery) of the left-turn driving state at different target vehicle driving trajectory granularities, and the waypoints are output, including the waypoint information corresponding to the temporal trajectory granularity of the left-turn driving state (temporal regheads): , ,……, , the waypoint information corresponding to the spatial trajectory granularity in the left-turn driving state (spatialreg heads): , ,……, , the waypoint information corresponding to different driving style trajectory granularity in the left turn state (driving-style reg heads): , ,……, .

[0102] Furthermore, based on MLP, the target vehicle driving trajectory information of each target vehicle driving trajectory granularity corresponding to each modality is predicted according to M fused plan queries.

[0103] Next, step S500 is executed to control the target vehicle to travel according to the multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities.

[0104] In one implementation of the present application, the target vehicle is controlled to travel according to multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities, including: determining a target driving state from multiple driving states, and controlling different driving execution devices of the target vehicle according to the target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities of the target driving state to control the travel of the target vehicle.

[0105] Exemplarily, a target driving state is determined from multiple driving states based on the vehicle's driving conditions on the current road, and different driving execution devices of the target vehicle are controlled based on target vehicle driving trajectory information of different target vehicle driving trajectory granularities corresponding to the target driving state to control the driving of the target vehicle.

[0106] For example, based on the vehicle's current driving conditions on the road, the target driving state is determined to be a straight driving state from multiple driving states. Based on the target vehicle driving trajectory information at different target vehicle driving trajectory granularities in the straight driving state, different driving execution devices of the target vehicle are controlled to control the driving of the target vehicle. For example, the steering wheel is controlled based on the target vehicle driving trajectory information corresponding to the spatial trajectory granularity with a high sampling interval in the straight driving state, and the throttle is controlled based on the target vehicle driving trajectory information corresponding to the temporal trajectory granularity with a high sampling interval in the straight driving state.

[0107] For another example, based on the vehicle's current driving conditions on the road, it is determined from multiple driving states that the target driving state is a left turn driving state, and it is determined that the user has selected an aggressive driving mode, then the vehicle throttle is controlled according to the target vehicle driving trajectory information corresponding to the aggressive driving style.

[0108] In the implementation of the present application, the target vehicle's driving trajectory is predicted and processed based on the multi-view images of the target vehicle in the current scene and the query information of each task. Therefore, the target vehicle's driving trajectory information determined is the target vehicle's driving trajectory information corresponding to each target vehicle's driving trajectory granularity under different driving states, determined based on the target vehicle's current driving needs. Therefore, after the driving state is determined, the target vehicle's driving trajectory information corresponding to each target vehicle's driving trajectory granularity under the same driving state may differ in driving angle and driving speed, but the driving direction is the same. Therefore, by controlling different execution devices of the vehicle based on the target vehicle's driving trajectory information corresponding to different target vehicle's driving trajectory granularity under the same driving state, the vehicle's driving trajectory can be obtained by fusing multiple target vehicle's driving trajectory information.

[0109] Furthermore, in one implementation of the present application, obstacle trajectory information (such as obstacle driving trajectory information, obstacle motion trajectory information, obstacle position information, etc.) is predicted based on the target query information corresponding to the obstacle trajectory prediction task based on a multilayer perceptron (MLP, also known as a multilayer perceptron), and map element information is predicted based on the target query information corresponding to the map element prediction task.

[0110] It should be noted that prediction through MLP is a mature technology and is widely used in methods such as vad and uniad.

[0111] Furthermore, in another implementation of the present application, vehicle driving is controlled based on the multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities, as well as obstacle trajectory information and / or map element information.

[0112] Exemplarily, each map element is displayed on a map interface so that when the target vehicle drives according to the target driving trajectory information, it drives according to the map composed of the map elements.

[0113] For example, each obstacle is displayed on a map interface so that when the target vehicle drives according to the target driving trajectory information, it can determine whether it can avoid each obstacle based on the position of each obstacle and the driving trajectory of each obstacle during driving.

[0114] For example, each map element and each obstacle is displayed on the map interface so that when the target vehicle is driving according to the target driving trajectory information, it can be determined that it is driving according to the map composed of map elements, and that each obstacle can be avoided based on the position of each obstacle and the driving trajectory of each obstacle during driving.

[0115] Furthermore, in the implementation method of the present application, the target query information corresponding to the obstacle trajectory prediction task, map element prediction task and target vehicle driving trajectory prediction task (the final q_od, q_map, q_plan) will be used as the historical query information corresponding to the obstacle trajectory prediction task, map element prediction task and target vehicle driving trajectory prediction task involved in the next control of the target vehicle driving (that is, the history_q_od, history_q_map, history_q_plan of the next frame).

[0116] It should be noted that in the implementation method of this application, each time an image is taken, a target vehicle driving prediction is performed, so the image feature information, the initial query information corresponding to each task, the time series query information, the context query information, and the target query information are all information of the current frame.

[0117] After obtaining the target query information of each task, it is used as the historical query information for the next frame.

[0118] The vehicle control method provided by the implementation of this application is actually an end-to-end planning method for assisted driving. By learning trajectories of various properties (including time trajectories, spatial trajectories, trajectories of different driving styles, trajectories of different time / space intervals, and trajectories of different driving scenarios), and through multi-task and image feature fusion methods, these driving trajectories can promote each other, so as to select different target vehicle driving trajectory information to control vehicle driving according to specific driving needs (including selecting time and space trajectories for lateral and longitudinal control of the vehicle, selecting target vehicle driving trajectory information for different driving scenarios or different driving styles based on scene or speed limit information, and selecting target vehicle driving trajectory information at different time intervals based on vehicle response time to control the vehicle). In this way, through multi-task prediction, the target vehicle driving trajectory information corresponding to the granularity of each target vehicle driving trajectory can be made more comprehensive and accurate, and then the vehicle driving can be controlled based on the target vehicle driving trajectory information, which can better and more accurately perform vehicle assisted driving control, better meet actual driving needs, and enhance user experience.

[0119] In another implementation of the present application, a deep learning model (e.g., a transform model) is used (as an example of a target machine learning model) to predict the target vehicle's trajectory. The deep learning model includes an image feature extraction module, a unified decoder layer, and a multilayer perceptron. The unified decoder layer includes a temporal interaction module, a collaborative interaction module, and a task deformable attention module.

[0120] like Figure 6 As shown, another vehicle control method provided by the implementation of the present application includes the following steps.

[0121] S10, determining multiple image information corresponding to different perspectives corresponding to the target vehicle, and determining query information corresponding to the target task, the target task including the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task.

[0122] S20, input multiple image information and query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task into the target machine learning model, so that the target machine learning model performs image feature extraction on the multiple image information to obtain multiple image feature information, and obtains target query information corresponding to the target vehicle driving trajectory prediction task based on the multiple image feature information and the query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task, and obtains multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities based on the target query information corresponding to the target vehicle driving trajectory prediction task.

[0123] S30 , controlling the target vehicle to travel according to the plurality of target vehicle driving trajectory information corresponding to the plurality of target vehicle driving trajectory granularities.

[0124] Among them, the target machine learning model obtains multiple image feature information in the following way: multiple image information is input into the image feature extraction module of the deep learning model to perform image feature extraction to obtain multiple image feature information.

[0125] The target machine learning model obtains the target query information corresponding to the target vehicle driving trajectory prediction task in the following way: the initial query information and historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task are input into the spatiotemporal interaction module of the deep learning model to obtain the temporal query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task; the temporal query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task is input into the collaborative interaction module to obtain the contextual query information corresponding to the target vehicle driving trajectory prediction task; the multiple image feature information and the contextual query information corresponding to the target vehicle driving trajectory prediction task are input into the deformable attention module of the deep learning model to obtain the target query information corresponding to the target vehicle driving trajectory prediction task.

[0126] Furthermore, based on the spatiotemporal interaction module, the time series query information of the obstacle trajectory prediction task is obtained according to the initial query information of the obstacle trajectory prediction task and the historical query information of the obstacle trajectory prediction task; the time series query information of the map element prediction task is obtained according to the initial query information of the map element prediction task and the historical query information of the map element prediction task; the time series query information of the target vehicle driving trajectory prediction task is obtained according to the initial query information of the target vehicle driving trajectory prediction task and the historical query information corresponding to the obstacle detection task, the map element prediction task, and the target vehicle driving trajectory prediction task.

[0127] Based on the collaborative interaction module, the context query information of the obstacle trajectory prediction task is obtained according to the time series query information corresponding to the obstacle trajectory prediction task, and the context query information of the map element prediction task is obtained according to the time series query information of the map element prediction task.

[0128] Based on the deformable attention module, the target query information corresponding to the obstacle trajectory prediction task is obtained according to multiple image feature information and the context query information of the obstacle trajectory prediction task, and the target query information corresponding to the map element prediction task is obtained according to multiple image feature information and the context query information of the map element prediction task.

[0129] Furthermore, based on the target query information corresponding to the target vehicle driving trajectory prediction task, multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities is obtained, including: inputting the target query information corresponding to the target vehicle driving trajectory prediction task into the multi-layer perceptron of the deep learning model, so that the multi-layer perceptron predicts multiple target vehicle driving trajectory information corresponding to multiple target vehicle driving trajectory granularities based on the target query information of the target vehicle driving trajectory prediction task.

[0130] Furthermore, the target query information corresponding to the obstacle trajectory prediction task is input into the multi-layer perceptron to obtain obstacle trajectory information, and the target query information corresponding to the map element prediction task is input into the multi-layer perceptron to obtain map element information. The target vehicle is controlled based on the multiple target vehicle driving trajectory information corresponding to the multiple target vehicle driving trajectory granularities, as well as the obstacle trajectory information and map element information.

[0131] For the specific implementation process of steps S10 to S30 , please refer to the specific description of steps S100 to S500 .

[0132] For example, Figure 7 As shown in the figure, the image feature extraction module includes a backbone network (Backbone) and a feature pyramid network (FPN) for multi-view images to extract image features, and obtain multiple image feature information (Multi-scale features), which are input into the Task Deformable Attention Module.

[0133] In addition, the anchors (i.e., task anchors: box, polyline, polyline) of the obstacle trajectory prediction task (agent), the map element prediction task (map), and the target vehicle driving trajectory prediction task (plan) and the query of each task (i.e., the initial query information of the task query: agent, map, plan) are input into the temporal interaction module, so that the temporal interaction module extracts the temporal information of the initial query information of each task and the historical query information of each task (temporal task query) to obtain the temporal query information corresponding to each task, and inputs the temporal query information corresponding to each task into the collaborative interaction module. The collaborative interaction module performs multi-head attention learning (Collaborative Attention Mask) based on the temporal query information of each task to extract contextual information and obtain the contextual query information corresponding to each task. The contextual query information of each task is input into the deformable attention module. Module), the deformable attention module extracts image information from the context query information and image feature information of each task respectively, obtains the target query information of each task, updates the latest query information of each task (update task query), and sends the target query information of each task to the multi-layer perceptron. The multi-layer perceptron predicts the obstacle information / obstacle trajectory information (agent (w / motion)) based on the obstacle detection head according to the final query information (agent) of the obstacle trajectory prediction task, and predicts the map element (map) based on the map detection head according to the final query information (map) of the map element prediction task. The granularity is divided based on the driving trajectory detection head (Planning Head) according to the final query information (plan) of the target vehicle driving trajectory prediction task (as shown in Figures A, B, and C), and the updated multi-granularity driving trajectory query information (updated multi-granularity planning query) is obtained, and the classification loss (cls heads) is calculated. , and calculate the regression loss (regheads) to get , and output As the corresponding outputs, so that The driving trajectory information of time trajectory granularity, spatial trajectory granularity, and driving style trajectory granularity under the same driving state is obtained, and the target vehicle (ego car) is controlled to drive based on the target vehicle driving trajectory information of different driving trajectory granularities.

[0134] Among them, Figure 7 As shown in Figure 1, the target vehicle driving trajectory prediction task is a multi-granularity planning query. A is the query information corresponding to the temporal trajectory granularity (Temporal planning query), B is the query information corresponding to the spatial trajectory granularity (Spatial planning query), and C is the query information corresponding to the driving style trajectory granularity (Driving-style planning query). are the waypoints corresponding to the temporal trajectory granularity (Temporal waypoints), are the spatial waypoints corresponding to the spatial trajectory granularity, These are the driving-style waypoints.

[0135] Furthermore, after a map is formed based on the map elements, obstacles are displayed on the map so that the user can clearly know the relative positional relationship between the planned route and the obstacles.

[0136] Furthermore, after obtaining the target query information of each task, the target query information is input back to the (topk) temporal interaction module as the historical query information (temporal task query) of each task for the next frame prediction.

[0137] Furthermore, when using the target vehicle driving trajectory information output by the deep learning model for target vehicle control, target vehicle driving trajectory information of different driving trajectory granularities can be selected based on actual conditions to control the target vehicle. For example, target vehicle driving trajectory information with a high sampling interval spatial trajectory granularity can be used to control the steering wheel, while target vehicle driving trajectory information with a high sampling interval temporal trajectory granularity can be used to control the throttle. Alternatively, target vehicle driving trajectory information with a specific driving style trajectory granularity can be used to control the throttle based on the road speed limit or user requirements.

[0138] Furthermore, the implementation of the present application also discloses a vehicle, which is used to execute the vehicle control method provided by the above implementation to achieve self-vehicle control.

[0139] Furthermore, in the implementation of the present application, the cloud can also perform obstacle trajectory prediction, map element prediction, and prediction of multiple target vehicle driving trajectory information corresponding to multiple driving trajectory granularities, so as to control the driving of the target vehicle according to the multiple target vehicle driving trajectory information, obstacle trajectory information, and map element information corresponding to the multiple driving trajectory granularities.

[0140] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the vehicle control method in the above embodiment.

[0141] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a vehicle processor, the vehicle processor executes the technical solution of the vehicle control method of the above embodiment.

[0142] In some possible implementations, various aspects of the method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a vehicle's processor, the program code is used to enable the vehicle's processor to execute the steps of the method according to various exemplary implementations of the present application described above in this specification. For example, the vehicle can execute the vehicle control method recorded in the embodiments of the present application.

[0143] The program product may employ any combination of one or more readable media. The readable medium may be a readable data medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CDROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0144] The implementation method of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the technical solution of the vehicle control method in the above embodiment can be implemented.

[0145] It should be noted that, in addition to the implementation of the present application described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application is introduced in conjunction with the preferred embodiment, this does not mean that the query of this invention is limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended from the present application. In order to provide an in-depth understanding of the present application, the above description contains many specific details, and the present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that, in the absence of conflict, the embodiments in the present application and the queries in the embodiments can be combined with each other.

[0146] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0147] It should be noted that in the accompanying drawings, some structure or method queries may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these queries may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structure or method queries in a particular figure does not imply that such queries are required in all embodiments, and in some embodiments, these queries may not be included or may be combined with other queries.

[0148] Although the present application has been illustrated and described with reference to certain preferred implementations of the present application, those skilled in the art should understand that the above description is provided as a further detailed explanation of the present application in conjunction with specific implementations, and that the specific implementation of the present application should not be limited to these descriptions. Those skilled in the art may make various changes in form and detail, including simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. A vehicle control method, characterized in that: The method comprises: Determining a plurality of image information corresponding to different viewing angles corresponding to the target vehicle, and performing image feature extraction on the plurality of image information to obtain a plurality of image feature information; Determining query information corresponding to target tasks, wherein the target tasks include obstacle trajectory prediction tasks, map element prediction tasks, and target vehicle driving trajectory prediction tasks; Obtaining target query information corresponding to the target vehicle driving trajectory prediction task based on the plurality of image feature information and the query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task; Performing driving trajectory granularity division on the target query information corresponding to the target vehicle driving trajectory prediction task to obtain target query information corresponding to a plurality of target vehicle driving trajectory granularities, wherein the target query information corresponding to each target vehicle driving trajectory granularity includes target query information corresponding to a plurality of driving states; Performing fusion processing on target query information corresponding to the same driving state at each target vehicle driving trajectory granularity to obtain target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state; According to the target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state, the target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities for each driving state is predicted, so as to obtain a plurality of target vehicle driving trajectory information corresponding to the plurality of target vehicle driving trajectory granularities; The target vehicle is controlled to travel according to the plurality of target vehicle driving trajectory information corresponding to the plurality of target vehicle driving trajectory granularities.

2. The vehicle control method according to claim 1, characterized in that: The multiple target vehicle driving trajectory granularities include time trajectories corresponding to different time intervals, spatial trajectories corresponding to different spatial intervals, trajectories corresponding to different driving styles, and trajectories corresponding to different driving scenarios.

3. The vehicle control method according to claim 2, characterized in that: Predicting target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities for each driving state according to target vehicle driving trajectory query information of each driving state includes: Based on the multi-layer perceptron and the target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state, the target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities for each driving state is predicted.

4. The vehicle control method according to claim 3, characterized in that: Controlling the target vehicle to travel according to the plurality of target vehicle driving trajectory information corresponding to the plurality of target vehicle driving trajectory granularities includes: A target driving state is determined from the multiple target vehicle driving states, and different driving execution devices of the target vehicle are controlled according to the target driving state corresponding to the target vehicle driving trajectory information of different target vehicle driving trajectory granularities to control the driving of the target vehicle.

5. The vehicle control method according to any one of claims 1 to 4, characterized in that: The query information includes initial query information and historical query information. Then, based on the plurality of image feature information and the query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle trajectory prediction task, target query information corresponding to the target vehicle trajectory prediction task is obtained, including: Obtaining time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task based on the initial query information and the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task; Obtaining contextual query information corresponding to the target vehicle trajectory prediction task based on the time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle trajectory prediction task based on a cross-attention mechanism; Based on the deformable attention mechanism, target query information of the target vehicle driving trajectory prediction task is obtained according to the context query information and the multiple image feature information of the target vehicle driving trajectory prediction task.

6. The vehicle control method according to claim 5, characterized in that: Obtaining time series query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task according to the initial query information and the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task, including: Obtaining the temporal query information of the obstacle trajectory prediction task based on the initial query information of the obstacle trajectory prediction task and the historical query information of the obstacle trajectory prediction task based on a cross-attention mechanism; obtaining the temporal query information of the map element prediction task based on the initial query information of the map element prediction task and the historical query information of the map element prediction task based on a cross attention mechanism; Based on the cross-attention mechanism, the temporal query information of the target vehicle driving trajectory prediction task is obtained according to the initial query information of the target vehicle driving trajectory prediction task and the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task.

7. The vehicle control method according to claim 6, characterized in that: The method further comprises: Obtaining the contextual query information of the obstacle trajectory prediction task based on the time series query information of the obstacle trajectory prediction task based on a self-attention mechanism, and obtaining the contextual query information of the map element prediction task based on the time series query information of the map element prediction task; Obtaining target query information corresponding to the obstacle trajectory prediction task based on the multiple image feature information and the context query information of the obstacle trajectory prediction task based on a deformable attention mechanism, and obtaining target query information corresponding to the map element prediction task based on the multiple image feature information and the context query information of the map element prediction task; Obstacle trajectory information is predicted based on the target query information corresponding to the obstacle trajectory prediction task based on a multilayer perceptron, and map element information is predicted based on the target query information corresponding to the map element prediction task; The target vehicle is controlled to travel according to the plurality of target vehicle driving trajectory information corresponding to the plurality of target vehicle driving trajectory granularities, the obstacle trajectory information, and the map element information.

8. The vehicle control method according to claim 7, characterized in that: The method further comprises: The target query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task obtained are respectively used as the historical query information corresponding to the obstacle trajectory prediction task, the map element prediction task, and the target vehicle driving trajectory prediction task involved in the next control of the target vehicle driving.

9. A vehicle control method, characterized in that: The method comprises: Determining multiple image information corresponding to different perspectives corresponding to a target vehicle, and determining query information corresponding to target tasks, the target tasks including an obstacle trajectory prediction task, a map element prediction task, and a target vehicle driving trajectory prediction task; The plurality of image information and the query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task are input into the target machine learning model, so that the target machine learning model performs image feature extraction on the plurality of image information to obtain a plurality of image feature information, and obtains the target query information corresponding to the target vehicle driving trajectory prediction task based on the plurality of image feature information and the query information corresponding to the obstacle trajectory prediction task, the map element prediction task and the target vehicle driving trajectory prediction task, performs driving trajectory granularity division on the target query information corresponding to the target vehicle driving trajectory prediction task, and obtains a plurality of target vehicles Target query information corresponding to a driving trajectory granularity, the target query information corresponding to each target vehicle driving trajectory granularity including target query information corresponding to a plurality of driving states, performing fusion processing on the target query information corresponding to the same driving state at each target vehicle driving trajectory granularity to obtain target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities for each driving state, and predicting target vehicle driving trajectory information corresponding to different target vehicle driving trajectory granularities for each driving state based on the target vehicle driving trajectory query information corresponding to different target vehicle driving trajectory granularities to obtain a plurality of target vehicle driving trajectory information corresponding to the plurality of target vehicle driving trajectory granularities; The target vehicle is controlled to travel according to the plurality of target vehicle driving trajectory information corresponding to the plurality of target vehicle driving trajectory granularities.

10. A vehicle, characterized in that: The vehicle is used to implement the vehicle control method according to any one of claims 1 to 8, or to implement the vehicle control method according to claim 9.

Images