Vehicle, control method and control device thereof and computer readable storage medium

Abstract

The invention discloses a vehicle, a control method and device thereof and a computer readable storage medium, and relates to the technical field of intelligent driving, and the method comprises the steps: obtaining video information obtained by a vehicle-mounted camera of the vehicle and external parameter information of the vehicle-mounted camera in response to a task execution instruction; inputting the video information, the external parameter information and a target task corresponding to the task execution instruction into a preset encoder module to output multi-dimensional feature information corresponding to the target task, and determining a query vector of the preset encoder module based on the target task; and determining a task detection result based on the multi-dimensional feature information so as to control the vehicle based on the task detection result. According to the method, the query vector is determined based on the target task, so that the preset encoder module outputs the corresponding multi-dimensional feature information in combination with the video information and the external parameter information to determine the task detection result for vehicle control, and thus the vehicle control precision is improved and the application cost is reduced based on an end-to-end intelligent driving algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to a vehicle, a control method and device thereof, and a computer readable storage medium. BACKGROUND

[0002] In the related art, intelligent driving algorithms rely on three independent modules of perception, prediction and planning, which are sequentially run to generate planning results to send to the control module, and then the control module generates control results to send to the vehicle execution mechanism. However, in this method, the three modules of perception, prediction and planning are independent and sequentially run, which can cause cumulative errors and ultimately affect the accuracy of the planning results. SUMMARY

[0003] The present application aims to at least partially solve one of the technical problems in the related art. To this end, the first object of the present application is to provide a vehicle control method, which determines a query vector based on a target task, so that a preset encoder module outputs corresponding multi-dimensional feature information in combination with video information and extrinsic parameter information to determine a task detection result for vehicle control, thereby improving vehicle control accuracy based on an end-to-end intelligent driving algorithm and reducing application costs.

[0004] The second object of the present application is to provide a vehicle control device.

[0005] The third object of the present application is to provide a vehicle.

[0006] The fourth object of the present application is to provide a computer readable storage medium.

[0007] To achieve the above objects, the first aspect of the present application provides a vehicle control method, which comprises: in response to a task execution instruction, acquiring video information obtained by a vehicle-mounted camera of a vehicle and extrinsic parameter information of the vehicle-mounted camera; inputting the video information, the extrinsic parameter information and a target task corresponding to the task execution instruction into a preset encoder module to output multi-dimensional feature information corresponding to the target task, wherein a query vector of the preset encoder module is determined based on the target task; determining a task detection result based on the multi-dimensional feature information, and controlling the vehicle based on the task detection result.

[0008] According to the vehicle control method provided in the embodiments of the present application, in response to a task execution instruction, video information obtained by a vehicle camera of a vehicle and extrinsic information of the vehicle camera are acquired, and then the video information, the extrinsic information, and a target task corresponding to the task execution instruction are input into a preset encoder module to output multi-dimensional feature information corresponding to the target task. A query vector of the preset encoder module is determined based on the target task, and then a task detection result is determined according to the multi-dimensional feature information, so as to control the vehicle based on the task detection result. Thus, the query vector is determined based on the target task, so that the preset encoder module outputs corresponding multi-dimensional feature information in combination with the video information and the extrinsic information, to determine the task detection result for vehicle control, thereby improving the vehicle control precision based on an end-to-end intelligent driving algorithm and reducing the application cost.

[0009] In addition, the vehicle control method according to the above embodiments of the present application can have the following additional technical features: According to an embodiment of the present application, the preset encoder module includes a task self-attention mechanism network, a time cross mechanism network, and a sensor cross mechanism network. The video information, the extrinsic information, and the target task corresponding to the task execution instruction are input into the preset encoder module to output the multi-dimensional feature information corresponding to the target task, including: pre-processing the video information, the extrinsic information, and the task information to obtain a sensor mark and a plurality of task query vectors corresponding to the target task; calling a memory queue based on the target task to determine a last frame time mark; inputting the plurality of task query vectors into the task self-attention mechanism network to output a plurality of initial task features that are globally optimized; inputting the plurality of initial task features and the last frame time mark into the time cross mechanism network to output a plurality of initial task features that are enhanced in time sequence; inputting the plurality of initial task features that are enhanced in time sequence and the sensor mark into the sensor cross mechanism network to output globally aligned fusion features to obtain the multi-dimensional feature information.

[0010] According to an embodiment of the present application, the pre-processing of the video information, the extrinsic information, and the task information to obtain the sensor mark and the plurality of task query vectors corresponding to the target task includes: determining the sensor mark based on the video information and the extrinsic information; and processing the target task in combination with a meta reference point and a map reference point to obtain the plurality of task query vectors.

[0011] According to an embodiment of the present application, the determination of the sensor mark based on the video information and the extrinsic information includes: inputting the video information into an image backbone network to output a corresponding semantic mapping graph; inputting the extrinsic information into a first position encoding network to output a first embedding vector corresponding to a 3D (Three-Dimensional) space position; and generating the sensor mark based on the semantic mapping graph and the first embedding vector.

[0012] According to one embodiment of the present application, a plurality of task query vectors are obtained by processing a plurality of tasks in combination with a meta reference point and a map reference point, including: inputting the meta reference point and the map reference point into a second position encoding network to output a second embedding vector corresponding to a 3D space position; performing semantic analysis processing on the plurality of tasks to obtain corresponding task labels; and generating a plurality of task query vectors based on the task labels and the second embedding vector.

[0013] According to one embodiment of the present application, the memory queue includes a plurality of historical features, and the memory queue is recalled based on a target task to determine a previous frame time mark, including: obtaining a corresponding previous frame historical feature in the memory queue; inputting the previous frame historical feature into a self-vehicle conversion network and performing bias correction based on a motivation compensation network to obtain a previous frame historical feature in a current frame coordinate system; and determining the previous frame time mark based on the previous frame historical feature.

[0014] According to one embodiment of the present application, the plurality of task query vectors include a target query vector, a map query vector, a prediction query vector and a planning query vector, the multi-dimensional feature information includes a target feature, a map feature and a self-vehicle feature, and the task detection result is determined based on the multi-dimensional feature information, including: determining a target detection result and an obstacle motivation prediction result based on the target feature; constructing an online map according to the map feature; and determining a planning trajectory according to the self-vehicle feature.

[0015] To achieve the above object, the second aspect embodiment of the present application proposes a vehicle control device, which comprises: an acquisition module configured to acquire video information obtained by a vehicle-mounted camera of a vehicle and extrinsic information of the vehicle-mounted camera in response to a task execution instruction; a feature detection module configured to input the video information, the extrinsic information and a target task corresponding to the task execution instruction into a preset encoder module to output multi-dimensional feature information corresponding to the target task, wherein a query vector of the preset encoder module is determined based on the target task; and a control module configured to determine a task detection result based on the multi-dimensional feature information, and control the vehicle based on the task detection result.

[0016] According to the vehicle control device provided in the embodiments of the present application, the video information acquired by the vehicle-mounted camera of the vehicle and the extrinsic information of the vehicle-mounted camera are acquired by the acquisition module in response to the task execution instruction, the video information, the extrinsic information and the target task corresponding to the task execution instruction are input into the preset encoder module by the feature detection module, so as to output the multi-dimensional feature information corresponding to the target task, wherein the query vector of the preset encoder module is determined based on the target task, the task detection result is determined based on the multi-dimensional feature information by the control module, and the vehicle is controlled based on the task detection result. Therefore, the query vector is determined based on the target task, so that the preset encoder module outputs the corresponding multi-dimensional feature information in combination with the video information and the extrinsic information, the task detection result is determined, and the vehicle control is used, so that the vehicle control precision is improved based on the end-to-end intelligent driving algorithm, and the application cost is reduced.

[0017] To achieve the above object, the third aspect of the present application provides a vehicle, comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the vehicle control method of any one of the above.

[0018] According to the vehicle provided in the embodiments of the present application, the steps of the vehicle control method of any one of the above are implemented by the processor executing the computer program stored in the memory, based on the above vehicle control method, the vehicle control precision is improved based on the end-to-end intelligent driving algorithm, and the application cost is reduced.

[0019] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of the vehicle control method of any one of the above.

[0020] According to the computer readable storage medium provided in the embodiments of the present application, the steps of the vehicle control method of any one of the above can be implemented when the computer program is executed by the processor, based on the above vehicle control method, the vehicle control precision is improved based on the end-to-end intelligent driving algorithm, and the application cost is reduced.

[0021] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Flowchart of the vehicle control method according to the embodiments of the present application; Figure 2 Network architecture diagram of the intelligent driving algorithm according to one embodiment of the present application; Figure 3 Network architecture diagram of the intelligent driving algorithm according to one specific embodiment of the present application; Figure 4 Connection diagram of a vehicle control device according to an embodiment of the application; Figure 5 Block diagram of a vehicle according to an embodiment of the application. DETAILED DESCRIPTION

[0023] Embodiments of the present application are described in detail below with reference to the attached drawings, which show by way of example, embodiments in which the same or similar elements or elements having the same or similar functions are denoted with the same reference numerals throughout the drawings. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0024] A vehicle and a control method, control device and computer readable storage medium thereof according to embodiments of the present application are described below with reference to the drawings.

[0025] Existing intelligent driving algorithms rely on three independent modules of perception, prediction and planning, which are sequentially run to generate planning results to the control module, and then the control module generates control results to the vehicle execution mechanism. Since the three modules of perception, prediction and planning are independent and sequentially run, errors are accumulated, which ultimately affects the accuracy of the planning results.

[0026] A kind of end-to-end intelligent driving algorithm is proposed in related technology, its input is panoramic video stream, output is planning result, the main architecture of the deep learning network designed is Transformer structure (deep learning model), and Transformer mechanism includes encoder (encoder) and decoder (decoder) architecture. The paper aims at encoder structure, inputs panoramic video stream and camera external parameter, designs BEV Query (Bird's Eye View Query, bird's eye view query) as the query input of encoder. BEV feature map is generated in encoder, and is used for obstacle tracking, map generation, obstacle motive prediction and trajectory generation of decoder. This method relies on BEV feature map, generates BEV feature map through BEV Query, is used for the prediction trajectory generated by decoder module, and in theory, the feature precision generated by the designed trajectory Query is not accurate.

[0027] To solve the theoretical structure deficiency of the end-to-end intelligent driving algorithm based on the BEV feature map in the related technology. A vehicle control method is provided, which determines the query vector of the preset encoder module based on the target task, so that the preset encoder module can output corresponding multi-dimensional feature information according to the received video information, extrinsic information and target task, to determine the task detection result, which is used for vehicle control, can solve the problems of segmented training of the end-to-end algorithm based on the BEV feature in the related technology, and the precision loss caused by the irrelevance of the task and the query vector, thereby improving the vehicle control precision and reducing the application cost. The vehicle control method of the present application will be described in detail below with reference to the accompanying drawings.

[0028] As shown in Figure 1 , the vehicle control method of the embodiment of the present application can include: S1, in response to a task execution instruction, obtaining video information obtained by a vehicle-mounted camera of a vehicle and extrinsic information of the vehicle-mounted camera.

[0029] Specifically, the triggering of the task execution instruction can be triggered by the user through the control option, or automatically triggered during the running of the vehicle, and the specific limitation is not limited. The video stream of the vehicle is obtained by the vehicle-mounted camera as the video information, and the extrinsic information of the vehicle-mounted camera refers to the position and attitude parameters of the camera in the world coordinate system, which is used to describe its spatial positioning and orientation, and can include position parameters, attitude parameters.

[0030] S2, inputting the video information, extrinsic information and target task corresponding to the task execution instruction into a preset encoder module to output multi-dimensional feature information corresponding to the target task, wherein the query vector of the preset encoder module is determined based on the target task; That is, the query vector Query of the preset encoder module is determined based on the target task, for example, the query vector includes detection Query, map Query, prediction Query and planning Query, which is input into the preset encoder module such as the encoder module.

[0031] The collected video information, extrinsic information of the camera and query vector are input into the preset encoder module, each task is controlled by its corresponding Query, and each task interacts with each other, thereby outputting multi-dimensional feature information corresponding to the target task, such as ego feature, map feature, etc.

[0032] S3, determining a task detection result based on the multi-dimensional feature information, and controlling the vehicle based on the task detection result.

[0033] That is, the result is output according to the multi-dimensional feature information, which is used for vehicle control. For example, as Figure 2As shown, the detection Query, the map Query, the prediction Query and the planning Query are designed based on the task execution instruction, and are taken as the input of the preset encoder module (i.e., the Transfoemer network), and then the map result, the detection result, the prediction result and the planning result are output in combination with the omnidirectional video stream and the external parameter information, so that the segmented training of the end-to-end algorithm based on the BEV feature and the precision loss problem caused by the irrelevant task and Query can be solved. Moreover, based on the preset encoder module structure, the training process can be converted from two-stage to one-stage, and the model is easier to train and converge, thereby improving the detection result precision. In combination Figure 3 As shown, in some embodiments of the present application, the preset encoder module includes a task self-attention mechanism network, a time cross mechanism network and a sensor cross mechanism network. The video information, the external parameter information and the target task corresponding to the task execution instruction are input into the preset encoder module to output the multi-dimensional feature information corresponding to the target task, including: pre-processing the video information, the external parameter information and the task information to obtain a sensor mark and a plurality of task query vectors corresponding to the target task; calling a memory queue based on the target task to determine a last frame time mark; inputting the plurality of task query vectors into the task self-attention mechanism network to output a plurality of initial task features that are globally optimized; inputting the plurality of initial task features and the last frame time mark into the time cross mechanism network to output a plurality of initial task features that are enhanced in time sequence; inputting the plurality of initial task features that are enhanced in time sequence and the sensor mark into the sensor cross mechanism network to output a globally aligned fusion feature to obtain the multi-dimensional feature information.

[0034] Specifically, based on the information conversion of the algorithm, all inputs are uniformly expressed and marked, i.e., the video information, the external parameter information and the task information are pre-processed to obtain a sensor mark and a plurality of task query vectors corresponding to the target task.

[0035] Taking the plurality of task query vectors including the meta Query (target query vector), the map Query (map query vector) and the ego Query (planning query vector) as an example, the task self-attention mechanism network realizes information exchange and feature enhancement among the three tasks of detection, prediction and planning, i.e., inputting three groups of task features and outputting three groups of enhanced features that are globally cooperatively optimized, thereby avoiding the problem of "sub-task optimization and global sub-optimization" caused by independent calculation of each task. Thus, the task self-attention mechanism network ensures direct interaction of any task without explicit constraints, and promotes the synergistic effect, such as perception based on planning awareness, or interactive prediction and planning based on game idea.

[0036] The time cross mechanism network integrates the observation history of the last frame, that is, aligns, weights, and fuses the current frame and the historical frame across time to identify the pose change of the target. Specifically, the current frame is taken as Query, and the aligned historical frame is taken as Key / Value. The historical features are weighted and injected into the current features through cross attention. The current + historical features and the pose change are input, and the current features with time sequence enhancement are output, so that the network has short-term motion memory, thereby improving the detection robustness and trajectory prediction accuracy of dynamic objects (pedestrians, vehicles, and cyclists). The existing example uses the BEV feature map of the last frame to time information. The present application uses a first-in-first-out queue to separately maintain each task Query, and a cross attention mechanism is formed for the historical Query. In this way, the time information is fused, and the execution accuracy is improved.

[0037] The sensor mark is input to the sensor cross mechanism network as spatial feature extraction of a preset encoder module. The sensor cross mechanism network establishes a channel between the task and the original sensor input, ensures end-to-end learning without information loss, specifically aligns, weights, and fuses the heterogeneous features generated by different sensors and different perspectives through cross attention, generates a unified feature map that is globally consistent and complementary across modalities, and provides the unified feature map for downstream detection, prediction, and planning. Here, compared with the end-to-end algorithm based on the BEV feature map, the embodiment is more efficient in this respect and has fewer learning gradient collapse problems.

[0038] Therefore, the embodiment combines video information, extrinsic information, and historical information (memory queue) to output multi-dimensional feature information, thereby improving detection accuracy.

[0039] In some embodiments of the present application, the video information, extrinsic information, and task information are preprocessed to obtain a sensor mark and a plurality of task query vectors corresponding to a target task, including: determining the sensor mark based on the video information and the extrinsic information; and processing the target task in combination with the meta reference point and the map reference point to obtain the plurality of task query vectors.

[0040] That is, the sensor mark is generated based on the video information and the extrinsic information, so that the sensor mark can include semantic information (object category) and 3D geometry (position in the real world) and sensor identity (encoded implicitly through extrinsic parameters), and then sent to a preset encoder module for task feature output. The plurality of task query vectors are obtained based on the meta reference point, the map reference point, and the target task, so that the task query vector can fuse the meta reference point (data-driven 3D anchor), the map reference point (knowledge-driven prior), and the task semantic information, and can be directly sent to the preset encoder module as an initial Query to participate in subsequent detection, trajectory prediction, and path planning processes.

[0041] In some embodiments of the present application, the plurality of task query vectors include a target query vector, a map query vector and a planning query vector, the multi-dimensional feature information includes target features, map features and ego vehicle features, and the task detection result is determined based on the multi-dimensional feature information, including: determining a target detection result and an obstacle motive prediction result based on the target features; constructing an online map according to the map features; and determining a planning trajectory according to the ego vehicle features.

[0042] Specifically, taking Figure 3 for example, the plurality of task query vectors include a meta Query (target query vector), a map Query (map query vector) and an ego Query (planning query vector). The meta Query represents a dynamic target (such as a vehicle, a pedestrian, etc.), which will be used for target detection and obstacle motive prediction, i.e., meta features are output based on the meta Query, which are used to determine the target detection result and the obstacle motive prediction result. The map Query represents a static element (such as a lane line, a traffic sign, etc.), which will be used for constructing an online map, i.e., map features are output based on the map Query, which are used for constructing the online map. The ego Query represents a potential behavior of the ego vehicle, which will be used for constructing a planning result, i.e., ego vehicle features are output based on the ego Query, which are used for generating the planning trajectory.

[0043] In some embodiments of the present application, the extrinsic information is labeled based on the video information to obtain a sensor label, including: inputting the video information into an image backbone network to output a corresponding semantic map; inputting the extrinsic information into a first position encoding network to output a first embedding vector corresponding to a 3D (Three-Dimensional) spatial position; and generating the sensor label based on the semantic map and the first embedding vector.

[0044] Specifically, a current video frame at a current time is extracted based on the video information, and the current video frame is input into the image backbone network to output a semantic map based on semantic segmentation of the current video frame by the image backbone network.

[0045] The camera extrinsic matrix at the same frame time, i.e., the extrinsic information, is input into the first position encoding network, the camera extrinsic data is projected onto a normalized camera coordinate by the first position encoding network according to the intrinsic parameters, and then converted to a world coordinate, and the coordinate is subjected to Fourier encoding and stitching convolution operation to obtain the first embedding vector.

[0046] The semantic mapping graph and the first embedding vector are processed based on a weighted global average pooling operation to obtain sensor labels mapped to a unified dimension. The sensor labels can have semantic information (object category) and 3D geometry (position in the real world) and sensor identity (encoded implicitly through extrinsic parameters) at the same time, and then are sent to a multi-task decoder, i.e., a preset encoder module, as Key / Value of Cross-Attention, to realize end-to-end training and reasoning.

[0047] In some embodiments of the present application, a plurality of task query vectors are obtained by processing a plurality of tasks in combination with a meta reference point and a map reference point, including: inputting the meta reference point and the map reference point into a second position encoding network to output a second embedding vector corresponding to a 3D space position; performing semantic analysis processing on the plurality of tasks to obtain corresponding task labels; and generating a plurality of task query vectors based on the task labels and the second embedding vector.

[0048] Specifically, the meta reference point predicted by the network is obtained, and the map reference point cut online from the high-definition map is read. After Fourier encoding processing and full connection dimension increasing and dimension decreasing processing of the meta reference point and the map reference point by the second position encoding network, the second embedding vector is obtained.

[0049] Learnable task embedding vectors are predefined for the three target tasks of detection, prediction and planning. For each task, the task embedding is copied multiple times and spliced with the rough result (category, speed, trajectory endpoint) of the previous frame of the task, and then sent to a task semantic analysis network to obtain a task label.

[0050] Cross-attention processing is performed with the task label as Query and the second embedding vector as Key / Value, and then residual connection, normalization processing are performed to obtain the final task query vector. At this time, the task query vector can integrate: the meta reference point (data-driven 3D anchor), the map reference point (knowledge-driven prior), and the task semantics and historical information. The three task query vectors can be directly sent to the corresponding decoder, i.e., the preset encoder module, as the initial Query to participate in the subsequent detection, trajectory prediction and path planning process, to realize end-to-end training and real-time reasoning.

[0051] In some embodiments of the present application, the memory queue includes a plurality of historical features of a plurality of frames, and the memory queue is recalled based on a target task to determine a previous frame time mark, including: obtaining a corresponding previous frame historical feature in the memory queue; inputting the previous frame historical feature into a self-vehicle conversion network and performing bias correction based on a motivation compensation network to obtain a previous frame historical feature in a current frame coordinate system; and determining a previous frame time mark based on the above-mentioned previous frame historical feature.

[0052] That is, a preset length memory queue corresponding to the task type is maintained in the system, and the memory queue stores the feature map (which can be BEV or image backbone output) that has been calculated, and the last frame history feature is read. Based on the ego transformation network, the relative pose of the last frame (i.e. the current frame) is determined, and then a standard coordinate network is generated according to the camera intrinsic parameter, the pose is back-projected to the world coordinate, and then projected to the last frame coordinate system to obtain a sampling network. Then, based on the micro-bilinear sampling, the last frame history feature is converted to the current frame coordinate system, but the dynamic target may still have a motion deviation. At this time, the target residual is dynamically corrected by the motion compensation network to obtain the last frame history feature in the current frame coordinate system. Finally, the last frame history feature is converted into a vector through average pooling and a fully connected layer to obtain the last frame time label that is spatio-temporally aligned with the current frame.

[0053] The embodiment introduces the last frame calculation result input time label input and outputs to the time cross mechanism network through the time label to meet the algorithm's demand for time feature extraction.

[0054] In summary, according to the vehicle control method of the embodiment of the present application, in response to a task execution instruction, video information obtained by a vehicle-mounted camera of a vehicle and extrinsic parameter information of the vehicle-mounted camera are acquired, and then the video information, the extrinsic parameter information and a target task corresponding to the task execution instruction are input into a preset encoder module to output multi-dimensional feature information corresponding to the target task. A query vector of the preset encoder module is determined based on the target task, and then a task detection result is determined according to the multi-dimensional feature information, so as to control the vehicle based on the task detection result. Thus, the method determines the query vector based on the target task, so that the preset encoder module outputs the corresponding multi-dimensional feature information in combination with the video information and the extrinsic parameter information, to determine the task detection result for vehicle control, thereby improving the vehicle control precision based on the end-to-end intelligent driving algorithm and reducing the application cost.

[0055] Corresponding to the above embodiment, the present application also proposes a vehicle control device.

[0056] As shown in Figure 4 The vehicle control device of the embodiment of the present application includes an acquisition module 10, a feature detection module 20 and a control module 30.

[0057] The acquisition module 10 is configured to acquire, in response to a task execution instruction, video information obtained by a vehicle-mounted camera of a vehicle and extrinsic parameter information of the vehicle-mounted camera. The feature detection module 20 is configured to input the video information, the extrinsic parameter information and a target task corresponding to the task execution instruction into a preset encoder module to output multi-dimensional feature information corresponding to the target task, wherein a query vector of the preset encoder module is determined based on the target task. The control module 30 is configured to determine a task detection result based on the multi-dimensional feature information, and control the vehicle based on the task detection result.

[0058] According to an embodiment of the present application, the preset encoder module includes a task self-attention mechanism network, a time cross mechanism network, and a sensor cross mechanism network. The feature detection module 20 inputs video information, external parameter information, and a target task corresponding to a task execution instruction into the preset encoder module to output multi-dimensional feature information corresponding to the target task, specifically for: pre-processing the video information, the external parameter information, and the task information to obtain a sensor mark and a plurality of task query vectors corresponding to the target task; calling a memory queue based on the target task to determine a last frame time mark; inputting the plurality of task query vectors into the task self-attention mechanism network to output a plurality of initial task features that are globally optimized; inputting the plurality of initial task features and the last frame time mark into the time cross mechanism network to output a plurality of initial task features that are enhanced in time sequence; inputting the plurality of initial task features that are enhanced in time sequence and the sensor mark into the sensor cross mechanism network to output globally aligned fusion features to obtain the multi-dimensional feature information.

[0059] According to an embodiment of the present application, the feature detection module 20 pre-processes the video information, the external parameter information, and the task information to obtain a sensor mark and a plurality of task query vectors corresponding to the target task, specifically for: determining the sensor mark based on the video information and the external parameter information; processing the target task in combination with a meta reference point and a map reference point to obtain the plurality of task query vectors.

[0060] According to an embodiment of the present application, the feature detection module 20 marks the external parameter information based on the video information to obtain a sensor mark, specifically for: inputting the video information into an image backbone network to output a corresponding semantic mapping graph; inputting the external parameter information into a first position encoding network to output a first embedding vector corresponding to a 3D (Three-Dimensional) spatial position; and generating the sensor mark based on the semantic mapping graph and the first embedding vector.

[0061] According to an embodiment of the present application, the feature detection module 20 processes a plurality of tasks in combination with a meta reference point and a map reference point to obtain a plurality of task query vectors, specifically for: inputting the meta reference point and the map reference point into a second position encoding network to output a second embedding vector corresponding to a 3D spatial position; performing semantic analysis processing on the plurality of tasks to obtain corresponding task marks; and generating the plurality of task query vectors based on the task marks and the second embedding vector.

[0062] According to an embodiment of the present application, the memory queue includes a plurality of frame history features, and the feature detection module 20 invokes the memory queue based on the target task to determine the last frame time mark, specifically for: obtaining the corresponding last frame history feature in the memory queue; inputting the last frame history feature into the ego vehicle conversion network, and performing bias correction based on the motivation compensation network to obtain the last frame history feature in the current frame coordinate system; determining the last frame time mark based on the last frame history feature.

[0063] According to an embodiment of the present application, the plurality of task query vectors includes a target query vector, a map query vector and a planning query vector, and the multi-dimensional feature information includes a target feature, a map feature and an ego vehicle feature. The control module 30 determines the task detection result based on the multi-dimensional feature information, specifically for: determining the target detection result and the obstacle motivation prediction result based on the target feature; constructing an online map according to the map feature; and determining a planning trajectory according to the ego vehicle feature.

[0064] It should be noted that the details of the vehicle control device not disclosed in the embodiments of the present application are referred to the details disclosed in the vehicle control method of the above embodiments of the present application, which will not be described here.

[0065] According to the vehicle control device of the embodiments of the present application, the acquisition module acquires the video information acquired by the vehicle-mounted camera of the vehicle and the extrinsic information of the vehicle-mounted camera in response to the task execution instruction. The feature detection module inputs the video information, the extrinsic information and the target task corresponding to the task execution instruction into the preset encoder module to output the multi-dimensional feature information corresponding to the target task, wherein the query vector of the preset encoder module is determined based on the target task. The control module determines the task detection result based on the multi-dimensional feature information, and controls the vehicle based on the task detection result. Therefore, the device determines the query vector based on the target task, so that the preset encoder module outputs the corresponding multi-dimensional feature information in combination with the video information and the extrinsic information to determine the task detection result for vehicle control, thereby improving the vehicle control precision based on the end-to-end intelligent driving algorithm and reducing the application cost.

[0066] Corresponding to the above embodiments, the present application also provides a vehicle.

[0067] As shown in Figure 5 The vehicle 100 of the embodiments of the present application includes a memory 110 for storing a computer program, and a processor 120 for executing the computer program to implement the steps of the vehicle control method of any one of the above embodiments.

[0068] According to the vehicle of the embodiment of the present application, the steps of the vehicle control method of any one of the above are implemented by the processor executing the computer program stored in the memory, and the vehicle control precision is improved and the application cost is reduced based on the end-to-end intelligent driving algorithm based on the vehicle control method.

[0069] Corresponding to the above embodiment, the present application also provides a computer readable storage medium.

[0070] The computer readable storage medium of the embodiment of the present application, the computer readable storage medium has a computer program stored therein, wherein the computer program is executed by the processor to implement the steps of the vehicle control method of any one of the above.

[0071] According to the computer readable storage medium of the embodiment of the present application, the computer program is executed by the processor to implement the steps of the vehicle control method of any one of the above, and the vehicle control precision is improved and the application cost is reduced based on the end-to-end intelligent driving algorithm based on the vehicle control method.

[0072] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from an instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices. For the purpose of this specification, "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices. More specific examples (non-exhaustive list) of computer readable medium include the following: electrical connections having one or more wires (electronic devices), portable computer diskette (magnetic devices), random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CD ROM). In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing the program as necessary, and then storing it in a computer memory.

[0073] It should be understood that portions of the present application can be realized with a hardware, software, firmware or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be realized with software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized with hardware, and as in another embodiment, it can be realized with any one or a combination of the following technologies known in the art: discrete logic circuit having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0074] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0075] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified and limited.

[0076] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection", "fixing" and other terms should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0077] Although the embodiments of the present application have been shown and described above, it can be understood that the above-described embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A vehicle control method, characterized in that, The method includes: In response to the task execution command, the system acquires video information obtained by the vehicle's onboard camera and external parameter information of the onboard camera. The video information, the extrinsic information, and the target task corresponding to the task execution instruction are input into a preset encoder module to output multi-dimensional feature information corresponding to the target task, wherein the query vector of the preset encoder module is determined based on the target task. The task detection result is determined based on the multi-dimensional feature information, and the vehicle is controlled based on the task detection result.

2. The vehicle control method according to claim 1, characterized in that, The preset encoder module includes a task self-attention mechanism network, a temporal crossover mechanism network, and a sensor crossover mechanism network. It inputs the video information, the extrinsic parameter information, and the target task corresponding to the task execution instruction into the preset encoder module to output multi-dimensional feature information corresponding to the target task, including: The video information, the extrinsic parameter information, and the task information are preprocessed to obtain sensor tags and multiple task query vectors corresponding to the target task; Based on the target task, the memory queue is retrieved to determine the time marker of the previous frame; Multiple task query vectors are input into the task self-attention mechanism network to output multiple initial task features that have been globally optimized. The initial task features and the previous frame time stamp are input into the temporal crossover mechanism network to output multiple initial task features that have undergone temporal enhancement. The initial task features enhanced by time sequence and the sensor labels are input into the sensor cross mechanism network, and globally aligned fused features are output to obtain the multi-dimensional feature information.

3. The vehicle control method according to claim 2, characterized in that, The video information, the extrinsic parameter information, and the task information are preprocessed to obtain sensor tags and multiple task query vectors corresponding to the target task, including: The sensor marker is determined based on the video information and the extrinsic parameter information; The target task is processed by combining the meta reference point and the map reference point to obtain multiple task query vectors.

4. The vehicle control method according to claim 3, characterized in that, Determining the sensor marker based on the video information and the extrinsic parameter information includes: The video information is input into the image backbone network to output the corresponding semantic map. The extrinsic parameter information is input into the first position encoding network to output a first embedding vector corresponding to the 3D spatial position; The sensor label is generated based on the semantic mapping graph and the first embedding vector.

5. The vehicle control method according to claim 4, characterized in that, By combining meta-reference points and map reference points to process multiple tasks, multiple task query vectors are obtained, including: The meta reference point and the map reference point are input into the second location encoding network to output a second embedding vector corresponding to the 3D spatial location; Semantic analysis is performed on multiple tasks to obtain corresponding task tags; Multiple task query vectors are generated based on the task tag and the second embedding vector.

6. The vehicle control method according to claim 3, characterized in that, The memory queue includes multiple frames of historical features. The memory queue is retrieved based on the target task to determine the time stamp of the previous frame, including: Obtain the historical features of the previous frame from the memory queue; The previous frame historical features are input into the vehicle conversion network, and the deviation is corrected based on the engine compensation network to obtain the previous frame historical features in the current frame coordinate system. The previous frame time marker is determined based on the aforementioned historical features of the previous frame.

7. The vehicle control method according to any one of claims 2-6, characterized in that, The multiple task query vectors include a target query vector, a map query vector, a prediction query vector, and a planning query vector. The multi-dimensional feature information includes target features, map features, and vehicle features. The task detection result is determined based on the multi-dimensional feature information, including: The target detection results and obstacle motivation prediction results are determined based on the target characteristics. Construct an online map based on the map features; The planned trajectory is determined based on the vehicle's characteristics.

8. A vehicle control device, characterized in that, The device includes: The acquisition module is used to acquire video information and external parameter information of the vehicle's onboard camera in response to a task execution command. The feature detection module is used to input the video information, the extrinsic information, and the target task corresponding to the task execution instruction into the preset encoder module to output multi-dimensional feature information corresponding to the target task, wherein the query vector of the preset encoder module is determined based on the target task. The control module is used to determine the task detection result based on the multi-dimensional feature information, and to control the vehicle based on the task detection result.

9. A vehicle, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the vehicle control method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the vehicle control method as described in any one of claims 1 to 7.

Images