Model training method, vehicle control method, device, equipment, medium and vehicle

Abstract

The present disclosure provides a model training method, a vehicle control method, an apparatus, a device, a medium and a vehicle, which relate to the field of artificial intelligence technology, in particular to the field of deep learning and computer vision, and can be applied to scenes such as autonomous driving and intelligent transportation. The specific implementation scheme is as follows: the deep learning model includes a processing sub-model and a control sub-model, and the processing sub-model is used to process the first sample set to obtain the first intermediate feature; the first intermediate feature is input into the control sub-model to obtain the first control parameter that characterizes the control method of the vehicle; the deep learning model is trained according to the difference between the first control parameter and the reference control parameter. The first sample set is obtained by collecting data on the environment in which the vehicle is located within the first historical period, and the first intermediate feature characterizes the first data obtained by collecting data on the environment at the first moment after the first historical period; the reference control parameter characterizes the control method for the vehicle at the first moment.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of deep learning and computer vision, and can be applied to scenes such as autonomous driving and intelligent transportation. More specifically, the present disclosure provides a deep learning model training method, a vehicle control method, a device, an electronic device, a storage medium, a computer program product, and an autonomous driving vehicle. Background Art

[0002] The vehicle needs to be equipped with an automatic driving system to achieve unmanned driving. The automatic driving system can include multiple modules in series. During the working process, the output of the previous module is used as the input of the next module, and each module performs data processing in turn. And plan the control method of the vehicle. Summary of the invention

[0003] The present disclosure provides a deep learning model training method, a vehicle control method, an apparatus, an electronic device, a storage medium, a computer program product, and an autonomous driving vehicle.

[0004] According to one aspect of the present disclosure, a training method for a deep learning model is provided, wherein the deep learning model includes a processing sub-model and a control sub-model, and the method includes: processing a first sample set using the processing sub-model to obtain a first intermediate feature; wherein the first sample set includes a plurality of sample data based on a time series, the first sample set is obtained by collecting data on an environment in which a vehicle is located within a first historical period, and the first intermediate feature characterizes the first data obtained by collecting data on the environment at a first moment after the first historical period; inputting the first intermediate feature into the control sub-model to obtain a first control parameter characterizing a control method of the vehicle; and training the deep learning model according to the difference between the first control parameter and a reference control parameter to obtain a trained deep learning model; wherein the reference control parameter characterizes the control method for the vehicle at the first moment.

[0005] According to another aspect of the present disclosure, a vehicle control method is provided, including: inputting multiple historical data based on time series into a deep learning model to obtain target control parameters; the deep learning model includes a processing sub-model and a control sub-model, and the multiple historical data are obtained by collecting data about the environment in which the vehicle is located; and based on the target control parameters, controlling the automatic driving of the vehicle; wherein, inputting multiple historical data based on time series into the deep learning model to obtain the target control parameters includes: inputting multiple historical data into the processing sub-model to obtain a second intermediate feature; inputting the second intermediate feature into the control sub-model to obtain the target control parameter; wherein the deep learning model is trained using the above-mentioned deep learning model training method.

[0006] According to another aspect of the present disclosure, a training device for a deep learning model is provided, wherein the deep learning model includes a processing sub-model and a control sub-model, and the device includes: a processing module, a first input module, and a training module. The processing module is used to process a first sample set using the processing sub-model to obtain a first intermediate feature; wherein the first sample set includes a plurality of sample data based on a time series, the first sample set is obtained by collecting data on the environment in which the vehicle is located within a first historical period, and the first intermediate feature characterization is the first data obtained by collecting data on the environment at a first moment after the first historical period. The first input module is used to input the first intermediate feature into the control sub-model to obtain a first control parameter that characterizes the control method of the vehicle. The training module is used to train the deep learning model according to the difference between the first control parameter and the reference control parameter to obtain a trained deep learning model; wherein the reference control parameter characterizes the control method for the vehicle at the first moment.

[0007] According to another aspect of the present disclosure, a vehicle control device is provided, comprising: a second input module and a control module. The second input module is used to input multiple historical data based on time series into a deep learning model to obtain target control parameters; the deep learning model includes a processing sub-model and a control sub-model, and the multiple historical data are obtained by collecting data about the environment in which the vehicle is located. The control module is used to control the automatic driving of the vehicle based on the target control parameters. The second input module comprises: a fourth input sub-module and a fifth input sub-module. The fourth input sub-module is used to input multiple historical data into the processing sub-model to obtain a second intermediate feature. The fifth input sub-module is used to input the second intermediate feature into the control sub-model to obtain the target control parameter. The deep learning model is trained using the training device of the above-mentioned deep learning model.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the deep learning model provided by the present disclosure.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vehicle control method provided by the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0012] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the above-mentioned electronic device.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 It is a schematic diagram of an application scenario of a deep learning model training method, a vehicle control method and a device according to an embodiment of the present disclosure;

[0016] Figure 2 is a schematic flow chart of a training method for a deep learning model according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of a pre-trained encoder according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of a pre-trained fusion network according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of a training control sub-model according to an embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram of the phased training of the deep learning model according to an embodiment of the present disclosure;

[0021] Figure 7 is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure;

[0022] Fig. 8A is a schematic flow chart of a vehicle control method according to an embodiment of the present disclosure;

[0023] Figure 8B is a schematic diagram of a vehicle control method according to an embodiment of the present disclosure;

[0024] Fig. 9 is a schematic structural block diagram of a training device for a deep learning model according to an embodiment of the present disclosure;

[0025] Fig.10is a schematic structural block diagram of a vehicle control device according to an embodiment of the present disclosure; and

[0026] Fig.11 It is a structural block diagram of an electronic device used to implement the deep learning model training method and / or vehicle control method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] In some embodiments, the automatic driving system includes multiple modules connected in series, and the multiple modules include, for example, a perception module, a planning module, and a control module. During operation, data can be collected through sensors such as cameras, lidars, millimeter-wave radars, and then the perception module performs perception processing such as static road structure detection, dynamic obstacle detection, signal light detection, and headlight detection through the collected data. Then, the planning module plans information such as paths based on the detection results output by the perception module. After that, the control module performs rule-based planning control based on the results output by the planning module to generate target control parameters, thereby controlling the actions of the accelerator pedal, brake pedal, and steering wheel. By connecting each module in series, each module can be optimized separately, and problems can be easily located. However, the error accumulation of each module in series is large, so the error of the target control parameter finally obtained is large.

[0029] The disclosed embodiment aims to propose a model training method, which can be used to train an end-to-end deep learning model. The input of the deep learning model includes data collected by sensors, and may also include other data such as maps and route information. The output of the deep learning model includes control parameters for the vehicle. Since the full link can be guided to eliminate error accumulation, the result output by the model is more accurate.

[0030] It should be noted that in practical applications, the end-to-end model lacks a certain degree of interpretability. In some embodiments provided in the present disclosure, it can also be trained through self-supervision to give the model stronger representation capabilities and allow the model to establish cognition of the future, thereby enhancing the interpretability of the model.

[0031] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Figure 1It is a schematic diagram of application scenarios of the deep learning model training method, vehicle control method and device according to the embodiments of the present disclosure.

[0033] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include sensors 101 , 102 , 103 , a vehicle 110 , a network 120 , and a server 130 .

[0035] Sensors 101, 102, 103 may interact with server 130 via network 120 to receive or send messages, etc. Sensors 101, 102, 103 may be functional elements integrated in vehicle 110, such as infrared sensors, ultrasonic sensors, millimeter wave radars, image acquisition devices, laser radars, inertial measurement units, etc. Sensors 101, 102, 103 may be used to collect data about the environment in which vehicle 110 is located, such as the positions and states of objects such as pedestrians, vehicles, and obstacles around the vehicle, as well as road data around the vehicle.

[0036] The vehicle 110 may be an autonomous vehicle or a manually driven vehicle with an autonomous driving mode.

[0037] The network 120 is used to provide a medium for communication links between the sensors 101, 102, 103 and the server 130, and may also be used to provide a medium for communication links between the vehicle 110 and the server 130. The network 120 may include various connection types, such as wired and / or wireless communication links, and the like.

[0038] The server 130 may be arranged at a remote end capable of establishing communication with the vehicle-mounted terminal, and may be specifically implemented as a distributed server cluster consisting of multiple servers, or may be implemented as a single server.

[0039] The server 130 may be a server that provides various services. For example, map applications, data processing applications, model training applications, etc. may be installed on the server 130. Taking the server 130 running the model training application as an example: the server 130 may receive collected data such as point cloud data and image data transmitted from the sensors 101, 102, and 103 through the network 120. The collected data may be stored in a database, and the deep learning model may be trained using the data.

[0040] After the server 130 completes the training task, the server 130 may send the trained deep learning model to the vehicle 110, so that the vehicle 110 processes the data collected by the sensors 101, 102, and 103 based on the deep learning model and obtains the target control parameters, thereby controlling the vehicle. Alternatively, the server 130 may also receive the collected data of the sensors 101, 102, and 103, and then use the trained deep learning model to process the collected data and obtain the target control parameters, and then output the target control parameters to the vehicle 110.

[0041] It should be noted that the training method of the deep learning model provided in the embodiment of the present disclosure can generally be executed by the server 130. Accordingly, the training device of the deep learning model provided in the embodiment of the present disclosure can also be set in the server 130. The vehicle control method provided in the embodiment of the present disclosure can generally be executed by the vehicle 110 or the server 130. Accordingly, the vehicle control device provided in the embodiment of the present disclosure can also be set in the vehicle 110 or the server 130.

[0042] Understandably, Figure 1 The number of sensors, vehicles, networks and servers in the embodiment is only for illustration. Any number of sensors, vehicles, networks and servers may be provided as required.

[0043] Figure 2 It is a schematic flowchart of a deep learning model training method according to an embodiment of the present disclosure.

[0044] like Figure 2 As shown, the deep learning model training method 200 may include operations S210 to S230, wherein the training model of the deep learning model may include a processing sub-model and a control sub-model.

[0045] In operation S210, a first sample set is processed using a processing sub-model to obtain a first intermediate feature.

[0046] In operation S220, the first intermediate feature is input into the control sub-model to obtain a first control parameter that characterizes the control method of the vehicle.

[0047] In operation S230, a deep learning model is trained according to a difference between the first control parameter and the reference control parameter to obtain a trained deep learning model.

[0048] For example, by continuously collecting data about the environment in which the vehicle is located using sensors such as laser radar and cameras, multiple data based on time series can be obtained. The data collected in the first historical period can be used as multiple sample data in the first sample set, and the data collected at the first moment after the first historical period can be used as the first data. For example, the first historical period can include the time T1-N1 to the time T1, and the first moment can include the time T1+1.

[0049] For example, the first sample set is input into the processing sub-model, and the processing sub-model outputs a first intermediate feature, and the first intermediate feature represents the first data. For example, the similarity between the feature extracted from the first data and the first intermediate feature is greater than a set threshold.

[0050] For example, the processing sub-model can be a pre-trained neural network. During the pre-training process, the input of the processing sub-model includes multiple collected data based on time series within a period of time. The label can use the data features corresponding to the data collected at the next moment of the period of time. The processing sub-model can be pre-trained based on the difference between the output data of the processing sub-model and the data features.

[0051] For example, the control sub-model may be implemented by a transformer. The first control parameter output by the control sub-model may include at least one of the following: at least one of steering wheel rotation information, vehicle acceleration, brake pedal travel, and accelerator pedal travel.

[0052] For example, the reference control parameter represents the control method for the vehicle at the first moment. For example, the real control parameter at the first moment in history can be used as the reference control parameter, or the reference control parameter can be manually marked.

[0053] For example, cross entropy loss or other loss functions can be used to train a deep learning model by minimizing the model loss based on the gradient descent method.

[0054] During the training process, the parameters of the control sub-model can be adjusted while keeping the parameters of the processing sub-model unchanged. Alternatively, the parameters of the control sub-model can be adjusted and the parameters of the processing sub-model can be fine-tuned.

[0055] The method provided by the embodiment of the present disclosure can train an end-to-end deep learning model. Compared with an autonomous driving system including multiple series modules, the cumulative error of the deep learning model in processing data is smaller and the processing result is more accurate.

[0056] Figure 3 is a schematic diagram of a pre-trained encoder according to an embodiment of the present disclosure.

[0057] According to another embodiment of the present disclosure, the processing sub-model may include an encoder, and the initial encoder M11 may be pre-trained using the third sample set, and the pre-trained initial encoder M11 is used as the encoder. The process of pre-training the initial encoder M11 mainly includes the following processes: encoding, decoding, and adjusting parameters.

[0058] First, the third sample set is described. The sample data 301 in the third sample set is obtained by collecting data about the environment in which the vehicle is located using a sensor. The sensor may include at least one of a laser radar, a camera, etc. Accordingly, a single sample data 301 in the third sample set may include at least one of point cloud data 3011 and image data 3012. It can be understood that, compared with the method of only including single modality data, the single sample data 301 includes multimodal data, which can provide more information for the model, thereby ensuring the model training effect.

[0059] Next, taking a single sample image including point cloud data 3011 and image data 3012 as an example, the encoding, decoding and parameter adjustment process in the encoder pre-training process is explained.

[0060] During the encoding process, the initial encoder M11 may be used to encode the sample data 301 in the third sample set to obtain the third encoding feature 303. For example, the initial encoder M11 that has not completed pre-training may include a point cloud encoder and an image encoder, and the point cloud encoder and the image encoder may be used to extract the point cloud features 3021 of the point cloud data 3011 and the image features 3022 of the image data 3012, respectively, and then the point cloud features 3021 and the image features 3022 are respectively converted to a bird's-eye view (BEV) perspective to obtain the BEV features of the image and the BEV features of the point cloud, and then the two BEV features are fused into the same BEV feature to obtain the third encoding feature 303.

[0061] During the decoding process, the third encoding feature 303 is input into the initial decoder M21, and the initial decoder M21 outputs initial decoded data 304. The initial decoded data 304 may include restored point cloud data 3041 and restored image data 3042.

[0062] In the process of adjusting the parameters, the initial encoder M11 and the initial decoder M21 may be pre-trained according to the difference between the third sample set and the initial decoded data 304 to obtain a pre-trained encoder and a pre-trained decoder.

[0063] For example, a part of the loss L11 is calculated based on the difference between the point cloud data 3011 in the third sample set and the point cloud data 3041 in the initial decoded data 304, and another part of the loss L12 is calculated based on the difference between the image data 3012 in the third sample set and the image data 3042 in the initial decoded data 304. Then, the two parts of the loss L11 and L12 are weighted and calculated or otherwise calculated to obtain the total loss L1, and the parameters of the initial encoder M11 and the initial decoder M21 are adjusted based on the total loss L1 to obtain the pre-trained encoder and the pre-trained decoder. It can be seen that this embodiment calculates the loss of multimodal data separately, and then calculates the total loss L1 through multiple parts of the loss, so the total loss L1 can accurately evaluate the training accuracy of the encoder and the decoder.

[0064] It can be seen that the pre-trained encoder and pre-trained decoder obtained in the above manner can achieve the following functions: the features of the original image data and point cloud data are extracted by the pre-trained encoder, and then the features can be restored to the original image data and point cloud data by the pre-trained codec.

[0065] By adopting the above technical solution, the historically collected data is used as the third sample set and also as the supervisory signal for the reconstruction task. That is, the above process adopts a self-supervisory paradigm and does not require additional annotation of the training labels of the third sample set. In addition, since there is a large amount of historically collected data as the third sample set, the encoder and decoder can learn stronger feature expression capabilities by using the massive data.

[0066] It should be noted that, in other embodiments, if a single sample image includes only one of the point cloud data 3011 and the image data 3012, the encoding process is similar to the above encoding process, except that the missing data is not processed. The initial decoded data 304 obtained after decoding includes only one of the image data 3012 and the point cloud data 3011. There is no need to calculate the loss based on the missing data during the parameter adjustment process.

[0067] Figure 4 It is a schematic diagram of a pre-trained fusion network according to an embodiment of the present disclosure.

[0068] According to another embodiment of the present disclosure, the processing sub-model may include a fusion network, and the initial fusion network M31 may be pre-trained using the second sample set, and the pre-trained initial fusion network M31 is used as the fusion network. The pre-training process for the initial fusion network M31 mainly includes the following processes: encoding, fusion, decoding, and adjusting parameters.

[0069] First, the second sample set is explained. By continuously collecting data about the environment in which the vehicle is located using sensors, multiple sample data 405 based on time series can be obtained. The sensor can include at least one of a laser radar, a camera, etc. The data collected during the second historical period can be used as the second sample set, and the data collected at the second moment after the second historical period can be used as the second data. For example, the second historical period can include the moment T2-N2 to the moment T2, and the second moment can include the moment T2+1, N2 is an integer greater than or equal to 1, and T2 is an integer greater than or equal to N2. Figure 4 Take N2 equal to 2 as an example for illustration, and N2 may also be other values. The single sample data 405 in the second sample set may include at least one of point cloud data and image data.

[0070] Next, taking a single sample image including point cloud data and image data as an example, the encoding, decoding and parameter adjustment process in the pre-training process of the initial fusion network M31 is described. The pre-training process requires the use of a pre-trained encoder M12 and a pre-trained decoder M22, wherein the pre-trained encoder M12 can be configured to convert input data into coding features, and the pre-trained decoder M22 can be configured to convert coding features into output data, and the similarity between the output data and the input data is greater than a threshold.

[0071] During the encoding process, the pre-trained encoder M12 can be used to encode the multiple sample data 405 in the second sample set, so as to obtain multiple second encoded features 406, and each sample data 405 corresponds to a second encoded feature 406. For example, the pre-trained encoder M12 can be used to extract the point cloud features of the point cloud data and the image features of the image data, and then the point cloud features and the image features are respectively converted to the bird's-eye view to obtain the BEV features of the image and the BEV features of the point cloud, and then the two BEV features are fused into one BEV feature to obtain the second encoded feature 406.

[0072] During the fusion process, multiple second encoded features 406 can be input into the initial fusion network M31, and the initial fusion network M31 outputs fused features 407, which represent images and point clouds of future frames, such as point cloud data and image data at time T2+1.

[0073] During the decoding process, the fused feature 407 may be input into the pre-trained decoder M22 to obtain decoded data 408. The pre-trained decoder M22 may restore the image and point cloud of the future frame, and the future frame may correspond to time T2+1, for example.

[0074] In the process of adjusting parameters, the image and point cloud of the future frame in the time sequence can be used as true value supervision, that is, the point cloud data and image data collected at time T2+1 are used as the true value (the second data above is the true value), and the initial fusion network M31 can be pre-trained according to the difference between the decoded data 408 and the true value. It can be understood that the image and point cloud of the future frame in the time sequence are used as the true value supervision, so there is no need for additional supervision, that is, there is no need to mark the labels required for training.

[0075] For example, in the process of adjusting parameters, a part of the loss L21 can be calculated according to the difference between the point cloud data in the decoded data and the point cloud data in the second data, and another part of the loss L22 can be calculated according to the difference between the image data in the decoded data and the image data in the second data, and then the total loss L2 is calculated according to the two parts of the loss L21 and L22, and the parameters of the initial fusion network M31 are adjusted based on the total loss L2. It can be seen that this embodiment calculates the loss based on the multi-modal data separately, and then calculates the total loss L2 through multiple parts of the loss, so the total loss L2 can accurately evaluate the training accuracy of the fusion network.

[0076] In this embodiment, after the encoder M12 processes the data (image data and / or point cloud data) into coding features, the decoder M22 can restore the coding features, and the restored data has a high degree of similarity to the data before encoding. And during training, the supervisory signal is the second data collected at the second moment after the second historical period. In addition, in the process of adjusting the parameters, the parameters of the pre-trained encoder M12 and the pre-trained decoder M22 are frozen, and only the parameters of the initial fusion network M31 are adjusted. Therefore, the pre-trained fusion network can generate future data based on multiple historical data, for example, based on the data collected from time T2-N2 to time T2, and the generated data is relatively similar to the data collected at time T2+1. Therefore, the pre-trained fusion network can better output the features of future data, thereby having the ability to infer future data based on historical data.

[0077] Figure 5 It is a schematic diagram of the training control sub-model according to an embodiment of the present disclosure.

[0078] In this embodiment, the processing sub-model may include an encoder M12 and a fusion network M32, and the fusion network M32 may be a Transformer network, for example. The control sub-model M4 may be trained using the first sample set, and the training process may include the following processes: encoding, fusion, and parameter adjustment.

[0079] First, the first sample set is described. The data collected from time T1-N1 to time T1 can be used as the multiple sample data 509 in the first sample set, and the data collected at time T1+1 can be used as the first data. N1 is an integer greater than or equal to 1, T1 is an integer greater than or equal to N1, Figure 5 Take N1 equal to 2 as an example for illustration, and N1 may also be other values. The single sample data 509 in the first sample set may include at least one of point cloud data and image data.

[0080] Next, taking a single sample image including point cloud data and image data as an example, the encoding, fusion, control parameter generation and parameter adjustment processes in the training control sub-model M4 are explained.

[0081] During the encoding process, the encoder M12 is used to encode the multiple sample data 509 in the first sample set respectively to obtain multiple first encoding features 510.

[0082] For example, the encoder M12 may include an image encoder and a point cloud encoder. For a single sample image, the image encoder is used to encode the image data in the sample data 509 to obtain image features. The point cloud encoder is also used to encode the point cloud data in the sample data 509 to obtain point cloud features. The image features and the point cloud features are then fused to obtain the first coded features 510. In this way, the sample data at three moments correspond to three first coded features 510. It can be understood that, compared to the method of only including single modal data, the single sample data 301 in this embodiment includes multi-modal data. Therefore, extracting features from data of different modalities and then fusing them can provide more information for the model, thereby ensuring the model training effect.

[0083] For example, in the process of determining the first coded feature 510 based on the image feature and the point cloud feature, the image feature can be converted into an image top view feature under a bird's-eye view perspective, and the point cloud feature can be converted into a point cloud top view feature under a bird's-eye view perspective, and then the image top view feature and the point cloud top view feature are fused to obtain the first coded feature 510. The features from a bird's-eye view perspective can provide a more intuitive representation of the world, which is conducive to data fusion.

[0084] During the fusion process, multiple first encoding features 510 can be input into the fusion network M32 to obtain a first intermediate feature 511.

[0085] In the process of generating control parameters, the first intermediate feature 511 can be input into the control sub-model M4, and the control sub-model M4 outputs the first control parameter 512.

[0086] In the process of adjusting parameters, the loss L3 can be calculated based on the difference between the first control parameter 512 and the reference control parameter 513, and then the parameters of the control sub-model M4 can be adjusted based on the loss L3, and the parameters of the encoder M12 and the fusion network M32 can also be adjusted.

[0087] Figure 6 It is a schematic diagram of the staged training of the deep learning model according to an embodiment of the present disclosure.

[0088] like Figure 6 As shown, the deep learning model provided in this embodiment includes an encoder, a decoder, a fusion network and a control sub-model, and the training method of the deep learning model includes a first stage Stage_1, a second stage Stage_2 and a third stage Stage_3.

[0089] like Figure 6 As shown, the first stage Stage_1 mainly uses the third sample set to pre-train the encoder and decoder to obtain a pre-trained encoder and a pre-trained decoder. The second stage Stage_2 mainly uses the second sample set to pre-train the fusion network on the basis of freezing the parameters of the pre-trained encoder and the pre-trained decoder. The third stage Stage_3 mainly uses the first sample set to train the control sub-model, and the parameters of the encoder and the fusion network can be adjusted again during the training process.

[0090] Figure 7 Schematic diagram of the training method of the deep learning model according to the embodiment of the present disclosure. Figure 7 The first stage Stage_1, the second stage Stage_2 and the third stage Stage_3 are described in detail below.

[0091] First, the first stage Stage_1 is explained. For example, a single sample data 701 in the third sample set includes image data and point cloud data. The image data and point cloud data are respectively encoded using the image encoder and point cloud encoder in the initial encoder M11 to obtain image features and point cloud features, and then the image features and point cloud features are converted into BEV features 703 (i.e., the third encoded features above). The BEV features 703 are then input into the initial decoder M21 to obtain initial decoded data 704, which includes restored image data and restored point cloud data. Then, the total loss L1 of the image data, point cloud data in the sample data 701 and the image and point cloud in the decoded data 704 is calculated, and the parameters of the initial encoder M11 and the initial decoder M21 are adjusted based on the total loss L1, thereby obtaining a pre-trained encoder M12 and a pre-trained decoder M22.

[0092] It can be seen that the first stage Stage_1 uses the original image data and point cloud data as the supervision information for the reconstruction task. There is no need for additional annotation of sample labels. In addition, the massive sample data can make the pre-trained encoder and pre-trained decoder have stronger feature expression capabilities.

[0093] The second stage Stage_2 is described below.

[0094] For example, the second sample set includes a plurality of sample data 705 collected from time T2-N2 to time T2. Figure 7 Take N2 equal to 2 as an example for explanation. The sample data 705 are input into the pre-trained encoder M12, so that N2+1 second encoded features 706 are obtained. The encoding process of the encoder M12 is similar to the working process of the first stage Stage_1 encoder M11, which will not be repeated here. Then the N2+1 second encoded features 706 are input into the initial fusion network M31, and the initial fusion network M31 outputs the fusion feature 707. The fusion feature 707 represents the image and point cloud at the future time T2+1. The fusion feature 707 is then input into the pre-trained decoder M22 to obtain the decoded data 708, which includes the restored image data and the restored point cloud data.

[0095] Next, based on the image data, point cloud data, restored image and restored point cloud total loss L2 in the sample data 705, and freezing the parameters of the pre-trained encoder M12 and the pre-trained decoder M22, only the parameters of the initial fusion network M31 are adjusted to obtain the pre-trained fusion network M32.

[0096] It can be seen that the second stage Stage_2 uses the image data and point cloud data of future frames in the time series as true value supervision. Therefore, similar to the first stage Stage_1, it is also self-supervised training without additional labeling of sample labels. The fusion network trained in the second stage Stage_2 has the ability to infer future data based on historical data, allowing the model to establish cognition of the future.

[0097] The third stage Stage_3 is described below.

[0098] For example, the data collected from time T1-N1 to time T1 are used as the plurality of sample data 709 in the first sample set. Figure 7 Take N1 equal to 3 as an example. The sample data 709 are input into the pre-trained encoder M12, so as to obtain N1+1 first encoding features 710. The encoding process of the encoder M12 is similar to the working process of the first stage Stage_1 encoder M11, which will not be repeated here.

[0099] Then, the N1+1 first encoding features 710 are input into the fusion network M32, and the fusion network M32 outputs the first intermediate features 711. Then, the first intermediate features 711 are input into the control sub-model M4, and the control sub-model M4 outputs the first control parameters 712. Then, the loss L3 is calculated based on the difference between the first control parameters 712 and the reference control parameters at time T1+1, and the parameters of the control sub-model M4 are adjusted based on the loss L3, and the encoder M12 and the fusion network M32 can also be fine-tuned.

[0100] The above is a detailed description of the first stage Stage_1, the second stage Stage_2 and the third stage Stage_3. It can be seen that the first stage Stage_1 uses the reconstruction task for self-supervised training, and the model has a strong feature expression ability. The second stage Stage_2 uses the self-supervised training method of reasoning about the future, so that the model can understand the future. Then, based on the training of the third stage Stage_3, the control sub-model has the ability to determine the vehicle control method based on the collected data.

[0101] By adopting the training method provided in this embodiment, by checking the restoration status of future frames, such as checking the decoded data, and determining the status of the restored image data and the restored point cloud data, the training status of the encoder, decoder and fusion network can be known, thereby enhancing the interpretability of the model.

[0102] It should be noted that the first stage Stage_1, the second stage Stage_2 and the third stage Stage_3 mentioned above all use sample data for training, that is, the difference between the three stages mainly lies in the different training methods, and there is no restriction on the training samples. Therefore, the sample data used in the three stages can be completely the same, completely different, or partially the same. Correspondingly, the first historical period and the second historical period mentioned above can be the same period, partially overlapping periods, or completely non-overlapping periods. Similarly, the first moment and the second moment can be the same moment or different moments.

[0103] Fig. 8A is a schematic flow chart of a vehicle control method according to an embodiment of the present disclosure, Figure 8B is a schematic diagram of a vehicle control method according to an embodiment of the present disclosure.

[0104] like Fig. 8A and Figure 8B As shown, the vehicle control method 800 may include operations S810 to S820.

[0105] In operation S810, a plurality of time-series-based historical data is input into a deep learning model to obtain a target control parameter.

[0106] In operation S820, the vehicle is controlled to automatically drive based on the target control parameter.

[0107] For example, multiple historical data are time series data obtained by collecting data about the environment in which the vehicle is located, and a single historical data may include at least one of image data and point cloud data.

[0108] For example, the deep learning model can be trained using the training method described above.

[0109] For example, the deep learning model includes a processing sub-model and a control sub-model. Multiple historical data can be input into the processing sub-model to obtain a second intermediate feature, and then the second intermediate feature can be input into the control sub-model to obtain the target control parameter.

[0110] For example, the processing sub-model may include an encoder and a fusion network, and the encoder may include an image encoder and a point cloud encoder. During operation, the encoder may extract point cloud features of point cloud data and image features of image data, and then convert the two features into encoded data from a bird's-eye view. Then, multiple encoded data corresponding to multiple historical data are input into the fusion network, and the fusion network outputs future data representations, which are then input into the control sub-model, and the control sub-model outputs target control parameters.

[0111] For example, the target control parameter may include at least one of steering wheel rotation information, vehicle acceleration, brake pedal travel, and accelerator pedal travel. The vehicle may be controlled according to the steering wheel angle, acceleration, travel, etc. indicated by the target control parameter.

[0112] It can be seen that the encoder can be trained in the process of training the deep learning model, while the encoder is not needed in the process of generating target control parameters using the deep learning model.

[0113] This embodiment uses an end-to-end deep learning model to process the data collected by the sensor and generate target control parameters. Compared with the autonomous driving system with multiple modules connected in series, it can reduce the cumulative error between modules and improve the control accuracy. In addition, the deep learning module has strong feature expression ability, future cognition ability, and strong interpretability, so it can ensure the accuracy of control.

[0114] Fig. 9 It is a schematic structural block diagram of a training device for a deep learning model according to an embodiment of the present disclosure.

[0115] like Fig. 9 As shown, the deep learning model includes a processing sub-model and a control sub-model, and the training device 900 of the deep learning model may include a processing module 910, a first input module 920 and a training module 930.

[0116] The processing module 910 is used to process the first sample set using the processing sub-model to obtain a first intermediate feature; wherein the first sample set includes multiple sample data based on time series, the first sample set is obtained by collecting data on the vehicle's environment within a first historical period, and the first intermediate feature represents: the first data obtained by collecting data on the environment at the first moment after the first historical period.

[0117] The first input module 920 is used to input the first intermediate feature into the control sub-model to obtain a first control parameter that characterizes the control method of the vehicle.

[0118] The training module 930 is used to train the deep learning model according to the difference between the first control parameter and the reference control parameter to obtain a trained deep learning model; wherein the reference control parameter represents the control method for the vehicle at the first moment.

[0119] In this embodiment, the processing sub-model includes an encoder and a fusion network. The processing module includes: a first encoding sub-module and a first input sub-module. The first encoding sub-module is used to use the encoder to encode multiple sample data in the first sample set respectively to obtain multiple first encoding features. The first input sub-module is used to input the multiple first encoding features into the fusion network to obtain the first intermediate feature.

[0120] In this embodiment, each sample data in the first sample set includes image data and point cloud data, and the encoder includes an image encoder and a point cloud encoder. The first encoding submodule includes: a first encoding unit, a second encoding unit and a fusion unit. The first encoding unit is used to encode the image data in the sample data using the image encoder for each sample data in the first sample set to obtain image features. The second encoding unit is used to encode the point cloud data in the sample data using the point cloud encoder to obtain point cloud features. The fusion unit is used to fuse the image features and the point cloud features to obtain the first encoding features.

[0121] In this embodiment, the fusion unit includes: a first conversion subunit, a second conversion subunit and a fusion subunit. The first conversion subunit is used to convert the image feature into an image top view feature under a bird's-eye view perspective. The second conversion subunit is used to convert the point cloud feature into a point cloud top view feature under a bird's-eye view perspective. The fusion subunit is used to fuse the image top view feature and the point cloud top view feature to obtain a first coding feature.

[0122] In this embodiment, the fusion network is obtained by pre-training the initial fusion network, and the device also includes a first pre-training module for pre-training operations on the initial fusion network, and the first pre-training module includes: a second encoding submodule, a second input submodule, a third input submodule and a first pre-training submodule. The second encoding submodule is used to encode multiple sample data in the second sample set using a pre-trained encoder to obtain multiple second encoding features. The second input submodule is used to input the multiple second encoding features into the initial fusion network to obtain fusion features. The third input submodule is used to input the fusion features into the pre-trained decoder to obtain decoded data. The first pre-training submodule is used to pre-train the initial fusion network according to the difference between the decoded data and the second data. The second sample set includes multiple sample data based on time series, the second sample set is obtained by collecting data on the environment in which the vehicle is located within the second historical period, and the second data is obtained by collecting data on the environment in which the vehicle is located at the second moment after the second historical period.

[0123] In this embodiment, the pre-trained encoder is configured to convert input data into coding features, and the pre-trained decoder is configured to convert the coding features into output data, and the similarity between the output data and the input data is greater than a threshold.

[0124] In this embodiment, the first pre-training submodule includes: a first adjustment unit, which is used to adjust the parameters of the initial fusion network according to the difference between the decoded data and the second data while keeping the parameters of the pre-trained encoder and the parameters of the pre-trained decoder unchanged.

[0125] In this embodiment, the decoded data includes image data and point cloud data, and the second data includes image data and point cloud data. The first pre-training submodule includes: a second adjustment unit, configured to adjust the parameters of the initial fusion network according to the difference between the image data in the decoded data and the image data in the second data, and the difference between the point cloud data in the decoded data and the point cloud data in the second data.

[0126] In this embodiment, the encoder is obtained by pre-training the initial encoder, and the device also includes a second pre-training module for pre-training the initial encoder, and the second pre-training module includes: a third encoding submodule, a decoding submodule, and a second pre-training submodule. The third encoding submodule is used to encode the sample data in the third sample set using the initial encoder to obtain a third encoding feature. The decoding submodule is used to decode the third encoding feature using the initial decoder to obtain initial decoded data. The second pre-training submodule is used to pre-train the initial encoder and the initial decoder according to the difference between the third sample set and the initial decoded data to obtain a pre-trained encoder and a pre-trained decoder.

[0127] In this embodiment, each sample data in the third sample set includes image data and point cloud data, and the initial decoded data includes image data and point cloud data. The second pre-training submodule includes: a third adjustment unit, which is used to adjust the parameters of the initial encoder and the initial decoder according to the difference between the image data in the third sample set and the image data in the initial decoded data, and the difference between the point cloud data in the third sample set and the point cloud data in the initial decoded data, so as to obtain a pre-trained encoder and a pre-trained decoder.

[0128] In this embodiment, the training module includes one of a first adjustment submodule and a second adjustment submodule. The first adjustment submodule is used to adjust the parameters of the processing submodel and the parameters of the control submodel. The second adjustment submodule is used to adjust the parameters of the control submodel while keeping the parameters of the processing submodel unchanged.

[0129] In this embodiment, the first control parameter includes at least one of the following: at least one of steering wheel rotation information, vehicle acceleration, brake pedal travel, and accelerator pedal travel.

[0130] Fig.10 It is a schematic structural block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0131] like Fig.10 As shown, the vehicle control device 1000 may include a second input module 1010 and a control module 1020 .

[0132] The second input module 1010 is used to input multiple historical data based on time series into the deep learning model to obtain target control parameters. The deep learning model includes a processing sub-model and a control sub-model, and the multiple historical data are obtained by collecting data about the environment in which the vehicle is located. The second input module 1010 includes: a fourth input sub-module and a fifth input sub-module, and the fourth input sub-module is used to input multiple historical data into the processing sub-model to obtain a second intermediate feature. The fifth input sub-module is used to input the second intermediate feature into the control sub-model to obtain the target control parameter. The deep learning model is trained using the training device of the above-mentioned deep learning model.

[0133] The control module 1020 is used to control the automatic driving of the vehicle based on the target control parameters.

[0134] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned deep learning model training method.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned vehicle control method.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides an autonomous driving vehicle, comprising the above-mentioned electronic device.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned deep learning model training method and / or vehicle control method.

[0138] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, including a computer program, which implements the above-mentioned deep learning model training method and / or vehicle control method when executed by a processor.

[0139] Fig.11 It is a block diagram of an electronic device for implementing the training method of the deep learning model and / or the vehicle control method of the embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0140] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0141] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0142] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the training method of the deep learning model and / or the vehicle control method. For example, in some embodiments, the training method of the deep learning model and / or the vehicle control method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the training method of the deep learning model and / or the vehicle control method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the deep learning model training method and / or the vehicle control method in any other suitable manner (e.g., by means of firmware).

[0143] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0145] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0148] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0149] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0150] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0151] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a deep learning model, wherein the deep learning model includes a processing sub-model and a control sub-model, wherein the processing sub-model includes an encoder and a fusion network; the method includes: Processing a first sample set using the processing sub-model to obtain a first intermediate feature; wherein the first sample set includes a plurality of sample data based on a time series, the first sample set is obtained by collecting data on an environment in which the vehicle is located within a first historical period, and the first intermediate feature representation is first data obtained by collecting data on the environment at a first moment after the first historical period; Inputting the first intermediate feature into the control sub-model to obtain a first control parameter characterizing a control method of the vehicle; and Training the deep learning model according to a difference between the first control parameter and a reference control parameter to obtain a trained deep learning model; wherein the reference control parameter represents a control method for the vehicle at the first moment; The fusion network is obtained by pre-training the initial fusion network, and the pre-training operation for the initial fusion network includes: using a pre-trained encoder to encode multiple sample data in the second sample set to obtain multiple second encoding features; inputting the multiple second encoding features into the initial fusion network to obtain fusion features; inputting the fusion features into the pre-trained decoder to obtain decoded data; and pre-training the initial fusion network according to the difference between the decoded data and the second data; wherein the second sample set includes multiple sample data based on time series, the second sample set is obtained by collecting data on the vehicle's environment within a second historical period, and the second data is obtained by collecting data on the vehicle's environment at a second moment after the second historical period.

2. The method according to claim 1, wherein: The using the processing sub-model to process the first sample set to obtain the first intermediate feature comprises: Using the encoder to encode the plurality of sample data in the first sample set respectively to obtain a plurality of first encoding features; and The multiple first encoding features are input into a fusion network to obtain the first intermediate features.

3. The method according to claim 2, wherein: Each sample data in the first sample set includes image data and point cloud data, and the encoder includes an image encoder and a point cloud encoder; and the encoder is used to encode the multiple sample data in the first sample set respectively to obtain multiple first encoding features, including: For each sample data in the first sample set, Encoding the image data in the sample data using the image encoder to obtain image features; Encoding the point cloud data in the sample data using the point cloud encoder to obtain point cloud features; and The image feature and the point cloud feature are fused to obtain the first coding feature.

4. The method according to claim 3, wherein: The fusing the image feature and the point cloud feature to obtain the first coding feature comprises: Converting the image features into top-view features of the image from a bird's-eye view perspective; Converting the point cloud features into point cloud top view features from a bird's eye view perspective; and The image top view feature and the point cloud top view feature are fused to obtain the first coding feature.

5. The method according to claim 1, wherein: The pre-trained encoder is configured to convert input data into encoding features, and the pre-trained decoder is configured to convert the encoding features into output data, and the similarity between the output data and the input data is greater than a threshold.

6. The method according to claim 5, wherein: Pre-training the initial fusion network according to the difference between the decoded data and the second data includes: While keeping the parameters of the pre-trained encoder and the parameters of the pre-trained decoder unchanged, the parameters of the initial fusion network are adjusted according to the difference between the decoded data and the second data.

7. The method according to claim 5, wherein: The decoded data includes image data and point cloud data, and the second data includes image data and point cloud data; and pre-training the initial fusion network according to the difference between the decoded data and the second data includes: Adjust the parameters of the initial fusion network according to the difference between the image data in the decoded data and the image data in the second data, and the difference between the point cloud data in the decoded data and the point cloud data in the second data.

8. The method according to any one of claims 2 to 7, wherein: The encoder is obtained by pre-training an initial encoder, and the pre-training operation for the initial encoder includes: Encoding the sample data in the third sample set using the initial encoder to obtain a third encoding feature; Decoding the third coding feature using an initial decoder to obtain initial decoded data; and According to the difference between the third sample set and the initial decoded data, the initial encoder and the initial decoder are pre-trained to obtain a pre-trained encoder and a pre-trained decoder.

9. The method according to claim 8, wherein: Each sample data in the third sample set includes image data and point cloud data, and the initial decoded data includes image data and point cloud data; and pre-training the initial encoder and the initial decoder according to the difference between the third sample set and the initial decoded data includes: According to the difference between the image data in the third sample set and the image data in the initial decoded data, and the difference between the point cloud data in the third sample set and the point cloud data in the initial decoded data, the parameters of the initial encoder and the initial decoder are adjusted to obtain a pre-trained encoder and a pre-trained decoder.

10. The method according to claim 1, wherein: Training the deep learning model according to a difference between the first control parameter and a reference control parameter comprises one of the following: adjusting parameters of the processing sub-model and parameters of the control sub-model; and On the basis of keeping the parameters of the processing sub-model unchanged, the parameters of the control sub-model are adjusted.

11. The method according to claim 1, wherein: The first control parameter includes at least one of the following: at least one of steering wheel rotation information, vehicle acceleration, brake pedal travel, and accelerator pedal travel.

12. A vehicle control method, comprising: Input multiple historical data based on time series into the deep learning model to obtain the target control parameters; The deep learning model includes a processing sub-model and a control sub-model, and the plurality of historical data are obtained by collecting data of the environment in which the vehicle is located; as well as Based on the target control parameters, control the vehicle's automatic driving; Among them, multiple historical data based on time series are input into the deep learning model, and the target control parameters obtained include: Inputting the plurality of historical data into the processing sub-model to obtain a second intermediate feature; Inputting the second intermediate feature into the control sub-model to obtain the target control parameter; Wherein, the deep learning model is trained using the method described in any one of claims 1 to 11.

13. A training device for a deep learning model, the deep learning model comprising a processing sub-model and a control sub-model, the processing sub-model comprising an encoder and a fusion network, the fusion network being obtained by pre-training an initial fusion network; the device comprising: a processing module, configured to process a first sample set using the processing sub-model to obtain a first intermediate feature; wherein the first sample set includes a plurality of sample data based on a time series, the first sample set is obtained by collecting data on an environment in which the vehicle is located within a first historical period, and the first intermediate feature representation is first data obtained by collecting data on the environment at a first moment after the first historical period; A first input module, configured to input the first intermediate feature into the control sub-model to obtain a first control parameter characterizing a control method of the vehicle; and A training module, configured to train the deep learning model according to a difference between the first control parameter and a reference control parameter to obtain a trained deep learning model; wherein the reference control parameter represents a control method for the vehicle at the first moment; A first pre-training module, configured to perform a pre-training operation on the initial fusion network, the first pre-training module; Wherein, the first pre-training module includes: A second encoding submodule, used to encode multiple sample data in the second sample set using a pre-trained encoder to obtain multiple second encoding features; A second input submodule, used for inputting the plurality of second encoding features into the initial fusion network to obtain fusion features; A third input submodule is used to input the fusion feature into a pre-trained decoder to obtain decoded data; and A first pre-training submodule, configured to pre-train the initial fusion network according to a difference between the decoded data and the second data; Among them, the second sample set includes multiple sample data based on time series, the second sample set is obtained by collecting data on the vehicle's environment within a second historical period, and the second data is obtained by collecting data on the vehicle's environment at a second moment after the second historical period.

14. The device according to claim 13, wherein: The processing module comprises: A first encoding submodule, configured to respectively encode the plurality of sample data in the first sample set using the encoder to obtain a plurality of first encoding features; and The first input submodule is used to input the multiple first encoding features into a fusion network to obtain the first intermediate feature.

15. The device according to claim 14, wherein: Each sample data in the first sample set includes image data and point cloud data, and the encoder includes an image encoder and a point cloud encoder; The first encoding submodule includes: A first encoding unit, configured to encode image data in each sample data in the first sample set by using the image encoder to obtain image features; A second encoding unit, configured to encode the point cloud data in the sample data using the point cloud encoder to obtain point cloud features; and A fusion unit is used to fuse the image feature and the point cloud feature to obtain the first coding feature.

16. The device according to claim 15, wherein: The fusion unit comprises: A first conversion subunit, configured to convert the image features into image top view features from a bird's-eye view perspective; A second conversion subunit is used to convert the point cloud features into point cloud top view features under a bird's eye view perspective; and The fusion subunit is used to fuse the image top view feature and the point cloud top view feature to obtain the first coding feature.

17. The device according to claim 13, wherein: The pre-trained encoder is configured to convert input data into encoding features, and the pre-trained decoder is configured to convert the encoding features into output data, and the similarity between the output data and the input data is greater than a threshold.

18. The device according to claim 17, wherein: The first pre-training submodule includes: The first adjustment unit is used to adjust the parameters of the initial fusion network according to the difference between the decoded data and the second data while keeping the parameters of the pre-trained encoder and the parameters of the pre-trained decoder unchanged.

19. The device according to claim 17, wherein: The decoded data includes image data and point cloud data, the second data includes image data and point cloud data; the first pre-training submodule includes: A second adjustment unit is used to adjust the parameters of the initial fusion network according to the difference between the image data in the decoded data and the image data in the second data, and the difference between the point cloud data in the decoded data and the point cloud data in the second data.

20. The device according to any one of claims 14 to 19, wherein: The encoder is obtained by pre-training an initial encoder, and the device further includes a second pre-training module for pre-training the initial encoder, wherein the second pre-training module includes: A third encoding submodule, used to encode the sample data in the third sample set using the initial encoder to obtain a third encoding feature; a decoding submodule, configured to decode the third coding feature using an initial decoder to obtain initial decoded data; and The second pre-training submodule is used to pre-train the initial encoder and the initial decoder according to the difference between the third sample set and the initial decoding data to obtain a pre-trained encoder and a pre-trained decoder.

21. The device according to claim 20, wherein: Each sample data in the third sample set includes image data and point cloud data, and the initial decoded data includes image data and point cloud data; The second pre-training submodule includes: A third adjustment unit is used to adjust the parameters of the initial encoder and the initial decoder according to the difference between the image data in the third sample set and the image data in the initial decoded data, and the difference between the point cloud data in the third sample set and the point cloud data in the initial decoded data, so as to obtain a pre-trained encoder and a pre-trained decoder.

22. The device according to claim 13, wherein: The training module includes one of the following: a first adjustment submodule, configured to adjust parameters of the processing submodel and parameters of the control submodel; and The second adjustment submodule is used to adjust the parameters of the control submodel on the basis of keeping the parameters of the processing submodel unchanged.

23. The device according to claim 13, wherein: The first control parameter includes at least one of the following: at least one of steering wheel rotation information, vehicle acceleration, brake pedal travel, and accelerator pedal travel.

24. A vehicle control device, comprising: The second input module is used to input multiple historical data based on time series into the deep learning model to obtain target control parameters; The deep learning model includes a processing sub-model and a control sub-model, and the plurality of historical data are obtained by collecting data of the environment in which the vehicle is located; as well as A control module, used for controlling the automatic driving of the vehicle based on target control parameters; Wherein, the second input module includes: a fourth input submodule, configured to input the plurality of historical data into the processing submodel to obtain a second intermediate feature; a fifth input submodule, configured to input the second intermediate feature into the control submodel to obtain the target control parameter; Wherein, the deep learning model is trained using the device described in any one of claims 13 to 23.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

26. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor so as to enable the at least one processor to perform the method of claim 12 .

27. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 12.

28. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

29. An autonomous driving vehicle comprising the electronic device described in claim 26.

Images