Vehicle control methods and devices, control prediction model training methods and devices

Abstract

This application relates to the field of vehicle control technology, and discloses a vehicle control method and apparatus, and a control prediction model training method and apparatus. The method includes: first, acquiring scene image information and control information of the vehicle. The control information is used to characterize the controllable items of the vehicle. Then, the scene image information is input into a first input sub-model in the control prediction model to obtain scene features; simultaneously, the control information is input into a second input sub-model in the control prediction model to obtain control features. Next, the scene features and control features are processed based on the output sub-model in the control prediction model to obtain a target control decision. Finally, the vehicle can be controlled according to the target control decision. Applying the technical solution of this invention can more accurately predict the control decisions of autonomous vehicles.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of vehicle control technology, specifically to a vehicle control method and apparatus, and a control prediction model training method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence and chip computing power, traditional automobiles are gradually moving towards new energy and intelligent directions, and autonomous driving technology is a breakthrough representative of this. Currently, autonomous driving control generally uses traditional feature extraction methods to first extract the environmental features and vehicle control features during the vehicle's driving process, and then uses pre-trained models to analyze them, thereby predicting control decisions as the basis for autonomous driving.

[0003] However, the accuracy of traditional feature extraction methods is dependent on the specific extraction method chosen by the user, making it difficult to guarantee good accuracy. This results in less accurate subsequent control decisions, leading to a poor user experience. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a vehicle control method and apparatus, and a control prediction model training method and apparatus, to solve the problem that the prediction of control decisions for autonomous driving in the prior art is not accurate enough.

[0005] Firstly, this application provides a vehicle control method, which includes: firstly acquiring scene image information and control information of the vehicle. The control information is used to characterize the controllable items of the vehicle. Then, the scene image information is input into a first input sub-model of a control prediction model to obtain scene features; simultaneously, the control information is input into a second input sub-model of the control prediction model to obtain control features. Next, the scene features and control features are processed based on the output sub-model of the control prediction model to obtain a target control decision. Finally, the vehicle can be controlled according to the target control decision.

[0006] Based on the technical solution provided in this application, scene image information and control information of the vehicle can be obtained first. Then, a pre-trained deep learning model, specifically the first and second input sub-models in the control prediction model, can be used to extract scene features and control features from the scene image information and control information, respectively. Subsequently, the output sub-model in the control prediction model can predict the target control decision based on the scene features and control features obtained from the two input sub-models. Because the control prediction model is pre-trained, its two input sub-models can acquire the ability to extract features from data based on a large number of training samples, which is more accurate and convenient compared to existing feature extraction methods. The output sub-model can acquire the ability to obtain control decisions based on scene features and control features based on a large number of samples. Combining the capabilities of the two input sub-models, the control decision can be accurately and quickly determined using the vehicle's scene image information and control information. Then, the vehicle can be controlled based on this decision. Therefore, based on the technical solution provided in this application, feature extraction (scene features and control features) can be completed conveniently and accurately, thereby quickly and conveniently obtaining the target control decision for the vehicle, achieving the goal of autonomous driving, and improving the user experience.

[0007] Furthermore, because the technical solution provided in this application contains three independently existing sub-models in the control prediction model, if a problem or malfunction occurs in a certain sub-model, one of them can be adjusted individually. This makes model adjustment in the technical solution provided in this application more convenient and easier for users to use.

[0008] In one possible design approach of the first aspect, acquiring scene image information and control information includes: acquiring scene image data and control data of the vehicle; and performing data preprocessing on the scene image data and control data of the vehicle to obtain scene image information and control information.

[0009] Understandably, the scene image data and internal control data of vehicles that can be acquired in practice are quite complex. For example, the scene image data can be scene images acquired by the vehicle's cameras or point cloud data acquired by external radar. The vehicle's control data can originate from the CAN (Controller Area Network) bus communication protocol messages defined for the entire vehicle in the autonomous driving controller, which can include control signal data from vehicle architecture domains such as the power domain, braking domain, body domain, and cockpit domain. For ease of use, this data needs to be preprocessed to obtain data that can be directly used for subsequent predictions. Based on this, in this application, the scene image information and control information can be obtained by: acquiring external image data and control data, performing data preprocessing to obtain scene image information and control information, and providing data support for subsequent control decision predictions.

[0010] In one possible design approach of the first aspect, the first input sub-model, the second input sub-model, and the output sub-model are all convolutional neural networks.

[0011] It is understandable that convolutional neural networks have strong feature extraction and classification capabilities, so in this embodiment, the three sub-models in the control prediction model can all be convolutional neural networks.

[0012] Secondly, this application provides a method for training a control prediction model. The method includes: acquiring multiple sets of sample data and corresponding sample control decisions; the sample data includes sample scene image information and sample control information; inputting the sample scene image information into a first input sub-model of an initial control prediction model to obtain sample scene features; inputting the sample control information into a second input sub-model of the initial control prediction model to obtain sample control features; processing the sample scene features and sample control features based on the output sub-model of the initial control prediction model to obtain predictive control decisions; using the predictive control decisions as the initial training output of the initial control prediction model and the sample control decisions as supervisory information, iteratively training the initial control prediction model to obtain the trained control prediction model.

[0013] Based on the technical solution corresponding to the above design approach, a control prediction model can be trained using deep learning. This model enables it to extract features from scene image information and control information, and predict control decisions based on the extracted features. Therefore, the vehicle control method provided in this application can conveniently and quickly predict vehicle control decisions using this model.

[0014] In one possible design approach of the first aspect, the predictive control decision is used as the initial training output of the initial control prediction model, and the sample control decision is used as the supervision information. The initial control prediction model is iteratively trained to obtain the trained control prediction model, including: determining the loss value based on the predictive control decision and the sample control decision; and iteratively updating the initial control prediction model based on the loss value to obtain the trained control prediction model.

[0015] Based on the above technical solution, the initial control prediction model can be continuously iterated and optimized based on training data and loss values, thereby obtaining the control prediction model.

[0016] Thirdly, this application provides a vehicle control device, which includes: an acquisition module, a processing module, and a control module.

[0017] The system comprises the following modules: an acquisition module for acquiring scene image information and control information of the vehicle; control information for characterizing controllable items of the vehicle; a processing module for inputting the scene image information acquired by the acquisition module into the first input sub-model of the control prediction model to obtain scene features; a processing module for inputting the control information acquired by the acquisition module into the second input sub-model of the control prediction model to obtain control features; a processing module for processing the scene features and control features based on the output sub-model of the control prediction model to obtain a target control decision; and a control module for controlling the vehicle according to the target control decision obtained by the processing module.

[0018] In one possible implementation of the third aspect, the acquisition module is specifically used to: acquire scene image data and control data of the vehicle; and perform data preprocessing on the scene image data and control data to obtain scene image information and control information of the vehicle.

[0019] In one possible implementation of the third aspect, the first input sub-model, the second input sub-model, and the output sub-model are all convolutional neural networks.

[0020] Fourthly, this application provides a control prediction model training device, which includes: an acquisition module for acquiring multiple sets of sample data and sample control decisions corresponding to the multiple sets of sample data; the sample data includes sample scene image information and sample control information; a training module for inputting the sample scene image information acquired by the acquisition module into a first input sub-model in an initial control prediction model to obtain sample scene features; the training module is further used to input the sample control information acquired by the acquisition module into a second input sub-model in the initial control prediction model to obtain sample control features; the training module is further used to process the sample scene features and sample control features based on the output sub-model in the initial control prediction model to obtain predictive control decisions; the training module is further used to iteratively train the initial control prediction model using the predictive control decisions as the initial training output of the initial control prediction model and the sample control decisions acquired by the acquisition module as supervision information to obtain a trained control prediction model.

[0021] In one possible implementation of the fourth aspect, the training module includes a computation unit and an optimization unit; the computation unit is used to determine the loss value based on the predictive control decision and the sample control decision; the optimization unit is used to iteratively update the initial control prediction model based on the loss value calculated by the computation unit to obtain the trained control prediction model.

[0022] Fifthly, an electronic device is provided, including a processor, a memory, a communication interface, and a communication bus. The processor, memory, and communication interface communicate with each other via the communication bus. The memory stores executable instructions. When the executable instructions are executed on the processor, they cause the processor to execute the vehicle control method provided in the first aspect and any possible implementation thereof, or to execute the control prediction model training method provided in the second aspect and any possible implementation thereof.

[0023] In a sixth aspect, a computer-readable storage medium is provided, which stores executable instructions that, when executed on an electronic device, cause the electronic device to perform a vehicle control method as provided in the first aspect and any possible implementation thereof, or cause a processor to perform a control prediction model training method as provided in the second aspect and any possible implementation thereof.

[0024] In a seventh aspect, a computer program product containing computer instructions is provided, which, when executed on an electronic device, causes the electronic device to execute the vehicle control method provided in the first aspect and any possible implementation thereof, or causes the processor to execute the control prediction model training method provided in the second aspect and any possible implementation thereof.

[0025] Understandably, the solutions provided in aspects three through seven above are all used to implement the corresponding methods provided in aspects one and two above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0026] It should be understood that, in this application, the names of the aforementioned vehicle control devices and electronic devices do not constitute a limitation on the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this invention, it falls within the scope of the claims of this disclosure and their equivalents. Furthermore, it should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this disclosure.

[0027] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0028] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0029] Figure 1 A schematic diagram illustrating the principle of a vehicle control method provided in an embodiment of this application;

[0030] Figure 2 A schematic diagram illustrating the implementation environment of a vehicle control method provided in this application embodiment;

[0031] Figure 3 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 1 ;

[0032] Figure 4 A schematic diagram illustrating the principle of another vehicle control method provided in this application embodiment;

[0033] Figure 5 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 2 ;

[0034] Figure 6 A flowchart illustrating a control prediction model training method provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the structure of an initial control prediction model provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of the structure of a vehicle control device provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of the structure of a control prediction model training device provided in an embodiment of this application;

[0038] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. "A and / or B" includes the following three combinations: only A, only B, and a combination of A and B.

[0041] With the development of technology, traditional cars are gradually moving towards new energy and intelligence, and automatic operation technology is a breakthrough representative of this.

[0042] In related technologies, autonomous driving control typically utilizes traditional feature extraction methods to first extract environmental features and vehicle control characteristics during vehicle operation. These are then analyzed using pre-trained models to predict control decisions that serve as the basis for autonomous driving. However, the accuracy of traditional feature extraction methods is highly dependent on the specific extraction method chosen by the user, making it difficult to guarantee high accuracy. This results in less accurate subsequent control decisions and a poor user experience.

[0043] To address the aforementioned problems, this application provides a vehicle control method. (Refer to...) Figure 1As shown, in this method, the vehicle's scene information and control information can be obtained first. The scene information and control information can be obtained from the vehicle's current data. Scene information characterizes the current scene the vehicle is in, while control information characterizes the items the vehicle can currently control. Then, the scene information and control information are input into the first input sub-model and the second input sub-model of the control prediction model, respectively, to obtain scene features and control features. The output sub-model of the control prediction model then processes the scene features and control features to predict the target control decision. The control prediction model is pre-trained and includes a first input sub-model, a second input sub-model, and an output sub-model. The first input sub-model has the ability to obtain scene features using scene image information, the second input sub-model has the ability to obtain control features using control information, and the output sub-model has the ability to obtain control decisions using scene features and control features. Therefore, based on this control prediction model, after extracting features from the vehicle's scene information and control information, the features can be processed to obtain a control prediction, i.e., a control decision, that the vehicle is currently likely to require. In this way, based on this scheme, features (scene features and control features) can be extracted conveniently and accurately, thereby quickly and easily obtaining control decisions for the vehicle, achieving the goal of autonomous driving, and improving the user experience.

[0044] Figure 2 This is a schematic diagram illustrating an implementation environment for a vehicle control method according to an exemplary embodiment. (Refer to...) Figure 2 As shown, the implementation environment may include terminal 01 and server 02, and electronic devices may communicate with the server via wired or wireless communication.

[0045] For example, the terminal 01 in this application embodiment can be an on-board computer (or autonomous driving controller) in a vehicle or a terminal capable of wirelessly communicating with the vehicle, such as a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, personal computer (PC), etc., which can acquire scene image data of the vehicle and make predictive control decisions based on the data. This application embodiment does not impose any special limitations on the specific form of the terminal.

[0046] For example, server 02 in this disclosure can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center; this disclosure does not limit this. Server 02 is primarily used for training control prediction models. After training the control prediction model, server 02 can send the model to terminal 01 for use, or it can obtain scene image data of the vehicle through terminal 01, use the model to predict vehicle control decisions, and send the prediction results to the vehicle (specifically, the vehicle's main control device) through terminal 01 to enable the vehicle to complete the corresponding control and achieve autonomous driving.

[0047] The vehicle control method provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0048] This application provides a vehicle control method applied to a vehicle control device. The vehicle control device may be an electronic device or a part of an electronic device, and the electronic device may be the aforementioned terminal or server. Taking an electronic device as an example, refer to... Figure 3 As shown, the method may include S301-S305:

[0049] S301. Electronic devices acquire scene image information and control information of the vehicle.

[0050] Control information is used to characterize the controllable aspects of a vehicle. Examples include the steering wheel, brakes, horn, lights, seats, and accelerator. Specifically, control information can be obtained from signals from the vehicle's autopilot controller.

[0051] Scene image information can be the current scene image information of the vehicle, and control information can also be the current control information of the vehicle. Here, "current" can be the current time (e.g., 12:28:06 on May 5, 2021) or a period of time (e.g., from 12:28:05 on May 5, 2021 to 12:58:06 on May 5, 2021).

[0052] In practice, control information can be fixed, meaning that the items that can be controlled by the vehicle at any given time are fixed. For example, regardless of the time, only the steering wheel and brakes are controllable items. Of course, control information can also be non-fixed. For example, if the vehicle is in a congested traffic scenario, the steering wheel, brakes, horn, seat, and lights are controllable items; if the vehicle is in a free-flowing traffic scenario, the steering wheel, brakes, horn, lights, and accelerator are controllable items.

[0053] In this embodiment, the scene image information and control information can be acquired in real time by an electronic device. When the control information remains constant, the electronic device does not need to acquire it in real time.

[0054] The method by which an electronic device acquires scene image information depends on the specific form of the electronic device. For example, if the electronic device is an onboard computer installed in a vehicle, it can acquire scene image information from various sensors controlling the vehicle. Alternatively, if the electronic device is a terminal communicating with the vehicle (or another vehicle), it can send an acquisition request to the vehicle, causing the vehicle to acquire its own scene image information and then send it back to the electronic device. Yet another example is if the electronic device is a server, which can send an acquisition request to the vehicle, causing the vehicle to acquire scene image information and then send it to the server via a terminal. Of course, any other feasible implementation method can be used in practice, and this application does not impose any specific limitations on this. The same applies to the acquisition of control information by the electronic device.

[0055] S302. The electronic device inputs the scene image information into the first input sub-model in the control prediction model to obtain scene features.

[0056] S303. The electronic device inputs the control information into the second input sub-model in the control prediction model to obtain the control characteristics.

[0057] It should be noted that S302 and S303 can be implemented simultaneously or sequentially, and this application does not impose specific restrictions on this.

[0058] In this embodiment, the control prediction model can be a pre-trained deep learning model. The electronic device can acquire the control prediction model either by acquiring sample data and training it itself, or by acquiring it from other devices. This application does not impose specific limitations in this regard.

[0059] During the pre-training process, the first and second input sub-models in the control prediction model can acquire the ability to extract features from data based on a large number of training samples. Specifically, the first input sub-model can be trained to obtain scene features from scene image information, while the second input sub-model can be trained to obtain control features from control information. The electronic device can then implement S302 and S303 as described above based on the first and second input sub-models in the control prediction model.

[0060] S304. Electronic devices use the output sub-model in the control prediction model to process scene features and control features to obtain target control decisions.

[0061] As described in S303, during the training process of the control prediction model, its output sub-model can acquire the ability to make control decisions based on scene features and control characteristics using a large number of training samples. Therefore, electronic devices can successfully implement S304 based on this control prediction model.

[0062] In one feasible approach, for ease of processing, it is combined with Figure 1 , refer to Figure 4 As shown, after obtaining the scene features and control features of the vehicle through the control prediction model, the electronic device can merge these two features into a merge set (or splice them together to obtain a feature sequence) based on the output sub-model in the control prediction model, and process the merge set (or feature sequence) to obtain the target control decision.

[0063] In this embodiment, the target control decision can specifically be a response value for each item corresponding to the control information. The response value can be 0 or 1, where 0 represents no response (i.e., no control is applied to the corresponding item), and 1 represents a response (i.e., control is applied to the corresponding item).

[0064] For example, taking a controllable device corresponding to a controllable feature as an example, including a steering wheel, brake, and accelerator, the target control decision can be [steering wheel v1, brake v2, accelerator v3]. Specifically,

[0065]

[0066] v1 can also add any feasible options such as turning left by X degrees or turning right by Y degrees.

[0067]

[0068] v2 can also include any feasible options such as braking a quarter turn.

[0069]

[0070] In v3, any feasible options can be added, such as increasing the throttle by X degree or decreasing the throttle by Y degree.

[0071] The 10 options corresponding to v1-v3 mentioned above can be the controllable items corresponding to the control features.

[0072] In some embodiments, after the electronic device inputs scene image information and control information into the control prediction model, it obtains 10 decision values ​​corresponding to the aforementioned v1-v3 pairs. The final control decision can be to select the largest value among the values ​​corresponding to v1-v3 as the final decision for v1-v3, thereby obtaining the target control decision. For example, the 10 decision values ​​corresponding to the v1-v3 pairs can be as follows:

[0073]

[0074] The final decisions obtained for v1-v3 can be left turn, full braking, and deceleration. The final target control decision can be [1000001010], where the three 1s from left to right represent responding to left turn, full braking, and deceleration, respectively, i.e., executing the relevant operations.

[0075] Of course, the above are just examples, and in practice, target control decisions can be implemented in any other feasible way.

[0076] S305. Electronic devices control the vehicle based on target control decisions.

[0077] Specifically, if the electronic device is an onboard computer (or an autonomous driving controller), it can directly control the vehicle based on the target control decision. If the electronic device is a terminal that can communicate with the vehicle or a server that communicates with the vehicle through a terminal, it achieves this by sending the target control decision to the vehicle, and the sending method can be any feasible method.

[0078] Based on the technical solution provided in this application, scene image information and control information of the vehicle can be obtained first. Then, a pre-trained deep learning model, specifically the first and second input sub-models in the control prediction model, can be used to extract scene features and control features from the scene image information and control information, respectively. Subsequently, the output sub-model in the control prediction model can predict the target control decision based on the scene features and control features obtained from the two input sub-models. Because the control prediction model is pre-trained, its two input sub-models can acquire the ability to extract features from data based on a large number of training samples, which is more accurate and convenient compared to existing feature extraction methods. The output sub-model can acquire the ability to obtain control decisions based on scene features and control features based on a large number of samples. Combining the capabilities of the two input sub-models, the control decision can be accurately and quickly determined using the vehicle's scene image information and control information. Then, the vehicle can be controlled based on this decision. Therefore, based on the technical solution provided in this application, feature extraction (scene features and control features) can be completed conveniently and accurately, thereby quickly and conveniently obtaining the target control decision for the vehicle, achieving the goal of autonomous driving, and improving the user experience.

[0079] Furthermore, because the technical solution provided in this application contains three independently existing sub-models in the control prediction model, if a problem or malfunction occurs in a certain sub-model, one of them can be adjusted individually. This makes model adjustment in the technical solution provided in this application more convenient and easier for users to use.

[0080] In some embodiments, refer to Figure 4 As shown, the acquisition of vehicle scene image information and control information can be achieved by first acquiring vehicle scene image data and vehicle control data, followed by certain data preprocessing. Based on this, Figure 5 A schematic flowchart of another embodiment of a vehicle control method provided in this application is shown. This vehicle control method is implemented by a vehicle control device, which can be an electronic device or part of an electronic device, such as the aforementioned terminal or server. (Refer to...) Figure 5 As shown, the vehicle control method includes S501-S507:

[0081] S501, Electronic devices acquire scene image data and control data of the vehicle.

[0082] The scene image data can be captured by cameras inside or outside the vehicle, or obtained by radar scanning inside or outside the vehicle. For example, the scene image data can include images inside and outside the vehicle.

[0083] Control data can be signal data from the vehicle's autonomous driving controller. For example, the signal data from the autonomous driving controller can include CAN (controller area network) bus communication protocol messages defined for the entire vehicle, which can include control of vehicle architecture domains such as the powertrain domain, braking domain, body domain, and cockpit domain.

[0084] Specifically, how the electronic device acquires scene image data and control data is similar to the acquisition method of scene image information in S301 of the aforementioned implementation, and will not be repeated here.

[0085] S502, The electronic device performs data preprocessing on the scene image data and control data of the vehicle to obtain scene image information and control information.

[0086] Data preprocessing can include two parts. The first part can be any feasible processing method, such as noise reduction, deduplication, filtering, feature extraction, normalization, etc. The second part can be a preset encoding method. For example, for scene image data, the preset encoding method can be to convert the content in the scene image data that indicates the vehicle scene into information that is easy to process, based on some image processing methods. The specific preset encoding method selected can be determined according to actual needs.

[0087] S503. The electronic device inputs the scene image information into the first input sub-model in the control prediction model to obtain scene features.

[0088] S504. The electronic device inputs the control information into the second input sub-model in the control prediction model to obtain the control characteristics.

[0089] It should be noted that S503 and S504 can be implemented simultaneously or sequentially, and this application does not impose specific restrictions on this.

[0090] S505. Electronic devices use the output sub-model in the control prediction model to process scene features and control features to obtain target control decisions.

[0091] The specific implementations of S503-S505 can be referred to the relevant descriptions of S302-S304 mentioned above, and will not be repeated here.

[0092] Understandably, convolutional neural networks possess strong feature extraction and classification capabilities. Therefore, in this embodiment, the three sub-models in the control prediction model can all be convolutional neural networks. Thus, in some embodiments, the first input sub-model, the second input sub-model, and the output sub-model can all be convolutional neural network (CNN) models.

[0093] S506. Electronic devices control the vehicle based on target control decisions.

[0094] Understandably, the vehicle scene image data and vehicle control data that can be acquired in practice are quite complex. For ease of use, this data needs to be preprocessed to obtain data that can be directly used for subsequent predictions. Based on this, in the technical solutions corresponding to S501-S506, scene image information and control information can be obtained by acquiring image data from outside the vehicle and vehicle control data, and then performing data preprocessing to obtain scene image information and control information. This provides strong data support for subsequent control decision predictions.

[0095] In this embodiment of the application, in order to successfully implement the vehicle control method provided in the foregoing embodiments, it is necessary to train in advance the control prediction model to be used in the vehicle control method.

[0096] Based on this, this application also provides a method for training a control prediction model. This method can be applied to a control prediction model training device, which can be an electronic device or a part of an electronic device, and the electronic device can be the aforementioned terminal or server. Taking an electronic device as an example, refer to... Figure 6 As shown, the training method for this control prediction model may include S601-S605:

[0097] S601, Electronic equipment acquires multiple sets of sample data and makes one-to-one sample control decisions based on the multiple sets of sample data.

[0098] The sample data includes sample scene image information and sample control information. Specifically, the sample control decision can be the response value for each controllable item corresponding to the sample control information.

[0099] In this embodiment, the sample scene image information can be obtained by preprocessing scene image data acquired by the electronic device from multiple vehicles. The sample control information can be obtained by preprocessing signal data from the autonomous driving controllers of multiple vehicles (the signal data of the autonomous driving controllers of each vehicle can be the same). The sample control decision can be the control decision implemented by the user corresponding to each scene image data acquired by the electronic device, that is, the response result of the user to the controllable items of the corresponding vehicle (executing the operation corresponding to a certain controllable item is a response, and not executing the operation corresponding to a certain controllable item is no response).

[0100] In this embodiment, the sample data and the corresponding sample control decisions can be a standard database composed of multiple pre-collected data sets (specifically, stored as data columns in an Excel spreadsheet). This standard database can be further divided into a first standard sub-database storing sample scene image information, a second standard sub-database storing sample control information, and a third sub-database storing sample control decisions. These three sub-databases can also contain three types of data relationships to facilitate use during training.

[0101] S602. The electronic device inputs the sample scene image information into the first input sub-model in the initial control prediction model to obtain the sample scene features.

[0102] S603. The electronic device inputs the sample control information into the second input sub-model in the initial control prediction model to obtain the sample control features.

[0103] S604. Electronic equipment processes sample scene features and sample control features based on the output sub-model in the initial control prediction model to obtain predictive control decisions.

[0104] The specific implementations of S602-S604 can be referred to the specific implementations of S503-S505 in the aforementioned embodiments, and will not be repeated here.

[0105] As can be seen, the initial control prediction model includes a first input sub-model, a second input sub-model, and an output sub-model. In this embodiment, the first input sub-model in the initial control prediction model has the ability to obtain scene features using scene image information, the second input sub-model has the ability to obtain control features using control information, and the output sub-model has the ability to obtain control decisions using scene features and control features. It should be noted that the capabilities of each sub-model in the initial control prediction model are only those that can be estimated based on the initial configuration, and these capabilities are not powerful. In other words, the output results obtained from the capabilities of each sub-model in the initial control prediction model are not accurate enough, and subsequent training is required to obtain the control prediction model needed in the aforementioned embodiments.

[0106] In addition, all three sub-models in the initial control prediction model can be CNN models.

[0107] In one possible implementation, refer to Figure 7 As shown, the first input sub-model includes: a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first flattening layer, and a first fully connected (dense) layer; the second input sub-model includes: a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, a second flattening layer, and a second fully connected layer; the output sub-model includes: a concatenate layer, a third fully connected layer, a fifth convolutional layer, a fifth pooling layer, a sixth convolutional layer, a sixth pooling layer, a seventh convolutional layer, a seventh pooling layer, a first dropout layer, a third flattening layer, a second dropout layer, and a fourth fully connected layer.

[0108] Of course, the final control prediction model trained by this control prediction model training method has the same specific structure as the initial control prediction model.

[0109] For example, the first input sub-model includes: a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first planarization layer, and a first fully connected layer; the second input sub-model includes: a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, a second planarization layer, and a second fully connected layer; the output sub-model includes: a concatenation layer, a third fully connected layer, a fifth convolutional layer, a fifth pooling layer, a sixth convolutional layer, a sixth pooling layer, a seventh convolutional layer, a seventh pooling layer, a first dropout layer, a third planarization layer, a second dropout layer, and a fourth fully connected layer. The specific parameters of the first input sub-model in the initial control prediction model are shown in Table 1 below. This table lists the input and output layers of each layer in detail, as well as the parameters of the output layer (the same applies to Tables 2 and 3). The specific parameters of the first input sub-model in the control prediction model are the same as those of the first input sub-model in the initial control prediction model. Of course, these specific parameters are hyperparameters that will not change during training.

[0110] Table 1 - First Input Submodel

[0111]

[0112]

[0113] A filter can include one or more convolutional kernels, the number of which depends on the required number of channels. The kernel size is specifically the number of rows multiplied by the number of columns. For example, if the kernel size is 100, then the kernel size could be 10*10. Of course, in practice, if the number of rows and columns of a convolutional kernel are the same, it can also be represented by either the number of rows or the number of columns. The pooling kernel size follows the same principle. The pooling kernels in Table 1 can all be pooling kernels with the same number of rows and columns.

[0114] In this embodiment, the convolutional layer is specifically used to perform convolution processing on the input matrix using a predefined convolutional kernel with a preset stride. Pooling is used to perform pooling operations on the input matrix using a predefined pooling kernel with a preset stride and a specific pooling method. The planarization layer is used to expand the input matrix into a one-dimensional matrix. The fully connected layer is used to perform convolution operations on the input matrix through a preset number (i.e., the number of nodes) filters (each filter includes at least one convolutional kernel, and the convolutional kernel and the input matrix are of the same size) to obtain a one-dimensional matrix; simultaneously, a certain number of features are randomly discarded according to a dropout rate. The number of elements in this one-dimensional matrix is ​​the number of nodes.

[0115] For example, taking the parameters shown in Table 1 above as the specific parameters of the first input sub-model, and the matrix channel number corresponding to the scene image information input to the first input sub-model being 1, if the matrix size obtained after processing by the first convolutional layer is (1024, 50), the matrix obtained by the first pooling layer is (204, 100), the matrix size obtained by the second convolutional layer is (41, 30), the matrix size obtained by the second pooling layer is (4, 60), the matrix size obtained by the first planarization layer is (240, 1), and the matrix size obtained by the first fully connected layer is (100, 1).

[0116] For example, the specific parameters of the second input sub-model in the initial control prediction model are shown in Table 2 below. The specific parameters of the second input sub-model in the control prediction model are the same as those of the second input sub-model in the initial control prediction model. Of course, these specific parameters are hyperparameters that will not change during training.

[0117] Table 2 - Second Input Sub-model

[0118]

[0119]

[0120] For example, the specific parameters of the output sub-model in the initial control prediction model are shown in Table 3 below. The specific parameters of the output sub-model in the control prediction model are the same as those of the output sub-model in the initial control prediction model. Of course, these specific parameters are hyperparameters that will not change during training.

[0121] Table 3 - Output Sub-model

[0122]

[0123]

[0124] In this embodiment of the application, the discarding layer (first discarding layer and second discarding layer) specifically discards a preset number of elements in the input matrix according to the discarding rate.

[0125] S605. The electronic device uses the predictive control decision as the initial training output of the initial control prediction model and the sample control decision as the supervision information to iteratively train the initial control prediction model to obtain the trained control prediction model.

[0126] After S605, if the electronic device that performs the above-described control prediction model training method is also used to perform the vehicle control method in the aforementioned embodiment, then the electronic device can implement the vehicle control method in the aforementioned embodiment based on the control prediction model. If the electronic device that performs the above-described control prediction model training method and the electronic device that performs the vehicle control method in the aforementioned embodiment are not the same device, then the control prediction model can be sent to the electronic device that performs the vehicle control method in the aforementioned embodiment so that it can successfully implement the vehicle control method.

[0127] In one possible implementation, S605 may specifically include: determining a loss value based on the predictive control decision and the sample control decision; iteratively updating the initial control prediction model based on the loss value to obtain the trained control prediction model.

[0128] Specifically, the loss value can be calculated using any feasible loss function. For example, the mean squared error (MSE) is used to calculate the Euclidean distance between the predicted value (predictive control decision) and the true value (sample control decision). This application does not impose any specific restrictions on this.

[0129] Subsequently, based on this loss value, the parameters in the initial control prediction model can be adjusted using any feasible parameter tuning method. Examples include stochastic gradient descent (SGD). This application does not impose specific limitations in this regard.

[0130] In this way, the initial control prediction model can be iteratively optimized based on the training data and loss value, thereby obtaining the control prediction model.

[0131] Based on the technical solutions corresponding to S601-S605 above, a control prediction model can be trained using deep learning. This model enables it to extract features from scene image information and control information, and predict control decisions based on the extracted features. Thus, the vehicle control method provided in this application can conveniently and quickly predict vehicle control decisions using this model.

[0132] In addition, based on the architecture of the initial control prediction model, parameter tuning and optimization can be performed on the three sub-models separately during the training of the control prediction model, rather than overall parameter tuning and optimization. This makes the training optimization process more reasonable and can improve training efficiency to a certain extent.

[0133] It is understood that, in order to achieve the aforementioned functions, the electronic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0134] In the case of dividing each function into corresponding modules, embodiments of this application also provide a vehicle control device. For example... Figure 8 The diagram shown is a structural schematic of a vehicle control device provided in an embodiment of this application. The device may include: an acquisition module 801, a processing module 802, and a control module 803.

[0135] The system includes an acquisition module 801 for acquiring scene image information and control information of the vehicle; the control information is used to characterize the controllable items of the vehicle; a processing module 802 for inputting the scene image information acquired by the acquisition module 801 into the first input sub-model of the control prediction model to obtain scene features; the processing module 802 is also used to input the control information acquired by the acquisition module 801 into the second input sub-model of the control prediction model to obtain control features; the processing module 802 is also used to process the scene features and control features based on the output sub-model of the control prediction model to obtain a target control decision; and a control module 803 for controlling the vehicle according to the target control decision obtained by the processing module 802.

[0136] In one possible implementation, the acquisition module 801 is specifically used to: acquire scene image data and control data of the vehicle; and perform data preprocessing on the scene image data and control data to obtain scene image information and control information of the vehicle.

[0137] In one possible implementation, the first input sub-model, the second input sub-model, and the output sub-model are all convolutional neural networks.

[0138] Regarding the vehicle control device in the above embodiments, the specific methods by which each module performs its operation and the corresponding beneficial effects have been described in detail in the embodiments of the vehicle control method described above, and will not be repeated here.

[0139] In the case of dividing each function into corresponding functional modules, embodiments of this application also provide a control prediction model training device. (Refer to...) Figure 9As shown, the device may include an acquisition module 901 and a training module 902.

[0140] The acquisition module 901 is used to acquire multiple sets of sample data and corresponding sample control decisions; the sample data includes sample scene image information and sample control information; the training module 902 is used to input the sample scene image information acquired by the acquisition module 901 into the first input sub-model of the initial control prediction model to obtain sample scene features; the training module 902 is also used to input the sample control information acquired by the acquisition module 901 into the second input sub-model of the initial control prediction model to obtain sample control features; the training module 902 is also used to process the sample scene features and sample control features based on the output sub-model of the initial control prediction model to obtain predictive control decisions; the training module 902 is also used to iteratively train the initial control prediction model using the predictive control decisions as the initial training output of the initial control prediction model and the sample control decisions acquired by the acquisition module 901 as supervision information to obtain the trained control prediction model.

[0141] In one possible implementation, the training module 902 includes a calculation unit 9021 and an optimization unit 9022; the calculation unit 9021 is used to determine the loss value based on the predictive control decision and the sample control decision; the optimization unit 9022 is used to iteratively update the initial control prediction model based on the loss value calculated by the calculation unit 9021 to obtain the trained control prediction model.

[0142] Regarding the control prediction model training device in the above embodiments, the specific methods of operation of each module and the corresponding beneficial effects have been described in detail in the embodiments of the control prediction model training method described above, and will not be repeated here.

[0143] Figure 10 This is a schematic diagram illustrating a possible structure of an electronic device according to an exemplary embodiment. The electronic device can be the aforementioned vehicle control device, or it can be a terminal or server including the vehicle control device. Alternatively, the electronic device can be the aforementioned control prediction model training device, or it can be a terminal or server including the control prediction model training device. Figure 10 As shown, the electronic device includes a processor 101 and a memory 102. The memory 102 stores instructions executable by the processor 101, which in turn implements the functions of the various modules in the hair transformation model generation device 03 and / or hair transformation device 04 described in the above embodiments. The memory 102 stores at least one instruction, which is loaded and executed by the processor 101 to implement the methods provided in the above method embodiments.

[0144] In a specific implementation, as one embodiment, processor 101 (101-1 and 101-2) may include one or more CPUs, for example... Figure 10 CPU0 and CPU1 are shown in the diagram. As one embodiment, the electronic device may include multiple processors 101, such as... Figure 10 The processors 101-1 and 101-2 are shown. Each CPU in these processors 101 can be a single-core processor or a multi-core processor. Here, processor 101 can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0145] The memory 102 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), disk computer storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 102 may exist independently and be connected to the processor 101 via a communication bus 103. The memory 102 may also be integrated with the processor 101.

[0146] The communication bus 103 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This communication bus 103 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0147] In addition, to facilitate information interaction between the electronic device and other devices (e.g., information interaction between the electronic device as a terminal and a server, or information interaction between the electronic device as a server and a terminal), the electronic device includes a communication interface 104. The communication interface 104 uses any transceiver-like device for communication with other devices or communication networks, such as control systems, radio access networks (RAN), wireless local area networks (WLAN), etc. The communication interface 104 may include a receiving unit to implement receiving functions and a transmitting unit to implement transmitting functions. The communication interface 104, processor 101, and memory 102 are connected via a communication bus 103 to complete mutual communication.

[0148] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the vehicle control method or the control prediction model training method in the above method embodiments.

[0149] For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0150] This application also provides a computer program product containing computer instructions, which, when executed on an electronic device, cause the electronic device to perform the vehicle control method or control prediction model training method in the above method embodiments.

[0151] In this application, the electronic device, computer-readable storage medium, or computer program product provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0152] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device (e.g., electronic device) can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device (e.g., electronic device) and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0153] In the embodiments provided in this application, it should be understood that the disclosed systems, devices (e.g., electronic devices) and methods can be implemented in other ways. For example, the device (e.g., electronic device) embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A vehicle control method characterized by, The method includes: Acquire scene image information and control information of the vehicle; the control information is used to characterize the controllable items of the vehicle; The scene image information is input into the first input sub-model in the control prediction model to obtain scene features; The control information is input into the second input sub-model in the control prediction model to obtain the control features; The scene features and the control features are processed based on the output sub-model in the control prediction model to obtain the target control decision. The first input sub-model, the second input sub-model and the output sub-model are all convolutional neural networks. The vehicle is controlled according to the target control decision; The acquisition of scene image information and control information of the vehicle includes: Acquire scene image data and control data of the vehicle; The scene image data and control data are preprocessed to obtain the scene image information and control information of the vehicle.

2. A control prediction model training method, characterized by, include: Acquire multiple sets of sample data and corresponding sample control decisions for each set of sample data; the sample data includes sample scene image information and sample control information. The sample scene image information is input into the first input sub-model in the initial control prediction model to obtain the sample scene features; The sample control information is input into the second input sub-model in the initial control prediction model to obtain the sample control features; Based on the output sub-model in the initial control prediction model, the sample scene features and the sample control features are processed to obtain predictive control decisions; The predicted control decision is used as the initial training output of the initial control prediction model, and the sample control decision is used as the supervision information. The initial control prediction model is iteratively trained to obtain the trained control prediction model. The step of using the predictive control decision as the initial training output of the initial control prediction model, and the sample control decision as supervision information, to iteratively train the initial control prediction model to obtain the trained control prediction model includes: The loss value is determined based on the predictive control decision and the sample control decision; The initial control prediction model is iteratively updated based on the loss value to obtain the trained control prediction model.

3. A vehicle control device, characterized in that, The device includes: The acquisition module is used to acquire scene image information and control information of the vehicle; the control information is used to characterize the controllable items of the vehicle. The processing module is used to input the scene image information acquired by the acquisition module into the first input sub-model in the control prediction model to obtain scene features; The processing module is further configured to input the control information acquired by the acquisition module into the second input sub-model in the control prediction model to obtain control features; The processing module is further configured to process the scene features and the control features based on the output sub-model in the control prediction model to obtain the target control decision, wherein the first input sub-model, the second input sub-model and the output sub-model are all convolutional neural networks; The control module is used to control the vehicle according to the target control decision obtained by the processing module; Specifically, the acquisition module is used to: acquire scene image data and control data of the vehicle; and perform data preprocessing on the scene image data and control data to obtain scene image information and control information of the vehicle.

4. A control prediction model training device, characterized in that, include: The acquisition module is used to acquire multiple sets of sample data and sample control decisions corresponding to each set of sample data; the sample data includes sample scene image information and sample control information. The training module is used to input the sample scene image information acquired by the acquisition module into the first input sub-model in the initial control prediction model to obtain sample scene features; The training module is also used to input the sample control information acquired by the acquisition module into the second input sub-model in the initial control prediction model to obtain sample control features; The training module is also used to process the sample scene features and the sample control features based on the output sub-model in the initial control prediction model to obtain predictive control decisions; The training module is further configured to use the predictive control decision as the initial training output of the initial control prediction model, and the sample control decision acquired by the acquisition module as supervision information to iteratively train the initial control prediction model to obtain the trained control prediction model. Specifically, the training module is used to: determine a loss value based on the predicted control decision and the sample control decision; and iteratively update the initial control prediction model based on the loss value to obtain the trained control prediction model.

5. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to execute the vehicle control method as described in claim 1, or to execute the control prediction model training method as described in claim 2.

6. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions, which, when executed on the electronic device, cause the electronic device to perform the vehicle control method as described in claim 1, or to perform the control prediction model training method as described in claim 2.

Images