Training method of automatic driving decision model, vehicle control method and device

Abstract

The present disclosure provides a training method of an automatic driving decision model, a vehicle control method and device, relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, reinforcement learning, automatic driving and intelligent transportation. The specific implementation scheme is: obtaining first vehicle state information and first environment information of a vehicle in a first traffic scene; using an automatic driving decision model to process the first vehicle state information and the first environment information multiple times to obtain multiple first control strategies for the vehicle in the first traffic scene; wherein the automatic driving decision model is trained according to second vehicle state information and second environment information of a vehicle in a second traffic scene; training an initial evaluation model according to the multiple first control strategies to obtain a target evaluation model; and training the automatic driving decision model according to the target evaluation model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning, reinforcement learning, autonomous driving, and intelligent transportation technology. Specifically, it relates to a training method and apparatus for an autonomous driving decision-making model, a vehicle control method, apparatus, electronic device, storage medium, and autonomous vehicle. Background Technology

[0002] When an autonomous vehicle is driving in a traffic scenario, it needs to make decisions in real time based on vehicle status information and surrounding environment information, and provide control strategies for the autonomous vehicle so as to control the driving status of the autonomous vehicle according to the control strategies.

[0003] For a given traffic scenario, a vehicle often has more than one correct way to drive. However, when making decisions based on vehicle status information and surrounding environment information, the relevant technologies can only provide a single control strategy, rather than the optimal control strategy, resulting in autonomous vehicles performing poorly when dealing with that particular traffic scenario. Summary of the Invention

[0004] This disclosure provides a training method and apparatus for an autonomous driving decision-making model, a vehicle control method, apparatus, electronic device, storage medium, and autonomous vehicle.

[0005] According to one aspect of this disclosure, a training method for an autonomous driving decision model is provided, comprising: acquiring first vehicle state information and first environmental information of a vehicle in a first traffic scenario; processing the first vehicle state information and first environmental information multiple times using the autonomous driving decision model to obtain multiple first control strategies for the vehicle in the first traffic scenario; wherein the autonomous driving decision model is trained based on second vehicle state information and second environmental information of the vehicle in a second traffic scenario; training an initial evaluation model based on the multiple first control strategies to obtain a target evaluation model; and training the autonomous driving decision model based on the target evaluation model.

[0006] According to another aspect of this disclosure, a vehicle control method is provided, comprising: acquiring vehicle state information and environmental information of a vehicle in a traffic scenario; processing the vehicle state information and environmental information using an autonomous driving decision model to obtain a control strategy for the vehicle in the traffic scenario; wherein the autonomous driving decision model is trained according to the training method of the autonomous driving decision model described above; and controlling the vehicle based on the control strategy.

[0007] According to another aspect of this disclosure, a training apparatus for an autonomous driving decision model is provided, comprising: a first acquisition module for acquiring first vehicle state information and first environmental information of a vehicle in a first traffic scenario; a first processing module for processing the first vehicle state information and first environmental information multiple times using the autonomous driving decision model to obtain multiple first control strategies for the vehicle in the first traffic scenario; wherein the autonomous driving decision model is trained based on second vehicle state information and second environmental information of a vehicle in a second traffic scenario; a first training module for training an initial evaluation model based on the multiple first control strategies to obtain a target evaluation model; and a second training module for training the autonomous driving decision model based on the target evaluation model.

[0008] According to another aspect of this disclosure, a vehicle control device is provided, comprising: a third acquisition module for acquiring vehicle state information and environmental information of a vehicle in a traffic scenario; a third processing module for processing the vehicle state information and environmental information using an autonomous driving decision model to obtain a control strategy for the vehicle in the traffic scenario; wherein the autonomous driving decision model is trained using the training device for the autonomous driving decision model described above; and a control module for controlling the vehicle based on the control strategy.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to this disclosure.

[0010] According to another aspect of this disclosure, an autonomous vehicle is provided, including electronic devices according to embodiments of this disclosure.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided according to this disclosure.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to this disclosure.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is an exemplary system architecture for which vehicle control methods and apparatus can be applied according to embodiments of this disclosure;

[0016] Figure 2 This is a flowchart of a training method for an autonomous driving decision-making model according to an embodiment of the present disclosure;

[0017] Figure 3A , Figure 3B and Figure 3C This is a schematic diagram of the training process of an autonomous driving decision-making model according to an embodiment of the present disclosure;

[0018] Figure 4 This is a flowchart of a vehicle control method according to an embodiment of the present disclosure;

[0019] Figure 5 This is a block diagram of a training apparatus for an autonomous driving decision-making model according to an embodiment of the present disclosure;

[0020] Figure 6 This is a block diagram of a vehicle control device according to an embodiment of the present disclosure;

[0021] Figure 7 This is a schematic diagram of an autonomous vehicle according to an embodiment of the present disclosure; and

[0022] Figure 8 This is a block diagram of an electronic device used to implement the training method of the autonomous driving decision model and the vehicle control method of the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] Figure 1 This is an exemplary system architecture for which vehicle control methods and apparatus can be applied according to embodiments of this disclosure.

[0025] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0026] like Figure 1As shown, the system architecture 100 according to this embodiment may include sensors 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the sensors 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0027] Sensors 101, 102, and 103 can interact with server 105 via network 104 to receive or send messages, etc.

[0028] Sensors 101, 102, and 103 can be functional components integrated on vehicle 106, such as infrared sensors, ultrasonic sensors, millimeter-wave radar, information acquisition devices, image acquisition equipment, etc. Sensors 101, 102, and 103 can be used to collect information about the surrounding environment and vehicle status of vehicle 106.

[0029] Server 105 can also be integrated into vehicle 106, but it is not limited to this. It can also be set up at a remote end that can establish communication with the vehicle terminal. Specifically, it can be implemented as a distributed server cluster composed of multiple servers, or it can be implemented as a single server.

[0030] Server 105 can be a server that provides various services. Applications such as map applications and information processing applications can be installed on server 105. Server 105 can analyze and process the received data and feed the processing results back to vehicle 106.

[0031] For example, server 105 receives vehicle status information and surrounding environment information transmitted from sensors 101, 102, and 103 via network 104. The surrounding environment information and vehicle status information are input into the autonomous driving decision model to obtain a control strategy for the vehicle, and the vehicle 106 is controlled based on the control strategy.

[0032] It should be noted that the vehicle control method provided in this embodiment can generally be executed by server 105. Correspondingly, the vehicle control device provided in this embodiment can also be located in server 105. However, it is not limited to this. The vehicle control method provided in this embodiment can also generally be executed by vehicle 106. Correspondingly, the vehicle control device provided in this embodiment can also be located in vehicle 106.

[0033] It should be understood that Figure 1 The number of sensors, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of sensors, networks, and servers can be included.

[0034] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0035] Figure 2 This is a flowchart of a training method for an autonomous driving decision-making model according to an embodiment of the present disclosure.

[0036] like Figure 2 As shown, the training method 200 for the autonomous driving decision model includes operations S210 to S240.

[0037] In operation S210, the first vehicle status information and the first environment information of the vehicle in the first traffic scenario are obtained.

[0038] In operation S220, the first vehicle state information and the first environment information are processed multiple times using the autonomous driving decision model to obtain multiple first control strategies for the vehicle under the first traffic scenario.

[0039] During operation of S230, an initial evaluation model is trained based on multiple first control strategies to obtain the target evaluation model.

[0040] During operation S240, the autonomous driving decision model is trained based on the target evaluation model.

[0041] According to embodiments of this disclosure, the first vehicle state information and the first environmental information are, for example, vehicle state information and environmental information surrounding the vehicle in a first traffic scene detected by sensors.

[0042] Vehicle status information includes, but is not limited to, vehicle location, speed and acceleration, direction of travel, steering angle, and distance between the vehicle and surrounding vehicles.

[0043] Environmental information includes, but is not limited to, the status information of vehicles in the surrounding environment, the status information of pedestrians in the surrounding environment, road information in the surrounding environment, and information about obstacles in the surrounding environment. Vehicle status information includes, for example, the location, speed, acceleration, direction, and turning angle of surrounding vehicles. Pedestrian status information includes, for example, the pedestrian's walking status, direction of travel, and speed. Road information includes, for example, lane markings, road connections, road attributes (e.g., road type, direction, and length), traffic signs, and traffic light status. Obstacle information includes, but is not limited to, the location, size, movement status, and direction of movement of obstacles.

[0044] According to embodiments of this disclosure, the autonomous driving decision-making model is obtained, for example, through supervised learning training based on second vehicle state information and second environmental information in a second traffic scenario. Training the autonomous driving decision-making model through supervised learning can avoid the model outputting unreasonable control strategies.

[0045] The second vehicle state information and the second environmental information are, for example, vehicle state information and surrounding environmental information in a second traffic scene detected by sensors. The second vehicle state information and the second environmental information have the same or similar definitions as the first vehicle state information and the first environmental information, respectively, and will not be elaborated here.

[0046] In this embodiment of the disclosure, an autonomous driving decision model is used to process vehicle state information and environmental information in a traffic scenario to obtain a control strategy for the vehicle in that traffic scenario. The control strategy can be used to indicate the operation (or action) to be performed by the vehicle, such as performing one or more of the following operations: acceleration, deceleration, steering, or lane changing. In some embodiments, the control strategy can also indicate the control mode or control system to be selected for the vehicle, such as selecting an emergency braking control system or a steering control system. Based on the control strategy, the vehicle can be controlled, thereby achieving autonomous driving.

[0047] For example, by using an autonomous driving decision model to process the first vehicle state information and the first environmental information multiple times, several first control strategies for the vehicle under the first traffic scenario can be obtained. These multiple first control strategies can all be used to control the vehicle to perform corresponding operations so that the vehicle can cope with the road conditions under the first traffic scenario.

[0048] For example, when a vehicle is driving in a first traffic scenario, it can acquire first vehicle state information and first environmental information in the first traffic scenario. If an obstacle is detected on the road ahead of the vehicle, the first vehicle state information and first environmental information can be processed multiple times using an autonomous driving decision model to obtain multiple first control strategies. These multiple first control strategies include, for example, overtaking, gentle braking to avoid an obstacle, lateral obstacle avoidance, and emergency braking.

[0049] It is understandable that the aforementioned primary control strategies can all control the vehicle to perform corresponding operations to avoid collisions with obstacles. However, for the vehicle, the operational complexity and risk level corresponding to these primary control strategies vary. Consequently, the effects achieved by controlling the vehicle's driving state based on these primary control strategies also differ.

[0050] Here, operational complexity refers to the number of operations the vehicle needs to perform based on the control strategy, or the complexity and difficulty of those operations. Risk level characterizes the impact on the vehicle's normal operation after the control strategy is applied. The greater the impact on the vehicle's normal operation, the higher the risk level.

[0051] For example, from a risk perspective, overtaking carries the risk of traffic violations and collisions, while emergency stopping carries the risk of rear-end collisions. Compared to gentle braking and lateral avoidance, overtaking and emergency stopping involve a higher level of risk.

[0052] For example, from an operational complexity perspective, overtaking requires controlling the vehicle to perform more operations or actions. Compared to other control strategies, overtaking involves a higher level of operational complexity.

[0053] While ensuring driving safety, and considering both risk level and operational complexity, gentle braking and lateral avoidance are better control strategies than overtaking and emergency stopping in the current road conditions. Accordingly, using gentle braking and lateral avoidance to control the vehicle's driving state allows the vehicle to perform better in this traffic scenario.

[0054] In order to enable the autonomous driving decision model to make more reasonable and reliable decisions for road conditions in the same traffic scenario, for example, in the example above where an obstacle is detected, the model can output a control strategy of slow braking or lateral avoidance to control the vehicle to avoid collision with the obstacle. In this embodiment of the present disclosure, multiple first control strategies can be used to train an initial evaluation model so as to use the trained target evaluation model to optimize the autonomous driving decision model.

[0055] In this embodiment of the disclosure, the target evaluation model is used to evaluate the merits of different control strategies output by the autonomous driving decision model for the same traffic scenario. The autonomous driving decision model can be optimized based on the evaluation results output by the target evaluation model using a policy-based reinforcement learning algorithm, such as Proximal Policy Optimization (PPO), thereby enabling the optimized autonomous driving decision model to output more reasonable and reliable control strategies.

[0056] According to embodiments of this disclosure, during model training, a target evaluation model is trained using multiple control strategies output by the autonomous driving decision-making model. This allows the target evaluation model to evaluate the merits of different control strategies within the same traffic scenario. Subsequently, based on a reinforcement learning algorithm, the target evaluation model is used to optimize the training of the autonomous driving decision-making model, enabling it to output a more reasonable control strategy for the same traffic scenario. This improves the rationality and reliability of the autonomous driving decision-making model's output. Compared to related technologies that only provide a single control strategy, the solution of this disclosure allows the model to provide the optimal control strategy suitable for the traffic scenario and gives the model greater flexibility, enabling it to adapt to different traffic scenarios.

[0057] According to embodiments of this disclosure, training an initial evaluation model based on multiple first control strategies to obtain a target evaluation model may include the following operations.

[0058] For example, multiple first control strategies can be ranked to obtain a ranking result. Then, based on the ranking result, the initial evaluation model is trained to obtain the target evaluation model.

[0059] As mentioned earlier, the operational complexity and risk level of multiple first control strategies vary. Furthermore, the effectiveness of controlling the vehicle's driving state based on these first control strategies also differs. To enable the initial evaluation model to better learn the advantages and disadvantages of each control strategy and thus accurately evaluate different control strategies in the same traffic scenario, the advantages and disadvantages of multiple first control strategies can be ranked. The initial evaluation model can then be trained based on the ranking results to obtain the target evaluation model.

[0060] For example, multiple primary control strategies can be ranked based on their operational complexity and risk level. In ranking the control strategies, the lower the risk level, the higher the ranking. Similarly, the lower the operational complexity, the higher the ranking. A higher ranking indicates that using that control strategy to control the vehicle is more reasonable and reliable in the same traffic scenario.

[0061] Furthermore, to ensure safe driving, ranking results based on risk level take precedence over ranking results based on operational complexity. For example, when two control strategies have similar operational complexity, if the risk level of one control strategy is lower than that of the other, then the first control strategy can be ranked before the second.

[0062] In some embodiments, the operational complexity and risk level values ​​of each control strategy can be determined based on user input. Both the operational complexity and risk level values ​​can range from 0 to 1. The complexity and risk level values ​​can then be multiplied, and the control strategies can be ranked based on the result. In one example, weighting coefficients can be set for the complexity and risk level values, and the control strategies can be ranked according to their respective weighting coefficients.

[0063] Using the aforementioned first control strategies, such as overtaking, gentle braking to avoid collisions, lateral collision avoidance, and emergency braking, and ranking them according to the sorting method described above, we can obtain the following ranking result: Gentle braking to avoid collisions ≥ Lateral collision avoidance > Emergency braking > Overtaking.

[0064] In some embodiments, multiple first control strategies can also be ranked according to risk level. The method of ranking by risk level is similar to the process described above, and will not be repeated here.

[0065] It should be noted that, in addition to the complexity of operation and the level of risk, this disclosure can also rank multiple first control strategies according to other ranking conditions. The specific ranking can be set according to actual needs, and no limitation is made here.

[0066] Based on the above ranking results, the initial evaluation model can be trained to obtain the target evaluation model.

[0067] For example, based on the ranking results, a first sample control strategy and a second sample control strategy with a correlation are determined from multiple first control strategies.

[0068] Next, the initial evaluation model is used to process the first sample control strategy and the second sample control strategy respectively, to obtain the first evaluation parameter corresponding to the first sample control strategy and the second evaluation parameter corresponding to the second sample control strategy.

[0069] Next, the evaluation parameter loss is determined based on the first and second evaluation parameters. Then, based on the evaluation parameter loss, the parameters of the initial evaluation model are adjusted to obtain the target evaluation model.

[0070] For example, given the ranking result: Mild braking avoidance ≥ Lateral avoidance > Emergency stopping > Overtaking, multiple sample control strategy pairs can be determined from overtaking, mild braking avoidance, lateral avoidance, and emergency stopping based on this ranking result. Each sample control strategy pair includes a first sample control strategy and a second sample control strategy that have a correlation relationship. The correlation relationship characterizes the order of superiority of the first and second sample control strategies.

[0071] For example, mild braking and lateral avoidance can be used as the first control strategy pair, lateral avoidance and emergency stopping as the second control strategy pair, and emergency stopping and overtaking as the third control strategy pair.

[0072] For each control strategy pair, the initial evaluation model is used to process the first sample control strategy and the second sample control strategy in the control strategy pair respectively, to obtain the first evaluation parameter corresponding to the first sample control strategy and the second evaluation parameter corresponding to the second sample control strategy.

[0073] For example, by using the initial evaluation model to process the braking avoidance and lateral avoidance in the first control strategy pair, the first evaluation parameter corresponding to braking avoidance and the second evaluation parameter corresponding to lateral avoidance can be obtained. Similarly, the evaluation parameters for each control strategy in other control strategy pairs can be obtained.

[0074] Next, for each control strategy pair, the evaluation parameter loss is determined based on the corresponding first and second evaluation parameters. Then, based on the evaluation parameter loss, the parameters of the initial evaluation model are adjusted to obtain the target evaluation model.

[0075] According to embodiments of this disclosure, a target evaluation model is trained using multiple control strategies output by an autonomous driving decision model. This allows the target evaluation model to evaluate the merits of different control strategies in the same traffic scenario, thereby determining a more reasonable control strategy for that traffic scenario. This, in turn, helps to improve the rationality and reliability of the output results of the autonomous driving decision model.

[0076] According to embodiments of this disclosure, after obtaining the target evaluation model, the autonomous driving decision model can be trained based on the target evaluation model.

[0077] For example, acquiring third-party vehicle state information and third-party environmental information in a third traffic scenario. The third-party vehicle state information and third-party environmental information are based on the vehicle's state information and the surrounding environment information detected by sensors in the third traffic scenario. The third-party vehicle state information and third-party environmental information have the same or similar definitions as the first-party vehicle state information and first-party environmental information, which will not be elaborated here.

[0078] Next, the autonomous driving decision model is used to process the third vehicle state information and the third environmental information to obtain the third control strategy for the vehicle in the third traffic scenario. Then, the target evaluation model is used to process the third control strategy to obtain the target evaluation parameters corresponding to the third control strategy.

[0079] The target evaluation parameter is used to evaluate the quality of the third control strategy. The closer the third control strategy is to the optimal control strategy, the larger the target evaluation parameter will be; conversely, the smaller the third control strategy will be. Adjusting the parameters of the autonomous driving decision-making model based on the target evaluation parameter is beneficial for obtaining a higher-quality autonomous driving decision-making model.

[0080] According to embodiments of this disclosure, the autonomous driving decision-making model described above can be trained in the following manner.

[0081] First, obtain the second vehicle status information and second environment information of the vehicle in the second traffic scenario, as well as the control strategy label for the vehicle in the second traffic scenario.

[0082] The second vehicle state information and the second environmental information are vehicle state information and surrounding environmental information in a second traffic scenario detected by sensors. The second vehicle state information and the second environmental information have the same or similar definitions as the first vehicle state information and the first environmental information, respectively, and will not be repeated here.

[0083] The control strategy label is determined based on the driver's actual driving data in the second traffic scenario. Driving data includes, for example, driver operation data and vehicle operation data.

[0084] Next, the second vehicle state information and the second environment information are input into the initial deep learning model to obtain the second control strategy for the vehicle in the second traffic scenario.

[0085] Next, based on the second control strategy and the control strategy label, the control strategy loss is determined. Then, based on the control strategy loss, the parameters of the initial deep learning model are adjusted to obtain the autonomous driving decision model.

[0086] In the embodiments of this disclosure, the autonomous driving decision-making model is trained through supervised learning, which can avoid the model outputting unreasonable control strategies.

[0087] Figure 3A , Figure 3B and Figure 3C This is a schematic diagram illustrating the training process of an autonomous driving decision-making model according to an embodiment of this disclosure. See below for reference. Figures 3A to 3C An example is provided to illustrate the scheme disclosed herein.

[0088] Figure 3A The diagram illustrates the training process of the initial deep learning model. For example... Figure 3AAs shown, after acquiring the second vehicle state information and second environmental information 310 of the vehicle in the second traffic scenario, the second vehicle state information and second environmental information 310 are input into the initial deep learning model to obtain the second control strategy 312 for the vehicle in the second traffic scenario. Next, based on the second control strategy 312 and the control strategy label 314, the control strategy loss 316 is determined. Then, the parameters of the initial deep learning model are adjusted based on the control strategy loss 316 to obtain the autonomous driving decision model. The autonomous driving decision model can be used to process the vehicle state information and environmental information of a vehicle in a certain traffic scenario to obtain the control strategy for the vehicle in that traffic scenario.

[0089] Figure 3B The diagram illustrates the training process of the initial evaluation model. For example... Figure 3B As shown, after obtaining the first vehicle state information and first environment information 320 of the vehicle in the first traffic scenario, the first vehicle state information and first environment information 320 can be input into the autonomous driving decision model obtained above to obtain multiple first control strategies 322 for the vehicle in the first traffic scenario. Then, the multiple first control strategies 322 are ranked according to their merits to obtain the ranking result 323.

[0090] Next, based on the ranking result 323, multiple sample control strategy pairs are determined from multiple first control strategies 322. Each sample control strategy pair includes a first sample control strategy 322_1 and a second sample control strategy 322_2 that are related.

[0091] Next, the initial evaluation model is used to process the first sample control strategy 322_1 and the second sample control strategy 322_2 in each sample control strategy pair to obtain the first evaluation parameter 325_1 corresponding to the first sample control strategy 322_1 and the second evaluation parameter 325_2 corresponding to the second sample control strategy 322_2.

[0092] Next, for each control strategy pair, the evaluation parameter loss 326 is determined based on the first evaluation parameter 325_1 and the second evaluation parameter 325_2 corresponding to that control strategy pair. Then, based on the evaluation parameter loss 326, the parameters of the initial evaluation model are adjusted to obtain the target evaluation model.

[0093] Figure 3C The diagram illustrates the optimization process of an autonomous driving decision-making model. For example... Figure 3C As shown, the third vehicle state information and third environment information 330 of the vehicle in the third traffic scenario are obtained. Then, the third vehicle state information and third environment information 330 obtained above are processed using the autonomous driving decision model to obtain the third control strategy 332 for the vehicle in the third traffic scenario.

[0094] Next, the target evaluation model obtained above is used to process the third control strategy 332 to obtain the target evaluation parameters 335 corresponding to the third control strategy 332. Then, the parameters of the autonomous driving decision-making model can be adjusted according to the target evaluation parameters 335.

[0095] According to embodiments of this disclosure, during model training, a target evaluation model is trained using multiple control strategies output by the autonomous driving decision-making model. This allows the target evaluation model to evaluate the merits of different control strategies within the same traffic scenario. Subsequently, based on a reinforcement learning algorithm, the target evaluation model is used to optimize and train the autonomous driving decision-making model, enabling it to output more reasonable control strategies for the same traffic scenario. This improves the rationality and reliability of the autonomous driving decision-making model's output results.

[0096] Figure 4 This is a flowchart of a vehicle control method according to an embodiment of the present disclosure.

[0097] like Figure 4 As shown, the vehicle control method 400 may include operations S410 to S430.

[0098] The S410 is used to obtain vehicle status information and environmental information in a traffic scenario.

[0099] When operating the S420, the autonomous driving decision model is used to process vehicle state information and environmental information to obtain control strategies for the vehicle in traffic scenarios.

[0100] When operating the S430, the vehicle is controlled based on the control strategy.

[0101] According to embodiments of this disclosure, the autonomous driving decision model is trained using the training method for the autonomous driving decision model described in the above embodiments. By inputting the vehicle state information and environmental information of a vehicle in a certain traffic scenario into the autonomous driving decision model, the optimal control strategy for the vehicle in that traffic scenario can be obtained. Based on this optimal control strategy, the vehicle can be controlled, thereby achieving autonomous driving.

[0102] In the embodiments of this disclosure, by using the autonomous driving decision model trained in the above manner to make decisions based on vehicle state information and environmental information, the optimal control strategy for the vehicle can be obtained, which can improve the rationality and reliability of the control strategy, thereby improving the control effect of the vehicle.

[0103] Figure 5 This is a block diagram of a training apparatus for an autonomous driving decision-making model according to an embodiment of the present disclosure.

[0104] like Figure 5As shown, the training device 500 for the autonomous driving decision model includes a first acquisition module 510, a first processing module 520, a first training module 530, and a second training module 540.

[0105] The first acquisition module 510 is used to acquire the first vehicle status information and the first environment information of the vehicle in the first traffic scenario.

[0106] The first processing module 520 is used to process the first vehicle state information and the first environment information multiple times using the autonomous driving decision model to obtain multiple first control strategies for the vehicle in the first traffic scenario; wherein, the autonomous driving decision model is trained based on the second vehicle state information and the second environment information of the vehicle in the second traffic scenario.

[0107] The first training module 530 is used to train an initial evaluation model based on multiple first control strategies to obtain a target evaluation model.

[0108] The second training module 540 is used to train the autonomous driving decision model based on the target evaluation model.

[0109] According to embodiments of this disclosure, the first training module 530 includes a ranking submodule and a training submodule. The ranking submodule is used to rank multiple first control strategies to obtain a ranking result; and the training submodule is used to train an initial evaluation model based on the ranking result to obtain a target evaluation model.

[0110] According to embodiments of this disclosure, the training submodule includes: a first determining unit, a processing unit, a second determining unit, and an adjustment unit. The first determining unit is used to determine a first sample control strategy and a second sample control strategy with a correlation relationship from multiple first control strategies based on the ranking results; the processing unit is used to process the first sample control strategy and the second sample control strategy respectively using an initial evaluation model to obtain a first evaluation parameter corresponding to the first sample control strategy and a second evaluation parameter corresponding to the second sample control strategy; the second determining unit is used to determine the evaluation parameter loss based on the first evaluation parameter and the second evaluation parameter; and the adjustment unit is used to adjust the parameters of the initial evaluation model based on the evaluation parameter loss to obtain a target evaluation model.

[0111] According to embodiments of this disclosure, the second training module 540 includes: an acquisition submodule, a first processing submodule, a second processing submodule, and an adjustment submodule. The acquisition submodule acquires third vehicle state information and third environment information of the vehicle in the third traffic scenario; the first processing submodule processes the third vehicle state information and third environment information using an autonomous driving decision model to obtain a third control strategy for the vehicle in the third traffic scenario; the second processing submodule processes the third control strategy using a target evaluation model to obtain target evaluation parameters corresponding to the third control strategy; and the adjustment submodule adjusts the parameters of the autonomous driving decision model based on the target evaluation parameters.

[0112] According to embodiments of this disclosure, the training device 500 for the autonomous driving decision model further includes: a second acquisition module, a second processing module, a determination module, and an adjustment module. The second acquisition module acquires second vehicle state information and second environmental information of the vehicle in a second traffic scenario, as well as a control policy label for the vehicle in the second traffic scenario. The second processing module inputs the second vehicle state information and the second environmental information into an initial deep learning model to obtain a second control policy for the vehicle in the second traffic scenario. The determination module determines a control policy loss based on the second control policy and the control policy label. The adjustment module adjusts the parameters of the initial deep learning model based on the control policy loss to obtain the autonomous driving decision model.

[0113] Figure 6 This is a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0114] like Figure 6 As shown, the vehicle control device 600 includes a third acquisition module 610, a third processing module 620, and a control module 630.

[0115] The third acquisition module 610 is used to acquire vehicle status information and environmental information of vehicles in traffic scenarios.

[0116] The third processing module 620 is used to process vehicle state information and environmental information using the autonomous driving decision model to obtain a control strategy for the vehicle in a traffic scenario. The autonomous driving decision model is trained using the training device for the autonomous driving decision model in the above embodiments.

[0117] The control module 630 is used to control the vehicle based on a control strategy.

[0118] It should be noted that the implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the device embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of each corresponding step in the method embodiments, and will not be repeated here.

[0119] The collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) involved in the technical solutions disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0120] In the technical solutions disclosed herein, the authorization or consent of the data owner is obtained before acquiring or collecting relevant data.

[0121] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, a computer program product, and an autonomous vehicle.

[0122] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in the embodiments of the present disclosure.

[0123] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform methods as described in embodiments of the present disclosure.

[0124] According to embodiments of the present disclosure, a computer program product includes a computer program that, when executed by a processor, implements the methods as described in embodiments of the present disclosure.

[0125] According to embodiments of this disclosure, an autonomous vehicle may include the electronic devices described in any of the above embodiments. The electronic devices may be located within the autonomous vehicle or remotely; this disclosure does not limit this. The electronic devices, when executed by their processor, can implement the methods described in the above embodiments.

[0126] Figure 7 This is a schematic diagram of an autonomous vehicle according to an embodiment of the present disclosure.

[0127] like Figure 7 As shown, this disclosure also provides an example autonomous driving vehicle 700, which may include the electronic equipment of any of the above embodiments. The autonomous driving vehicle 700 can control its movement according to a control strategy input by the electronic equipment. It is understood that the electronic equipment may be located within the autonomous driving vehicle 700 or remotely; this disclosure does not limit this.

[0128] In some embodiments, the autonomous vehicle 700 may also include sensing devices to acquire various sensing data for autonomous driving. Examples of sensing devices include, but are not limited to, radar, cameras, Global Positioning System (GPS), BeiDou Navigation Satellite System, or other positioning systems, as well as various sensors such as angle sensors, speed sensors, and proximity sensors.

[0129] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0130] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0131] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as methods for training autonomous driving decision models and methods for vehicle control. For example, in some embodiments, methods for training autonomous driving decision models and methods for vehicle control can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the methods for training autonomous driving decision models and methods for vehicle control described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to execute training methods for autonomous driving decision models and vehicle control methods.

[0133] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for an autonomous driving decision-making model, comprising: Obtain the first vehicle status information and the first environmental information of the vehicle in the first traffic scenario; The first vehicle state information and the first environment information are processed multiple times using an autonomous driving decision model to obtain multiple first control strategies for the vehicle in the first traffic scenario; wherein, the autonomous driving decision model is trained based on the second vehicle state information and the second environment information of the vehicle in the second traffic scenario. An initial evaluation model is trained based on the multiple first control strategies to obtain a target evaluation model; and The autonomous driving decision-making model is trained based on the target evaluation model. The step of training an initial evaluation model based on the plurality of first control strategies to obtain a target evaluation model includes: The plurality of first control strategies are sorted to obtain a sorting result; and Based on the ranking results, the initial evaluation model is trained to obtain the target evaluation model.

2. The method according to claim 1, wherein, The step of training the initial evaluation model based on the ranking results to obtain the target evaluation model includes: Based on the ranking results, a first sample control strategy and a second sample control strategy with a correlation are determined from the plurality of first control strategies; The initial evaluation model is used to process the first sample control strategy and the second sample control strategy respectively to obtain a first evaluation parameter corresponding to the first sample control strategy and a second evaluation parameter corresponding to the second sample control strategy. Based on the first evaluation parameter and the second evaluation parameter, determine the evaluation parameter loss; and Based on the loss of the evaluation parameters, the parameters of the initial evaluation model are adjusted to obtain the target evaluation model.

3. The method according to claim 1, wherein, The step of training the autonomous driving decision model based on the target evaluation model includes: Obtain third-party vehicle status information and third-party environmental information in the third traffic scenario; The autonomous driving decision model is used to process the third vehicle state information and the third environment information to obtain a third control strategy for the vehicle in the third traffic scenario. The target evaluation model is used to process the third control strategy to obtain target evaluation parameters corresponding to the third control strategy; and The parameters of the autonomous driving decision-making model are adjusted based on the target evaluation parameters.

4. The method according to claim 1, further comprising: Acquire the second vehicle status information and second environment information of the vehicle in the second traffic scenario, as well as the control strategy label for the vehicle in the second traffic scenario; The second vehicle state information and the second environment information are input into the initial deep learning model to obtain the second control strategy for the vehicle in the second traffic scenario. Based on the second control strategy and the control strategy label, determine the control strategy loss; as well as Based on the loss of the control strategy, the parameters of the initial deep learning model are adjusted to obtain the autonomous driving decision model.

5. A vehicle control method, comprising: Obtain vehicle status information and environmental information in traffic scenarios; The vehicle state information and the environmental information are processed using an autonomous driving decision model to obtain a control strategy for the vehicle in the traffic scenario; wherein the autonomous driving decision model is trained using the method according to any one of claims 1 to 4; and The vehicle is controlled based on the control strategy described above.

6. A training device for an autonomous driving decision-making model, comprising: The first acquisition module is used to acquire the first vehicle status information and the first environment information of the vehicle in the first traffic scenario. The first processing module is used to process the first vehicle state information and the first environment information multiple times using an autonomous driving decision model to obtain multiple first control strategies for the vehicle under the first traffic scenario; wherein, the autonomous driving decision model is trained based on the second vehicle state information and the second environment information of the vehicle in the second traffic scenario. A first training module is used to train an initial evaluation model based on the plurality of first control strategies to obtain a target evaluation model; and The second training module is used to train the autonomous driving decision-making model based on the target evaluation model. The first training module includes: The sorting submodule is used to sort the plurality of first control strategies to obtain a sorting result; and The training submodule is used to train the initial evaluation model based on the ranking results to obtain the target evaluation model.

7. The apparatus according to claim 6, wherein, The training submodule includes: The first determining unit is configured to determine, based on the sorting result, a first sample control strategy and a second sample control strategy that have a correlation relationship from the plurality of first control strategies; The processing unit is used to process the first sample control strategy and the second sample control strategy using the initial evaluation model to obtain a first evaluation parameter corresponding to the first sample control strategy and a second evaluation parameter corresponding to the second sample control strategy. The second determining unit is configured to determine the evaluation parameter loss based on the first evaluation parameter and the second evaluation parameter; and An adjustment unit is used to adjust the parameters of the initial evaluation model according to the evaluation parameter loss, so as to obtain the target evaluation model.

8. The apparatus according to claim 6, wherein, The second training module includes: The acquisition submodule is used to acquire the third vehicle status information and third environment information of vehicles in the third traffic scenario; The first processing submodule is used to process the third vehicle state information and the third environment information using the autonomous driving decision model to obtain a third control strategy for the vehicle in the third traffic scenario. The second processing submodule is used to process the third control strategy using the target evaluation model to obtain target evaluation parameters corresponding to the third control strategy; and The adjustment submodule is used to adjust the parameters of the autonomous driving decision-making model according to the target evaluation parameters.

9. The apparatus according to claim 6, further comprising: The second acquisition module is used to acquire the second vehicle status information and the second environment information of the vehicle in the second traffic scenario, as well as the control strategy label for the vehicle in the second traffic scenario. The second processing module is used to input the second vehicle state information and the second environment information into the initial deep learning model to obtain the second control strategy for the vehicle under the second traffic scenario. The determination module is used to determine the control strategy loss based on the second control strategy and the control strategy label; as well as An adjustment module is used to adjust the parameters of the initial deep learning model based on the control strategy loss, thereby obtaining the autonomous driving decision model.

10. A vehicle control device, comprising: The third acquisition module is used to acquire vehicle status information and environmental information of vehicles in traffic scenarios; The third processing module is used to process the vehicle state information and the environmental information using an autonomous driving decision model to obtain a control strategy for the vehicle in the traffic scenario; wherein the autonomous driving decision model is trained by the device according to any one of claims 6 to 9. as well as A control module is used to control the vehicle based on the control strategy.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

12. An autonomous vehicle, including the electronic equipment as claimed in claim 11.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 5.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images