Automatic driving method and related model training method, electronic device, and storage medium
By incorporating timestamp training methods from road test scenario segments into the autonomous driving learning model, and utilizing the state information of the target vehicle and dynamic objects in the environment, as well as reward values, the model is optimized. This solves the problem of low prediction accuracy of reinforcement learning models under multiple environmental factors, and achieves more stable and accurate autonomous driving predictions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN DEEPROUTE AI CO LTD
- Filing Date
- 2022-07-29
- Publication Date
- 2026-07-24
AI Technical Summary
Existing reinforcement learning models have failed to effectively integrate with specific scenarios in the field of autonomous driving, resulting in low prediction accuracy under multiple environmental factors.
By acquiring the state information of target vehicles and dynamic objects in the road test scene segments, the system uses timestamps for training and combines reward values to optimize the autonomous driving learning model, including prediction-assisted verification of dynamic objects in the environment.
It improves the stability and prediction accuracy of autonomous driving learning models and enhances their predictive capabilities under multiple environmental factors.
Smart Images
Figure CN115358416B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving method and related model training methods, electronic devices, and storage media. Background Technology
[0002] Autonomous driving technology is a crucial technology in the automotive industry and a key research focus for major manufacturers. Applying reinforcement learning to autonomous driving can serve as an important technological foundation. However, the reinforcement learning models in related technologies are trained using historical experience data without being integrated with specific application scenarios, resulting in low prediction accuracy when facing application scenarios with multiple environmental factors. Summary of the Invention
[0003] This application provides at least one autonomous driving method and related model training method, electronic device, and storage medium.
[0004] The first aspect of this application provides a method for training an autonomous driving learning model, comprising: acquiring a road test scene segment for training, wherein the road test scene segment describes an autonomous driving road test scene, the autonomous driving road test scene including a target vehicle and environmental dynamic objects; acquiring a first state information set of the target vehicle at the current time stamp and the next time stamp, a second state information set of the environmental dynamic objects at the current time stamp and the next time stamp, and a reward value of the target vehicle during the time period between the current time stamp and the next time stamp, wherein the environmental dynamic objects include multiple dynamic vehicles at a preset distance from the target vehicle, and the second state information set includes state information of the multiple dynamic vehicles obtained using an autonomous driving learning model corresponding to the current time stamp; and using the first state information set, the second state information set, and the reward value to train the autonomous driving learning model corresponding to the current time stamp for use in the autonomous driving learning model corresponding to the next time stamp, thereby obtaining a trained autonomous driving learning model.
[0005] The road test scene segment includes road test data located at multiple timestamps; obtaining the second state information set of the dynamic objects in the environment at the current timestamp and the next timestamp includes: obtaining the current state information of each of the multiple dynamic vehicles at the current timestamp from the road test data at the current timestamp; obtaining the next state information of each dynamic vehicle at the next timestamp based on the current state information of each dynamic vehicle at the current timestamp and using the autonomous driving learning model corresponding to the current timestamp; obtaining the current state information of the remaining dynamic objects in the environment at the current timestamp and the next state information at the next timestamp from the road test data at the current timestamp and the next state information at the next timestamp.
[0006] The road test scenario segment further includes trajectory data of each of the multiple dynamic vehicles; based on the current state information of the multiple dynamic vehicles at the current timestamp, and using the autonomous driving learning model corresponding to the current timestamp, the next state information of the multiple dynamic vehicles at the next timestamp is obtained, including: based on the current state information of each dynamic vehicle at the current timestamp, and using the autonomous driving learning model corresponding to the current timestamp, predicting the first action information of each dynamic vehicle; obtaining the second action information of each dynamic vehicle from the trajectory data of each dynamic vehicle; and performing weighted processing on the first action information and the second action information to obtain the next state information of each dynamic vehicle at the next timestamp.
[0007] The weighting coefficient of the first action information increases as the current timestamp moves forward among the plurality of timestamps, while the weighting coefficient of the second action information decreases as the current timestamp moves forward among the plurality of timestamps.
[0008] The road test scenario segment includes road test data located at multiple timestamps; obtaining the first state information set of the target vehicle at the current timestamp and the next timestamp includes: obtaining the current state information of the target vehicle at the current timestamp from the road test data at the current timestamp; and obtaining the next state information of the target vehicle at the next timestamp based on the current state information of the target vehicle at the current timestamp, the predetermined action information of the target vehicle, and the time period.
[0009] The step of obtaining the reward value of the target vehicle during the time period between the current timestamp and the next timestamp includes: obtaining the reward value of the target vehicle during the time period based on the first state information set and the second state information set.
[0010] The autonomous driving road test scenario also includes traffic lights; the method further includes: obtaining a third state information set of the traffic lights at the current timestamp and the next timestamp; using the first state information set, the second state information set, and the reward value to train the autonomous driving learning model corresponding to the current timestamp, so as to use the autonomous driving learning model corresponding to the next timestamp, further including: using the first state information set, the second state information set, the third state information, and the reward value to train the autonomous driving learning model corresponding to the current timestamp, so as to use the autonomous driving learning model corresponding to the next timestamp.
[0011] A second aspect of this application provides an autonomous driving method applied to a target vehicle, comprising:
[0012] The first state information of the target vehicle at its current timestamp in the autonomous driving road test scenario and the second state information of dynamic objects in the environment in the autonomous driving road test scenario are obtained; the first state information and the second state information are input into the autonomous driving learning model to obtain the driving action of the target vehicle, so that the target vehicle drives according to the driving action; wherein, the autonomous driving learning model is trained using the training method of the autonomous driving learning model in the first aspect above.
[0013] A third aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the training method of the autonomous driving learning model in the first aspect above, or to implement the autonomous driving method in the second aspect above.
[0014] The fourth aspect of this application provides a non-volatile computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the training method of the autonomous driving learning model in the first aspect above, or implement the autonomous driving method in the second aspect above.
[0015] The above scheme obtains a first state information set of the target vehicle at the current time stamp and the next time stamp, a second state information set of environmental dynamic objects at the current time stamp and the next time stamp, and a reward value of the target vehicle during the time period between the current time stamp and the next time stamp. Using the first state information set, the second state information set, and the reward value, the autonomous driving learning model corresponding to the current time stamp is trained to obtain the autonomous driving learning model corresponding to the next time stamp. Thus, a trained autonomous driving learning model is obtained. In this way, when training the autonomous driving learning model, the prediction of environmental dynamic objects is combined to assist in verifying the prediction of the target vehicle, making the trained autonomous driving learning model more stable and more accurate in making predictions.
[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.
[0018] Figure 1 This is a flowchart illustrating an embodiment of the training method for the autonomous driving learning model of this application;
[0019] Figure 2 This is a schematic diagram illustrating the change process of the weighting coefficients in the training method of the autonomous driving learning model of this application;
[0020] Figure 3 This is a flowchart illustrating another embodiment of the training method for the autonomous driving learning model of this application;
[0021] Figure 4 This is a flowchart illustrating an embodiment of the autonomous driving method of this application;
[0022] Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;
[0023] Figure 6 This is a schematic diagram of a framework of an embodiment of the non-volatile computer-readable storage medium of this application. Detailed Implementation
[0024] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0025] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0026] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0027] Please see Figure 1 , Figure 1 This is a flowchart illustrating a training method for an autonomous driving learning model according to an embodiment of this application. The method is applied to an electronic device, which can be installed in a vehicle and can be an in-vehicle device. In some possible implementations, the training method for the autonomous driving learning model can be implemented by a processor calling computer-readable instructions stored in memory. Specifically, it may include the following steps:
[0028] Step S100: Obtain road test scene segments for training, wherein the road test scene segments are used to describe autonomous driving road test scenarios, and the autonomous driving road test scenarios include target vehicles and dynamic objects in the environment.
[0029] In this embodiment, the road test scene segment can be a segment ranging from 10 to 30 seconds, and can include autonomous driving road test scenarios such as going straight, overtaking, turning, and detouring. Dynamic objects in the environment include other vehicles and pedestrians besides the target vehicle.
[0030] To facilitate training, road test scene segments are divided into multiple smaller segments based on timestamps. These timestamps are then combined sequentially to obtain a corresponding time array, such as [0, 0.1, 0.2, ..., 9.9, 1.0]. The first smaller segment is between 0 and 0.1 seconds. The unit of timestamps is seconds, and a value of 0 represents a timestamp of 0 seconds. The number of elements and the time interval in the time array can be set uniformly or as needed.
[0031] When used for training, road test scenario clips must include the following information:
[0032] 1) Trajectory data of dynamic objects in the environment. For ease of querying, each dynamic object in the environment is assigned a corresponding ID and type. The trajectory data of dynamic objects in the environment includes a dataset composed of the state information of the dynamic object at each time stamp. The state information includes: position coordinates (x, y, z), velocity (v...). x v y v z The dimensions (l, w, h) and orientation angle (theta) are used to determine the target vehicle's position. The position coordinates are obtained from the map information corresponding to the road test scene segment. Subsequent state information for the target vehicle and dynamic objects in the environment includes position coordinates (x, y, z) and velocity (v). x v y v z ), length, width, and height (l, w, h), and orientation angle (theta).
[0033] 2) Static map information, specifically including: road information in the road test scenario segment. The road test scenario segment contains several roads, each with speed limits, a center line, and edge lines. The driving trajectories of the target vehicle and other vehicles in the dynamic environment along the road center line and edge lines are generated as a series of position coordinates (x, y, z) based on timestamps, used for subsequent reward value calculation.
[0034] Step S101: Obtain the first state information set of the target vehicle at the current time stamp and the next time stamp, the second state information set of the environmental dynamic objects at the current time stamp and the next time stamp, and the reward value of the target vehicle during the time period between the current time stamp and the next time stamp. The environmental dynamic objects include multiple dynamic vehicles at a preset distance from the target vehicle, and the second state information set includes the state information of multiple dynamic vehicles obtained by using the autonomous driving learning model corresponding to the current time stamp.
[0035] The time interval between the current timestamp and the next timestamp can be any time interval within a road test scenario segment. The first state information set includes the current state information of the target vehicle at the current timestamp and the next state information of the target vehicle at the next timestamp; the second state information set includes the current state information of environmental dynamic objects at the current timestamp and the next state information of environmental dynamic objects at the next timestamp.
[0036] The current state information of the target vehicle at the current timestamp and the current state information of the dynamic objects in the environment at the current timestamp are used as the state, the next state information of the target vehicle at the next timestamp and the next state information of the dynamic objects in the environment at the next timestamp are used as the new-state, and the reward value is used as the reward. The autonomous driving learning model at the current timestamp is trained to obtain the autonomous driving learning model at the next timestamp.
[0037] Multiple dynamic vehicles in the environment are positioned at preset distances from the target vehicle; for example, the dynamic vehicle closest to the target vehicle. These multiple dynamic vehicles can be three vehicles positioned in front of, to the left of, and to the right of the target vehicle.
[0038] Step S102: Using the first state information set, the second state information set, and the reward value, train the autonomous driving learning model corresponding to the current timestamp, so as to use it for the autonomous driving learning model corresponding to the next timestamp, thereby obtaining the trained autonomous driving learning model.
[0039] Training the autonomous driving learning model at the current timestamp means training the autonomous driving learning model at the current timestamp. In other words, the information at the current timestamp is input into the current autonomous driving learning model to perform training, so that it can be used for the autonomous driving learning model at the next timestamp. The autonomous driving learning model at the current timestamp and the autonomous driving learning model at the next timestamp have the same model architecture, but their model parameters are different.
[0040] Specifically, the autonomous driving learning model corresponding to the current timestamp is trained using a first state information set, a second state information set, and a reward value to obtain the autonomous driving learning model corresponding to the next timestamp. The specific training steps include: inputting the state information of the current timestamp into the autonomous driving learning model for that timestamp, causing it to output the predicted action for the current timestamp; driving the target vehicle according to the predicted action to obtain the target vehicle's state information at the next timestamp; assigning a corresponding reward value based on the state information at the next timestamp; and then adjusting the model parameters of the autonomous driving learning model based on the reward value to obtain the autonomous driving learning model corresponding to the next timestamp.
[0041] In this embodiment, by acquiring the first state information set of the target vehicle at the current time stamp and the next time stamp, the second state information set of the environmental dynamic objects at the current time stamp and the next time stamp, and the reward value of the target vehicle during the time period between the current time stamp and the next time stamp, the autonomous driving learning model corresponding to the current time stamp is trained using the first state information set, the second state information set, and the reward value to obtain the autonomous driving learning model corresponding to the next time stamp. Thus, a trained autonomous driving learning model is obtained. Therefore, when training the autonomous driving learning model, the prediction of environmental dynamic objects is combined to assist in verifying the prediction of the target vehicle, making the trained autonomous driving learning model more stable and more accurate in making predictions.
[0042] In an exemplary embodiment, the road test scenario segment includes road test data located at multiple timestamps; the step of obtaining the second state information set in the above embodiment may further include:
[0043] From the road test data at the current timestamp, obtain the current state information of each dynamic vehicle among multiple dynamic vehicles at the current timestamp; based on the current state information of each dynamic vehicle at the current timestamp, use the autonomous driving learning model corresponding to the current timestamp to obtain the next state information of each dynamic vehicle at the next timestamp; from the road test data at the current timestamp and the next timestamp, obtain the current state information of the remaining dynamic objects in the environment at the current timestamp and the next state information at the next timestamp.
[0044] In this embodiment, the current state information of the dynamic vehicle at the current timestamp can be obtained from the road test data corresponding to the current timestamp. In order to achieve prediction of dynamic objects in the autonomous driving learning model, the next state information of the dynamic vehicle at the next timestamp is calculated based on the trajectory data of the dynamic vehicle between time periods and the autonomous driving learning model at the current timestamp.
[0045] In this embodiment, the remaining dynamic objects include other environmental vehicles and pedestrians besides the three dynamic vehicles. The current state information of the remaining dynamic objects at the current timestamp can be obtained from the road test data at the current timestamp; the next state information of the remaining dynamic objects at the next timestamp can be obtained from the road test data at the next timestamp. Specifically, the next state information of each remaining dynamic object at the next timestamp can be obtained from the trajectory data of each remaining dynamic object between the current timestamp and the next timestamp.
[0046] In an exemplary embodiment, the road test scenario segment further includes trajectory data of each of the multiple dynamic vehicles; the step of obtaining the second state information set in step S101 above may further include:
[0047] From the road test data at the current timestamp, obtain the current state information of each dynamic vehicle among multiple dynamic vehicles at the current timestamp; based on the current state information of each dynamic vehicle at the current timestamp, use the autonomous driving learning model corresponding to the current timestamp to predict the first action information of each dynamic vehicle; from the trajectory data of each dynamic vehicle, obtain the second action information of each dynamic vehicle; perform weighted processing on the first action information and the second action information to obtain the next state information of each dynamic vehicle at the next timestamp; and from the road test data at the current timestamp and the next timestamp, obtain the current state information of the remaining dynamic objects in the environment at the current timestamp and the next state information at the next timestamp.
[0048] In this embodiment, to predict dynamic vehicles in the autonomous driving learning model, during training, the actual second action information at the current timestamp is obtained based on the trajectory data of the dynamic vehicle. First action information is predicted based on the current state information of the dynamic vehicle at the current timestamp. The prediction of the first action information uses the autonomous driving enhancement model at the current timestamp, ensuring the accuracy of the data prediction. Then, the first and second action information are weighted according to a preset weighting rule to obtain the weighted action information of the dynamic vehicle at the current timestamp. Finally, the vehicle kinematics model is used to calculate the next state information of the dynamic vehicle at the next timestamp using the weighted action information and the current state information of the dynamic vehicle at the current timestamp.
[0049] In an exemplary embodiment, the rules for the weighting coefficients may include: the weighting coefficient of the first action information increases as the current timestamp moves forward among multiple timestamps, and the weighting coefficient of the second action information decreases as the current timestamp moves forward among multiple timestamps.
[0050] In this embodiment, the training rule for predicting dynamic vehicles in the autonomous driving learning model is as follows: At the start of training, the weighting coefficient of the first action information is 0, and the weighting coefficient of the second action information is 1.0. At this point, the weighted action information of the dynamic vehicle obtained is close to the action information in the real trajectory data. As training progresses, the weighting coefficient of the first action information is gradually increased, making the weighting coefficient of the first action information larger and larger, while the weighting coefficient of the second action information is made smaller and smaller. Ideally, the weighting coefficient of the first action information is 1, and the weighting coefficient of the second action information is 0. At this point, the weighted action information of the dynamic vehicle obtained is close to the action information predicted by the autonomous driving learning model. Please refer to [link / reference] for details. Figure 2 This diagram illustrates the change process of the weighting coefficients in this application. In the diagram, steer represents the front wheel steering angle, acceleration represents longitudinal acceleration (acceleration along the direction of the target vehicle), real represents the second value corresponding to the second action information, and prediction represents the first value corresponding to the first action information. The first action information includes a first action and a corresponding first value; the first value is a two-dimensional array including the corresponding acceleration and steer. The second action information includes a second action and a corresponding second value; the second value is a two-dimensional array including the corresponding acceleration and steer. The closer the weighted average of the first and second values is to real, the greater the weight of the second action; the closer the weighted average of the first and second values is to prediction, the greater the weight of the first action. Based on the change of the weighting coefficients, the actual actions of the dynamic vehicle are gradually transformed into predicted actions output by the autonomous driving learning model, thereby realizing the training process of the dynamic vehicle's self-play. In this embodiment, during the change of the weighting coefficients, the sum of the weighting coefficients of the first action information and the weighting coefficients of the second action information remains constant at 1.
[0051] In an exemplary embodiment, the road test scene segment includes road test data located at multiple timestamps; the step of obtaining the first state information set of the target vehicle in step S101 above may further include:
[0052] Obtain the current status information of the target vehicle at the current timestamp from the road test data at the current timestamp; based on the current status information of the target vehicle at the current timestamp, the target vehicle's planned action information and time period, obtain the next status information of the target vehicle at the next timestamp.
[0053] In this embodiment, the vehicle kinematics model calculates the next state information of the target vehicle at the next timestamp based on predetermined action information, current state information, and time period. To make the next state information more accurate, the vehicle kinematics model can also incorporate data from the target vehicle itself, traffic lights, and roads, such as the target vehicle's length, width, and height, the duration of its red light wait, and road speed limits.
[0054] In an exemplary embodiment, the step of obtaining the reward value of the target vehicle between the current timestamp and the next timestamp in step S101 may further include:
[0055] Based on the first state information set and the second state information set, obtain the reward value of the target vehicle within the time period.
[0056] In this embodiment, reward and penalty functions are preset, and the reward value is obtained by calculating the first state information set and the second state information set according to the reward and penalty functions.
[0057] In this embodiment, to make the obtained reward value more accurate, auxiliary information is calculated based on the target vehicle's first state information set. Then, the auxiliary information is calculated using reward and penalty functions to obtain an auxiliary reward value. The reward value also includes the auxiliary reward value. The auxiliary information includes the target vehicle's current road location information, whether it has crossed traffic light start and end lines, whether it has run a red light, and whether it is speeding. The current road location information includes the center line location information, edge line location information, and the current road speed limit information. Based on the center line information, the reward value for whether the target vehicle crossed the line can be calculated; based on the speed limit information, the reward value for whether it is speeding can be calculated, and other auxiliary reward values.
[0058] Please see Figure 3 , Figure 3 This is a flowchart illustrating another method for training an autonomous driving learning model according to an embodiment of this application. The method is applied to an electronic device, which can be installed in a vehicle and can be an in-vehicle device. In some possible implementations, the training method for the autonomous driving learning model can be implemented by a processor calling computer-readable instructions stored in memory. Specifically, it may include the following steps:
[0059] Step S300: Obtain road test scene segments for training, wherein the road test scene segments are used to describe autonomous driving road test scenarios, and autonomous driving road test scenarios include target vehicles, dynamic objects in the environment, and traffic lights.
[0060] In this embodiment, the road test scenario segment includes the following information:
[0061] 1) Trajectory information of dynamic objects in the environment. Each dynamic object in the environment has a corresponding ID and type. The trajectory data of the dynamic objects is a dataset composed of the state information of the dynamic object at each time stamp. The state information includes: position coordinates (x, y, z), velocity (v) x v y v z The dimensions (l, w, h) and orientation angle (theta) are used to determine the target vehicle's position. The position coordinates are obtained from the map information corresponding to the road test scene segment. Subsequent state information for the target vehicle and dynamic objects in the environment includes position coordinates (x, y, z) and velocity (v). x v y v z ), length, width, and height (l, w, h), and orientation angle (theta).
[0062] 2) Static map information, specifically including: road information in the road test scenario segment. The road test scenario segment contains several roads, each with speed limits, a center line, and edge lines. The driving trajectories of the target vehicle and other vehicles in the dynamic environment along the road center line and edge lines are generated as a series of (x, y, z) position coordinates based on timestamps, used for subsequent reward value calculation.
[0063] 3) Dynamic map information, specifically including: traffic light information for each timestamp; each traffic light information includes the corresponding color status, i.e., red, yellow, and green, as well as the start and end points of the corresponding stop line, which can be represented by location coordinates.
[0064] Step S301: Obtain the first state information set of the target vehicle at the current time stamp and the next time stamp, the second state information set of the environmental dynamic objects at the current time stamp and the next time stamp, the reward value of the target vehicle during the time period between the current time stamp and the next time stamp, and the third state information set of the traffic light at the current time stamp and the next time stamp. The environmental dynamic objects include multiple dynamic vehicles at a preset distance from the target vehicle, and the second state information set includes the state information of multiple dynamic vehicles obtained by using the autonomous driving learning model corresponding to the current time stamp.
[0065] In this embodiment, the third state information set of the traffic light at the current timestamp and the next timestamp can be obtained from the road test data between the current timestamp and the next timestamp. The current state information of the traffic light at the current timestamp is obtained from the road test data at the current timestamp; the next state information of the traffic light at the next timestamp is obtained from the road test data at the next timestamp.
[0066] In some embodiments, the extension of step S301 has been described in the above embodiments and will not be repeated here.
[0067] Step S302: Using the first state information set, the second state information set, the third state information, and the reward value, train the autonomous driving learning model corresponding to the current timestamp so that it can be used for the autonomous driving learning model corresponding to the next timestamp.
[0068] When training an autonomous driving learning model, traffic lights and dynamic environmental objects are incorporated for auxiliary training. When making predictions, the model can anticipate the next action of these dynamic objects and adjust the predicted next action of the target vehicle based on this information, along with the state of the traffic lights. This makes the action predictions of the trained autonomous driving learning model more accurate.
[0069] Please see Figure 4 , Figure 4 This is a flowchart illustrating an autonomous driving method according to an embodiment of this application, applied to a target vehicle. Specifically, it can be applied to an in-vehicle device of the target vehicle. The autonomous driving method may include the following steps:
[0070] Step S400: Obtain the first state information of the target vehicle at the current timestamp in the autonomous driving road test scenario and the second state information of the dynamic objects in the environment in the autonomous driving road test scenario.
[0071] In this embodiment, the target vehicle can be an autonomous vehicle, and a camera can be installed on the target vehicle to collect the first state information of the target vehicle at the current timestamp and the second state information of dynamic objects in the environment at the current timestamp. The information collection in this embodiment is not limited to a camera; it can also be an information collection device such as a depth camera.
[0072] Step S401: Input the first state information and the second state information into the autonomous driving learning model to obtain the driving action of the target vehicle, so that the target vehicle drives according to the driving action; wherein, the autonomous driving learning model is trained using the training method of the autonomous driving learning model in the above embodiment.
[0073] In this embodiment, the autonomous driving learning model is trained to predict corresponding driving actions based on the state information of the current timestamp. The target vehicle then drives according to the driving actions, obtains the state information and reward value of the next timestamp, and subsequently predicts the driving actions for the next timestamp. The driving actions in this embodiment include acceleration and steering angle.
[0074] In this embodiment, an autonomous driving learning model is trained using a reinforcement learning model training method for autonomous driving. Then, the autonomous driving learning model is combined with the state data of the target vehicle and dynamic objects in the environment to predict the driving route of the target vehicle in actual operation. This can effectively avoid traffic accidents and improve the accuracy of prediction.
[0075] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0076] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the electronic device 50 of this application. The electronic device 50 is installed in a vehicle and can be an in-vehicle device. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps of any of the above-described embodiments of the training method for the autonomous driving learning model, or to implement the steps of any of the above-described embodiments of the autonomous driving method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server.
[0077] Specifically, processor 52 controls itself and memory 51 to implement the steps of any of the above-described autonomous driving learning model training method embodiments, or to implement the steps of any of the above-described autonomous driving method embodiments. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.
[0078] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the non-volatile computer-readable storage medium 60 of this application. The non-volatile computer-readable storage medium 60 stores program instructions 601 that can be executed by a processor. The program instructions 601 are used to implement the steps of any of the above-described embodiments of the training method for the autonomous driving learning model, or to implement the steps of any of the above-described embodiments of the autonomous driving method.
[0079] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0080] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0081] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A training method for an autonomous driving learning model, characterized in that, include: Acquire road test scene segments for training, wherein the road test scene segments are used to describe autonomous driving road test scenarios, and the autonomous driving road test scenarios include target vehicles and dynamic objects in the environment; The system acquires a first state information set of the target vehicle at the current time and the next time, a second state information set of the environmental dynamic objects at the current time and the next time, and a reward value of the target vehicle during the time period between the current time and the next time. The environmental dynamic objects include multiple dynamic vehicles at a preset distance from the target vehicle, and the second state information set includes the state information of the multiple dynamic vehicles obtained using the autonomous driving learning model corresponding to the current time. Using the first state information set, the second state information set, and the reward value, the autonomous driving learning model is trained for the current timestamp, so as to be used for the autonomous driving learning model for the next timestamp, thereby obtaining the trained autonomous driving learning model. Obtaining the reward value of the target vehicle during the time period between the current timestamp and the next timestamp includes: By using reward and penalty functions, the reward value of the target vehicle within the time period is calculated on the first state information set and the second state information set.
2. The method according to claim 1, characterized in that, The road test scenario segment includes road test data located at multiple timestamps; Obtaining the second state information set of the dynamic environmental object at the current timestamp and the next timestamp includes: From the road test data at the current timestamp, obtain the current status information of each of the multiple dynamic vehicles at the current timestamp; Based on the current state information of each dynamic vehicle at the current timestamp, the next state information of each dynamic vehicle at the next timestamp is obtained using the autonomous driving learning model corresponding to the current timestamp. Based on the road test data at the current timestamp and the next timestamp, obtain the current state information of the remaining dynamic objects in the environment at the current timestamp and the next state information at the next timestamp.
3. The method according to claim 2, characterized in that, The road test scenario segment also includes trajectory data of each of the multiple dynamic vehicles; Based on the current state information of the multiple dynamic vehicles at the current timestamp, and using the autonomous driving learning model corresponding to the current timestamp, the next state information of the multiple dynamic vehicles at the next timestamp is obtained, including: Based on the current state information of each dynamic vehicle at the current timestamp, the first action information of each dynamic vehicle is predicted using the autonomous driving learning model corresponding to the current timestamp. From the trajectory data of each of the dynamic vehicles, the second motion information of each of the dynamic vehicles is obtained; The first action information and the second action information are weighted to obtain the next state information of each dynamic vehicle at the next timestamp.
4. The method according to claim 3, characterized in that, The weighting coefficient of the first action information increases as the current timestamp moves forward among the multiple timestamps, while the weighting coefficient of the second action information decreases as the current timestamp moves forward among the multiple timestamps.
5. The method according to claim 1, characterized in that, The road test scenario segment includes road test data located at multiple timestamps; Obtaining the first state information set of the target vehicle at the current timestamp and the next timestamp includes: From the road test data at the current timestamp, obtain the current status information of the target vehicle at the current timestamp; Based on the current status information of the target vehicle at the current timestamp, the predetermined action information of the target vehicle, and the time period, the next status information of the target vehicle at the next timestamp is obtained.
6. The method according to claim 1, characterized in that, The autonomous driving road test scenarios also include traffic lights; The method further includes: Obtain the third state information set of the traffic light at the current timestamp and the next timestamp; Using the first state information set, the second state information set, and the reward value, the autonomous driving learning model is trained for the current timestamp, and then used for the autonomous driving learning model for the next timestamp, further including: Using the first state information set, the second state information set, the third state information, and the reward value, the autonomous driving learning model is trained for the current timestamp, so that it can be used for the autonomous driving learning model for the next timestamp.
7. An autonomous driving method, characterized in that, Applied to target vehicles, including: Obtain the first state information of the target vehicle at its current timestamp in the autonomous driving road test scenario and the second state information of dynamic objects in the environment of the autonomous driving road test scenario; The first state information and the second state information are input into the autonomous driving learning model to obtain the driving action of the target vehicle, so that the target vehicle drives according to the driving action; The autonomous driving learning model is trained using the training method for the autonomous driving learning model according to any one of claims 1 to 6.
8. An electronic device, characterized in that, It includes a memory and a processor that are coupled to each other, the processor being used to execute program instructions stored in the memory to implement the training method of the autonomous driving learning model according to any one of claims 1 to 6, or to implement the autonomous driving method according to claim 7.
9. A non-volatile computer-readable storage medium storing program instructions thereon, characterized in that, When the program instructions are executed by the processor, they implement the training method of the autonomous driving learning model according to any one of claims 1 to 6, or the autonomous driving method according to claim 7.