An end-to-end automatic driving method and system, storage medium

Abstract

This invention relates to an end-to-end autonomous driving method and system, and a storage medium, comprising: during vehicle driving, receiving a current-time image of the vehicle's front, and performing semantic recognition on the current-time image to obtain a first semantic segmentation map; inputting the current-time image of the vehicle's front and the first semantic segmentation map as model inputs to a pre-trained end-to-end autonomous driving model for processing, to obtain a second semantic segmentation map and vehicle motion control information; wherein, the vehicle motion control information includes steering wheel angle and vehicle speed. This invention addresses the technical problem that traditional end-to-end autonomous driving models cannot understand how the driving strategy interprets the driving environment, and that the input samples for driving strategy learning include a variety of information, making the process quite complex.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to an end-to-end autonomous driving method and system, and a storage medium. Background Technology

[0002] Traditional autonomous driving employs a rule-based modular approach, typically divided into perception, decision-making and planning, and control. Modular solutions rely on the intricate and complex design of each module, but human-designed systems often cannot cover all driving scenarios. For example, the perception module needs a meaningful representation to perfectly characterize the external environment, aiding the decision-making and control modules. In contrast, end-to-end autonomous driving solutions are closer to human nature, representing a pure AI approach. End-to-end autonomous driving solutions possess artificial intelligence characteristics, learning perception while simultaneously implementing control actions through purely data-driven learning.

[0003] In the process of realizing this invention, the inventors discovered that traditional end-to-end autonomous driving models cannot know how driving strategies interpret the driving environment. Furthermore, the input samples for driving strategy learning include a variety of information, which is quite complex. Summary of the Invention

[0004] The purpose of this invention is to propose an end-to-end autonomous driving method and system to solve the technical problems that traditional end-to-end autonomous driving models cannot know how the driving strategy interprets the driving environment, and the input samples for driving strategy learning include a variety of information, which is quite complex.

[0005] To achieve the above objectives, embodiments of the present invention propose an end-to-end autonomous driving method, comprising the following steps:

[0006] Step S100: During vehicle driving, receive the current image of the front of the vehicle and perform semantic recognition on the current image of the front of the vehicle to obtain the first semantic segmentation map of the current moment.

[0007] Step S200: Input the current vehicle front image and the current first semantic segmentation map as model inputs into the pre-trained end-to-end autonomous driving model for processing to obtain the second semantic segmentation map and the vehicle motion control information; wherein, the vehicle motion control information includes steering wheel angle and vehicle speed.

[0008] The end-to-end autonomous driving model includes a probabilistic graphical model and a driving strategy model. The probabilistic graphical model is used to infer potential state variables based on the model input at the current moment, and to decode the potential state variables to obtain a second semantic segmentation map. The driving strategy model is used to make driving decisions based on the potential state variables output by the probabilistic graphical model to obtain vehicle action control information.

[0009] Preferably, the probabilistic graphical model is specifically used to infer potential state variables based on the model input at the current moment, the model input at historical moments, and vehicle actions.

[0010] Preferably, the probabilistic graphical model includes a filtering module, a latent dynamics module, and a generation module. The filtering module is used to filter the model input variables input to the probabilistic graphical model. The latent dynamics module is used to infer latent state variables based on the filtering results of the filtering module. The generation module is used to decode the latent state variables to obtain a second semantic segmentation graph.

[0011] Preferably, the driving strategy model is trained based on the stochastic policy learning algorithm in the maximum entropy reinforcement learning algorithm.

[0012] Preferably, the optimization objective function of the end-to-end autonomous driving model is f = f1 + f2, the optimization objective function of the probabilistic graphical model is f1, and the optimization objective function of the driving strategy model is f2.

[0013] in:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019] This indicates that the generation module decodes the potential state variables to obtain the image in front of the vehicle. This is the second semantic segmentation map. The potential state variables obtained by reasoning for the potential dynamic module. x represents the filtering result of the filtering module. 1:t+1 The image shows the front view of the vehicle from the initial moment of driving to time t+1. 1:t For the vehicle's actions from the initial moment of driving to time t, m 1:t+1 The semantic segmentation map from the initial moment of vehicle driving to time t+1, z 1:t+H Let z be the potential state variable from the initial moment of vehicle driving to time t+H. 1:t+1Let a be the potential state variable from the initial moment of vehicle driving to moment t+1. t+1:t+H This refers to the vehicle's actions from time t+1 to time t+H, where t+1 is the current time, H is greater than 0, and a t+H For the vehicle's motion at time t+H, z t+H Let π(a) be the potential state variable at time t+H. t+H |z t+H Let π(a) be the driving strategy at time t+H. t |z t Let P(z) be the driving strategy at time t. t+1 |z t ,a t Let z be the latent state variable at time t+1 predicted based on the latent state variables at time t and the vehicle's actions. t+1 Let z be the potential state variable at time t+1. t Let a be the potential state variable at time t. t Let t be the vehicle's movement at time t.

[0020] Embodiments of the present invention also propose an end-to-end autonomous driving system, comprising:

[0021] The information receiving unit is used to receive the current image of the front of the vehicle during vehicle driving, and to perform semantic recognition on the current image of the front of the vehicle to obtain the first semantic segmentation map of the current moment.

[0022] An autonomous driving decision unit is used to input the current image of the vehicle in front and the current first semantic segmentation map as model inputs into a pre-trained end-to-end autonomous driving model for processing, to obtain a second semantic segmentation map and the vehicle motion control information; wherein, the vehicle motion control information includes steering wheel angle and vehicle speed.

[0023] The end-to-end autonomous driving model includes a probabilistic graphical model and a driving strategy model. The probabilistic graphical model is used to infer potential state variables based on the model input at the current moment, and to decode the potential state variables to obtain a second semantic segmentation map. The driving strategy model is used to make driving decisions based on the potential state variables output by the probabilistic graphical model to obtain vehicle action control information.

[0024] Preferably, the probabilistic graphical model is specifically used to infer potential state variables based on the model input at the current moment, the model input at historical moments, and vehicle actions.

[0025] Preferably, the probabilistic graphical model includes a filtering module, a latent dynamics module, and a generation module. The filtering module is used to filter the model input variables input to the probabilistic graphical model. The latent dynamics module is used to infer latent state variables based on the filtering results of the filtering module. The generation module is used to decode the latent state variables to obtain a second semantic segmentation graph.

[0026] Preferably, the driving strategy model is trained based on the stochastic policy learning algorithm in the maximum entropy reinforcement learning algorithm.

[0027] Embodiments of the present invention also propose a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the end-to-end autonomous driving method described above.

[0028] The embodiments of the present invention have at least the following beneficial effects:

[0029] The end-to-end autonomous driving model of this embodiment takes as input an image of the vehicle's front and a first semantic segmentation map obtained by semantic recognition based on the image. In other words, it can make decisions and output vehicle action control information based solely on the image of the vehicle's front captured by the front camera. Compared to traditional autonomous driving models that include multi-dimensional information such as the front image, LiDAR image, and vehicle status, this model reduces the complexity of the input and lowers the cost of autonomous driving. Furthermore, the end-to-end autonomous driving model of this embodiment also outputs a second semantic segmentation map, which can be used to explain how the driving strategy of the end-to-end autonomous driving model interprets the vehicle environment.

[0030] Other features and advantages of embodiments of the present invention will be set forth in the following description. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating an end-to-end autonomous driving method according to an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the probabilistic graphical model in an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of the driving decision-making model in an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram of the end-to-end autonomous driving model in an embodiment of the present invention.

[0036] Figure 5 This is a schematic diagram of an end-to-end autonomous driving system according to an embodiment of the present invention. Detailed Implementation

[0037] The various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. Furthermore, numerous specific details are set forth in the following detailed embodiments to better illustrate the invention. Those skilled in the art will understand that the invention can be practiced without certain specific details. In some instances, means well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.

[0038] See Figure 1 An embodiment of the present invention proposes an end-to-end autonomous driving method, comprising the following steps:

[0039] Step S100: During vehicle driving, receive the current image of the front of the vehicle and perform semantic recognition on the current image of the front of the vehicle to obtain the first semantic segmentation map of the current moment.

[0040] Specifically, the image in front of the vehicle refers to the RGB environment image captured by the front camera of the vehicle. The first semantic segmentation image can be identified using a pre-trained image semantic segmentation model. Semantic segmentation is to classify each pixel in the image and determine the category of each point (such as belonging to the background, people or vehicles, etc.) to divide the region. It has been widely used in autonomous driving scenarios. For example, network structures such as VGG and ResNet can be used for image semantic segmentation.

[0041] Step S200: Input the current vehicle front image and the current first semantic segmentation map as model inputs into the pre-trained end-to-end autonomous driving model for processing to obtain the second semantic segmentation map and the vehicle motion control information; wherein, the vehicle motion control information includes steering wheel angle and vehicle speed.

[0042] The end-to-end autonomous driving model includes a probabilistic graphical model (PGM) and a driving strategy model. The PGM is used to infer potential state variables based on the model input at the current moment, and to decode the potential state variables to obtain a second semantic segmentation map.

[0043] Furthermore, the probabilistic graphical model is specifically used to infer potential state variables based on the model input at the current moment, the model input at historical moments, and vehicle actions.

[0044] It should be understood that the latent state variables include future latent state variables. The probabilistic graphical model described in this embodiment is based on historical state sequences (including the vehicle's frontal image up to the current time and the vehicle's actions up to the previous time) to infer all latent state variables up to the current time, and then predicts future latent state variables. The latent state at each time step can be inferred from the latent state at the previous time step and the vehicle's actions. The principle can be found in [reference needed]. Figure 2 This can be understood as summarizing historical conditions to analyze environmental changes during vehicle operation and potential future environmental changes during vehicle operation, using this as a basis for driving decisions, thereby improving the reliability and safety of driving decisions.

[0045] Specifically, the potential state at each moment can be understood as the probabilistic graphical model (PGM)'s understanding and interpretation of the vehicle's driving environment. That is, after inputting the model input, how does the probabilistic graphical model understand the information of the model input when making driving decisions based on the model input? In order to increase the interpretability of the autonomous driving system's decisions, this embodiment proposes to decode the potential state variables to the camera end to obtain the second semantic segmentation map, showing how the system understands the semantics of the environment.

[0046] Furthermore, the probabilistic graphical model includes a filtering module, a latent dynamics module, and a generation module. The filtering module is used to filter the model input variables input to the probabilistic graphical model. The latent dynamics module is used to infer latent state variables based on the filtering results of the filtering module. The generation module is used to decode the latent state variables to obtain a second semantic segmentation graph.

[0047] Specifically, in this embodiment, the generation module comprises two parts, used to decode the RGB environment image and the second semantic segmentation map based on the latent state variables, respectively. Both contain 5 deconvolutional network layers, with (filters, kernel size, strides) of (512, 3, 1), (256, 3, 2), (128, 3, 2), (64, 3, 2), and (3, 3, 1), respectively. The latent dynamic module consists of two fully connected layers of size 512, connected to a Gaussian output layer. The filtering module comprises two parts, each containing 5 convolutional network layers, with (filters, kernel size, strides) of (64, 3, 2), (128, 3, 2), (256, 3, 2), (512, 3, 2), and (512, 3, 1), respectively. It encodes the model input into a feature size of 512, then connects two fully connected layers of size 512, and finally connects to a Gaussian output layer.

[0048] The driving strategy model is used to make driving decisions based on the potential state variables output by the probabilistic graphical model to obtain vehicle action control information; the driving strategy model is trained based on the stochastic policy learning algorithm in the maximum entropy reinforcement learning algorithm.

[0049] Specifically, in this implementation, the driving strategy model includes a strategy evaluation part and a strategy optimization part; the strategy evaluation part consists of two fully connected layers of size 512, and is finally connected to a linear output layer; the strategy optimization part consists of two fully connected layers of size 512, and is finally connected to a Gaussian output layer and a two-dimensional vector Tanh function activation.

[0050] like Figure 3 As shown, z t Indicates state; a t Indicates an action; O t It is a binary random variable, representing whether the action taken at time t is optimal. Its conditional probability can be defined as:

[0051]

[0052] Where r(z) t, a t Let P(O) be the reward function. In policy learning through reinforcement learning, a higher reward indicates higher optimality. Therefore, to optimize policy behavior, the entire trajectory P(O) should be maximized. 1:H The optimal probability of ) is derived through a series of formulas, namely:

[0053]

[0054] Where t=1:H represents the time period from the start of vehicle driving to a future time H, π(a t |z t ) represents the driving strategy function, which maximizes the leftmost side by maximizing the right side of the inequality, and the rightmost inequality is exactly consistent with the MaxEnt RL objective function;

[0055] The objective function of the standard reinforcement learning (RL) algorithm is to find the policy that collects the maximum cumulative reward, expressed as:

[0056]

[0057] In MaxEnt RL, entropy maximization is introduced, with the objective policy being:

[0058]

[0059] in, is the variable π(a t |z t The entropy of ρ; π This represents the distribution of state-action pairs that the intelligent agent will encounter under policy π control. α is a hyperparameter used to adjust the degree of emphasis on entropy.

[0060] Ultimately, the likelihood of the optimal variable in the PGM can be maximized by using MaxEnt RL, and the reinforcement learning problem can be defined as the PGM learning problem.

[0061] Furthermore, the optimization objective function of the end-to-end autonomous driving model is f = f1 + f2, the optimization objective function of the probabilistic graphical model is f1, and the optimization objective function of the driving strategy model is f2.

[0062] in:

[0063]

[0064]

[0065]

[0066]

[0067]

[0068] This indicates that the generation module decodes the potential state variables to obtain the image in front of the vehicle. This is the second semantic segmentation map. The potential state variables obtained by reasoning for the potential dynamic module. x represents the filtering result of the filtering module. 1:t+1 The image shows the front view of the vehicle from the initial moment of driving to time t+1. 1:t For the vehicle's actions from the initial moment of driving to time t, m 1:t+1 The semantic segmentation map from the initial moment of vehicle driving to time t+1, z 1:t+H Let z be the potential state variable from the initial moment of vehicle driving to time t+H. 1:t+1 Let a be the potential state variable from the initial moment of vehicle driving to moment t+1. t+1:t+H This refers to the vehicle's actions from time t+1 to time t+H, where t+1 is the current time, H is greater than 0, and a t+H For the vehicle's motion at time t+H, z t+HLet π(a) be the potential state variable at time t+H. t+H |z t+H Let π(a) be the driving strategy at time t+H. t |z t Let P(z) be the driving strategy at time t. t+1 |z t ,a t Let z be the latent state variable at time t+1 predicted based on the latent state variables at time t and the vehicle's actions. t+1 Let z be the potential state variable at time t+1. t Let a be the potential state variable at time t. t Let t be the vehicle's movement at time t.

[0069] Furthermore, the reward function r in this embodiment is as follows:

[0070] r = w1r collision +v lon +w2r fast +r out -w3α 2 +w4r lat -0.1

[0071] Where, r collision This is a collision-related reward; it is set to -1 if a collision occurs and 0 otherwise. lon It is the longitudinal speed of the vehicle; r fast This is a reward related to driving too fast; if the desired target speed is exceeded, it is set to -1, otherwise 0. out If the vehicle runs out of its lane, set it to -1; otherwise, set it to 0. α is the vehicle's steering wheel angle, expressed in radians; r lat It's lateral acceleration. The last constant term 0.1 is to prevent the vehicle from standing still; the constant terms w1, w2, w3, w4 in front of the variables need to be manually adjusted according to different autonomous driving scenarios, different data, and different models.

[0072] like Figure 4 As shown, Figure 2 and Figure 3 The probabilistic graphical model (for environment model learning) and the driving strategy model (for policy learning) shown are combined to obtain the end-to-end autonomous driving model in this embodiment of the invention. In the probabilistic graphical model, each node z... t All include sensor input x t and semantic segmentation graph m tThis allows us to understand how the model processes the input by examining the semantic segmentation graph of each node, even in end-to-end models, thus avoiding completely black-box end-to-end models and making them interpretable.

[0073] The method in this embodiment is based on a maximum entropy reinforcement learning algorithm and employs a stochastic strategy for learning. By introducing maximum entropy, it has the following advantages:

[0074] 1) The learned strategy can be used as an initialization for more complex and specific tasks; through maximum entropy, the strategy learns not just one way to solve a task, but all of them; therefore, such a strategy is more conducive to learning new tasks; for example, initially learning to walk, and then learning to walk in a specific direction.

[0075] 2) It has stronger exploration capabilities and can more easily find better patterns under multimodal rewards; for example, it requires the robot to walk well and save energy at the same time.

[0076] 3) More robust and stronger generalization; it explores various optimal possibilities from different angles and can make adjustments more easily when faced with interference; interference is a state seen during the neural network learning process. Since it has been explored and learned, it can react better and continue to obtain high rewards.

[0077] Based on the above embodiments, the method of this invention proposes to combine environment learning (probabilistic graphical model) and policy learning (driving decision model) to construct an end-to-end autonomous driving model. The input to environment learning is historical sensor data and vehicle actions. Through latent sequence environment modeling using the PGM method, future potential state variables are inferred. These potential state variables are used to describe the current situation, summarizing historical information such as location, speed, the intentions of other road users, drivable areas, and road signs, which serve as the input to maximum entropy reinforcement learning, i.e., policy learning. Simultaneously, the potential states generate the original camera input and semantic segmentation (decoding), providing interpretability for the end-to-end autonomous driving model. The input to policy learning is the potential state variables. Through maximum entropy reinforcement learning, stochastic policy learning is adopted, which is divided into two parts: policy evaluation and policy optimization. Finally, through self-exploration and reinforcement, the system continuously interacts with the environment to generate the optimal policy and execute the optimal action.

[0078] See Figure 5 Embodiments of the present invention also propose an end-to-end autonomous driving system, comprising:

[0079] Information receiving unit 1 is used to receive the image in front of the vehicle at the current moment during vehicle driving, and to perform semantic recognition on the image in front of the vehicle at the current moment to obtain the first semantic segmentation map at the current moment.

[0080] The autonomous driving decision unit 2 is used to input the current image of the vehicle in front and the current first semantic segmentation map as model inputs into a pre-trained end-to-end autonomous driving model for processing, to obtain a second semantic segmentation map and the vehicle motion control information; wherein, the vehicle motion control information includes steering wheel angle and vehicle speed.

[0081] The end-to-end autonomous driving model includes a probabilistic graphical model and a driving strategy model. The probabilistic graphical model is used to infer potential state variables based on the model input at the current moment, and to decode the potential state variables to obtain a second semantic segmentation map. The driving strategy model is used to make driving decisions based on the potential state variables output by the probabilistic graphical model to obtain vehicle action control information.

[0082] Furthermore, the probabilistic graphical model is specifically used to infer potential state variables based on the model input at the current moment, the model input at historical moments, and vehicle actions.

[0083] Furthermore, the probabilistic graphical model includes a filtering module, a latent dynamics module, and a generation module. The filtering module is used to filter the model input variables input to the probabilistic graphical model. The latent dynamics module is used to infer latent state variables based on the filtering results of the filtering module. The generation module is used to decode the latent state variables to obtain a second semantic segmentation graph.

[0084] Furthermore, the driving strategy model is trained based on the stochastic policy learning algorithm in the maximum entropy reinforcement learning algorithm.

[0085] The systems described in the embodiments above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the system solution in the embodiments, depending on actual needs.

[0086] It should be noted that the system in the above embodiments corresponds to the method in the above embodiments. Therefore, the parts of the system in the above embodiments that are not described in detail can be obtained by referring to the content of the method in the above embodiments. That is, the specific steps recorded in the method in the above embodiments can be understood as the functions that the system in the above embodiments can achieve, and will not be described again here.

[0087] Furthermore, if the end-to-end autonomous driving system described above is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0088] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the end-to-end autonomous driving method described in the above embodiments.

[0089] Specifically, the computer-readable storage medium may include any entity or device capable of carrying the computer program instructions, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0090] The embodiments of the present invention have the following advantages:

[0091] The end-to-end autonomous driving model of this embodiment takes as input an image of the vehicle's front and a first semantic segmentation map obtained by semantic recognition based on the image. In other words, it can make decisions and output vehicle action control information based solely on the image of the vehicle's front captured by the front camera. Compared to traditional autonomous driving models that include multi-dimensional information such as the front image, LiDAR image, and vehicle status, this model reduces the complexity of the input and lowers the cost of autonomous driving. Furthermore, the end-to-end autonomous driving model of this embodiment also outputs a second semantic segmentation map, which can be used to explain how the driving strategy of the end-to-end autonomous driving model interprets the vehicle environment.

[0092] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An end-to-end autonomous driving method, characterized in that, Includes the following steps: Step S100: During vehicle driving, receive the current image of the front of the vehicle and perform semantic recognition on the current image of the front of the vehicle to obtain the first semantic segmentation map of the current moment. Step S200: Input the current vehicle front image and the current first semantic segmentation map as model inputs into the pre-trained end-to-end autonomous driving model for processing to obtain the second semantic segmentation map and the vehicle motion control information; wherein, the vehicle motion control information includes steering wheel angle and vehicle speed. The end-to-end autonomous driving model includes a probabilistic graphical model and a driving strategy model. The probabilistic graphical model is used to infer potential state variables based on the model input at the current moment, and to decode the potential state variables to obtain a second semantic segmentation map. The driving strategy model is used to make driving decisions based on the potential state variables output by the probabilistic graphical model to obtain vehicle action control information. Specifically, the probabilistic graphical model is used to infer potential state variables based on the model input at the current moment, the model input at historical moments, and vehicle actions. The probabilistic graphical model includes a filtering module, a latent dynamics module, and a generation module. The filtering module is used to filter the model input variables input to the probabilistic graphical model. The latent dynamics module is used to infer latent state variables based on the filtering results of the filtering module. The generation module is used to decode the latent state variables to obtain a second semantic segmentation graph.

2. The method according to claim 1, characterized in that, The driving strategy model is trained based on the stochastic policy learning algorithm in the maximum entropy reinforcement learning algorithm.

3. The method according to claim 2, characterized in that, The optimization objective function of the end-to-end autonomous driving model is: The objective function of the probabilistic graphical model is: The objective function of the driving strategy model is: ; in: This indicates that the generation module decodes the potential state variables to obtain the image in front of the vehicle. This is the second semantic segmentation map. The potential state variables obtained by reasoning for the potential dynamic module. The result of the filtering process by the filtering module. This is an image of the front of the vehicle from the initial moment of driving to time t+1. The vehicle's actions from the initial moment of driving to time t. This is a semantic segmentation map from the initial moment of vehicle driving to time t+1. These are the potential state variables from the initial moment of vehicle driving to time t+H. These are the potential state variables from the initial moment of vehicle driving to time t+1. This refers to the vehicle's actions from time t+1 to time t+H, where t+1 is the current time and H is greater than 0. Let t+H be the vehicle's movement. Let H be the potential state variables at time t+H. The driving strategy at time t+H. Let t be the driving strategy. The latent state variables at time t+1 are predicted based on the latent state variables at time t and the vehicle's actions. Let be the potential state variables at time t+1. Let be the potential state variable at time t. Let t be the vehicle's movement at time t.

4. An end-to-end autonomous driving system, characterized in that, include: The information receiving unit is used to receive the current image of the front of the vehicle during vehicle driving, and to perform semantic recognition on the current image of the front of the vehicle to obtain the first semantic segmentation map of the current moment. An autonomous driving decision unit is used to input the current image of the vehicle in front and the current first semantic segmentation map as model inputs into a pre-trained end-to-end autonomous driving model for processing, to obtain a second semantic segmentation map and the vehicle motion control information; wherein, the vehicle motion control information includes steering wheel angle and vehicle speed. The end-to-end autonomous driving model includes a probabilistic graphical model and a driving strategy model. The probabilistic graphical model is used to infer potential state variables based on the model input at the current moment, and to decode the potential state variables to obtain a second semantic segmentation map. The driving strategy model is used to make driving decisions based on the potential state variables output by the probabilistic graphical model to obtain vehicle action control information. Specifically, the probabilistic graphical model is used to infer potential state variables based on the model input at the current moment, the model input at historical moments, and vehicle actions. The probabilistic graphical model includes a filtering module, a latent dynamics module, and a generation module. The filtering module is used to filter the model input variables input to the probabilistic graphical model. The latent dynamics module is used to infer latent state variables based on the filtering results of the filtering module. The generation module is used to decode the latent state variables to obtain a second semantic segmentation graph.

5. The system according to claim 4, characterized in that, The driving strategy model is trained based on the stochastic policy learning algorithm in the maximum entropy reinforcement learning algorithm.

6. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the end-to-end autonomous driving method according to any one of claims 1-3.

Images