A driving control method based on self-supervised imitation learning

Abstract

The application relates to a driving control method based on self-supervised imitation learning, and belongs to the technical field of automatic driving reinforcement learning. The method comprises the following steps: generating an automatic driving task data set; determining a t moment latent representation of an anchor node in an observation image and a latent representation of a sample related to the anchor node; reserving effective information between a t moment observation image s ≤t and the c t by using a density ratio machine; generating the t moment context latent representation c t by performing integration; reserving effective information between a t+k moment observation image s t+k and the c t by using a density ratio machine; calculating a cost function after the different automatic driving scene observation images pass through a trained encoder; and updating the policy network by inputting the updated cost function into an imitation learning target formula. The application can relieve the dimension disaster faced in high-dimensional data learning strategy by dimension reduction mapping, extract high-density information with stronger task correlation in image observation data, and improve the sampling efficiency of model samples.

Description

Technical Field

[0001] This invention relates to a driving control method based on self-supervised imitation learning, belonging to the field of reinforcement learning technology for autonomous driving. Background Technology

[0002] As a crucial component of artificial intelligence, reinforcement learning (RL) simulates the general thinking process humans use to handle sequential decision-making problems in real life. It employs a learning method where an agent continuously interacts with its environment, receiving feedback and then performing a series of actions to maximize overall gains. With the development of deep learning, reinforcement learning has been widely applied to many academic problems and practical business decision-making tasks, achieving better results than other algorithms such as supervised learning. Reinforcement learning aims to design a computational paradigm that enables agents to possess learning processes and abilities similar to humans. It doesn't simply fit a mapping of training data to function inputs and outputs; instead, it establishes a method to help the agent achieve its goals by executing actions. Reinforcement learning is a goal-oriented algorithm that learns how to accomplish a complex goal or how to maximize a specific metric. The agent perceives its environment, processes data through reinforcement learning algorithms, and then takes actions to attempt to achieve a predefined goal, similar to our everyday behavioral patterns.

[0003] Imitation learning, an important branch of reinforcement learning, also known as demonstration learning, uses relatively simple methods to imitate demonstrated behaviors, extract useful knowledge, reproduce those behaviors in similar environments, and ultimately enable the agent to make decisions like an "expert." Imitation learning demonstrates good policy training efficiency in real-world scenarios, aiding in the research of autonomous control systems and the design of artificial intelligence agents. In fact, in the late 1980s, Professor Pomerleau of Carnegie Mellon University successfully trained a shallow neural network to control a driverless car across North America using examples of human experts. AlphaGo, the first robot to defeat top human players, also used imitation learning methods to learn from human chess games.

[0004] One important application of imitation learning is in autonomous driving. In autonomous driving, the agent's observations contain rich environmental information (such as weather conditions and roadside building information), making it difficult for the agent to directly correlate the complex information within the observed images with decision-making and control. Therefore, a crucial technique for reinforcement learning agents to learn how to perform complex control tasks from observations requires the use of deep neural networks to extract key information. One intuitive and basic method is to use convolutional neural networks to learn from raw pixels. However, empirical observation shows that using traditional deep neural networks for reinforcement learning from high-dimensional observations has low sample efficiency. If state information exists in the pixel data, then in principle, relevant information can be extracted by learning the representation of the state. Therefore, reinforcement learning methods that combine representation learning are particularly important.

[0005] Existing imitation learning methods can be divided into two categories. One category consists of traditional imitation learning methods containing action information, such as behavior cloning and GAIL, where the expert trajectory consists of multiple state-action pairs, typically in the form of physical states. The main idea of ​​this type of method is to minimize the action difference between the generative policy and the expert policy. However, this method lacks a module for handling high-dimensional inputs. To use the GAIL method in a high-dimensional environment, significant modifications to its discriminator structure are required to process the raw visual data. Therefore, this type of method cannot learn in pixel-based high-dimensional environments. The other category consists of imitation learning methods without action information, such as behavior cloning from observations and adversarial generative imitation learning from observations. In this type, the expert trajectory contains only observation information and no action information. This type of method mainly uses convolutional neural networks or generative models to handle high-dimensional inputs, which leads to low sample efficiency in obtaining state representations. Summary of the Invention

[0006] The purpose of this invention is to provide a driving control method based on self-supervised imitation learning to alleviate the curse of dimensionality faced in high-dimensional data learning strategies, and to extract contextual information that is highly relevant to the task from image observation data, thereby improving the sampling efficiency of model samples.

[0007] This invention provides a driving control method based on self-supervised imitation learning, comprising the following steps: Generate an expert policy dataset for autonomous driving tasks, the dataset including observation images of agent expert policies; The image observed at time t at the anchor node in the dataset The nonlinear encoder, after data augmentation, is input to the strided convolutional layer. The resulting latent representation of the anchor node ; The anchor node-related samples in the dataset are... Observational images at time Image data obtained by using different random blocks and a data augmentation method of the same type as the anchor node is input into a nonlinear encoder of a strided convolutional layer. In this process, the latent representation of the relevant samples is obtained. ; The latent representations in the latent space before time t Integrate to produce the aforementioned Latent representation of time context , using the Extract the Relevant contextual information, among which, It is an autoregressive model; Use a density ratio meter to retain the observed image at time t+k and stated The effective information between them, using the above inverse vector The encoder is trained using a contrastive learning-based method to learn the mapping from high-dimensional observations to low-dimensional features in an unsupervised manner. The cost function is calculated by passing the observed images of different autonomous driving scenarios through a trained encoder. The cost function is then updated by an expert policy, a policy network, and a policy regularization term, and then fed into the imitation learning objective formula to update the policy network and learn the control policy.

[0008] The present invention also provides a driving control system based on imitation learning, wherein the driving control system employs the aforementioned method and includes the following modules: The environment modeling module is based on the open-source autonomous driving simulator Carla platform for driving control application and verification, with the simulator of the platform serving as the main simulation environment. The action modeling module is used to model the decision-making actions of the intelligent agent; The imitation learning module takes driving data collected from the agent's interactions with the environment, gathered by expert policies, and inputs the driving observation data into a nonlinear encoder. The low-dimensional vectors of the latent representations of the observed anchor nodes and their associated samples are obtained, and the observation image at time t+k is preserved using a density ratio analyzer. and stated The effective information between them is used to calculate the cost function after the observation images of different autonomous driving scenarios are passed through the trained encoder. The cost function is updated by expert policy, policy network and policy regularization term and then passed into the imitation learning objective formula to update the policy network. During the test, the agent is put into the autonomous driving simulator Carla platform for policy evaluation. The agent's observation image is input into the updated policy network and the decision action obtained by the agent is obtained as the output of the agent. The environment response module is used by the agent to select decision actions based on the driving scene observations input at the current moment, using the imitation learning module and the action modeling module, and output response information based on the decision actions. The effectiveness of the agent is determined based on the magnitude of the final reward function value.

[0009] The present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0010] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the aforementioned method.

[0011] Beneficial effects: This invention maps high-dimensional image observation data into low-dimensional vector features using a self-supervised contrastive learning module. First, the image data is preprocessed using data augmentation to indirectly enhance the robustness of the obtained state representation. Second, a context encoding representation is introduced to extract the contextual information of the observation, and the density ratio is incorporated into the contrastive learning loss function for training the state representation encoder. The method for obtaining state representations employed in this invention effectively alleviates the curse of dimensionality in high-dimensional data learning strategies through dimensionality reduction mapping—that is, the increase in dimensionality leads to excessively sparse sample data in high-dimensional space, resulting in degraded performance. Furthermore, it extracts contextual information from image observation data that is more relevant to the task, thereby improving the sampling efficiency of the model samples. Attached Figure Description

[0012] Figure 1 This is an overall framework diagram of the driving control method model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the driving control method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of encoder self-supervised training according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the cost function update in an embodiment of the present invention; Figure 5 This is an example diagram of a simulation environment for an autonomous driving control task according to an embodiment of the present invention. Detailed Implementation

[0013] To introduce this invention, we will first introduce the variables that need to be used and their mathematical representations: In reinforcement learning, the agent operates within a Markov decision process framework M, represented by a 6-tuple M = {S, A, Z, T, c, γ}. Here, S and A represent the state space and action space of the framework, respectively, and s and a are elements in S and A. Z represents the latent space after the state space is mapped by the encoder, and z is an element in Z. In this embodiment, three identical sensors are deployed in the agent-controlled vehicle. Each sensor has a 60-degree horizontal field of view, capable of observing image data from the front, left front, and right front, respectively. Therefore, the agent can receive three images as its state input. The agent uses a PID controller (proportional-integral-derivative control) to implement driving decisions, primarily including three control operations: steering, throttle, and braking. This represents the state transition function, indicating the agent's state transition. Take action After transitioning to state In the context of autonomous driving, the probability of transitions represents the distribution of the agent's actions after observing the current sensor output and implementing a driving action via a PID controller. c is the cost function, representing the immediate feedback the agent can obtain by taking an action in a given state. γ is the discount factor, indicating the influence of future rewards on the current situation. The agent's behavior can be described by a policy function: S→A represents the action an agent should perform in a specific state, i.e., the distribution of actions. (Used...) This represents the expectation of all trajectories generated by a strategy.

[0014] The objective of this invention is achieved through the following technical solution: Generate an expert policy dataset for autonomous driving tasks, the dataset including observation images of agent expert policies; The image observed at time t at the anchor node in the dataset The nonlinear encoder, after data augmentation, is input to the strided convolutional layer. The resulting latent representation of the anchor node ; The anchor node-related samples in the dataset are... Observational images at time Image data obtained by using different random blocks and a data augmentation method of the same type as the anchor node is input into a nonlinear encoder of a strided convolutional layer. In this process, the latent representation of the relevant samples is obtained. ; The latent representations in the latent space before time t Integrate to produce the aforementioned Latent representation of time context , using the Extract the Relevant contextual information, among which, It is an autoregressive model; Use a density ratio meter to retain the observed image at time t+k and stated The effective information between them, using the above inverse vector The encoder is trained using a contrastive learning-based method to learn the mapping from high-dimensional observations to low-dimensional features in an unsupervised manner. The cost function is calculated by passing the observed images of different autonomous driving scenarios through a trained encoder. The cost function is then updated by an expert policy, a policy network, and a policy regularization term, and then fed into the imitation learning objective formula to update the policy network and learn the control policy.

[0015] The core idea of ​​this invention is to accelerate the learning progress of model-free reinforcement learning methods by using an auxiliary self-supervised task. Based on a self-supervised representation learning method, it learns low-dimensional representations of image information during driving from observations of a high-dimensional driving environment. Its goal is to capture potential sequence relationships within expert trajectories while learning expert demonstration representations, thus facilitating downstream driving tasks. First, a contrastive predictive coding representation learning method is employed to learn an encoder capable of extracting high-dimensional observation features using existing expert trajectories. Then, a cost function as close to reality as possible is obtained through the extracted representations. Finally, a high-quality policy network is iteratively obtained using the learned cost function to learn a control policy, resulting in a highly robust driving control strategy.

[0016] Figure 1 The diagram shows the overall framework of the driving control method model. Based on self-supervised learning, the imitation learning method first samples a batch of trajectories from expert trajectories, then encodes them using an encoder. The encoded representation is passed to a contrastive learning module for gradient updates, training an encoder capable of extracting high-dimensional features. Finally, the low-dimensional representation is passed to a policy network to learn the control policy. Figure 2 This is a schematic diagram of the driving control method according to an embodiment of the present invention.

[0017] refer to Figures 1-5 A preferred embodiment of the present invention includes the following steps: Step 1: Ensuring that the driving route is covered for most of the time, a well-defined autonomous driving controller can guarantee vehicle stability and robustness in various complex situations. A manually coded controller ensures continuous driving while avoiding collisions with other vehicles. This manually coded controller consists of a hand-designed controller that simulates human vehicle operation through hard coding, and is defined as an expert policy.

[0018] Step 2: Collect trajectories for autonomous driving scenarios using the expert strategy to generate a complex autonomous driving task dataset. The expert strategy uses the observed images in the current scene as input to determine the control actions to be taken, and marks the collected trajectories as expert trajectories, adding them to the expert dataset. In order to enable the agent to be trained; the expert dataset The dataset includes a sequence of observation images generated by the expert strategy, wherein the observation images are RGB images of size (48,48,3), and the total dataset consists of 100,000 observation data. The size, type, and number of the observation images can be adjusted according to the specific scenario and are not intended to limit the implementation of this invention.

[0019] Step 3: Determine the latent representations of the self-supervised learning anchor nodes (also known as anchor points) in the observed images for autonomous driving, as well as the latent representations of samples related to the anchor nodes. In this method, the autonomous driving observed image data at time t of the anchor node is first sampled. The images observed at time t of the anchor node in the dataset The nonlinear encoder, after data augmentation, is input to the strided convolutional layer. The resulting latent representation of the anchor node The dataset contains the anchor node-related samples, i.e., the observed images at time k after time t. Image data obtained by using different random blocks and a data augmentation method of the same type as the anchor node is input into a nonlinear encoder of a strided convolutional layer. In this process, the latent representation of the relevant samples is obtained. Regarding data augmentation, there are generally several recognized solutions, such as: flipping, which involves flipping the entire image horizontally and vertically, or a combination of both; rotation, which involves rotating the entire image by an angle; cropping, which involves cutting out a portion of the image; and shifting, which involves translating the image horizontally or vertically. Considering that redundant information in images is irrelevant to policy control in reinforcement learning, the data augmentation method involved in the samples of this invention is random cropping. That is, a rectangular sub-image of a fixed size is randomly cropped from the original image observation as the augmented image data. It should be noted that although both data augmentation methods involved in the anchor node and the anchor node-related samples are random cropping, they differ due to the different positions of the random cropping blocks.

[0020] Step 4, as follows Figure 3 As shown, in an autonomous driving expert trajectory Among them represent Observations obtained from the interaction between the agent and the environment at any given moment Since the observations are image data, which contains decision-related information as well as decision-independent environmental information, this invention considers using a nonlinear encoder with strided convolutional layers containing ResNet blocks. The Data augmented image mapped to latent representation Considering that temporal information from images often has a significant impact on decision-making, this model uses an autoregressive model based on the PixelCNN module. All representations in the latent space that are less than time t Integrate to generate contextual latent representations This allows for the extraction of useful information from driving images. The autoregressive model considers the entire trajectory to follow a certain data distribution, thus modeling it as follows: The autoregressive model is trained using maximum likelihood. The PixelCNN module consists of a 7x7 masked convolutional layer, followed by multiple residual modules consisting of 1x1, 3x3 masked convolutional layers, and 1x1 convolutional layers, then two more 1x1 masked convolutional layers, and finally a prediction layer consisting of a 1x1 convolutional layer combined with a softmax activation layer.

[0021] Step 5: Existing technical solutions often neglect the relevant information between the current observation's context and the state features obtained from future short-term observations when modeling the relationship between the current observation and contextual information. This invention obtains the latent representation of the context through Step 4. Then, we considered using a density ratio analyzer to preserve the observed image at time t+k. and stated The effective information between them is used to learn the encoder network. The density ratio is modeled as follows:

[0022] The symbol “∝” indicates a direct proportion. For the The following observations The probability distribution, To observe the The prior probability distribution, These are the encoder weights that need to be learned, each Step using different To represent the prediction, For the The inverse vector of .

[0023] Step Six: To fully extract observation information and map the high-dimensional observation space to a more semantically informative latent representation space, representation learning methods are generally used for feature extraction. Common representation learning methods include clustering-based methods, which aggregate similar high-dimensional observations using clustering algorithms and separate dissimilar high-dimensional observations; reconstruction-based methods, which map high-dimensional images to low-dimensional representations using structures such as autoencoders, and then reconstruct high-dimensional images from low-dimensional representations; and contrastive learning-based methods, which input images into loss functions such as InfoNCE through data augmentation to learn the mapping from high-dimensional observations to low-dimensional features unsupervised. Considering the high complexity of clustering-based methods and the often poor performance of reconstruction-based methods, a preferred embodiment of this invention considers using a contrastive learning method based on the InfoNCE loss function for training. Specifically, given time t, A batch consists of N random samples. For this batch of samples, the loss function is defined as follows:

[0024] in, Indicates the expected value. The numerator part within the function is the latent representation of the context. The observed image at time t+k The corresponding density ratio; The denominator within the function is as described Compared with all samples in the current batch The sum of the corresponding density ratios.

[0025] Combined with loss function Gradient descent optimization of the encoder:

[0026] in, The learning rate is used for gradient updates.

[0027] Step 7: In the above steps, the semantic information between temporal observation data and contextual latent representations in high-dimensional environments has been considered. Considering that the temporal relationships between observations in different autonomous driving scenarios also contain rich semantic information when observing in high-dimensional environments, this semantic information can be extracted by modeling the latent representation between the current observation and the next observation. To address this issue, existing technologies often consider modeling by adding a weight matrix or directly integrating the two latent representations through addition or concatenation. Unlike existing solutions, in this embodiment of the invention, a cost function c is defined as: Z×Z→ R, and the update formula for the cost function c is:

[0028] in, To collect expert strategies from the trajectory, The entire feasible space for policy networks learned by the agent. This is the policy network learned by the current agent. Its input is the encoded representation of the current state, and its output is the action taken by the current agent. This is a policy regularization term used to provide additional constraints on the cost function, thus aiding in its training. The cost function is parameterized as a neural network. Its input consists of two different observed image data. and The result obtained after the encoder is output as the cost function value. Indicates a policy-based network Construct trajectory data and sample two different observation images from the trajectory data. and The expectation of the cost function is calculated after encoding the two observed images by the encoder. Indicates based on expert strategy Construct trajectory data and sample two different observation images from the trajectory data. and The expectation of the cost function is calculated after encoding the two observed images by the encoder.

[0029] In a preferred embodiment of the present invention, the for:

[0030] in, Indicating expert strategy The trajectory data is upsampled from two different observation images. and After encoding the two observed images using an encoder, the cost function is calculated, and then processed by the formula... The result obtained.

[0031] Based on the above steps, the formula for the imitation learning objective is defined as follows: , in, For policy networks The frequency of access to the following states For expert strategies The frequency of access to the status under the following conditions. Indicates a policy-based network Construct trajectory data and sample two different observation images from the trajectory data. and After being encoded by the encoder, the value is input into the cost function c, and its logarithmic expectation is calculated. Indicates based on expert strategy Construct trajectory data and sample two different observation images from the trajectory data. and After being encoded by the encoder, the data is input into the cost function c, and the expected logarithm of the difference between the value and 1 is calculated. The collected expert dataset is... The expert trajectories used in the update process are from the dataset. The result of sampling, i.e. ,in The expert trajectory is represented, the target formula is calculated and backpropagated, and the policy network is updated.

[0032] like Figure 4 This is a schematic diagram illustrating the cost function update in an embodiment of the present invention. The expert policy and policy network first generate a set of data samples. and Then, the cost function of the data sample set is updated using the cost function update formula, and the updated cost function result is fed into the imitation learning objective formula to update the policy network. The output of the policy network is the final policy obtained by the agent.

[0033] The training process of the agent policy network is as follows: An autonomous driving agent virtual controller training architecture is constructed; this architecture includes an expert policy dataset, an agent policy network, an encoder network, and a data processing module. The expert policy dataset stores driving observation images sampled by the expert policy operation. The data processing module preprocesses the driving observation images, specifically by performing data augmentation; in this embodiment, random cropping is preferred for data augmentation. The encoder network maps high-dimensional image observations to low-dimensional feature representations and subsequently inputs these low-dimensional feature representations into the agent policy network. The agent policy network receives state feature representation information and provides specific actions, iteratively training the policy network parameters together with the encoder network using the expert policy dataset to obtain a highly robust final policy.

[0034] In a preferred embodiment of the present invention, a driving control system based on imitation learning is constructed using the above method, comprising the following components: The environment modeling module is based on the open-source autonomous driving simulator Carla platform for driving control application and verification. The simulator on this platform forms the core of the simulation environment. Figure 5In the real-world driving environment, entities correspond to entities in the simulator, including vehicles, pedestrians, roads and bridges, traffic lights, signs, weather, buildings, etc. The simulator aims to replicate the realistic physical world. The engine provides various sensors to be mounted at different locations on the vehicle for data collection and agent decision-making training. The vehicle's kinematic model is determined by using the dynamic equations of vehicle movement and collisions to model the vehicle's operation. This simulation environment considers the test scenario as a highway where weather conditions may change; therefore, the coefficient of friction between the vehicle and the ground in rainy weather is also taken into account.

[0035] The action modeling module is used to model the decision-making actions of the intelligent agent. We use a PID controller to implement the decision-making actions, which mainly include three control operations: steering, throttle, and braking. The imitation learning module takes driving data collected from the agent's interactions with the environment, gathered by expert policies, and inputs the driving observation data into a nonlinear encoder. The low-dimensional vectors of the latent representations of the observed anchor nodes and their associated samples are obtained, and the observation image at time t+k is preserved using a density ratio analyzer. and stated The effective information between them is used to calculate the cost function after the observation images of different autonomous driving scenarios are passed through the trained encoder. The cost function is updated by expert policy, policy network and policy regularization term and then passed into the imitation learning objective formula to update the policy network. During the test, the agent is put into the autonomous driving simulator Carla platform for policy evaluation. The agent's observation image is input into the updated policy network and the decision action obtained by the agent is obtained as the output of the agent. The environment response module involves the agent inputting the updated policy network obtained by the imitation learning module based on the current driving scene observations. The agent then selects a decision action using the imitation learning module and the action modeling module. Based on the decision action, the agent outputs response information, which includes the reward value for the next frame's environment observation and the current decision action. Considering a highway driving scenario, the reward value is defined as a formula related to speed and collision severity, namely:

[0036] in, To control the vehicle's velocity vector, a unit vector is mapped to the highway. Multiply by the discretized time. To assess the speed of an agent on a highway, the impact index is used, expressed in Newtonian terms. The severity of the collision is measured in seconds. Finally, the formula adds a penalty for vehicle steering steer to ensure the vehicle avoids steering as much as possible. In this formula... and These are the weight coefficients for the corresponding items, which are hyperparameters specified by the user.

[0037] The simulation environment is constructed based on the vehicle kinematics model and the reward function. During testing, the agent is placed in the autonomous driving engine platform for policy evaluation. The agent's observation images in the engine are input into the deep imitation learning framework, and the decision actions obtained by the agent are used as the agent's output. The agent interacts with the decision-making environment according to the aforementioned steps. The effectiveness of the agent is determined based on the final reward function value, and the agent's driving ability and performance are evaluated.

[0038] Overall, the imitation learning module uses training data to update the agent's policy. After the update is completed, it participates in the interaction between the agent and the environment in conjunction with the action modeling module. Each time, it makes a decision action based on the current observation of the environment modeling module by the agent, and inputs the decision action and the environmental observation into the environment response module. The environment response module feeds back to the agent based on the simulation environment composed of the dynamic model and the reward function to evaluate the agent's performance.

[0039] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the present invention.

Claims

1. A driving control method based on self-supervised imitation learning, characterized in that, Includes the following steps: Generate an expert policy dataset for autonomous driving tasks, the dataset including observation images of agent expert policies; The image observed at time t at the anchor node in the dataset The nonlinear encoder, after data augmentation, is input to the strided convolutional layer. The resulting latent representation of the anchor node ; The relevant samples of the anchor nodes in the dataset are... Observational images at time Image data obtained by using different random blocks and a data augmentation method of the same type as the anchor node is input into a nonlinear encoder of a strided convolutional layer. In this process, the latent representation of the relevant samples is obtained. ; The latent representations in the latent space before time t Integrate to produce the aforementioned Latent representation of time context , using the Extract the Relevant contextual information, among which, It is an autoregressive model; Use a density ratio meter to retain the observed image at time t+k and stated The effective information between them, using the above inverse vector The encoder is trained using a contrastive learning-based method to learn the mapping from high-dimensional observations to low-dimensional features in an unsupervised manner. The density ratio analyzer is modeled as follows: ， The symbol "∝" represents a direct proportion. For the The following observations The probability distribution, To observe the The prior probability distribution, These are the encoder weights that need to be learned, each Step using different To represent the prediction, For the The inverse vector; The cost function is calculated by passing the observed images of different autonomous driving scenarios through a trained encoder. The cost function is then updated by an expert policy, a policy network, and a policy regularization term, and then fed into the imitation learning objective formula to update the policy network and learn the control policy. The anchor nodes and related samples are all augmented using a random cropping method, that is, a rectangular sub-image of a fixed size is randomly cropped from the original image observation as the augmented image data. Define the cost function c: Z×Z→R, where Z represents the latent space after the state space is mapped by the encoder. The update formula for the cost function c is: ， in, To collect expert strategies from the trajectory, The entire feasible space for policy networks learned by the agent. This is the policy network learned by the current agent. Its input is the encoded representation of the current state, and its output is the action taken by the current agent. This is a policy regularization term, used to provide additional constraints on the cost function and assist in its training. The cost function is parameterized as a neural network, with its input being two different observed image data. and The result obtained after encoding is the output representing the cost function value. Indicates a policy-based network Construct trajectory data and sample two different observation images from the trajectory data. and The expectation of the cost function is calculated after encoding the two observed images by the encoder. Indicates based on expert strategy Construct trajectory data and sample two different observation images from the trajectory data. and The expectation of the cost function is calculated after encoding the two observed images by the encoder.

2. The driving control method based on self-supervised imitation learning as described in claim 1, characterized in that, Given time t, A batch consists of N random samples. For this batch of samples, the loss function is defined. , in, Indicates the expected value. The numerator part within the function is the latent representation of the context. With the The corresponding density ratio; The denominator within the function is as described Compared with all samples in the current batch The sum of the corresponding density ratios; Combined with the loss function Gradient descent optimization of the encoder: ， in, The learning rate is used for gradient updates.

3. The driving control method based on self-supervised imitation learning as described in claim 1, characterized in that, The strategy regularization term Represented as: ， in, Indicating expert strategy The trajectory data is upsampled from two different observation images. and After encoding the two observed images using an encoder, the cost function is calculated, and then processed by the formula... The result obtained.

4. The driving control method based on self-supervised imitation learning as described in claim 3, characterized in that, The objective formula for imitation learning is defined as follows: ， in, For policy networks The frequency of access to the following states For expert strategies The frequency of access to the following states Indicates a policy-based network Construct trajectory data and sample two different observation images from the trajectory data. and After being encoded by the encoder, the data is input into the cost function c, and its logarithmic expected value is calculated. Indicates based on expert strategy Construct trajectory data and sample two different observation images from the trajectory data. and After being encoded by the encoder, it is input into the cost function c, and the logarithmic expected value of the difference between it and 1 is calculated.

5. A driving control system based on self-supervised imitation learning, wherein the method described in any one of claims 1-4 is characterized in that, Includes the following modules: The environment modeling module is based on the open-source autonomous driving simulator Carla platform for driving control application and verification, with the simulator of the platform serving as the main simulation environment. The action modeling module is used to model the decision-making actions of the intelligent agent; The imitation learning module takes driving data collected from the agent's interactions with the environment, gathered by expert policies, and inputs the driving observation data into a nonlinear encoder. The low-dimensional vectors of the latent representations of the observed anchor nodes and their associated samples are obtained, and the observation image at time t+k is preserved using a density ratio analyzer. and The effective information between them is used to calculate the cost function after the observation images of different autonomous driving scenarios are passed through the trained encoder. The cost function is updated by expert policy, policy network and policy regularization term and then passed into the imitation learning objective formula to update the policy network. During the test, the agent is put into the autonomous driving simulator Carla platform for policy evaluation. The agent's observation image is input into the updated policy network and the decision action obtained by the agent is obtained as the output of the agent. The environment response module is used by the agent to select decision actions based on the driving scene observations input at the current moment, using the imitation learning module and the action modeling module, and output response information based on the decision actions. The effectiveness of the agent is determined based on the magnitude of the final reward function value.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-4.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method described in any one of claims 1-4.

Images