Autonomous learning method based on edge computing and reasoning of autonomous driving smart car

Abstract

The present invention provides an autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car, which has the following characteristics and includes the following steps: step S1, obtaining sensor data of the real car and lane in real time through multiple sensors set on the real car and lane; step S2, performing lightweight compression and distillation on the first observation data and the second observation data to construct a simulation environment; step S3, obtaining change events in the simulation environment based on the nearest mean classifier of class increment; step S4, training the virtual car based on the change event, and using the training result as an incremental data set; step S5, updating the decision model of the real car based on iterative weights in combination with the incremental data set; step S6, obtaining the optimal strategy based on the first observation data combined with the decision model and controlling the real car to execute. This method can well solve the problems of limitations of road condition observation data and real-time deployment of optimal response, and has great application prospects.

Description

Technical Field

[0001] The present invention relates to an autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car, and belongs to the technical field of artificial intelligence autonomous driving. Background Art

[0002] In recent years, the application of artificial intelligence (AI) in the real world, such as robots and autonomous driving, has focused on how to make more optimized real-time decisions when interacting with scenes. In artificial intelligence, edge AI aims to enable racing cars to autonomously learn how to deploy real-time and continuous reasoning and decision-making through multi-dimensional perception and multi-modal recognition, such as Sony GTsophy racing simulation game AI.

[0003] The edge computing field of edge AI is mainly divided into incremental learning paradigms based on regularization and replay. Among them, the regularization method based on data constraints (LwF) algorithm has the ability to integrate new knowledge from new data and distill existing knowledge. However, the LwF algorithm is highly dependent on the correlation between new and old tasks. When the task difference is too large, task confusion (inter-task confusion) will occur, and the training time of a task will increase linearly with the number of learning tasks. At the same time, the introduced regularization term often cannot effectively constrain the optimization process of the model on the new task. Around this problem, the parameter constraint algorithm (EWC) based on the Bayesian framework proposed an improvement strategy to correct the importance of different parameters to fit the model trained on the new task, but its performance on multiple data sets is not as robust as LwF. On the other hand, the incremental classifier and representation learning algorithm (iCaRL) is a classic model based on replay. The incremental classifier is selected by the nearest mean rule of the sample, and the network parameters are updated based on the loss function of the sum of classification and distillation losses. However, this method may cause the model to overfit the retained old data.

[0004] The incremental learning paradigm of existing autonomous driving technology generally collects data through the car's camera, and then the server trains the model based on the collected data to obtain an optimized model to control the car's autonomous driving. However, the data collected by the car's camera has limitations, and a large amount of real-time data will impose a heavy computing burden on the server, resulting in the car's inability to handle emergencies in real time. Summary of the invention

[0005] The present invention is made to solve the above-mentioned problems, and its purpose is to provide an autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car.

[0006] The present invention provides an autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car, which has the following characteristics: a simulation environment is constructed and a corresponding virtual car model is constructed in the simulation environment based on a real car, including the following steps: step S1, through multiple sensors set on the real car and the lane, the first-perspective data of the real car is obtained in real time as the first observation data, and the observation data of each perspective of the lane is obtained as the second observation data; step S2, the first observation data and the second observation data are lightweight compressed and distilled to construct the simulation environment; step S3, based on the nearest mean classifier of the class increment, the change event in the simulation environment is obtained; step S4, based on the confidence, the optimal strategy of the virtual car for the change event is obtained, based on the different positions of the virtual car in the change event, the first-perspective feature image of the change event is obtained, and the first-perspective feature image and the corresponding optimal strategy are used as an incremental data set; step S5, based on the incremental data set, the decision model of the real car is updated based on the iterative weight; step S6, according to the first observation data combined with the decision model, the optimal strategy is obtained based on the target distribution of different change events in the lane and the execution of the real car is controlled.

[0007] In the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car provided by the present invention, it can also have the following characteristics: wherein, the first observation data is the first-perspective image data of the real car, the second observation data is the image data of each perspective of the lane, and the sensor of the real car is a multi-dimensional vehicle-mounted camera module including: lens equipment, a single camera module, a dual-camera module with enhanced depth ranging, an external 3D-sensitive triple-camera module and a time-of-flight TOF sensor device, and the sensors at various locations in the lane include: a fisheye camera above the lane and multiple conventional cameras in the lane.

[0008] In the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car provided by the present invention, it can also have the following characteristics: wherein, in step S2, the first observation data is a data structure w(x, y) of size J*K, and the second observation data is a data structure f(x, y) of size M*N. The first observation data and the second observation data are grayed and normalized to obtain global mapping environment data to construct a simulation environment. The graying formula is as follows: Where R, G, and B are the pixel values ​​collected by the sensor, and the normalized correlation coefficient c(x, y) is as follows: Where x=0, 1, 2, ...NK, y=0, 1, 2, ...MJ, Δx is the offset of x, Δy is the offset of y, and the normalized function P is used to calculate the probability that the positioning range best matches the transformed target in the changing scene. The function P formula is as follows: Where D = D GRAY, k is a positive coefficient obtained from the distribution of all pixels D of the second observation data (k>0).

[0009] In the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car provided by the present invention, it can also have the following characteristics: wherein step S4 includes the following sub-steps: step S4-1, identifying the distribution probability of the highest affinity in the simulation environment A set of strategies for changing events combined with the virtual car t and through external cumulative rewards r = Q n = confidence derived iteratively from Q(s,u) Based on the state s of the virtual car at time t, the optimal strategy of the virtual car is obtained is sampled from the set A t (s,u t |τ t )'s best response, Q n For global rewards, Beta b (win, lose) is a variant of the self-directed (WSLS) algorithm, u t is the action of the real car at time t, τ t is the action-observation history of the virtual car at time t; step S4-2, taking the first-perspective images of the virtual car at the far, middle, near, left and right angles of the change event as the first-perspective feature images; step S4-3, taking the first-perspective feature images of each change event and the optimal strategy as the incremental data set.

[0010] In the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car provided by the present invention, it can also have the following characteristics: wherein, in step S6, based on different change events in continuous time The target distribution obtains the optimal strategy for multiple change events. Different change events X (k) The ergodic joint probability distribution of multiple change events The initial sequence X is sampled by Gibbs n ={x i :i=1,2,…,n} and conditional random distribution estimation continuous iterative sampling fitting reconstruction, conditional random distribution estimation for: Chinese is the best response action generated in real time at time t+1, is the optimal strategy collected from time 0 to t, is the change event x at location j that can be retrieved under the confidence b condition.

[0011] In the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car provided by the present invention, it can also have the following characteristics: wherein, in step S6, the optimal strategy is composed of multiple best response actions A * (s,u t |τ t ) constitutes the best response action A * (s,u t |τ t ) Based on generating a sequence of fitted observations and action estimates from time 0 to time t The iterative update is as follows: Where u -t represents the utility of choosing other random actions in the strategy, (u -t , τ -t ) represents the dominant strategy utility of the global joint action space corresponding to the first-person perspective of the real car in the state space s, Q(s,u) is the maximum global reward, b t is the confidence level at time t.

[0012] In the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car provided by the present invention, it can also have the following characteristics: wherein, by learning parameter a i Training the best response action A * , learning parameter a i The formula is: Where O s is the basic consumption per unit time, O e To explore the overhead, O g is the task loss, k is the number of change events, α is the learning rate, ∑ i is the coordinates of the comparable reference objects in all the second observation data, ∑ g By exploring the high-dimensional strategy gradient direction The parallel gradient update strategy implemented.

[0013] Functions and Effects of the Invention

[0014] According to the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car involved in the present invention, firstly, through multiple sensors at various locations of the real car and the lane, the first-perspective observation data of the car is obtained in real time as the first observation data, the observation data of each perspective of the lane is obtained as the second observation data, and the first-perspective observation data of the virtual car is obtained as the third observation data. Secondly, multiple groups of first observation data, second observation data and third observation data are merged and updated in real time to obtain simulation training data. Then, the virtual car is trained with the simulation training data to obtain the preferred strategy sequence of the change event as the strategy data set. Then, under the road environment, the change event in the first observation data is feature matched with the strategy data set to obtain the closest change event. Finally, the belief score of the preferred strategy sequence of the change event is calculated by combining the first observation data and the third observation data, and the strategy with the largest belief score is selected as the optimal strategy of the real car and executed. Therefore, the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car of the present invention can form a preprocessing data structure beyond visual range and reduce memory consumption while ensuring the same training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car in an embodiment of the present invention.

[0016] Figure 2 It is a flowchart of an autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car in an embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a simulation environment in an embodiment of the present invention; and

[0018] Figure 4 It is a schematic diagram of identifying change events and marking optimal strategies in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car of the present invention.

[0020] Figure 1 It is a flow chart of the autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car of the present invention.

[0021] like Figure 1As shown, the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car of the present invention is implemented according to the following principle: real-time image data of the lane is collected by the lane sensor, and the position, orientation, target position and other information of the real car in the lane are obtained based on the unsupervised change detection algorithm, and then the simulation platform is constructed in combination with the first-person image data of the real car sensor, and the lane change event is identified in the simulation platform, and the virtual car is trained to obtain the optimal strategy, and then the feature image of the change event is obtained based on the perspective of the virtual car, and the optimal strategy and the feature image are used as incremental data sets. Iteration is performed on the basis of the original decision model of the real car, and the iterated decision model is deployed on the real car in real time to realize the autonomous response of the real car to the change event.

[0022] Figure 2 It is a flow chart of the autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car of the present invention.

[0023] like Figure 2 As shown, this embodiment provides an autonomous learning method based on edge computing and reasoning of an autonomous driving intelligent car, constructs a simulation environment and constructs a corresponding virtual car model based on a real car in the simulation environment, including the following steps:

[0024] Step S1, by means of multiple sensors arranged on the real car and the lane, first-view data of the real car is obtained in real time as first observation data, and observation data of each view of the lane is obtained as second observation data.

[0025] Among them, the first observation data is the first-view image data of the real car, and the second observation data is the image data of each view of the lane. The sensor of the real car is a multi-dimensional on-board camera module including: a lens device with a focal length of 2.8mm, a single camera module, a dual-camera module with enhanced depth ranging, an external 3D photosensitive triple-camera module and a highly integrated time-of-flight TOF sensor device. The time-of-flight ToF sensor is based on the reflection of modulated near-infrared light after encountering an object. By calculating the time difference or phase difference between light emission and reflection, the travel position of the real car in the lane and the corresponding depth information of the obstacles and road signs in the current environment are obtained. Its measurement range can reach several meters, the accuracy is less than 2cm, and the update frequency per second can reach 30 to 60fps. Sensors in various places in the lane include: fisheye cameras above the lane and multiple conventional cameras in the lane.

[0026] Figure 3 is a schematic diagram of a simulation environment in an embodiment of the present invention.

[0027] like Figure 3 As shown, the global mapping environment data is obtained through the first observation data and the second observation data of the real environment to construct a simulation environment based on the real environment.

[0028] Step S2, performing lightweight compression and distillation on the first observation data and the second observation data to construct a simulation environment.

[0029] Among them, in step S2, the first observation data is a data structure w(x, y) of size J*K, and the second observation data is a data structure f(x, y) of size M*N. The first observation data and the second observation data are normalized and grayscaled to obtain global mapping environment data and construct a simulation environment.

[0030] The grayscale formula is as follows: Where R, G, and B are the pixel values ​​collected by the sensor. Grayscale the image data collected by the sensor can improve the matching efficiency, avoid low-speed floating-point calculations, and thus reduce the complexity.

[0031] The formula for the normalized correlation coefficient c(x, y) in the high grayscale area of ​​function f is as follows: Where x=0, 1, 2, ...NK, y=0, 1, 2, ...MJ, Δx is the offset of x, Δy is the offset of y, and the normalized function P is used to calculate the probability that the positioning range best matches the transformed target in the changing scene. The function P formula is as follows: Where D = D GRAY , k is a positive coefficient obtained from the distribution of all pixels D of the second observation data (k>0).

[0032] Step S3, obtaining the change events in the simulation environment based on the nearest mean classifier of the class increment.

[0033] Step S4, based on the confidence level, the optimal strategy of the virtual car for the change event is obtained, and the first-perspective feature image of the change event is obtained based on the different positions of the virtual car in the change event, and the first-perspective feature image and the corresponding optimal strategy are used as an incremental data set.

[0034] Wherein, step S4 includes the following sub-steps:

[0035] Step S4-1, identifying the distribution probability of the highest affinity in the simulation environment A set of strategies for changing events combined with the virtual car t and through external cumulative rewards r = Q n = confidence derived iteratively from Q(s,u) Based on the state s of the virtual car at time t, the optimal strategy of the virtual car is obtained is sampled from the set A t (s,u t |τ t )'s best response, Q n For global rewards, Betab (win, lose) is a variant of the self-directed (WSLS) algorithm, u t is the action of the real car at time t, τ t is the action-observation history of the virtual car at time t.

[0036] If and only if the α-finite measure of the measurable space in the non-stationary environment satisfies the return time t = inf (n> 0: X n ∈A), the conditional random field distribution can approximately fit and generate a first-person-view target distribution:

[0037]

[0038] In the formula, the joint action space (u t |τ t ) is used to characterize the fitting sequence formed by the fusion of the second observation data f(x, y) at the previous time t and the first perspective action u selected by the first observation data w(x, y) at the time t.

[0039] In step S4-2, the first-person perspective images of the virtual car at the far, middle, near, left, and right angles of the change event are used as first-person perspective feature images, and the fine-grained features of the lanes are quickly extracted for the current scene and lightweight distillation is performed, so that the cloud server can compress a large number of redundant and repeated fuzzy images, ensuring the same training effect, i.e., the first-person perspective image, while reducing memory consumption.

[0040] Step S4-3: taking the first-view feature image and the optimal strategy of each change event as an incremental data set.

[0041] Step S5, updating the decision model of the real car based on the iterative weights in combination with the incremental data set.

[0042] Among them, the iterative weight of the decision model is adjusted according to the correlation between the incremental data set and the scene in the original decision model, the turning and detour behavior labeling of the current incremental data set is completed, and the trained incremental preprocessing model is deployed in the smart car to achieve efficient recognition and obstacle avoidance.

[0043] Step S6, obtaining the optimal strategy based on the first observation data combined with the decision model based on the target distribution of different change events in the lane and controlling the actual car to execute it.

[0044] Among them, in step S6, based on different change events in continuous time The target distribution obtains the optimal strategy for multiple change events. Different change events X (k) The ergodic joint probability distribution of multiple change events The initial sequence X is sampled by Gibbsn ={x i :i=1,2,…,n} and conditional random distribution estimation continuous iterative sampling fitting reconstruction, conditional random distribution estimation for: In the formula is the best response action generated in real time at time t+1, is the optimal strategy collected from time 0 to t, is the change event x at location j that can be retrieved under the confidence b condition.

[0045] The optimal strategy consists of multiple best response actions A * (s,u t |τ t ) constitutes the best response action A * (s,u t |τ t ) Based on generating a sequence of fitted observations and action estimates from time 0 to time t The iterative update is as follows: Where u -t represents the utility of choosing other random actions in the strategy, (u -t , τ -t ) represents the dominant strategy utility of the global joint action space corresponding to the first-person perspective of the real car in the state space s, Q(s,u) is the maximum global reward, b t is the confidence level at time t.

[0046] By learning the parameter a i Training the best response action A * , learning parameter a i The formula is: Where O s is the basic consumption per unit time, O e To explore the overhead, O g is the task loss, k is the number of change events, α is the learning rate, ∑ i is the coordinates of the comparable reference objects in all the second observation data, ∑ g By exploring the high-dimensional strategy gradient direction The parallel gradient update strategy implemented.

[0047] Figure 4 It is a schematic diagram of identifying change events and marking optimal strategies in an embodiment of the present invention.

[0048] like Figure 4 As shown in the figure, the fisheye camera monitors in real time and updates the current position, direction and target positioning of the real car in the global bird's-eye view based on the unsupervised detection algorithm. Figure 3The left side is a bird's-eye view of the actual car at any time. Figure 3 The middle part is the lane change event, as shown in the blue dots in the figure, and the "new objects" detected by the lane sensor device are displayed in a binary way, that is, the change events are white dots. The white dots here indicate a mutation at that point, and the black dots are unchanged. The output is Figure 3 On the right, the white dot marked position is the position of the change event in the first-person perspective of the real car, so that the real car has the change detection ability of new scene recognition, sudden scene recognition, and uncommon scene recognition. By converting the obtained optimal strategy into a steering or detour annotation based on the first-person perspective, when a change event occurs in the first-person perspective of the real car, the annotation of the corresponding optimal strategy is also presented in the first-person perspective of the real car. The real car performs corresponding operations according to the annotation, such as Figure 3 As shown on the right, the blue and white columns are the change events that occur during the real car's movement. In the first-person perspective of the real car, the label of the optimal strategy for the change event appears, which is a left turn sign. The real car calls the preset response action for the sign and executes it, which is a left turn, to avoid the blue and white columns.

[0049] In this embodiment, the test results of 100 laps in a real lane are used as data samples. The real-time labeling and accuracy statistics of lane change events based on the real car and lane sensor are as follows:

[0050]

[0051] The first row is the type of each change event in the lane, the second row is the real-time annotation of the change event, and the third row is the accuracy of the annotation in line with the actual situation. For example, the cell in the second column of the third row indicates that the probability of correctly annotating the blue and white cylinder in the lane is 100%. As shown in the table above, in this embodiment, the annotation accuracy of the change event in the lane is greater than or equal to 94%, which has a high recognition accuracy.

[0052] In this embodiment, the cloud server controls the virtual car in the virtual environment based on the TS model (Teacher-Student model) of the distillation algorithm, obtains an incremental data set based on the optimal strategy of the virtual car for the change event, and the real car takes corresponding actions for the change event based on the incremental data set. In this embodiment, there are three countermeasure models, namely the teacher model (Teacher) obtained by the cloud server through complex calculations based on the large amount of data, the student model (Student) obtained by training the teacher model, i.e. the virtual car side, and the distillation model obtained based on the incremental data set trained by the virtual car, i.e. the real car side. The test results of one hundred laps on the real lane are used as data samples, and the statistical results of the recognition accuracy of each model for change events based on different observation data are as follows:

[0053]

[0054] The first row in this table is the name of each model, the second row is the probability of each model accurately identifying the change event when only the first observation data is used, the third row is the probability of each model accurately identifying the change event when only the second observation data is used, and the fourth row is the probability of each model accurately identifying the change event when the first observation data and the second observation data are combined. For example, the second cell in the fourth row indicates that the probability of the cloud server's teacher model accurately identifying the change event when combined with the first observation data and the second observation data is 97.77%. It can be seen from this table that compared with only using the first observation data or the second observation data, the probability of each model accurately identifying the change event when combined with the first observation data and the second observation data is higher. In the above table, the accuracy rates of the teacher model and the distillation model combined with the first observation data and the second observation data are 97.77% and 95.14% respectively. It can be seen that the probability of the real car accurately identifying the change event through the incremental data set is higher, and is closer to the accuracy of the cloud server teacher model.

[0055] Functions and Effects of the Embodiments

[0056] According to the autonomous learning method based on edge computing and reasoning of the autonomous driving intelligent car involved in this embodiment, the car and lane sensors obtain real-time road condition image data, build a simulation environment based on the image data, identify the change events in the simulation environment and generate the optimal strategy, use the optimal strategy and the characteristic image of the change event as the training result, iterate the car decision model, so as to achieve the real-time optimal response of the car to the change event in the lane. Therefore, on the one hand, the lane perception data and the car perception data can be combined to build comprehensive perception data, so as to better generate the optimal strategy. On the other hand, by compressing and distilling the training results, the car can complete the iterative update of the decision model faster, thereby improving the real-time response speed of the car to the change event in the lane. In short, this method can well solve the problems of road condition observation data limitations and real-time deployment of optimal response, and has great application prospects.

[0057] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.

Claims

1. An autonomous learning method based on edge computing and reasoning of an autonomous driving smart car, characterized in that: Constructing a simulation environment and building a corresponding virtual car model based on the real car in the simulation environment includes the following steps: Step S1, obtaining first-viewing angle data of the real car as first observation data and obtaining observation data of each viewing angle of the lane as second observation data in real time through a plurality of sensors arranged on the real car and the lane; Step S2, performing lightweight compression and distillation on the first observation data and the second observation data to construct the simulation environment; Step S3, obtaining a change event in the simulation environment based on a nearest mean classifier of class increment; Step S4, obtaining the optimal strategy of the virtual car for the change event based on the confidence level, obtaining a first-view feature image of the change event based on different positions of the virtual car in the change event, and using the first-view feature image and the corresponding optimal strategy as an incremental data set; Step S5, updating the decision model of the real car based on the iterative weight in combination with the incremental data set; Step S6, obtaining an optimal strategy based on the target distribution of different change events in the lane in combination with the decision model according to the first observation data and controlling the real car to execute it, The first observation data is the first-view image data of the real car, and the second observation data is the image data of each view of the lane. The sensor of the real car is a multi-dimensional vehicle-mounted camera module including: Lens equipment, single camera module, dual camera module with enhanced depth measurement, triple camera module with external 3D sensitivity and time-of-flight TOF sensor equipment, The sensors at various locations of the lane include: A fisheye camera above the lane and a plurality of conventional cameras of the lane, In the step S2, the first observation data is a data structure w(x, y) of size J*K, and the second observation data is a data structure f(x, y) of size M*N. The first observation data and the second observation data are grayed and normalized to obtain global mapping environment data to construct the simulation environment. The grayscale formula is as follows: Where R, G, and B are the pixel values ​​collected by the sensor. The normalized correlation coefficient c(x, y) formula is as follows: Where x = 0, 1, 2, ... NK, y = 0, 1, 2, ... MJ, Δx is the offset of x, Δy is the offset of y, The normalized function P is used to calculate the probability that the positioning range best matches the transformed target in the changing scene. The formula of function P is as follows: Where D = D GRAY , k is a positive coefficient obtained from the distribution of all pixels D of the second observation data (k>0), The step S4 includes the following sub-steps: Step S4-1, identifying the distribution probability of the highest affinity in the simulation environment Combine the set A of strategies that the virtual car makes for the change event and the external cumulative reward r=Q n = confidence derived iteratively from Q(s,u) Based on the state s of the virtual car at time t, the optimal strategy of the virtual car is obtained is sampled from the set A t (s,u t |τ t )'s best response, Q n For global rewards, Beta b (win, lose) is a variant of the self-directed (WSLS) algorithm, u t is the action of the real car at time t, τ t is the action-observation history of the virtual car at time t; Step S4-2, taking the first perspective images of the virtual car at the far, middle, near, left and right angles of the change event as the first perspective feature images; Step S4-3, taking the first viewing angle feature image and the optimal strategy of each change event as an incremental data set, In step S6, based on the different change events in continuous time The target distribution obtains the optimal strategy for multiple change events, and the different change events X (k) The ergodic joint probability distribution of multiple change events The initial sequence X is sampled by Gibbs n ={x i :i=1,2,…,n} and conditional random distribution estimation continuous iterative sampling fitting reconstruction, The conditional random distribution estimates for: In the formula is the best response action generated in real time at time t+1, is the optimal strategy collected from time 0 to t, is the change event x at location j that can be retrieved under the confidence b condition, In step S6, the optimal strategy is composed of multiple best response actions A * (s,u t |τ t ) is composed of the best response action A * (s,u t |τ t ) Based on generating a sequence of fitted observations and action estimates from time 0 to time t The iterative update of is as follows: Where u -t represents the utility of choosing other random actions in the strategy, (u -t , τ -t ) represents the dominant strategy utility of the global joint action space corresponding to the first-person perspective of the real car in the state space s, Q(s,u) is the maximum global reward, b t is the confidence level at time t, By learning the parameter a i Train the best response action A * , the learning parameter a i The formula is: Where O s is the basic consumption per unit time, O e To explore the overhead, O g is the task loss, k is the number of change events, α is the learning rate, Σ i is the coordinates of the comparable reference objects in all the second observation data, ∑ g By exploring the high-dimensional strategy gradient direction The parallel gradient update strategy implemented.

Images