An autonomous learning method for efficiently constructing beyond-horizon global perception

Abstract

The present invention provides an autonomous learning method for efficiently constructing beyond-horizon global perception. A virtual platform is constructed and a corresponding virtual car model is constructed within the virtual platform based on a real car. Step S1 acquires the car's first-person perspective observation data, lane observation data from various perspectives, and first-person perspective observation data of the virtual car in real time. Step S2 merges and updates multiple sets of observation data to obtain simulated training data. Step S3 uses the simulated training data to obtain a sequence of optimal strategies for different change events. Step S4 matches the features of the change events in the first observation data with the strategy dataset under a road surface environment to obtain the closest change event. Finally, Step S5 combines the observation data and obtains the optimal strategy for the real car based on the confidence level and executes it. This method can effectively address the problems of catastrophic forgetting and falling into a local optimal strategy, and has great application prospects.

Description

Technical Field

[0001] The present invention relates to an autonomous learning method for efficiently constructing beyond-visual-range global perception, and belongs to the technical field of artificial intelligence autonomous driving. Background Art

[0002] Currently, research on whether artificial intelligence systems can successfully operate and adapt in the real world has emerged. This challenge is called lifelong learning and has driven many systems with lifelong learning capabilities, including fully autonomous vehicles and smart city transportation.

[0003] An algorithmic improvement that enhances a model's ability to learn throughout life typically means utilizing limited resources to continuously learn new knowledge, apply previous knowledge to new tasks, and rapidly adapt to changes in tasks or the environment. Furthermore, the use of event memory replay—the replay of daytime experiences during dreams—can effectively overcome catastrophic forgetting, meaning that learning a new task doesn't degrade performance on previously learned tasks. This demonstrates the ability to learn from experience, predict future events, and adapt to new challenges.

[0004] Many existing systems can achieve good autonomous learning capabilities through training tasks, but improving and enhancing the model's lifelong learning capabilities still requires focusing on one or a combination of the following categories, such as being able to only complete the transfer and adaptation between similar tasks, but forgetting previously learned knowledge when training new tasks, which leads to catastrophic forgetting, and the problem of incorrectly executing local observations as the optimal strategy. Summary of the Invention

[0005] The present invention is made to solve the above-mentioned problems, and its purpose is to provide an autonomous learning method for efficiently constructing beyond-horizon global perception.

[0006] The present invention provides an autonomous learning method for efficiently constructing beyond-visual-range global perception, which has the following characteristics: constructing a virtual platform and constructing a corresponding virtual car model based on a real car in the virtual platform, including the following steps: step S1, through multiple sensors at various locations of the real car and lanes, obtaining the first-perspective observation data of the car as the first observation data, obtaining the observation data of each perspective of the lane as the second observation data, and obtaining the first-perspective observation data of the virtual car as the third observation data; step S2, merging and updating multiple groups of first observation data, second observation data and third observation data in real time to obtain simulated training data; step S3, obtaining a set of high-confidence strategies for different positions of the virtual car in different change events through simulated training data as the preferred strategy sequence for change time; step S4, comparing the change events in the first observation data with the strategy data set under the road environment Perform feature matching to obtain the closest change event; and step S5, combining the first observation data and the third observation data, based on the confidence level of the preferred strategy sequence of the change event, select a strategy with a high confidence level as the optimal strategy for the real car and execute it, wherein step S3 includes the following sub-steps: step S3-1, analyzing the simulated training data to obtain the change events therein; step S3-2, setting the virtual car at different positions of the change event, and obtaining observation data based on the first perspective of the virtual car as virtual observation data; step S3-3, according to the virtual observation data of the same change event, obtaining different strategies for the corresponding change event based on the virtual car; step S3-4, sorting the confidence levels of different strategies, and selecting at least one strategy with a high confidence level as the preferred strategy sequence of the change event; step S3-5, storing the change event and the corresponding preferred strategy sequence in the strategy data set.

[0007] The autonomous learning method for efficiently constructing beyond-horizon global perception provided by the present invention may also have the following features: wherein, step S2 includes the following operations: merging the first observation data, the second observation data, and the third observation data at the same moment into a set of merged data; buffering and storing t sets of merged data at t consecutive moments as simulated training data; and whenever a new set of merged data is obtained, removing the oldest merged data in the simulated training data and adding the new merged data.

[0008] The autonomous learning method for efficiently constructing beyond-horizon global perception provided by the present invention may also have the following features: wherein the data merging process in step S2 includes occasional replay of empirical experiences to generate simulated training data that best represents overall environmental perception:

[0009] {s0,u0,τ0,A 1 ,b1,s1,u1,τ1,A 2 ,b2,…,s t-1 ,ut-1 ,τ t-1 ,A t ,b t}, where s t-1 is the state of the virtual car at time t-1, u t-1 is the action of the real car at time t-1, τ t-1 is the action-observation history of the virtual car at time t-1, A t is the response action of the virtual car at time t, b t is the confidence level at time t.

[0010] The autonomous learning method for efficiently constructing beyond-horizon global perception provided by the present invention may also have the following characteristics: wherein the generation of different strategies includes the following process: in each change event t, based on the strategy π(u t |τ t ) Get the real car's action u t : The action of the virtual car is the observation history τ t Expressed as: τ t =(s1,a1,…,a t-1 ,s t ), the goal is to generate the different strategies within a bounded maximum period T (Confidence: ), thereby achieving cumulative rewards maximize: Where a t is the action of the virtual car at time t, s t is the state of the virtual car at time t, u t is the action of the real car at time t, γ is the discount coefficient, r t is the environmental reward obtained by the virtual car for its action at time t.

[0011] The autonomous learning method for efficiently constructing beyond-horizon global perception provided by the present invention may also have the following characteristics: wherein the generation of the optimal strategy sequence includes the following process: according to the timeline simulation training data o t ∈O(s,a), from which the cumulative reward is obtained The action-observation history τ of the virtual car t =(o0,u0,r1,…,r t-1 ,o t ) and execute A t (s,u t |τ t ) as the preferred strategy The following n-step Monte Carlo Markov transfer matrix structure is used, so that t=inf(n>0:Xn ∈A) moment, there is as follows:

[0012]

[0013] Where s is the state of the virtual car, a is the action of the virtual car, r t-1 For u t To realize the action of the car at time t, Indicates that when The optimal strategy obtained by sampling Able to reach the global reward Q within a limited number of n steps n (or any random reward Q A～E ), and the process is independent of state s, then the optimal strategy sequence is obtained.

[0014] The autonomous learning method for efficiently constructing beyond-horizon global perception provided by the present invention may also have the following characteristics: wherein, in step S5, the preferred strategy sequence of the change event is calculated by calculating the current confidence level And fine-tune the reasoning of the first observation data based on the conditional random field matrix, when the global reward of the environment is satisfied ≥Action or status reward When the current optimal strategy replaces the random optimal strategy As the optimal strategy, Beta(win,lose) is a variant of the self-directed (WSLS) algorithm, which adopts a win-lose game strategy. If the generated global reward of the environment reaches its expected level, the previous optimal strategy is repeated; otherwise, the strategy is adjusted accordingly according to the current state.

[0015] Functions and effects of the invention

[0016] According to the autonomous learning method for efficiently constructing beyond-visual-range global perception involved in the present invention, first, through multiple sensors at various locations of the real car and lane, the first-perspective observation data of the car is obtained in real time as the first observation data, the observation data of each perspective of the lane is obtained as the second observation data, and the first-perspective observation data of the virtual car is obtained as the third observation data. Secondly, multiple groups of first observation data, second observation data, and third observation data are merged and updated in real time to obtain simulation training data. Then, the virtual car is trained with the simulation training data to obtain the preferred strategy sequence of the change event as the strategy data set. Then, under the road environment, the change event in the first observation data is feature-matched with the strategy data set to obtain the closest change event. Finally, the belief score of the preferred strategy sequence of the change event is calculated by combining the first observation data and the third observation data, and the strategy with the largest belief score is selected as the optimal strategy of the real car and executed. Therefore, the autonomous learning method for efficiently constructing beyond-visual-range global perception of the present invention can overcome catastrophic forgetting and avoid falling into a local optimal strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a technical principle diagram of an autonomous learning method for efficiently constructing beyond-horizon global perception in an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of constructing a virtual environment and a virtual car in an embodiment of the present invention; and

[0019] Figure 3 It is a flowchart of an autonomous learning method for efficiently constructing beyond-horizon global perception in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the present invention's efficient autonomous learning method for constructing beyond-visual-range global perception.

[0021] Figure 1 This is a technical principle diagram of the present invention's efficient autonomous learning method for constructing beyond-visual-range global perception.

[0022] like Figure 1 As shown, the autonomous learning method of the present invention is implemented based on the following principles: first, data is collected by the sensors of the smart car and the lane according to the real road conditions to obtain virtual observation data, and a virtual car and a virtual environment are constructed in the virtual platform of the cloud server through the virtual observation data. Then, the virtual observation data is used as training data for data processing and strategy generation to obtain the car's preferred strategy set for change events. The server group provides the optimal strategy to the smart car through control signals, and the smart car takes action on the real change events through decision-making instructions.

[0023] Figure 2 It is a schematic diagram of constructing a virtual environment and a virtual car according to the present invention.

[0024] like Figure 2 As shown, multiple sets of data are collected and fitted through sensors, a virtual platform is constructed, a virtual environment is constructed in the virtual platform, and a corresponding virtual car model is constructed based on the real car.

[0025] Figure 3 It is a flow chart of the autonomous learning method for efficiently constructing beyond-horizon global perception according to the present invention.

[0026] like Figure 3 As shown, this embodiment provides an autonomous learning method for efficiently constructing beyond-horizon global perception, including the following steps:

[0027] In step S1, the first-perspective observation data of the real car, i.e., the smart car, and multiple sensors at various locations in the lane are used to obtain in real time the first-perspective observation data of the car as the first observation data, the observation data of each perspective of the lane as the second observation data, and the first-perspective observation data of the virtual car as the third observation data.

[0028] Step S2: merging and updating multiple sets of first observation data, second observation data, and third observation data in real time to obtain simulation training data.

[0029] Step S2 includes the following operations:

[0030] The first observation data, the second observation data and the third observation data at the same moment are merged into a set of merged data, and the t sets of merged data at t consecutive moments are buffered and stored as simulated training data. Whenever a new set of merged data is obtained, the oldest merged data in the simulated training data is removed and the new merged data is added.

[0031] The data merging process includes the following steps: t The generated real-time strategy has more similar patterns to the objective environment. In the replay phase, empirical experiences are occasionally replayed to generate simulated training data that best represents the overall environmental perception:

[0032]

[0033] Where s t-1 is the state of the virtual car at time t-1, u t-1 is the action of the real car at time t-1, τ t-1 is the action-observation history of the virtual car at time t-1, For b t is the confidence level at time t.

[0034] Step S3: obtaining a set of high-confidence strategies for different positions of the virtual car in different change events by simulating training data as a preferred strategy sequence for the change time.

[0035] The principle of step S3 is: the state of the virtual car is s∈S, the action of the virtual car is a∈A, and the environmental reward of the virtual car action is r t ∈Q(s t ,a t ), The observation sequence of the virtual car is o t ∈O(s,a), according to the conditional probability S×A×S→[0,1], obtain the optimal strategy sequence,

[0036] Among them, S is, A is, Q(s t ,a t )for, is a non-periodic ergodic transfer matrix, that is, there exists a positive integer n such that the optimal strategy It can be obtained from the combination of environmental rewards r t With the experience of confidence b The confidence b in this process is obtained by sampling, and the global reward Q is obtained within a limited n steps n Or from any random reward such as Q A～E , which is derived iteratively and has nothing to do with the state space S.

[0037] Wherein, step S3 includes the following sub-steps:

[0038] Step S3-1: Analyze the simulated training data to obtain the change events therein.

[0039] Step S3-2: setting the virtual car to different positions of the change event, and obtaining observation data based on the first-person perspective of the virtual car as virtual observation data.

[0040] Step S3-3: obtaining different strategies corresponding to the change event based on the virtual car according to the virtual observation data of the same change event.

[0041] The generation of different strategies includes the following process: in each change event t, based on the strategy π(u t |τ t ) Get the real car's action u t :

[0042]

[0043] The action of the virtual car is the observation history τ t Expressed as: τ t=(s1,a1,…,a t-1 ,s t ), the goal is to generate the different strategies within a bounded maximum period T (Confidence: ), thereby achieving cumulative rewards maximize:

[0044]

[0045] Among them, a t is the action of the virtual car at time t, s t is the state of the virtual car at time t, u t is the action of the real car at time t, γ is the discount coefficient, r t for.

[0046] Step S3-4: sort the confidences of different strategies and select at least one strategy with a high confidence as a preferred strategy sequence for the change event.

[0047] The generation of the optimal strategy sequence includes the following process:

[0048] Simulate training data on the timeline t ∈O(s,a), from which the cumulative reward is obtained The action-observation history τ of the virtual car t =(o0,u0,r1,…,r t-1 ,o t ) and execute A t (s,u t |τ t ) as the preferred strategy

[0049] The following n-step Monte Carlo Markov transfer matrix structure is used, so that t=inf(n>0:X n ∈A) moment, there is as follows:

[0050]

[0051] Where s is the state of the virtual car, a is the action of the virtual car, r t-1 For u t To realize the action of the car at time t, Indicates that when The optimal strategy obtained by sampling Able to reach the global reward Q within a limited number of n steps n , and the process is independent of state s, then the optimal strategy sequence is obtained.

[0052] Step S3-5: storing the change event and the corresponding preferred strategy sequence into the strategy data set.

[0053] Step S4: Under the road environment, feature matching is performed between the change event in the first observation data and the strategy data set to obtain the closest change event.

[0054] Step S5: combining the first observation data and the third observation data, calculating the belief scores of the preferred strategy sequence of the change event, and selecting the strategy with the largest belief score as the optimal strategy for the real car and executing it.

[0055] In step S5, the optimal strategy sequence of the change event is calculated by calculating the current confidence level And fine-tune the reasoning of the first observation data based on the conditional random field matrix, when the global reward of the environment is satisfied ≥Action or status reward When the current optimal strategy replaces the random optimal strategy As the optimal strategy, Beta(win,lose) is a variant of the self-directed (WSLS) algorithm, which adopts a win-lose game strategy. If the generated global reward of the environment reaches its expected level, the previous optimal strategy is repeated; otherwise, the strategy is adjusted accordingly according to the current state.

[0056] Functions and Effects of the Embodiments

[0057] According to the autonomous learning method for efficiently constructing beyond-horizon global perception involved in this embodiment, by merging and updating the observation data of multiple sensors at various locations of the real car and lanes in real time, simulated training data is obtained. Then, through the simulated training data, a set of high-confidence strategies for different positions of the virtual car in different change events is obtained in the virtual platform as the preferred strategy sequence for the change time. The change event in the first observation data of the real car is feature-matched with the strategy data set in the road environment to obtain the closest change event. Finally, the belief score of the preferred strategy sequence of the change event is calculated by combining the first observation data and the third observation data, and the strategy with the largest belief score is selected as the optimal strategy for the real car and executed. The preferred strategy sequence for the change event obtained by training with simulated training data can effectively solve the problem of catastrophic forgetting. Matching the first-person perspective observation data of the car with the change event in the strategy data set can enable the real car to execute the optimal strategy for the change event based on a global perspective. In short, this method can effectively solve the problems of catastrophic forgetting and falling into local optimal strategies, and has great application prospects.

[0058] The above embodiments are preferred examples of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. An autonomous learning method for efficiently constructing beyond-horizon global perception, characterized in that: Constructing a virtual platform and building a corresponding virtual car model based on the real car in the virtual platform includes the following steps: Step S1: acquiring, in real time, first-perspective observation data of the real car and lane using multiple sensors at various locations, as first observation data; acquiring observation data from various perspectives of the lane as second observation data; and acquiring first-perspective observation data of the virtual car as third observation data. Step S2, merging and updating in real time the multiple sets of the first observation data, the second observation data, and the third observation data to obtain simulation training data; Step S3, obtaining a set of high-confidence strategies for different positions of the virtual car in different change events through the simulated training data as a preferred strategy sequence for the change time; Step S4, performing feature matching on the change event in the first observation data and the strategy data set in the road environment to obtain the closest change event; and Step S5: combining the first observation data and the third observation data, based on the confidence level of the preferred strategy sequence of the change event, selecting the strategy with the high confidence level as the optimal strategy for the real car and executing it. Wherein, the step S3 includes the following sub-steps: Step S3-1, analyzing the simulated training data to obtain change events therein; Step S3-2, setting the virtual car to different positions of the change event, and obtaining observation data based on the first perspective of the virtual car as virtual observation data; Step S3-3, obtaining different strategies corresponding to the change event based on the virtual car according to the virtual observation data of the same change event; Step S3-4, sorting the confidences of the different strategies, and selecting at least one strategy with a high confidence as a preferred strategy sequence for the change event; Step S3-5, storing the change event and the corresponding preferred strategy sequence into a strategy data set, In step S5, the preferred strategy sequence of the change event is calculated by calculating the current confidence level. And fine-tune the reasoning of the first observation data based on the conditional random field matrix, when the global reward of the environment is satisfied Action or status rewards When the current optimal strategy replaces the random optimal strategy As the optimal strategy, Where Beta(win,lose) is a variant algorithm of the self-guided WSLS, which adopts a win-lose game strategy. If the global reward of the environment reaches its expected level, the previous optimal strategy is repeated; otherwise, the strategy is adjusted accordingly according to the current state.

2. The autonomous learning method for efficiently constructing beyond-horizon global perception according to claim 1, Its characteristics are: Wherein, the step S2 includes the following operations: Merging the first observation data, the second observation data, and the third observation data at the same time into a set of merged data; Buffering and storing t groups of combined data at t consecutive moments as the simulated training data; Whenever a new set of the merged data is obtained, the oldest merged data in the simulated training data is removed and the new merged data is added.

3. The autonomous learning method for efficiently constructing beyond-horizon global perception according to claim 2 is characterized by: in, The data merging process in step S2 includes occasionally replaying empirical experiences to generate the simulated training data that best represents the overall environmental perception. The simulated training data is: {s0,u0,τ0,A 1 ,b1,s1,u1,τ1,A 2 ,b2,…,s t-1 ,u t-1 ,t t-1 ,A t ,b t }, Where s t-1 is the state of the virtual car at time t-1, u t-1 is the action of the real car at time t-1, τ t-1 is the action-observation history of the virtual car at time t-1, A t is the response action of the virtual car at time t, b t is the confidence level at time t.

4. The autonomous learning method for efficiently constructing beyond-horizon global perception according to claim 1 is characterized by: in, The generation of different strategies in step S3-3 includes the following process: in each change event t, based on the strategy π(u t |τ t ) Get the action u of the real car t : The action-observation history τ of the virtual car t Expressed as: τ t =(s1,a1,…,a t-1 ,s t ), the goal is to generate the different strategies within a bounded maximum period T To achieve cumulative rewards maximize: Where a t is the action of the virtual car at time t, s t is the state of the virtual car at time t, u t is the action of the real car at time t, γ is the discount coefficient, r t It is the environmental reward obtained by the virtual car for its action at time t.

5. The autonomous learning method for efficiently constructing beyond-horizon global perception according to claim 1 is characterized by: in, The generation of the preferred strategy sequence in step S3-4 includes the following process: Simulate training data as described on the timeline t ∈O(s,a), from which the cumulative reward is obtained The action-observation history τ of the virtual car t =(o0,u0,r1,…,r t-1 ,o t ) and execute A t (s,u t |τ t ) as the preferred strategy The following n-step Monte Carlo Markov transfer matrix structure is used, so that t=inf(n>0:X n ∈A) moment, there is as follows: Where s is the state of the virtual car, a is the action of the virtual car, r t-1 is the environmental reward obtained by the virtual car at time t-1, u t is the action of the real car at time t, Indicates that when The optimal strategy obtained by sampling Able to reach the global reward Q within a limited number of n steps n , and the process is independent of the state s, then the preferred strategy sequence is obtained.

Images