Adaptive control method based on curiosity mechanism and priority experience replay mechanism
By employing an adaptive control method that integrates multi-source perception data fusion and semantic feature extraction, the problem of poor adaptability of autonomous vehicles in dynamically changing scenarios is solved, enabling more efficient adaptive decision-making and task completion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2023-02-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing adaptive decision-making methods for autonomous vehicles rely on a large amount of shared perception data, resulting in low training efficiency, poor adaptability and stability, and particularly poor performance in dynamically changing scenarios.
An adaptive control method based on curiosity mechanism and priority experience replay mechanism is adopted. Multi-source sensing data is obtained by using multiple heterogeneous sensors, sensing feature vectors are extracted by convolutional neural network, and semantic feature fusion vectors are obtained by combining curiosity mechanism and priority experience replay mechanism. Finally, the decision model parameters are optimized by evaluation network and demonstration network.
It improves the autonomous vehicle's adaptability and task completion efficiency in dynamically changing scenarios, and enhances the success rate and stability in obstacle avoidance scenarios.
Smart Images
Figure CN116205272B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of adaptive decision-making and reinforcement learning for autonomous vehicles, and in particular to an adaptive control method for autonomous vehicles based on a curiosity mechanism and a priority experience replay mechanism. Background Technology
[0002] Autonomous vehicle adaptive learning is a sub-topic combining deep reinforcement learning with intelligent unmanned systems. It involves learning basic skills from a meta-task based on the autonomous decision-making of the vehicle, and then transferring these skills to new tasks to enable adaptive capabilities. Currently, mainstream methods include domain adaptation, domain randomization, and parameter tuning.
[0003] Domain adaptation maps the state information of virtual and real environments to a common latent variable space. In the virtual environment, the algorithm is trained using data from this mapped latent variable space. When transferring to the real environment, the perceived state in the real environment is similarly mapped to the latent variable space, allowing the model trained in the virtual environment to be directly transferred to the real-world scenario. Autonomous vehicles are trained in the source domain, and then the trained model is transferred to the target domain. This process requires carefully setting constraints to achieve good decision-making results.
[0004] Domain randomization involves randomizing visual information or physical parameters in a virtual environment. For example, in obstacle avoidance tasks, randomizing wall color, floor color, target color, ground friction, and light intensity allows autonomous vehicles to learn in dynamically changing scenarios.
[0005] Parameter tuning involves transferring the adaptive decision-making model trained in the virtual scenario to the real scenario. Based on the feedback from the autonomous vehicle's decision-making actions in the real scenario, relevant parameters are manually adjusted in a timely manner to optimize the decision-making actions.
[0006] In summary, current autonomous vehicle adaptive decision-making methods rely on a large amount of perception data, and this perception data is all from the same source with a limited field of view. Due to these problems, autonomous vehicles suffer from low training efficiency, poor adaptability to unseen and dynamically changing scenarios, and low stability. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an adaptive decision-making method for autonomous vehicles based on the curiosity mechanism and the priority experience playback mechanism.
[0008] The objective of this invention can be achieved through the following technical solutions:
[0009] An adaptive control method based on curiosity mechanism and priority experience replay mechanism is proposed. The control method uses multiple heterogeneous sensors to obtain multi-source perception data, inputs the perception data into a trained adaptive decision model, and the adaptive decision model outputs adaptive decisions to the autonomous vehicle.
[0010] The adaptive decision model includes the following components:
[0011] Convolutional neural networks, based on input perceptual data, obtain perceptual feature vectors;
[0012] The curiosity mechanism and the priority experience replay mechanism obtain semantic feature fusion vectors based on the input perceptual feature vectors, respectively.
[0013] The evaluation network and the demonstration network output adaptive decisions based on the input semantic feature fusion vectors, respectively.
[0014] Furthermore, the heterogeneous sensor includes vision, radar, and position sensors;
[0015] After obtaining multi-source perception data, the image dimension output by the vision sensor and the radar dimension of the radar data are preprocessed respectively; the position state information vector P of the autonomous vehicle is obtained from the position sensor data. w This includes the speed of the autonomous vehicle, the steering angle of the autonomous vehicle, the horizontal coordinate of the autonomous vehicle, the vertical coordinate of the autonomous vehicle, the horizontal coordinate of the target, and the vertical coordinate of the target.
[0016] Furthermore, the perceptual data is input into the convolutional neural network to obtain perceptual feature vectors, specifically including the following process:
[0017] Image feature extraction: The image data output from the vision sensor is input into a convolutional layer, then into a fully connected layer, and the output is an image feature vector I. w ;
[0018] Radar feature extraction: After passing the radar data through convolutional layers and fully connected layers, the radar feature vector R is output. w ;
[0019] Obtaining the perceptual feature vector: The perceptual fusion feature vector Z is obtained by adding the image, radar, and location feature vectors according to different weight parameters. total Its formula is:
[0020] Z total =F1×I w +F2×R W +F3×P W
[0021] Where F1, F2, F2 are weight parameters, P w This is a vector of location and state information.
[0022] Furthermore, the visual sensors include left, front, and right cameras deployed on the autonomous vehicle. The images from the three cameras are processed through convolutional layers and fully connected layers, ultimately outputting three feature vectors, namely the left feature vector I. wl Pre-eigenvectors I wf and right eigenvector I wr ;
[0023] The three feature vectors are combined into an image fusion feature vector I according to different weight values. wtotal Then I wtotal After convolutional and fully connected layers, the output image feature vector I is obtained. w .
[0024] Furthermore, the perceptual feature vectors are input into the curiosity mechanism and the priority experience replay mechanism, respectively. The curiosity mechanism and the priority experience replay mechanism are as follows:
[0025] Priority experience replay mechanism: The perceptual feature vector Z... total As the current state G t The input is fed into the experience replay pool; the experience replay pool is a collection of perceptual feature vectors, and then the state difference value S is obtained through TD-Error. error Its formula is:
[0026] S error =R t+1 +r×V(g t+1 )-V(G t )
[0027] Among them, R t+1 The reward value R is the value to be calculated for the next moment. The formula for the reward value R is set to represent successfully avoiding obstacles, reaching the designated location, and finding the target as completing the obstacle avoidance task. r is a hyperparameter, and V(G) is the reward value for the next moment. t+1 ) is the next moment G t+1 The evaluation value, V(G) t ) represents the current time G t The evaluation value, based on the state difference value S error Iteration V(G) t The expression is: V(G) t )←V(G t )+α×S error
[0028] Where α is the learning rate, V(x) is the value function, and the system includes convolutional layers, activation functions, and fully connected layers. G... t After being input into the convolutional layer, the signal passes through the ReLU activation function, then through a fully connected layer, and outputs the current time step G. t Evaluation value V(G) t );
[0029] According to V(G) t The semantic feature vector G is obtained by filtering out state vectors with high semantic features based on the magnitude of the values. v ;
[0030] Curiosity Mechanism: Fusing Perceptual Features into Vector Z total H, the current state vector t The feature vector b(H) is obtained by inputting it into a convolutional layer. t );
[0031] The eigenvector b(H) t ) and the action A at the current moment t =[speed,steer], are input together into the NET network, which consists of convolutional layers and fully connected layers, and outputs the predicted feature b′(H t+1 ) and predict action A' t speed is the speed of the autonomous vehicle, and steer is the steering angle of the autonomous vehicle;
[0032] The state H at the next moment t+1 The feature vector b(H) is obtained by inputting it into the convolutional layer. t+1 );
[0033] The predicted feature vector b′(H) t+1 ) and eigenvector b(H t+1 The internal reward value r was calculated. t The formula is:
[0034]
[0035] Where n>0, n is the normalization factor, and r t This represents the internal reward value at the current moment.
[0036] According to r t The semantic feature vector H is obtained by filtering out the state vectors with high semantic features based on the magnitude of the values. b ;
[0037] Semantic Feature Fusion Vector: Obtaining a semantic feature vector G from the perceptual feature vector through a priority experience replay mechanism and a curiosity mechanism. v and H b The formula for merging them is:
[0038] G total =e1×G v +e2×H b
[0039] Where e1 and e2 are hyperparameters, G total This is a semantic feature fusion vector.
[0040] Furthermore, the evaluation network and demonstration network are as follows:
[0041] Evaluation Network: Semantic Fusion Feature Vector G total The next state and internal reward value r t The input is fed into the evaluation network, and the output is the evaluation value using the current policy;
[0042] The evaluation network includes convolutional layers, Tanh activation functions, and fully connected layers;
[0043] Demonstration network: semantic feature vector G total The current state and predicted action are input into the demonstration network to obtain the action value and policy probability, respectively; the demonstration network includes convolutional layers, ReLU activation function and fully connected layers.
[0044] Furthermore, the training steps of the adaptive decision model include:
[0045] Construct training scenarios;
[0046] The adaptive decision-making model is trained based on the constructed training scenario;
[0047] Calculate the cumulative reward value output by the adaptive decision model;
[0048] When the cumulative reward value converges, the adaptive decision model is output.
[0049] Furthermore, the construction of the training scenario specifically includes the following process:
[0050] Task set initialization: Construct a task set D = {D1, D2, ..., D...} n}, where n is the total number of tasks; the set of subtasks is divided into multiple groups, including a group with fixed light intensity parameter values and a group with light intensity parameter values that change once every set time interval between 0 and 1, and a dynamic scene is constructed by the random change of light intensity parameter values;
[0051] Obstacle avoidance scenario construction: Construct the boundary of the obstacle avoidance scenario; the initial positions of the object, static obstacle, dynamic obstacle and target positions in the same group of subtasks are the same, and the positions of the object, static obstacle and dynamic obstacle in each round are randomly generated during training.
[0052] Furthermore, the cumulative reward value calculation step is as follows: set the learning rate h in the obstacle avoidance scenario. d Number of rounds e d Maximum number of steps D step Then calculate the maximum cumulative reward value R. dynamic The formula is:
[0053]
[0054] Among them, R j It is the sum of the reward value R obtained in the j-th round of training in the obstacle avoidance scenario, and step is the number of steps used to complete each round in the obstacle avoidance scenario;
[0055] The formula for the reward value R1 is set so that successfully avoiding obstacles, reaching the designated location, and finding the target represent the completion of the obstacle avoidance task.
[0056] Furthermore, the process of outputting an adaptive decision model when the cumulative reward value converges specifically includes the following steps:
[0057] Evaluation metric: Maximum cumulative reward value R in obstacle avoidance scenarios dynamic As the number of training rounds increases, its evaluation metric R... index If its value remains unchanged after a set number of rounds, it indicates that convergence has been achieved in the obstacle avoidance scenario, and the adaptive decision model is output. The reward evaluation index R is... index The formula is:
[0058]
[0059] in, This represents the reward value at the current moment in the obstacle avoidance scenario. This is the reward value for the next moment in the obstacle avoidance scenario.
[0060] Compared with the prior art, the present invention has the following beneficial effects:
[0061] 1) This invention obtains semantic fusion feature vectors through a curiosity mechanism and a priority experience replay mechanism. Then, it uses an evaluation network and a demonstration network to optimize and update the parameters of the autonomous vehicle's adaptive decision-making model, obtaining the optimal decision action to ensure the autonomous vehicle can efficiently complete tasks in dynamically changing scenarios. This method can be transferred to other unmanned systems such as unmanned surface vessels, drones, and submersibles, enabling them to make adaptive decisions in dynamically changing environments.
[0062] 2) Compared with other autonomous vehicle adaptive decision-making methods, the method described in this invention randomizes the training of each component in the scene and performs multi-task learning simultaneously, so that it has good adaptability. This solves the problem of poor adaptability of autonomous vehicles to unseen and dynamically changing scenes, thereby improving the success rate of autonomous vehicles in obstacle avoidance scenarios. Attached Figure Description
[0063] Figure 1 This is a schematic diagram illustrating the construction and training process of the autonomous vehicle adaptive decision-making method provided in this embodiment of the invention. Detailed Implementation
[0064] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0065] Example 1
[0066] This invention proposes an adaptive decision-making method for autonomous vehicles based on a curiosity mechanism and a priority experience replay mechanism. It constructs a complete perception environment by fusing data from homogeneous and heterogeneous sensors. Then, it extracts semantic fusion feature vectors of the autonomous vehicle using the curiosity mechanism and the priority experience replay mechanism, which are used as input to the decision network. Finally, it outputs the adaptive decision-making model of the autonomous vehicle through a demonstration network and an evaluation network. The main features of this invention are as follows: first, it constructs an obstacle avoidance training scenario for the autonomous vehicle; second, it extracts semantic feature fusion vectors of the autonomous vehicle using the curiosity mechanism and the priority experience replay mechanism; and third, it optimizes and updates the parameters of the autonomous vehicle's adaptive decision-making model through a demonstration network and an evaluation network.
[0067] like Figure 1 The diagram shown illustrates the construction and training flowchart of the autonomous vehicle adaptive decision-making method based on the curiosity mechanism and the priority experience replay mechanism provided in this invention. The specific implementation steps of this process include the following:
[0068] Step 1: Construct the training scenario for the autonomous vehicle. The specific process is as follows:
[0069] (1-1) Initialization of the autonomous vehicle task set: Construct an autonomous vehicle task set D = {D1, D2, ..., D...} using the Unity virtual engine. n}, where n is the total number of tasks and is even, and m1 and m2 are the number of subtasks in set D, with specific values shown in formula (1). The lighting intensity in the task is strong light, and the lighting intensity parameter WQ = 1. The illumination intensity parameter value in the task changes every 10 seconds, ranging from 0 to 1. WQ = 0, WQ = 0.5, and WQ = 1 represent weak light, medium-weak light, and strong light, respectively. A dynamically changing scene is constructed through the random changes in the illumination intensity parameter value. The number of dynamic sub-tasks in the obstacle scene is m1 = 5 and m2 = 6.
[0070]
[0071] Where n is the total number of tasks in the scene and is an even number equal to 10, m i Let i be the i-th subtask.
[0072] (1-2) Obstacle avoidance scenario construction: a(x a =0,ya =0), b(x) b =2,y b =0), c(x) c =2,y c =2), d(x) d =0,y d =2) Connect the four points in sequence to form an obstacle avoidance scenario with a square boundary, where a is the origin. and The initial positions of the autonomous vehicle, static obstacles, dynamic obstacles, and target are the same. During training, the positions of the autonomous vehicle, static obstacles, and dynamic obstacles are randomly generated in each round to improve the autonomous vehicle's adaptive capabilities. The initial position of the autonomous vehicle is P. D (X ca =1,Y ca =0), the target position is O D (X ta =1.5,Y ta =1), the positions of static obstacle G and static obstacle H are respectively D s1 (X obs1 =0,Y obs1 =0.5) and D s2 (X obs2 =1.8,Y obs2 =0), the positions of dynamic obstacle J and dynamic obstacle U are respectively D D1 (X dyn1 =1,Y dyn1 =0.5) and D D2 (X dyn2 =1.7,Y dy =0.5). The autonomous vehicle successfully avoids obstacles, reaches the designated location, and finds the target, indicating that the autonomous vehicle has completed the obstacle avoidance task. The reward function formula for the obstacle avoidance scenario is:
[0073]
[0074] Step 2: Obtain multi-source sensing data using vision, radar, and position sensors. The specific process is as follows:
[0075] The image output from the vision sensor is preprocessed to have a dimension of 84×84×3. Radar data is preprocessed to have a radar dimension of 402. Detailed position and status information of the autonomous vehicle is collected through the position sensor, with a dimension of P. w For 6, P w =[speed,steer,car] x ,car y ,target x ,target y[speed] is the speed of the autonomous vehicle, steer is the steering angle of the autonomous vehicle, and car x Let car be the x-axis of the driverless car. y The target is the longitudinal coordinate of the autonomous vehicle. x The x-coordinate of the target is . y The vertical axis represents the target coordinate.
[0076] Step 3: Input the perceptual data into the convolutional neural network to obtain the perceptual feature vector. The specific process is as follows:
[0077] (3-1) Image Feature Extraction: The image data from step 2 is input into two convolutional layers, then into one fully connected layer, outputting an 8-dimensional feature vector. Images from the left, front, and right cameras deployed on the autonomous vehicle are processed through two convolutional layers and one fully connected layer, respectively, finally outputting three 8-dimensional feature vectors, namely the left feature vector I. wl Pre-eigenvectors I wf and right eigenvector I wr The three feature vectors above are combined according to different weight values to form the image fusion feature vector I. wtotal Then I wtotal After three convolutional layers and one fully connected layer, the output image feature vector I is obtained. w Its dimension is 8. wtotal The formula is:
[0078] I wtotal =l1×I wl +l2×I wf +l3×I wr (3)
[0079] Where l1 = 0.5, l2 = 0.3, and l3 = 0.2 are weight parameters.
[0080] (3-2) Radar Feature Extraction: The radar data from step 3 is processed through two convolutional layers and one fully connected layer to output an 8-dimensional feature vector R. W .
[0081] (3-3) Obtaining the perception feature vector: The image, radar, and location feature vectors are added together according to different weight parameters to obtain the perception fusion feature vector Z. total Its formula is:
[0082] Z total =F1×I w +F2×R W +F3×P W (4)
[0083] Where F1 = 0.6, F2 = 0.2, and F2 = 0.2 are weight parameters.
[0084] Step 4: The perceptual feature vectors are input into the curiosity mechanism and the priority experience replay mechanism respectively to obtain the semantic feature fusion vector. The process is as follows:
[0085] (4-1) Priority Experience Replay Mechanism: The perceptual feature vector Z from step 3 is used as a reference. total As the current state G t The features are input into the experience replay pool. The experience replay pool is a collection of feature vectors with a maximum storage size of max-size. The size of the perceptual feature vectors stored in the experience replay pool each time is batch-size. Then, the state difference value S is obtained through TD-Error. error Its formula is:
[0086] S error =R t+1 +r×V(G t+1 )-V(G t (5)
[0087] Among them, R t+1 Let V(G) be the reward value for the next time step, r be a hyperparameter equal to 0.98, and V(G) be the reward value for the next time step. t+1 ) is the next moment G t+1 The evaluation value, V(G) t ) represents the current time G t The evaluation value. V(x) is the value function, which consists of convolutional layers, activation functions, and fully connected layers, and G... t After being input into a single convolutional layer, the signal passes through a ReLU activation function, followed by a fully connected layer, and outputs the current time step G. t Evaluation value V(G) t According to formula (5), the state difference value S error Iteration V(G) t The formula is:
[0088] V(G t )←V(G t )+α×S error (6)
[0089] Where α is the learning rate, which is 0.0003.
[0090] According to V(G) t The magnitude of the value filters out semantic state vectors G with high semantic features. v V(G) t The larger the value, the better the semantic feature vector G is obtained. v The more conducive it is to decision-making.
[0091] (4-2) Curiosity Mechanism: Fusing the perceptual features from step 3 into vector Z totalAs the current state H t The feature vector b(H) is obtained by inputting it into a convolutional layer. t ), and then the feature vector b(H) t ) and the action A at the current moment t =[speed,steer], are input together into the NET network, which consists of one convolutional layer and two fully connected layers, and outputs the predicted feature b′(H t+1 ) and predict action A' t The state H at the next moment. t+1 The feature vector b(H) is obtained by inputting it into the convolutional layer. t+1 ). b′(H t+1 ) and eigenvector b(H t+1 Input into the internal reward function r t In the formula, it is:
[0092]
[0093] Where n>0, n is the normalization factor equal to 0.95, and r t Let r be the internal reward value at the current moment. t The larger the value, the better the semantic feature vector H is obtained. b The higher the quality, the greater the internal reward value r. t The size of the semantic state vector H is used to filter out the semantic state vectors with high semantic features. b .
[0094] (4-3) Semantic Feature Fusion Vector: The perceptual feature vector is used to obtain the semantic feature vector G through a priority experience playback mechanism and a curiosity mechanism. v and H b The formula for merging them is:
[0095] G total =e1×G v +e2×H b (8)
[0096] Where e1 = 0.7 and e2 = 0.3 are hyperparameters, G total This is a fusion vector of comprehensive semantic features.
[0097] Step 5: The semantic feature fusion vector is input into the evaluation network and the demonstration network respectively to output the cumulative reward value. The specific process is as follows:
[0098] (5-1) Evaluate the network: Use the comprehensive semantic feature fusion vector G obtained in step 4 to evaluate the network. total The next state and reward R tThe input is fed into the evaluation network, which outputs the evaluation value using the current policy. The evaluation network consists of two convolutional layers, one Tanh activation function, and two fully connected layers. This network structure outputs the evaluation value of the current policy.
[0099] (5-2) Demonstration Network: The comprehensive semantic feature fusion vector G obtained in step 4 is used to demonstrate the network. total The current state and predicted action input are used to obtain the action value and policy probability in the demonstration network, respectively. The demonstration network consists of 2 convolutional layers, 1 ReLU activation function, and 2 fully connected layers.
[0100] (5-3) Loss Function: Input the current state and reward value R into the mean squared loss function, and output the loss value. The input data dimensions must be consistent. The formula is as follows:
[0101] Loss(x,y)=(xy) 2 (9)
[0102] Where x is the current state G at the input time. total y is the reward value R1 at the current moment.
[0103] (5-4) Cumulative reward value: Set the learning rate h in the obstacle avoidance scenario of the autonomous vehicle. d =0.0003, number of rounds e d =1000, maximum number of steps D step =200, then calculate the maximum cumulative reward value R. dynamic The formula is:
[0104]
[0105] Among them, R j It is the sum of the reward values R obtained by the autonomous vehicle in the j-th round of training in the obstacle avoidance scenario, and step is the number of steps taken by the autonomous vehicle to complete each round in the obstacle avoidance scenario.
[0106] Step 6: When the cumulative reward value converges, output the autonomous vehicle adaptive decision model. The specific process is as follows:
[0107] Evaluation metric: Maximum cumulative reward value R in obstacle avoidance scenarios dynamic As the number of training rounds increases, its evaluation metric R... index If its value remains unchanged after 10 rounds, it indicates that the autonomous vehicle has converged in the obstacle avoidance scenario, and the autonomous vehicle adaptive decision-making model is output. Reward evaluation metric. The formula is:
[0108]
[0109] in, This represents the reward value at the current moment in the obstacle avoidance scenario. The reward for the next moment in the obstacle avoidance scenario is calculated using equation (2).
[0110] Experimental Description and Results: The experimental scenario was the obstacle avoidance scenario described in step 1. The verification scenario was an obstacle avoidance verification scenario that the autonomous vehicle had never encountered before. Table 1 shows the comparison results between the method of this invention and other benchmark methods in the obstacle avoidance verification scenario. There are two evaluation metrics: success rate and number of collisions. The success rate is calculated by dividing the number of successful tasks by the total number of tasks and then multiplying by the percentage. A successful task indicates that the vehicle finds the target quickly without colliding with any object. The number of collisions indicates the number of times the autonomous vehicle collides with surrounding walls, static obstacles, and dynamic obstacles during the task execution.
[0111] The autonomous vehicle adaptive decision-making method based on curiosity and priority experience replay mechanisms is the approach of this invention. It constructs a complete perception environment by fusing data from homogeneous and heterogeneous sensors, obtains semantic feature vectors through curiosity and priority experience replay mechanisms, and optimizes the autonomous vehicle's decision-making actions using evaluation and demonstration networks to improve its adaptive capabilities. MAML stands for Model-Free Meta-Learning, enabling rapid learning of new tasks with limited sample data; EPG stands for Evolutionary Policy Gradient Method, which learns a loss function to achieve rapid learning in new tasks; RL... 2 PEARL represents rapid learning through slow reinforcement learning, followed by gaining experience using an RNN neural network; PEARL represents probabilistic embedding for demonstration evaluation, using hidden layer features to infer the task; PPO represents proximal policy optimization, designing a new objective function and updating small batches of data.
[0112] Table 1. Experimental Results (Average results from 1000 tests)
[0113]
[0114]
[0115] It can be seen that the present invention has the highest success rate and the fewest collisions in obstacle avoidance scenarios, with an average success rate of 75%, which is 30 percentage points, 14 percentage points, 20 percentage points, 40 percentage points and 30 percentage points higher than other methods, respectively.
[0116] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. An adaptive control method based on curiosity mechanism and priority experience replay mechanism, characterized in that, The control method utilizes multiple heterogeneous sensors to obtain multi-source perception data, inputs the perception data into a trained adaptive decision-making model, and the adaptive decision-making model outputs adaptive decisions to the autonomous vehicle. The adaptive decision model includes the following parts: Convolutional neural networks, based on input perceptual data, obtain perceptual feature vectors; The curiosity mechanism and the priority experience replay mechanism, respectively, obtain semantic feature fusion vectors based on the input perceptual feature vectors, as specifically implemented as follows: Priority experience replay mechanism: This mechanism uses perceptual feature vectors... As the current state The input is fed into the experience replay pool; the experience replay pool is a collection of perceptual feature vectors, and then the state difference value is obtained through TD-Error. Its formula is: in, The reward value is the value at the next moment. The formula is set to indicate the completion of the obstacle avoidance task by successfully avoiding obstacles, reaching the designated location, and finding the target. For hyperparameters, For the next moment The evaluation value, For the current moment The evaluation value, based on the state difference value iteration The expression is: in, For learning rate, Value functions include convolutional layers, activation functions, and fully connected layers. After being input into the convolutional layer, it passes through The activation function, passed through a fully connected layer, outputs the current time step. Evaluation value ; according to The semantic feature vector is obtained by filtering out the state vectors with high semantic features based on the magnitude of the values. ; Curiosity Mechanism: Fusing Perceptual Features into Vectors As the current state vector The feature vector is obtained by inputting it into a convolutional layer. ; eigenvectors and the action at the current moment , input together In the middle, the said The network consists of convolutional layers and fully connected layers, and outputs... and predicting actions , For the speed of driverless cars, This refers to the steering angle of the driverless car. The state at the next moment The feature vector is obtained by inputting into the convolutional layer. ; Will and eigenvectors Calculate the internal reward value The formula is: = in, >0, As a normative factor, This represents the internal reward value at the current moment. according to The semantic feature vector is obtained by filtering out the state vectors with high semantic features based on the magnitude of the values. ; Semantic Feature Fusion Vector: Obtaining semantic feature vectors from perceptual feature vectors through a priority experience replay mechanism and a curiosity mechanism. and The formula for merging them is: in, , For hyperparameters, This is a semantic feature fusion vector; The evaluation network and the demonstration network output adaptive decisions based on the input semantic feature fusion vectors, respectively.
2. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 1, characterized in that, The heterogeneous sensors include vision, radar, and position sensors; After obtaining multi-source perception data, the image dimension output by the vision sensor and the radar dimension of the radar data are preprocessed respectively; the position state information vector of the autonomous vehicle is obtained from the position sensor data. This includes the speed of the autonomous vehicle, the steering angle of the autonomous vehicle, the horizontal coordinate of the autonomous vehicle, the vertical coordinate of the autonomous vehicle, the horizontal coordinate of the target, and the vertical coordinate of the target.
3. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 2, characterized in that, The perceptual data is input into the convolutional neural network to obtain the perceptual feature vector, specifically including the following process: Image feature extraction: The image data output from the vision sensor is input into a convolutional layer, then into a fully connected layer, and the output is an image feature vector. ; Radar feature extraction: Radar data is processed through convolutional and fully connected layers to output radar feature vectors. ; Obtaining the perceptual feature vector: The perceptual fusion feature vector is obtained by adding the image, radar, and location feature vectors according to different weight parameters. Its formula is: = in, , , For weight parameters, This is a vector of location and state information.
4. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 3, characterized in that, The visual sensors include cameras deployed on the left, front, and right sides of the autonomous vehicle. Images from the three cameras are processed through convolutional and fully connected layers, ultimately outputting three feature vectors, namely the left feature vector. Pre-eigenvectors and right eigenvectors ; The three feature vectors are combined into an image fusion feature vector according to different weight values. Then After convolutional and fully connected layers, the output image feature vector is obtained. .
5. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 1, characterized in that, The evaluation network and demonstration network are as follows: Evaluation Network: Integrating Semantic Feature Vectors The next state and internal reward value The input is fed into the evaluation network, and the output is the evaluation value using the current policy; The evaluation network includes convolutional layers, Tanh activation functions, and fully connected layers; Demonstration Network: Integrating semantic feature vectors The current state and predicted action input are used to obtain action values and policy probabilities in the demonstration network; the demonstration network includes convolutional layers, Activation functions and fully connected layers.
6. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 1, characterized in that, The training steps for the adaptive decision-making model include: Construct training scenarios; The adaptive decision-making model is trained based on the constructed training scenario; Calculate the cumulative reward value output by the adaptive decision model; When the cumulative reward value converges, the adaptive decision model is output.
7. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 6, characterized in that, The construction of the training scenario specifically includes the following process: Task set initialization: Building the task set , The total number of tasks is defined as follows: The set of subtasks is divided into multiple groups, including a group with fixed light intensity parameter values and a group with light intensity parameter values that change once every set time interval between 0 and 1. Dynamic scenes are constructed by randomly changing the light intensity parameter values. Obstacle avoidance scenario construction: Construct the boundary of the obstacle avoidance scenario; the initial positions of the object, static obstacle, dynamic obstacle and target positions in the same group of subtasks are the same, and the positions of the object, static obstacle and dynamic obstacle in each round are randomly generated during training.
8. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 6, characterized in that, The cumulative reward value calculation steps are as follows: setting the learning rate in the obstacle avoidance scenario. Number of rounds Maximum number of steps Then calculate the maximum cumulative reward value. The formula is: in, It is the first training in obstacle avoidance scenarios. Reward value per round The sum of, It is the number of steps used to complete each round in an obstacle avoidance scenario; The reward value The formula is set to indicate the completion of the obstacle avoidance task by successfully avoiding obstacles, reaching the designated location, and finding the target.
9. The adaptive control method based on curiosity mechanism and priority experience playback mechanism according to claim 6, characterized in that, When the cumulative reward value converges, the adaptive decision model is output, which specifically includes the following process: Evaluation metric: Maximum cumulative reward value in obstacle avoidance scenarios As the number of training rounds increases, its evaluation metrics If its value remains unchanged after a set number of rounds, it indicates that convergence has been achieved in the obstacle avoidance scenario, and the adaptive decision model is output, with the reward evaluation index as the output. The formula is: in, This represents the reward value at the current moment in the obstacle avoidance scenario. This is the reward value for the next moment in the obstacle avoidance scenario.