An Evaluation Network Design Method for a Reinforcement Learning-Based Regulation Algorithm for Unmanned Clamping Vehicles

Abstract

This invention discloses an evaluation network design method for a reinforcement learning-based regulation and control algorithm for unmanned gripper vehicles. This invention accelerates the convergence of the reinforcement learning regulation and control algorithm and reduces trial-and-error exploration time by incorporating high-quality offline positive samples into the training process of the evaluation network, thereby improving operational efficiency. Simultaneously, a dedicated evaluation network fusion layer is constructed, dynamically adjusting weights based on the credibility of offline positive samples and online training, fusing offline positive sample information with online training features to fully explore effective operational strategies, enhance the accuracy and stability of the evaluation network, and ultimately achieve a significant improvement in the performance of the entire reinforcement learning regulation and control algorithm, effectively promoting the efficient and precise operation of unmanned gripper vehicles in complex logistics environments.

Description

Technical Field

[0001] This invention relates to the field of unmanned gripper vehicle control technology, and in particular to an evaluation network design method for a reinforcement learning control algorithm for unmanned gripper vehicles, so as to improve the control accuracy and efficiency of unmanned gripper vehicles in logistics operation scenarios. Background Technology

[0002] In today's era, the logistics industry is experiencing rapid and vigorous development, and the efficiency and accuracy of cargo handling have become core competitive factors among companies in the industry. Unmanned cargo handling vehicles, as key equipment in the logistics automation system, play an extremely important role in cargo handling operations and transportation processes.

[0003] Currently, there are many innovative patent achievements in the field of autonomous driving technology. "Autonomous Driving Method and System Based on Offline Reinforcement Learning and Teacher-Student Intelligent Agents" (Publication No.: CN118915537A) collects real traffic driving data to construct a dataset, and uses imitation learning to train student intelligent agents to achieve autonomous driving; "A Control Method and Device for Unmanned Driving Equipment" (Publication No.: CN114167857B) uses a pre-trained autoencoder to extract decoupled environmental features, inputs them into a reinforcement learning decision model, and outputs a decision.

[0004] However, traditional reinforcement learning algorithms for unmanned forklift truck control suffer from numerous problems. In complex logistics environments, cargo layout, warehouse structure, and operational requirements are highly variable, generating a massive number of state and action combinations. Relying solely on traditional online reinforcement learning requires the agent to undergo extensive trial and error, consuming significant time and computational resources, hindering its efficient application and rapid deployment. Existing reinforcement learning training methods still have shortcomings, failing to adequately utilize high-quality offline positive samples from unmanned forklift truck operations. In actual operations, high-quality samples obtained from human expert operations or specific collections contain valuable strategies and represent the culmination of experience, such as obstacle avoidance techniques for experienced forklift operators handling specific goods and their experience in clamping fragile items. However, traditional algorithms often overlook or poorly integrate these strategies, limiting algorithm convergence and overall performance improvement, becoming a bottleneck for development. Summary of the Invention

[0005] The purpose of this invention is to address the technical shortcomings of existing technologies, such as slow learning processes and insufficient utilization of high-quality offline positive samples, by providing an evaluation network design method for reinforcement learning control algorithms for unmanned gripper vehicles.

[0006] The technical solution adopted to achieve the purpose of this invention is:

[0007] A method for designing an evaluation network for a reinforcement learning-based regulation algorithm for unmanned vehicles with a gripper, comprising the following steps:

[0008] Step 1: Analyze and determine the unmanned gripper truck operation scenario S, and determine the data type D and the importance weight w of the data type under different operation scenarios;

[0009] Step 2: Based on the operation scenario S and data type D determined in Step 1, collect offline positive samples of the unmanned gripper vehicle through multiple channels, merge them to form a total offline positive sample set P, and preprocess the offline positive sample set P.

[0010] Step 3, for each sample p in the preprocessed offline positive sample set P i After cleaning and normalization, we get p i clean Extract each sample Feature vector The combined feature vector of the offline positive samples is obtained.

[0011] Step 4, Building the Evaluation Network: A multi-layer fully connected neural network is used to build the basic architecture of the evaluation network. The input layer, hidden layer, fusion layer, and output layer are designed sequentially. The fusion layer employs a fusion strategy to combine the feature vectors of offline positive samples. and the online new sample feature vector output by the learning regulation algorithm The fusion is performed to obtain the fused feature vector.

[0012] Step 5: Use the offline positive sample feature vector from Step 3. Pre-train the evaluation network, select and initialize the reinforcement learning control algorithm, update the evaluation network parameters, and then use fused feature vectors. The evaluation network is trained online, and a loss function is constructed using the Adam optimizer for backpropagation. The error E between the state value predicted by the evaluation network for each new online sample and the actual reward obtained by the unmanned gripper is calculated. err We continuously monitor the prediction ability of the evaluation network after training, and monitor the convergence of the reinforcement learning regulation algorithm by calculating the entropy H(π) of the policy function, thus providing value assessment and policy guidance for the reinforcement learning regulation algorithm.

[0013] In the above technical solution, step 1 involves analyzing the unmanned gripper vehicle's operating scenarios: the set of operating scenarios for the unmanned gripper vehicle is S = {s1, s2, ..., s...}. m}, where each scene s i Including cargo type G i Warehouse layout L i and task requirements T i , i = 1, 2, ..., m.

[0014] The data type set D = {d1, d2, ..., d...} under different work scenariosr}, where d k The data type is represented by k = 1, 2, ..., r, and the data type includes operation parameter data d1, environmental parameter data d2, and cargo data d3.

[0015] The operational parameter data d1 includes the position (x, y, z) of the clamping vehicle and its speed v = (v... x ,v y ,v z The clamping vehicle state θ is d1 = ((x,y,z),v,θ); the environmental parameter data d2 includes temperature T, humidity H and light intensity I, i.e., d2 = (T,H,I); the cargo data d3 includes cargo information G, which includes the weight W and dimensions (a,b,c) of the cargo, i.e., d3 = (W,(a,b,c)).

[0016] In the above technical solution, in step 1, the set of weight vectors for different types of data under different work scenarios is w = (w1, w2, ..., w r ).

[0017] In the above technical solution, step 2 specifically includes the following steps:

[0018] Step 2.1: Collect offline positive samples of the unmanned gripper vehicle from the following three aspects:

[0019] Collect offline positive samples of human-operated autonomous gripper vehicles: the set H = {h1, h2, ..., h n1}, where each sample h in set H j (j = 1, 2, ..., n1) includes the job scenarios. Operation sequences of data types corresponding to the job scenario Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, where l = 1, 2, ..., k;

[0020] Collect offline positive samples from a simulated environment: its set M = {m1, m2, ..., m} n2}, where each sample m in set M i Including work scenarios and the corresponding operation sequence Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, where r = 1, 2, ..., q;

[0021] Collect offline positive samples from similar devices: their set E = {e1, e2, ..., e} n3}, where each sample e in set E t Including work scenarios and the corresponding operation sequence Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, u = 1, 2, ..., p;

[0022] Merge the three types of offline positive samples into a total offline positive sample set P = H∪M∪E = {p1, p2, ..., p n}, where n = n1 + n2 + n3, and each sample in the total offline positive sample set P. For the work scenario, For the corresponding operation sequence, operation sequence Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, where l = 1, 2, ..., k;

[0023] Step 2.2 involves preprocessing the total offline positive sample set P, including checking the integrity of the samples to ensure that each sample contains a complete scene. and operation sequence Mark or perform preliminary repairs on missing or erroneous data in the sample.

[0024] In the above technical solution, in step 3, for each sample p in the offline positive sample set P... i After cleaning and normalizing the data (i = 1, 2, ..., n), we obtain the cleaned and normalized data. Use a feature extraction function to extract the features of each sample. The eigenvectors in the data, i.e.: The extracted feature vector, d is the dimension of the feature vector, and the feature vectors of all data are... The offline positive sample feature vectors are obtained by merging the samples.

[0025] In the above technical solution, in step 4, the number N of input layer nodes is determined according to the operating status of the unmanned gripper vehicle. inputAs a guide for evaluating the network's training, the job state includes position (x, y, z) and velocity v = (v... x ,v y ,v z The clamping vehicle state θ and cargo information G are given, where the position is represented by three-dimensional coordinates, the velocity by a three-dimensional vector, the clamping vehicle state θ by m parameters, and the cargo information G by n features. Then N input =3 + 3 + m + n;

[0026] The hidden layer has L layers, and the number of neurons in the l-th layer of L layers is N. l And N1>N2>…>N L Let the input of the l-th layer be... The output is The weight matrix is ​​W l The bias vector is B l The calculation process for the l-th layer is as follows: That In the middle, f ReLU (x) = max(0,x) is the ReLU activation function, which takes the larger of 0 and x. The The output layer has only one node, used to estimate the state value V of the output layer. Among them, W L+1 B is the weight vector of the output layer. L+1 For the bias term of the output layer, add a constant bias to the neurons in the output layer. This is the output vector of the last hidden layer.

[0027] In the above technical solution, in step 4, the fusion strategy is a weighted summation fusion strategy or a concatenation fusion strategy. When it is a weighted summation fusion strategy, the offline positive sample feature vector is... and online new sample feature vector (Real-time online collection of new samples during the clamping vehicle's operation {p new1 ,p new2 ,…,p newq}, the and All include state vectors and action vectors The online training feature vector is obtained by preprocessing and extracting the data in the same way as the offline positive samples, and then fused along the dimensionality to obtain the fused feature vector. Where w1 and w2 are the fusion weights, and the fusion weights are based on the offline positive sample confidence C. off And online sample credibility C online Sure: w2 = 1 - w1, where ∈ is a small constant to prevent the denominator from being zero, C off =β1C src +β2C val Csrc To assess the reliability of the sample source, C val To determine the validation confidence of the sample in the simulated environment, β1 and β2 are weighting coefficients, and C... online This is obtained by calculating the fluctuation level of the value function of the current fusion strategy;

[0028] When using a splicing and fusion strategy, the offline positive sample feature vectors will be... and online new sample feature vector Concatenate the vectors along the dimensional lines to obtain the feature vectors. Then, the concatenated feature vector is processed by a fully connected layer (FC). Mapping to the appropriate dimension yields the fused feature vector.

[0029] In the above technical solution, the weight matrix and bias vector of the fully connected neural network can be determined through training. Let the output of the fusion layer be... The input for evaluating subsequent layers of the network is then... Continue with the forward propagation calculations for the network.

[0030] In the above technical solution, in step 5, the preprocessed offline positive samples are divided into a training set T. off ={p t1 ,p t2 ,…,p tm} and validation set V off ={p v1 ,p v2 ,…,p vk};

[0031] The evaluation network is trained using the training set, and the loss on the training set is calculated: Where m is the number of samples in the training set, V NN (p ti To evaluate the network's performance on each sample p in the offline positive sample training set ti State value, V true (p ti For each sample p in the offline positive sample training set ti The known state values ​​are given by i = 1, 2, ..., m; the state values ​​are obtained by transforming the feature vectors of offline positive samples. The input is a multi-layer fully connected neural network. After forward propagation and processing through hidden layers, the output layer outputs an estimate of the state value for each sample p. ti The prediction of its value.

[0032] After each training epoch, the network performance is evaluated using the validation set, and the loss on the validation set is calculated:

[0033]

[0034] Where k is the number of samples in the validation set, V NN (p vj To evaluate the network's performance on each sample p in the offline positive sample validation set vj State value, V true (p vj For each sample p in the offline positive sample training set vj The known state values ​​in the training set, where j = 1, 2, ..., k; when the loss on the validation set no longer decreases or reaches the preset number of training rounds N. max At that time, pre-training was completed.

[0035] In the above technical solution, in step 5, the update formula for the Adam optimizer is: Where, θ t To evaluate network parameters, η is the network learning rate. and For the estimation of the first and second moments of the corrected gradient, ∈ is a small constant to prevent the denominator from being zero;

[0036]

[0037] Where, m t For first-order moment estimation, m t =β1m t-1 +(1-β1)g t β1 is the decay rate estimated by the first moment; v t For second-order moment estimation, β² is the decay rate estimated by the second moment, g t To evaluate network parameters θ t gradient,

[0038] Fuse feature vectors using step 4. The network is trained and evaluated online, employing a loss function based on policy gradient combined with value calculation error. in, R represents the expectation. t Let γ be the reward at time t, and γ be the discount factor. Let be the state vector at time t. Let V be the action vector at time t, and V(·) be the state value. The policy function is defined, and the loss is propagated through the Adam optimizer to update the parameters of the evaluation network.

[0039] The error The network's predictive ability is continuously monitored and evaluated using this error. To evaluate the state vector of each sample in the new online samples predicted by the network, R new,i The reward for each sample in the given online new samples is calculated by determining the entropy of the policy function. The changes in [the parameters] are used to regulate the convergence of the reinforcement learning control algorithm.

[0040] In the above technical solution, based on the monitoring results, in each online training iteration, the confidence level C of the offline positive samples is determined. off And the credibility of online training C online Adjust the fusion weights w1 and w2 in the fusion of offline positive samples and online new samples.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] This invention collects and deeply processes offline positive samples from multiple sources to construct an evaluation network that integrates their features with online training features. It adopts a method of pre-training followed by dynamic online training to comprehensively optimize data utilization, network architecture, and training process. This significantly improves the accuracy of the evaluation network, accelerates reinforcement learning convergence, and enhances adaptability to complex scenarios. It effectively reduces operational errors and improves operational efficiency. In the application of unmanned gripper trucks in logistics operations, it demonstrates superior performance and competitive advantages, and powerfully promotes the progress and development of logistics automation technology. Attached Figure Description

[0043] Figure 1 This is a flowchart of the design method of the present invention.

[0044] Figure 2 This is a structural diagram of the design method of the present invention. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0046] An evaluation network design method for reinforcement learning regulation algorithms using offline positive samples includes the following steps:

[0047] like Figure 1 and Figure 2 As shown, in step 1, analyze and determine the unmanned gripper vehicle operation scenario S, and determine the data type D and the importance weight w of the data type under different operation scenarios;

[0048] Specifically, the analysis of unmanned gripper vehicle operation scenarios: The set of operation scenarios for the unmanned gripper vehicle is S = {s1, s2, ..., s...} m}, where each job scenario s i Including cargo type G i Warehouse layout Li and task requirements T i Specifically, G i This includes feature vector representations of the goods, such as shape, weight, and volume, i.e., G. i =(g i1 ,g i2 ,…,g in One of its features is g. ij (j = 1, 2, ..., n) represents the j-th feature of the goods in the i-th scenario; L i This includes information such as warehouse dimensions, shelf locations, and aisle information, i.e., L i =(l i1 ,l i2 ,…,l ip );T i This includes requirements such as the destination and sequence of transport, i.e., T i =(t i1 ,t i2 ,…,t iq ), then for each job scenario s i =(G i ,L i ,T i ), where i = 1, 2, ..., m.

[0049] The data type set D = {d1, d2, ..., d...} under different work scenarios r}, where d k The data type is represented by k = 1, 2, ..., r, and the data type includes operation parameter data d1, environmental parameter data d2, and cargo data d3.

[0050] The operational parameter data d1 includes the position (x, y, z) of the clamping vehicle and its speed v = (v... x ,v y ,v z The clamping vehicle state θ is d1 = ((x,y,z),v,θ); the environmental parameter data d2 includes T, humidity H and light intensity I, i.e., d2 = (T,H,I); the cargo data d3 includes cargo information G, which includes the weight W and dimensions (a,b,c) of the cargo, i.e., d3 = (W,(a,b,c)).

[0051] For example, in this embodiment, the importance weight of the data: in the scenario of handling fragile items s fragile Below, the clamping force data d F Importance weight w F It may be higher, especially in narrow passage driving scenarios. narrow Below, position and velocity data d (x,y,z,v) weight w (x,y,z,v)It might be too high. You could use a vector w = (w1, w2, ..., w...). r The vector represents the weights of different data types in different scenarios. The specific values ​​of the weight vectors depend on the degree of influence of each data type on the operation of the unmanned gripper vehicle. Operational parameter data includes parameters such as the gripper vehicle's position, speed, and status. These parameters directly affect the gripper vehicle's operational performance and safety, and therefore are assigned high weights. Environmental parameter data includes parameters such as temperature, humidity, and light intensity. These parameters affect the performance of the gripper vehicle's sensors and the comfort of the operating environment, but their direct impact is not as significant as that of the operational parameters, and therefore they are assigned medium weights. Cargo data refers to cargo information G, which includes parameters such as the weight and dimensions of the cargo. These parameters determine the size and weight of the load that the gripper vehicle needs to handle, and have a direct impact on the gripper vehicle's operation, therefore they are assigned high weights.

[0052] Step 2: Based on the operation scenario S and data type D determined in Step 1, collect offline positive samples of the unmanned gripper vehicle through multiple channels, merge them to form a total offline positive sample set P, and preprocess the offline positive sample set P, specifically including the following steps:

[0053] Step 2.1: Collect offline positive samples of the unmanned gripper vehicle from the following three aspects:

[0054] Collect offline positive samples of human-operated autonomous gripper vehicles: the set H = {h1, h2, ..., h n1}, where each sample h in set H j (j = 1, 2, ..., n1) includes job scenario information. Operation sequences of data types corresponding to the job scenario Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, where l = 1, 2, ..., k;

[0055] Collect offline positive samples from a simulated environment: its set M = {m1, m2, ..., m} n2}, where each sample m in set M i Including work scenarios and the corresponding operation sequence Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, where r = 1, 2, ..., q;

[0056] Collect offline positive samples from similar devices: their set E = {e1, e2, ..., e}n3}, where each sample e in set E t Including work scenarios and the corresponding operation sequence Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, u = 1, 2, ..., p;

[0057] Merge the three types of offline positive samples into a total offline positive sample set P = H∪M∪E = {p1, p2, ..., p n}, where n = n1 + n2 + n3, and each sample in the total offline positive sample set P. For the work scenario, For the corresponding operation sequence, operation sequence Operation sequence Each operation in in, This is the operation performed by the clamping vehicle. This is the result or reward corresponding to the operation, where l = 1, 2, ..., k.

[0058] Step 2.2 involves preprocessing the total offline positive sample set P, including checking the integrity of the samples to ensure that each sample contains a complete scene. and operation sequence Mark or perform preliminary repairs on missing or erroneous data in the sample.

[0059] Step 3, for each sample p in the preprocessed offline positive sample set P i After cleaning and normalizing the data (i = 1, 2, ..., n), we obtain the cleaned and normalized samples. Use a feature extraction function to extract the features of each sample. The eigenvectors in the data, i.e.: The extracted feature vector, d is the dimension of the feature vector, and the feature vectors of all data are... The offline positive sample feature vectors are obtained by merging the samples.

[0060] Specifically, data cleaning: the normal position range is [x min ,x max ]×[y min ,y max ]×[z min ,z max If xpij <x min or x pij >x max If the data at that location is outlier, it should be corrected or removed. For example, for a velocity v = (v x ,v y ,v z Let its normal speed range be [v] xmin ,v xmax ]×[v ymin ,v ymax ]×[v zmin ,v zmax Similarly, abnormal speed values ​​are handled. The samples after data cleaning are... Data normalization: For location (x, y, z), let the maximum size of the warehouse be (X...). max ,Y max Z max If ), then the normalized location data is For velocity v = (v x ,v y ,v z Let its mean be ). Standard deviation is The normalized speed is

[0061]

[0062] Step 4: A multi-layer fully connected neural network is used to build the evaluation network infrastructure. The input layer, hidden layer, fusion layer, and output layer are designed sequentially. The number of nodes N in the input layer is determined based on the operating status of the unmanned gripper vehicle. input As a guide for evaluating the network's training, the job state includes position (x, y, z) and velocity v = (v... x ,v y ,v z The clamping vehicle state θ and cargo information G are given, where the position is represented by three-dimensional coordinates, the velocity by a three-dimensional vector, the clamping vehicle state by m parameters, and the cargo information by n features. Then N input =3 + 3 + m + n.

[0063] The design evaluation network has L hidden layers, and the number of neurons in the l-th layer of L layers is N. l And N1>N2>·>N L Let the input of the l-th layer be... The output is The weight matrix is ​​W l The bias vector is B l The calculation process for the l-th layer is as follows: Among them, f ReLU(x) = max(0,x) is the ReLU activation function, which takes the larger of 0 and x. The output layer has only one node, used to output the state value estimate V of the output layer. Among them, W L+1 B is the weight vector of the output layer. L+1 For the bias term of the output layer, add a constant bias to the neurons in the output layer. This is the output vector of the last hidden layer.

[0064] The fusion layer employs a fusion strategy to combine the feature vectors of offline positive samples. The learning-based control algorithm outputs online new sample feature vectors. The fusion is performed to obtain the fused feature vector.

[0065] Specifically, the fusion strategy is a weighted summation fusion strategy or a concatenation fusion strategy. The weighted summation fusion strategy involves combining the offline positive sample feature vectors... and online new sample feature vector (Real-time online collection of new samples during the clamping vehicle's operation {p new1 ,p new2 ,…,p newq}, the and All include state vectors and action vectors The feature vector of the new online sample is obtained by preprocessing it in the same way as the offline positive sample and extracting it. Then, it is fused in dimensionality to obtain the fused feature vector. Where w1 and w2 are the fusion weights, and the fusion weights are based on the offline positive sample confidence C. off And online sample credibility C online Sure: w2 = 1 - w1, where ∈ is a small constant to prevent the denominator from being zero, C off =β1C src +β2C val C src To assess the reliability of the sample source, C val To determine the validation confidence of the sample in the simulated environment, β1 and β2 are weighting coefficients, and C... online This is obtained by calculating the degree of fluctuation in the value function of the current fusion strategy.

[0066] Specifically, the splicing and fusion strategy will combine the offline positive sample feature vectors and online new sample feature vector Concatenate the vectors along the dimensional lines to obtain the feature vectors. Then, the concatenated feature vector is processed by a fully connected layer (FC). Mapping to the appropriate dimension yields the fused feature vector.

[0067] In this embodiment, the weight matrix and bias vector of the fully connected neural network can be determined through training to extract information from the input vector and estimate the state value. Let the output of the fusion layer be... The input for evaluating subsequent layers of the network is then... Continue with the forward propagation calculations for the network.

[0068] Step 5: Use the offline positive sample feature vector from Step 3. The evaluation network is pre-trained, and a loss function is constructed. The loss is backpropagated using the Adam optimizer. A reinforcement learning regulation algorithm is selected and initialized (the PPO algorithm is selected in this embodiment). The evaluation network parameters are updated, and then the fused feature vectors are used. The evaluation network is trained online, and a loss function is constructed using the Adam optimizer for backpropagation. The error E between the state value of each sample predicted by the evaluation network in new online samples and the actual reward obtained by the unmanned gripper is calculated. err We continuously monitor the prediction ability of the evaluation network after training, and monitor the convergence of the reinforcement learning regulation algorithm by calculating the entropy H(π) of the policy function, thus providing value assessment and policy guidance for the reinforcement learning regulation algorithm.

[0069] Specifically, the preprocessed offline positive samples are divided into a training set T. off ={p t1 ,p t2 ,…,p tm} and validation set V off ={p v1 ,p v2 ,…,p vk The ratio of the two is 8:2; the evaluation network is trained using the training set, and the loss on the training set is calculated: Where m is the number of samples in the training set, V NN (p ti To evaluate the network's performance on each sample p in the offline positive sample training set ti State value, V true (p ti For each sample p in the offline positive sample training set ti The known state values ​​are denoted as , where i = 1, 2, ..., m; the state values ​​are obtained by transforming the feature vectors of offline positive samples. The input is a multi-layer fully connected neural network. After forward propagation and processing through hidden layers, the output layer outputs an estimate of the state value, which is the value of the given state p. ti The prediction of its value.

[0070] After each training epoch, the network performance is evaluated using the validation set, and the loss on the validation set is calculated: Where k is the number of samples in the validation set, V NN (p vj To evaluate the network's performance on each sample p in the offline positive sample validation set vj State value, V true (p vj For each sample p in the offline positive sample training set vj The known state values ​​in the training set, where j = 1, 2, ..., k; when the loss on the validation set no longer decreases or reaches the preset number of training rounds N. max At that time, pre-training was completed.

[0071] The update formula for the Adam optimizer is: Where, θ t Here are the network parameters, and η is the learning rate. and For the corrected gradient first and second moments estimates, ∈ is a small constant to prevent the denominator from being zero.

[0072]

[0073] Where, m t For first-order moment estimation, m t =β1m t-1 +(1-β1)g t β1 is the decay rate estimated by the first moment; v t For second-order moment estimation, β² is the decay rate estimated by the second moment, g t To evaluate network parameters θ t gradient,

[0074] Fuse feature vectors using step 4. The network is trained and evaluated online, employing a loss function based on policy gradient combined with value calculation error. in, R represents the expectation. t Let γ be the reward at time t, and γ be the discount factor. Let be the state vector at time t. Let V(t) be the action vector at time t, and V(t) be the state value. The policy function is defined, and the loss is propagated through the Adam optimizer to update the parameters of the evaluation network.

[0075] Specifically, in this embodiment, the reward mechanism includes:

[0076] Successfully grabbing cargo: If the unmanned grabbing vehicle successfully grabs cargo, a positive reward of +10 is given; otherwise, -10 is given.

[0077] Collision Avoidance: If the unmanned vehicle successfully avoids collisions with other objects or vehicles during its journey, it will receive a positive reward of +5; otherwise, it will receive a negative reward of -5.

[0078] Accurate placement: If the unmanned clamp truck can accurately place the goods in the designated location, a positive reward of +8 will be given; otherwise, a negative reward of -8 will be given.

[0079] Efficiency Improvement: If the unmanned gripper operates faster or more energy-efficiently than scheduled, a positive reward of +3 is given; otherwise, a negative reward of -3 is given.

[0080] Safety Violation: If the operation of the unmanned clamping vehicle violates safety regulations, such as speeding or taking the wrong route, a negative reward of -20 will be given.

[0081] By calculating the error between the state value and the actual reward received. Continuously monitor and evaluate the predictive ability of the network, among which, To evaluate the state vector of each sample in the new online samples predicted by the network, R new,i The reward for each sample in the given online new samples is calculated by determining the entropy of the policy function. The changes in [the parameters] are used to regulate the convergence of the reinforcement learning control algorithm.

[0082] Based on the monitoring results, in each online training iteration, the confidence level C of the offline positive samples is determined. off And the credibility of online training C online Adjust the fusion weights w1 and w2 in the fusion of offline positive samples and online new samples.

[0083] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An evaluation network design method for a human-free hug car reinforcement learning regulation algorithm, characterized in that, Comprising the following steps: Step 1, analyze and determine the unmanned clamping car operation scene S, determine the data type D and the importance weight w of the data type under different operation scenes; Step 2, according to the operation scene S and the data type D determined in step 1, multi-channel collection of unmanned clamping car offline positive samples, merging to form a total offline positive sample set P, preprocessing the offline positive sample set P; Step 3, for each sample p in the pre-processed offline positive sample set P i perform cleaning and normalization, obtaining extracting each sample feature vector merging to obtain the total offline positive sample feature vector Step 4, building an evaluation network: a multi-layer fully connected neural network is used to build an evaluation network infrastructure, and an input layer, a hidden layer, a fusion layer and an output layer are designed in sequence, the fusion layer uses a fusion strategy to fuse offline positive sample feature vectors and online new sample feature vectors output by the learning control algorithm, to obtain a fusion feature vector Step 5, using the offline positive sample feature vector of step 3 Pre-training the evaluation network, selecting and initializing the reinforcement learning control algorithm, updating the evaluation network parameters, and using the fusion feature vector Online training of the evaluation network and construction of the loss function, using the Adam optimizer to backpropagate the loss, and calculating the error E between the state value predicted by the evaluation network and the actual reward obtained by the unmanned gripper car for each sample in the online new sample err Continuously monitor the prediction ability of the trained evaluation network, monitor the convergence of the reinforcement learning control algorithm by calculating the entropy H(π) of the policy function, and provide value evaluation and strategy guidance for the reinforcement learning control algorithm; the update formula of the Adam optimizer is Where θ t is the evaluation network parameter, η is the evaluation network learning rate, and are the first and second moment estimates of the corrected gradient, and ∈ is a small constant to prevent the denominator from being zero; wherein m t is a first moment estimate, m t = β1m t-1 + (1-β1)g t , β1 is a decay rate of the first moment estimate; v t is a second moment estimate, β2 is a decay rate of the second moment estimate, g t is a gradient of the evaluation network parameter θ t ​ Fusing feature vectors with step 4 Online training of the critic network with a policy gradient-based loss function combined with a value calculation error wherein, denotes expectation, R t is the reward at time t, and γ is the discount factor, is the state vector at time t, is the action vector at time t, and V(·) is the state value, is the policy function, and the loss is propagated in the direction of the Adam optimizer, updating the parameters of the critic network; the error The prediction ability of the evaluation network is continuously monitored with this error, wherein For evaluating the prediction of the evaluation network for the state vector of each sample of the set of online new samples, R new,i For the reward of each sample of the set of online new samples, the entropy of the policy function is calculated The convergence of the reinforcement learning control algorithm is regulated by the change of the error 2. The method for evaluating a network design of claim 1, wherein, In step 1, the unmanned forklift operation scene analysis: the operation scene set of the unmanned forklift is S={s1, s2, …, s m}, wherein each scene s i includes goods type G i , warehouse layout L i and task requirement T i , i=1, 2, …, m; The data type set D = {d1, d2, ..., d...} under different work scenarios r }, where d k The data type is represented by k = 1, 2, ..., r, and the data type includes operation parameter data d1, environmental parameter data d2, and cargo data d3. The operation parameter data d1 includes the position (x, y, z) of the clamp car, the speed v = (v x ,v y ,v z ), and the clamp car state θ, that is, d1 = ((x, y, z), v, θ); the environment parameter data d2 includes the temperature T, the humidity H, and the illumination intensity I, that is, d2 = (T, H, I); and the cargo data d3 includes cargo information G, which includes the weight W and the size (a, b, c) of the cargo, that is, d3 = (W, (a, b, c)).

3. The method for evaluating a network design of claim 1, wherein, The step 2 specifically includes the following steps: Step 2.1, collect the unmanned clamping car offline positive samples from the following three aspects: Offline positive samples of human operating unmanned forklifts are collected: a set H = {h1, h2, …, h n1}, where each sample h j (j = 1, 2, …, n1) in the set H includes a job scene and an operation sequence of the data type corresponding to the job scene The operation sequence Each operation in the operation sequence is an operation performed by the forklift, is the result or reward corresponding to the operation, l = 1, 2, …, k. Offline positive samples in the simulation environment are collected: a set M = {m1, m2, …, m n2}, where each sample m i includes a job scene and a corresponding operation sequence Operation sequence Each operation in the operation sequence is an operation performed by a forklift truck, is the corresponding result or reward of the operation, r = 1, 2, …, q. Offline positive samples in similar equipment are collected: a set E = {e1, e2, …, e n3}, wherein each sample e t includes a job scene and a corresponding operation sequence Operation sequence Each operation in the operation sequence is an operation performed by a clamp truck, is the corresponding result or reward of the operation, u = 1, 2, …, p; The three offline positive samples are combined into a total offline positive sample set P = H U M U E = {p1, p2, …, pn}, where n = n1+ n2+ n3, and each sample in the total offline positive sample set P n} where n = n1+ n2+ n3, and each sample in the total offline positive sample set P for a job scene, for a corresponding operation sequence, the operation sequence operation sequence each operation wherein, is an operation performed by the forklift truck, is a corresponding result or reward of the operation, l = 1, 2, …, k; Step 2.

2. Pre-processing of the total offline positive sample set P, including checking the integrity of the samples, ensuring that each sample contains a complete scene and sequences of operations Marking or preliminary repair of missing or erroneous data in the samples.

4. The method for evaluating a network design of claim 1, wherein, In step 3, each sample p in the offline positive sample set P i After data cleaning and normalization are performed on (i = 1, 2, …, n), the cleaned and normalized data are obtained The feature vectors in each sample are extracted by using a feature extraction function, that is, For the extracted feature vectors, d is the dimension of the feature vector, and the feature vectors of all data are merged to obtain the offline positive sample feature vector ​​ 5. The method for evaluating a network design of claim 1, wherein, In step 4, the number N of input layer nodes is determined according to the operation state of the unmanned straddle carrier input , which includes position (x, y, z), velocity v = (v x , v y , v z ), straddle carrier state θ, and goods information G, and the position is represented by three-dimensional coordinates, the velocity is represented by a three-dimensional vector, the straddle carrier state θ is represented by m parameters, and the goods information G is represented by n features, so N input = 3 + 3 + m + n. The hidden layers have L layers, the number of neurons in the lth layer is N l , and N1>N2>…>N L , let the input of the lth layer be , the output be , the weight matrix be W l , and the bias vector be B l , then the calculation process of the lth layer is , where f ReLU (x)=max(0,x) is a ReLU activation function, taking the larger one of 0 and x, The output layer has only one node, which is used to output the state value estimate V of the output layer, , where W L+1 is the weight vector of the output layer, B L+1 is the bias term of the output layer, and is a constant bias added to the neuron of the output layer, is the output vector of the last hidden layer.

6. The method for evaluating a network design of claim 1, wherein, In the step 4, the fusion strategy is a weighted summation fusion strategy or a splicing fusion strategy; When using a weighted summation fusion strategy, the offline positive sample feature vectors will be... and online new sample feature vector (Real-time online collection of new samples during the clamping vehicle's operation {p new1 ,p new2 ,…,p newq }, the and All include state vectors and action vectors The online training feature vector is obtained by preprocessing and extracting the data in the same way as the offline positive samples, and then fused along the dimensionality to obtain the fused feature vector. Where w1 and w2 are the fusion weights, and the fusion weights are based on the offline positive sample confidence C. off And online sample credibility C online Sure: w2 = 1 - w1, where ∈ is a small constant to prevent the denominator from being zero, C off =β1C src +β2C val C src To assess the reliability of the sample source, C val To determine the validation confidence of the sample in the simulated environment, β1 and β2 are weighting coefficients, and C... online This is obtained by calculating the fluctuation level of the value function of the current fusion strategy; When using a splicing and fusion strategy, the offline positive sample feature vectors will be... and online new sample feature vector Concatenate the vectors along the dimensional lines to obtain the feature vectors. Then, the concatenated feature vector is processed by a fully connected layer (FC). Mapping to the appropriate dimension yields the fused feature vector.

7. The method for evaluating a network design of claim 1, wherein, The weight matrix and bias vector of the fully connected neural network can be determined by training, assuming the output of the fusion layer is The input to the subsequent layer of the evaluation network is then The forward propagation calculation of the network is continued.

8. The method for evaluating a network design of claim 1, wherein, In step 5, the preprocessed offline positive samples are divided into a training set T. off ={p t1 ,p t2 ,…,p tm } and validation set V off ={p v1 ,p v2 ,…,p vk }; The evaluation network is trained using the training set, and the loss on the training set is calculated: where m is the number of samples in the training set, V NN (p ti ) is the state value of each sample p ti in the offline positive sample training set evaluated by the evaluation network, V true (p ti ) is the known state value of each sample p ti in the offline positive sample training set, where i = 1, 2, …, m; the state value is obtained by inputting the feature vector of the offline positive sample into a multi-layer fully connected neural network, which is forward propagated, processed by a hidden layer, and outputs an estimated value of the state value in the output layer, which is a prediction of the value of each sample p ti . After the end of each training epoch, the network performance is evaluated using the validation set to calculate the loss on the validation set: where k is the number of samples in the validation set, V NN (p vj ) is the state value of each sample p vj in the offline positive sample validation set evaluated by the network, V true (p vj ) is the known state value of each sample p vj in the offline positive sample training set, where j = 1, 2, …, k; when the loss on the validation set no longer decreases or reaches the preset training number of rounds N max , the pre-training is completed.

9. The method for evaluating a network design of claim 1, wherein, According to the monitoring result, in each online training iteration, the fusion weights w1 and w2 in the fusion of the offline positive samples and the online new samples are adjusted according to the credibility C of the offline positive samples off and the credibility C of the online training online .

Images