Robotic decision-making method based on deep reinforcement learning

Abstract

The present application relates to the technical field of robot control, in particular to a robot decision method based on deep reinforcement learning, comprising the following steps: obtaining a rate of change data sequence, calculating a frame difference value, screening data frames greater than a threshold value, extracting a mean vector, generating a feature vector set, splicing a remaining time and distance ratio, constructing an encoding matrix, adjusting a weight parameter, screening a task group, mapping and sorting, generating a task batch set, screening an adaptive task, and generating an execution sequence list.The present application filters adjacent frame differences of multiple rate of change sequences in environmental perception data and extracts mutation segment features, dynamically adjusts strategy priority by screening specific inner product results during strategy evaluation, improves strategy adaptability, screens the task combination with the smallest conflict and the highest priority, screens the task batch with higher matching degree according to the calculation ability difference, and enhances the autonomous response capability and execution reliability of the robot system in complex task scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to a robot decision-making method based on deep reinforcement learning. BACKGROUND

[0002] The technical field of robot control mainly involves motion control, actuator driving, perception information processing, path planning, task scheduling and coordinated control of robot systems. This field covers the core control technology required by different types of robots such as industrial robots, service robots and special robots when performing predetermined tasks, usually including sensor data acquisition and fusion, control algorithm design, real-time feedback regulation, kinematics and dynamics modeling, stability control and error compensation mechanism. The control goal is usually to achieve high-precision, high-efficiency and high-reliability robot action execution and multi-robot cooperation, and the application scenarios cover automatic production lines, logistics transportation, medical assistance, disaster rescue and other fields.

[0003] Among them, the robot decision-making method refers to the process of generating suitable action or behavior sequence by a specific algorithm and model according to environmental perception data, internal state and preset target during the execution of the task by the robot system. Its purpose is to enable the robot to have autonomous judgment ability, dynamically adjust the behavior strategy according to the real-time changing environmental conditions and task requirements, and realize functions such as autonomous navigation, path selection, task priority judgment, action selection, and is widely used in new energy vehicle production and processing, intelligent manufacturing, etc.

[0004] Although the traditional decision-making method can achieve real-time feedback regulation through sensor data acquisition and fusion, it usually processes the environmental state with static models, which makes it difficult to effectively distinguish key change segments in dynamic environments, causing a lag in responding to environmental mutation events. In addition, the coordinated control of traditional methods is usually driven by fixed priority or static strategy table, which lacks dynamic adjustment mechanism for task priority during execution, and the processing of computational resource conflicts and dependency relationships in the task scheduling process is insufficient, resulting in uneven resource allocation when multiple tasks are concurrent, and easy to appear the situation that part of the high-priority tasks are blocked by low-priority tasks, affecting the overall task completion efficiency and the real-time and reliability of the robot system. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art, and a robot decision-making method based on deep reinforcement learning is proposed.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a robot decision-making method based on deep reinforcement learning, comprising the following steps:

[0007] S1: obtain real-time environment data of the robot in the new energy vehicle generation, calculate the adjacent frame difference value, screen the data frames greater than the change rate threshold, extract the data segment feature mean vector in the corresponding time period, and generate a mutation segment feature mean vector set;

[0008] S2: based on the mutation segment feature mean vector set, calculate the ratio between the remaining time and the shortest distance of the obstacle, call the ratio and the mutation segment feature mean vector set for splicing, construct a state input encoding matrix, and generate a strategy input encoding matrix;

[0009] S3: based on the strategy input encoding matrix, call the existing strategy branch weight parameter vector in the deep reinforcement learning strategy network, calculate the inner product between the encoding matrix and the weight parameter vector, select the strategy branch smaller than the reference value, adjust the corresponding weight parameter value, and generate a strategy branch updated weight set;

[0010] S4: obtain task dependency completion identification data in the task queue, screen the task group with no conflict identification, calculate the task group strategy mapping score based on the strategy branch updated weight set, sort according to the score, screen the task group ranking in the front, and generate a strategy mapping task batch set.

[0011] The application improves that the mutation segment feature mean vector set includes time period number index, change rate statistical vector, feature segment duration, feature segment start and end timestamp and feature segment weight score, the strategy input encoding matrix includes time period input encoding index, feature splicing vector, remaining time proportion item, obstacle distance proportion item and state vector length identification, the strategy branch updated weight set specifically is strategy number index, adjusted weight value, original weight difference value, update trigger identification and update timestamp, and the strategy mapping task batch set includes batch number, task index in the batch, mapping score sorting in the batch, batch execution priority identification and batch available resource index.

[0012] The application improves that the acquisition step of the mutation segment feature mean vector set is specifically:

[0013] S111: obtain real-time environment data of the robot in the new energy vehicle generation, the real-time environment data includes position change rate data sequence, environment light intensity change rate sequence and end effector contact force change rate sequence, calculate the difference value between adjacent data frames of the data sequence, call the difference value and the set change rate threshold for comparison, judge whether the difference value is greater than the change rate threshold, screen the data frames greater than the threshold, and generate a threshold overrun frame index set;

[0014] S112: Based on the threshold exceeding frame index set, the original position change rate data sequence, the ambient light intensity change rate data sequence and the end effector contact force change rate data sequence are called to filter the data frames in the corresponding time period, the average change rate of each segment is calculated, and the average change rate set of each segment is obtained.

[0015] S113: According to the average change rate set of each segment, the corresponding time period number index, the change rate statistical value, the duration of each segment and the start and end time stamp of each segment are extracted, the segment change rate statistical value is compared with the set segment weight score benchmark value, the segment characteristics meeting the conditions are filtered, and the average vector set of the mutation segment characteristics is generated.

[0016] The application improves that the acquisition step of the strategy input coding matrix is specifically:

[0017] S211: Based on the average vector set of the mutation segment characteristics, the current task remaining execution time recorded by the robot task scheduling unit is called, the time difference value between the task remaining time value corresponding to the current time and the task estimated completion time is collected, the unit progress time matching value is calculated combined with the actual task execution progress data, and the task urgency value is generated;

[0018] S212: The shortest distance sequence of the obstacle collected by the current environment perception unit is called, the nearest obstacle distance value under the current time stamp is obtained, the ratio calculation is carried out based on the task urgency value and the shortest distance value of the obstacle, the ratio parameter between the task urgency and the space obstruction is calculated, and the task environment correlation ratio information is generated.

[0019] S213: The task environment correlation ratio information and the average vector set of the mutation segment characteristics are called, the two are combined by vector splicing, the splicing index mapping is carried out according to the frame number dimension, the joint representation structure of each segment state input information is constructed, and the strategy input coding matrix is established.

[0020] The application improves that the acquisition step of the strategy branch update weight set is specifically:

[0021] S311: Based on the strategy input coding matrix, the weight parameter vector of each strategy branch in the deep reinforcement learning strategy network is called, the inner product calculation is carried out on each row vector in the strategy input coding matrix and the corresponding strategy weight parameter vector according to the index position, the coding response score of each strategy branch is obtained, and the strategy response score sequence is generated.

[0022] S312: According to the strategy response score sequence, the strategy priority adjustment benchmark value is called, whether each strategy response score is less than the benchmark value is compared item by item, the strategy number less than the benchmark value is filtered, the strategy identification meeting the condition is counted, and the strategy weight adjustment index set is generated.

[0023] S313: Call the policy weight adjustment index set, locate the original weight value in the weight parameter vector, call the corresponding response score and modify in combination with the policy input matrix average value, calculate the weight variation range, adjust the weight value and reconstruct the weight parameter of the policy branch, and obtain the policy branch update weight set.

[0024] The application improves that the obtaining step of the policy mapping task batch set is specifically:

[0025] S411: Obtain task dependency completion identification data in the task queue, call the calculation resource occupation conflict identification data in the current calculation node, filter the task numbers with no conflict and the dependency completion state as completed, count the task number set meeting the condition, and generate a non-conflict task number set;

[0026] S412: Based on the non-conflict task number set, call the weight parameter value corresponding to each policy number in the policy branch update weight set, map the task number and the policy number, calculate the policy mapping score of each task, and obtain the task mapping score sequence;

[0027] S413: According to the task mapping score sequence, sort the task numbers in descending order of scores, filter the task number group ranking in the front, establish the corresponding batch number mapping relationship according to the sorting order, and establish the policy mapping task batch set.

[0028] The application improves that the method further comprises the following steps:

[0029] S5: Based on the policy mapping task batch set, call the remaining calculation capability of the execution unit, calculate the difference value between the policy mapping task batch set and the remaining calculation capability coefficient, judge whether the difference value is less than the calculation capability adaptation threshold, filter the task batch meeting the adaptation condition, and generate a policy execution sequence list;

[0030] The policy execution sequence list comprises a sequence number, a task mapping index in the sequence, a corresponding execution unit number, a mapping priority score and an available computing power adaptation record.

[0031] The application improves that the obtaining step of the policy execution sequence list is specifically:

[0032] S511: Based on the policy mapping task batch set, call the remaining calculation capability data of each execution unit, extract the batch task corresponding allocation unit number, match the execution unit and call the current remaining calculation capability value, compare with the batch task average computing power demand value, calculate the numerical difference between the task batch demand and the execution unit capability, and generate a batch capability difference set;

[0033] S512: According to the batch capability difference value set, the computing capability adaptation threshold is called, it is judged whether each difference value is less than the adaptation threshold, the task batch number meeting the adaptation condition is screened, the batch information meeting the condition and the task list are integrated, and the adaptation task batch index set is generated;

[0034] S513: Based on the adaptation task batch index set, the task number and the execution unit mapping information are extracted, the execution list is generated according to the batch number arrangement order, the mapping task index and the execution unit number of each record are called, the task batch mapping stability score is obtained by operation, the sorting weight and the position relationship are determined according to the score, and the strategy execution sequence list is established.

[0035] Compared with the prior art, the advantages and positive effects of the present application are that:

[0036] In the present application, by screening the adjacent frame difference of the multiple change rate sequence in the environmental perception data and extracting the mutation segment feature, the capture ability of the key change trend in the dynamic environment can be enhanced, the ratio information calculated by combining the task remaining execution time and the obstacle distance is used to optimize the data structure of the decision input, the strategy priority is dynamically adjusted by screening the specific inner product result in the strategy evaluation process, the strategy adaptability is improved, the task dependency completion identifier and the computing resource occupation conflict identifier are fused in the task scheduling, the task combination with the smallest conflict and the highest priority is screened, the task batch with higher matching degree is selected according to the computing capability difference value, the flexibility of strategy adjustment and the efficiency of task scheduling in the dynamic environment are improved as a whole, and the autonomous response ability and the execution reliability of the robot system in the complex task scene are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The method flowchart of the present application is shown in the figure;

[0038] Figure 2 The flowchart for obtaining the mutation segment feature mean vector set of the present application is shown in the figure;

[0039] Figure 3 The flowchart for obtaining the strategy input coding matrix of the present application is shown in the figure;

[0040] Figure 4 The flowchart for obtaining the strategy branch update weight set of the present application is shown in the figure;

[0041] Figure 5 The flowchart for obtaining the strategy mapping task batch set of the present application is shown in the figure;

[0042] Figure 6 The flowchart for obtaining the strategy execution sequence list of the present application is shown in the figure. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0044] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0045] Please refer to Figure 1 The present application provides a technical solution: a robot decision-making method based on deep reinforcement learning, comprising the following steps:

[0046] S1: acquiring real-time environment data of the robot in new energy vehicle generation, the real-time environment data including position change rate data sequence, environment light intensity change rate sequence, end effector contact force change rate sequence, calculating the difference between adjacent frames based on the three data between adjacent data frames, calling the difference and the set change rate threshold for comparison, judging whether the difference is greater than the change rate threshold, screening the data frames greater than the threshold, extracting the data segment feature mean vector in the corresponding time period based on the screening result, generating a set of mutation segment feature mean vectors;

[0047] S2: based on the set of mutation segment feature mean vectors, calling the current task remaining execution time of the robot, the current obstacle shortest distance sequence, calculating the ratio between the remaining time and the obstacle shortest distance, calling the ratio and the set of mutation segment feature mean vectors for splicing, constructing a state input encoding matrix, generating a strategy input encoding matrix;

[0048] S3: based on the strategy input encoding matrix, calling the existing strategy branch weight parameter vector in the deep reinforcement learning strategy network, calculating the inner product between the encoding matrix and the weight parameter vector, judging whether the inner product value is less than the set strategy priority adjustment reference value, screening the strategy branches less than the reference value, adjusting the corresponding weight parameter value according to the screened strategy branches, generating a set of strategy branch updated weights;

[0049] S4: acquiring task dependency completion identification data in the task queue, calling the computing resource occupation conflict identification, screening the task group with no conflict, calculating the task group strategy mapping score based on the set of strategy branch updated weights, sorting by score, screening the task group with high ranking, generating a set of strategy mapping task batches;

[0050] S5: Based on the strategy mapping task batch set, call the remaining computing capacity of the execution unit, calculate the difference between the strategy mapping task batch set and the remaining computing capacity coefficient, judge whether the difference is less than the computing capacity adaptation threshold, screen the task batch that meets the adaptation condition, and generate a strategy execution sequence list;

[0051] The position change rate is the position change value of the end effector or the robot reference point per unit time; the environmental light intensity change rate is the brightness change rate per unit time collected by the light sensor; the end effector contact force change rate is the end force change rate per unit time obtained through the force sensor; the remaining execution time is the difference between the current time and the scheduled completion time of the task; the shortest distance of the obstacle is the minimum distance between the robot and the obstacle detected by the robot; the weight parameter vector is the trainable weight corresponding to each action in the strategy network; the strategy priority adjustment reference value is a weight adjustment threshold set by the system during design, usually a value in the normalized interval 0-1; the task dependency completion identifier marks whether there is an incomplete dependency item, and is a Boolean value; the computing resource occupation conflict identifier marks whether the current resource can be used for task scheduling, and is a Boolean value; the remaining computing capacity of the execution unit is the remaining capacity score calculated by CPU occupancy rate, memory remaining, I / O load, etc.; the computing capacity adaptation threshold is the minimum schedulable capacity reference value set for the execution unit.

[0052] The mutation segment feature mean vector set includes a time period number index, a change rate statistical vector, a feature segment duration, a feature segment start and end timestamp, and a feature segment weight score; the strategy input encoding matrix includes a time period input encoding index, a feature splicing vector, a remaining time proportion item, an obstacle distance proportion item, and a state vector length identifier; the strategy branch update weight set specifically includes a strategy number index, an adjusted weight value, an original weight difference value, an update trigger identifier, and an update timestamp; the strategy mapping task batch set includes a batch number, a task index within the batch, a batch mapping score ranking, a batch execution priority identifier, and a batch available resource index; and the strategy execution sequence list includes a sequence number, a sequence task mapping index, a corresponding execution unit number, a mapping priority score, and an available computing power adaptation record.

[0053] Please refer to Figure 2 , the acquisition steps of the mutation segment feature mean vector set are as follows:

[0054] S111: Obtain real-time environment data of the robot in the new energy vehicle generation, the real-time environment data including a position change rate data sequence, an environmental light intensity change rate sequence, and an end effector contact force change rate sequence, calculating the difference between adjacent data frames of the data sequence, calling the difference and the set change rate threshold for comparison, judging whether the difference is greater than the threshold, screening the data frames greater than the threshold, and generating a threshold overrun frame index set;

[0055] The position change rate data sequence, the ambient light intensity change rate data sequence, and the end effector contact force change rate data sequence in the robot environment perception unit are obtained, and for the position change rate data sequence, the difference between adjacent data frames is calculated frame by frame, and the sampling period is The collected position change rate data sequence is The calculation result of the adjacent difference is Similarly, the ambient light intensity change rate data sequence is The difference is calculated as The end effector contact force change rate data sequence is The difference is calculated as For each set of differences, the corresponding change rate threshold is called for comparison, the position change rate threshold is Set to The ambient light intensity change rate threshold is The contact force change rate threshold is The threshold is set according to the position change rate abnormal value monitoring interval under the high-speed motion state of the robot , and the middle section is taken ; the light intensity change rate is according to the measured fluctuation interval of the illumination mutation scene , and the median is taken ; the contact force change rate is measured according to the impact in motion , and is selected; the differences are filtered according to the above threshold, the 3rd and 4th items of the position change rate difference satisfy, the 2nd item of the light intensity change rate difference satisfies, the 1st and 3rd items of the contact force change rate difference satisfy, and the filtering result corresponds to the frame index , and after merging and deduplication, it is arranged as , and the threshold overrun frame index set is obtained.

[0056] S112: Based on the threshold overrun frame index set, the original position change rate data sequence, the ambient light intensity change rate data sequence, and the end effector contact force change rate data sequence are called to filter the data frames in the corresponding time period, and the formula is used:

[0057] ;

[0058] The average change rate of each segment is obtained by operation, and the average change rate set in the segment is obtained.

[0059] Wherein, represents the average change rate of the th segment, represents the normalized value of the th frame position change rate, represents the normalized value of the Frame ambient light intensity rate of change normalized value, Frame end effector contact force rate of change normalized value, Frame ambient light intensity rate of change normalized value, Frame ambient light intensity rate of change normalized value, Frame ambient light intensity rate of change normalized value;

[0060] Based on the threshold exceeding frame index set, call the original position rate of change data sequence , ambient light intensity rate of change data sequence , end effector contact force rate of change data sequence , filter out the exceeding frame segment, segment division window , normalize the data in the segment, position rate of change normalized interval , normalized result , light intensity rate of change normalized interval , normalized , contact force rate of change normalized interval , normalized , into the formula: ;

[0061] Formula parameter description: The average rate of change of the first segment, The normalized value of the position rate of change of the first frame, The normalized value of the ambient light intensity rate of change of the first frame, The normalized value of the contact force rate of change of the first frame, The number of frames in the first segment, the operation logic is to weight three types of rate of change parameters, normalize and process spatial change (position), external perception (light), and force feedback (contact) respectively, enhance the data distribution difference through square root and logarithm, and improve the ability to distinguish abnormal segments.

[0062] The innovation of the formula lies in that, by using and operations on three types of normalized values of sensors, the influence of different physical quantities is strengthened, and the recognition ability for abnormal segments is enhanced.

[0063] Calculation process: ;

[0064] The calculation process is the same, the average of segment 1 after calculation is 0.795, and the average of segment 2 is 0.934, generating the average rate of change set in the segment.

[0065] .

[0066] As shown in Table 1, the normalized parameter results provide basic data support for the segment mean value calculation process.

[0067] S113: According to the segment change rate mean set, the corresponding time segment number index, change rate statistical value, segment duration, segment start and end timestamp are extracted, the segment change rate statistical value is compared with the set segment weight score benchmark value, the segment characteristics meeting the conditions are screened, and the mutation segment characteristic mean vector set is generated.

[0068] According to the segment change rate mean set , the time segment number , the segment duration per segment , the segment start and end timestamp segment 1 is , segment 2 is , the segment weight score benchmark value is called , Set according to: robot high dynamic running scene monitoring mean difference interval , select the median , this value fluctuates with the standard deviation of environmental disturbance, when the standard deviation is greater than 0.15, it is recommended to take , the current test standard deviation is 0.12, select , compare the segment change rate mean, segment 1 , segment 2 , filter segment 2 that meets the conditions, establish segment feature vector, number 2, change rate statistical value 0.934, duration , start and end timestamp , generate mutation segment characteristic mean vector set.

[0069] Please refer to Figure 3 , the acquisition steps of the strategy input encoding matrix are as follows:

[0070] S211: Based on the mutation segment characteristic mean vector set, the current task remaining execution time recorded by the robot task scheduling unit is called, the time difference value between the task remaining time value corresponding to the current time and the task expected completion time is collected, combined with the task actual execution progress data, the unit progress time matching value is calculated, and the task urgency value is generated;

[0071] Based on the mutation segment characteristic mean vector set, the current task remaining execution time recorded in the robot task scheduling module is called, the current timestamp corresponding to the task remaining time , the time difference value between and the task expected completion time is calculated , combined with the task completion progress recorded by the task execution module , the unit progress time matching value , , for example, current timestamp , task estimated completion time , task completed progress , calculate , unit progress time matching value . Task urgency value By normalization, , wherein is the maximum unit progress time matching value in the current batch, in the whole batch monitoring , then . Urgency normalization coefficient The setting is based on the fact that the task set is collected several times in actual operation, and it is found that the interval is the most distinctive, and the reference interval is adjusted by the actual operation scene task density, if the robot is high-density scheduling scene, it is recommended to take the interval . This complies with the reference interval, and finally generates the task urgency value.

[0072] S212: Call the obstacle shortest distance sequence collected by the current environment perception unit, get the nearest obstacle distance value under the current timestamp, based on the task urgency value and the obstacle shortest distance value, calculate the ratio of the task urgency and the space obstruction, generate the task environment association ratio information;

[0073] Call the obstacle shortest distance sequence collected in the current environment perception module, the current timestamp The nearest distance of the obstacle The read value is 1.8m, according to the task urgency value obtained in paragraph 1 , two parameters perform division operation, calculate the ratio of the task urgency and the space obstruction , then . Obstacle distance threshold The setting is based on the robot safe running distance standard, according to ISO10218-1 standard , the current , so enter the normal judgment process. Space obstruction ratio The reference interval is induced after field data collection, and the normalized reference interval is , the current is effective within the interval, and the task environment association ratio information is generated. As shown in Table 2, the urgency value and the environment obstruction ratio data monitoring situation under the current different batch tasks are summarized.

[0074] ;

[0075] As shown in Table 2, each batch 、 Both are within the reference interval, effectively reflecting the relationship between different task urgency and obstacles.

[0076] S213: Call the task environment correlation ratio information and the average value vector set of the mutation segment feature, perform vector splicing combination, perform splicing index mapping according to the frame number dimension, construct the joint representation structure of each segment state input information, and establish the strategy input encoding matrix;

[0077] Call the task environment correlation ratio information and the average value vector set of the mutation segment feature, perform vector splicing on the frame number dimension, construct the joint representation structure of the state input information, and adopt horizontal splicing for the splicing operation. Each frame corresponds to a splicing field , for example, segment number corresponds to , and the splicing result is . The splicing mapping weight coefficient is adopted , and the setting basis is that the influence proportion of the mutation segment feature in the task state input feature is given priority, and is recommended after field test verification . The joint encoding value of the state input is calculated after splicing and jointing: ; the data is brought in . The innovation of the splicing joint weight is that the strategy input encoding matrix is more sensitive to spatial mutation situations by weighting the mutation segment feature and the spatial obstacle feature, which improves the stability of strategy convergence. The formula result is higher than the normalized input benchmark 0.8, which indicates that the state input of the current segment number will focus on strategy coding, and finally the strategy input encoding matrix is established.

[0078] Please refer to Figure 4 , and the steps for obtaining the strategy branch update weight set are as follows:

[0079] S311: Based on the strategy input encoding matrix, call the weight parameter vector of each strategy branch in the deep reinforcement learning strategy network, and perform inner product calculation on each row vector in the strategy input encoding matrix and the corresponding strategy weight parameter vector according to the index position, to obtain the coding response score of each strategy branch, and generate the strategy response score sequence;

[0080] Based on the strategy input encoding matrix, call the weight parameter vector of each strategy branch in the deep reinforcement learning strategy network. The number of strategy branches in the strategy network is set to , and the dimension of each branch is , and the weight parameter vectors are respectively , for example , , and so on. The current policy input encoding matrix is obtained from the previous step, and the matrix row vector corresponds to . For each row vector , the corresponding policy branch weight parameter is executed, and the calculation formula is:

[0081] That is, for the policy branch of , there is . The same inner product calculation is performed for the remaining policy branches to obtain the corresponding encoding response scores of all policy branches, which are summarized to form the policy response score sequence . The purpose of the inner product operation using this calculation logic is to directly measure the matching degree of the policy input state on different policy branches, and the weight parameter interval is set according to the average policy weight value at the th round of convergence during the policy training phase, and the interval is taken to ensure that after each training, the policy input state will not be affected by a single bias that is too large to affect multi-policy matching.

[0082] S312: According to the policy response score sequence, call the policy priority adjustment benchmark value, compare each policy response score in turn to see if it is less than the benchmark value, filter the policy numbers that are less than the benchmark value, count the policy identifiers that meet the conditions, and generate a policy weight adjustment index set;

[0083] According to the policy response score sequence , call the policy priority adjustment benchmark value for filtering. The setting logic is as follows: in the initial training phase of the policy network, the mean and standard deviation of the normalized policy response score of each round are counted, for example , and the adjustment is taken . The purpose of this setting interval is to prevent the filtering effect from being weakened when all policy response scores tend to be consistent, and to dynamically adjust to enhance the adjustment probability of low-active policies. For each policy response score , compare it in turn, if , then filter the policy number, for example, if , then are all lower than , and the number is filtered, and the policy weight adjustment index set is summarized and counted. The setting basis of the benchmark value is derived from the actual running data interval ​, the offset standard deviation is 0.15, which is derived from the actual variance mean in the 10-round iteration process, reasonably reflects the response activity between strategy branches, adjusts the screening standard to dynamically converge, and improves the robustness of the strategy network.

[0084] S313: Call the strategy weight adjustment index set, locate the original weight value in the weight parameter vector, call the corresponding response score and modify it in combination with the strategy input matrix mean value, using the formula: ;

[0085] Calculate the weight variation range, adjust the weight value and reconstruct the weight parameter of the strategy branch, and obtain the updated weight set of the strategy branch;

[0086] wherein, represents the weight variation range of the th strategy branch, represents the normalized value of the encoding response score corresponding to the th strategy branch, represents the mean normalized value of all row vectors of the strategy input encoding matrix, represents the mean normalized value of all vector variances of the strategy input encoding matrix, represents the strategy response distribution adjustment factor, represents the normalized value of the historical variation standard deviation of the th strategy branch, represents the strategy variation suppression factor, is the normalized strategy weight variation range, is the base of the natural logarithm.

[0087] Call the strategy weight adjustment index set strategy number , respectively locate its corresponding weight parameter , and call the corresponding response score and the row mean and the variance mean of the strategy input encoding matrix, for example. The strategy response distribution adjustment factor is currently set to 1.0, derived from the strategy input variance sensitivity analysis, this item is set by the strategy active state monitoring module, when the active strategy fluctuation degree > 10%, set , otherwise default . The historical variation standard deviation takes a value of , the strategy variation suppression factor , derived from the strategy cumulative error control mechanism, interval . The modified calculation formula is: ;

[0088] Formula parameter description: Strategy weight variation range, Strategy response score normalization value, Encoding matrix mean, Encoding matrix variance mean, Strategy response distribution adjustment factor, Historical variation standard deviation normalization value, Strategy variation suppression factor, Take the absolute value to ensure that the variation range is positive, and the denominator item Used to smooth the low variance state, The innovation lies in the composite adjustment mechanism which considers response volatility and historical error accumulation at the same time, enhancing the convergence of strategy branch dynamic adjustment.

[0089] Example calculation, ;

[0090] ;

[0091] The result In the adjustment interval The numerical meaning indicates that the second branch of the current strategy branch is active but lower than the expected active level, and the branch weight update needs to be moderately improved. The update result will be written into the strategy branch update weight set for the next round of strategy training parameter adjustment.

[0092] Please refer to Figure 5 The acquisition steps of the strategy mapping task batch set are as follows:

[0093] S411: Acquire task dependency completion identification data in the task queue, call the calculation resource occupation conflict identification data in the current calculation node, filter the task numbers with no conflict and the dependency completion status as completed, count the task number set meeting the conditions, and generate a non-conflict task number set;

[0094] Acquire task dependency completion identification data in the task queue. The current task queue contains items of tasks. The task dependency completion identification is a binary state, 1 indicating completed dependency and 0 indicating incomplete dependency. The example task dependency state data is . Synchronously call the calculation resource occupation conflict identification data in the current calculation node. The conflict identification is a binary state, 1 indicating conflict and 0 indicating no conflict. The example conflict state data is . The task numbers that meet the conditions of dependency completion state = 1 and conflict identification = 0 are sequentially screened out, and the example screening process is as follows: task 1 dependency completion, no conflict; task 4 dependency completion, no conflict; task 6 dependency not completed, skip; task 7 dependency completion, no conflict; task 10 dependency completion, no conflict. The task number set that meets the conditions after screening is summarized . The conflict determination logic uses binary intersection screening to avoid conflict tasks occupying resources and improve scheduling real-time performance. The conflict identification is determined based on the task load analysis module, and the state synchronization period is 1 second, so the current screening result has high stability. Finally, the conflict-free task number set is generated.

[0095] S412: Based on the conflict-free task number set, call the strategy branch to update the weight parameter value corresponding to each strategy number in the weight set, map the task number and the strategy number, calculate the strategy mapping score of each task, and obtain the task mapping score sequence;

[0096] Based on the conflict-free task number set , call the strategy branch to update the weight parameter value corresponding to each strategy number in the weight set, the current updated weight set contains strategy branches, and the weight example is . The task number and the strategy number are mapped in a one-to-one manner, and the mapping rule is indexed according to the task number modulo the number of strategy numbers (the number of strategy numbers in this embodiment is 6) remainder + 1, for example, task number 1 maps strategy 1 + 1 = 2, that is , task number 4 maps strategy 4 + 1 = 5, that is , task number 7 maps strategy 7 + 1 = 2, that is , task number 10 maps strategy 10 + 1 = 5, that is . Calculate the strategy mapping score of each task, and use the summation average method. If task number 1 maps , task number 4 maps , and the mapping score of task number 1 is 0.72, if the task is mapped to multiple strategy numbers, the average value is calculated. The mapping process uses the latest parameters in the weight set of the strategy branch, and the weight parameter interval is derived from the latest convergence state after dynamic training of the strategy network, and the update period is adjusted according to the strategy training rounds, and the regular setting is 5 rounds once. After the mapping is summarized, the task mapping score sequence is obtained, for example .

[0097] S413: According to the task mapping score sequence, sort the task numbers in descending order of scores, select the task number group with high ranking, and establish the corresponding batch number mapping relationship according to the sorting order to establish the strategy mapping task batch set;

[0098] According to the task mapping score sequence , the tasks are ranked from high to low according to the scores, and if the same score appears, the tasks are ranked in ascending order according to the task number. The current ranking result is task 1, task 7 (both 0.72), task 4, task 10 (both 0.66). The top-ranked task number group is screened, and all ranking results are taken to establish a batch number mapping relationship. The batch number is numbered in ascending order according to the ranking order. Task 1 is numbered batch 1, task 7 is numbered batch 2, task 4 is numbered batch 3, and task 10 is numbered batch 4. The reference standard for sorting mapping is the average response score standard interval of the strategy network , the batch mapping preferentially selects tasks higher than the interval median to enter the front position, improving the execution response priority. The batch number mapping result is shown in Table 3. The strategy mapping task batch set is established for subsequent strategy scheduling stage.

[0099] .

[0100] As shown in Table 3, the strategy mapping number and mapping score of each task number are clear, and the subsequent scheduling can be executed according to the batch priority order.

[0101] Please refer to Figure 6 , the acquisition steps of the strategy execution sequence list are as follows:

[0102] S511: Based on the strategy mapping task batch set, the remaining computing capability data of each execution unit is called, the batch task corresponding allocation unit number is extracted, the execution unit is matched and the current remaining computing capability value is called, and the value difference between the task batch demand and the execution unit capability is calculated. The batch capability difference set is generated;

[0103] Based on the strategy mapping task batch set, the remaining computing capability data of each execution unit is called. The current system deployment has execution units, the example number , the remaining computing capability data unit is , and the real-time collection value is . The batch task number mapping is: batch 1 corresponds to , batch 2 corresponds to , batch 3 corresponds to , and batch 4 corresponds to . The batch task average computing power demand value is calculated from the task execution history data, and the unit is , for example, batch 1 demand , batch 2 demand , batch 3 demand , and batch 4 demand . For each batch, the current remaining computing capability value of the corresponding execution unit is called and directly compared with the batch task average computing power demand value, and the difference value For example, batch 1 difference , batch 2 , batch 3 , batch 4 . Calculate the difference in computing power The positive and negative values reflect whether the current execution unit has sufficient capacity to perform the task batch, Indicates that there is a surplus, Then the current capacity is insufficient. Finally, a batch of capacity difference sets .

[0104] S512: According to the batch capacity difference set, call the computing power adaptation threshold, judge whether each difference is less than the adaptation threshold, filter the task batch number that meets the adaptation condition, integrate the batch information and the corresponding task list that meets the condition, and generate an adaptation task batch index set;

[0105] According to the batch capacity difference set, call the computing power adaptation threshold , judge whether each difference meets the adaptation condition. The setting is based on the execution unit load management module optimization strategy, combined with the historical task throughput rate standard, the current setting , the interval is , according to the resource fluctuation standard deviation, the current system standard deviation . The difference value screening rule is: if Then the screening is passed, for example, the judgment is as follows: batch 1 , pass; batch 2 , pass; batch 3 , not pass; batch 4 , pass. The batch number that meets the adaptation condition is screened out . Integrate the screening result, combine the task list corresponding to the batch number, and form an adaptation task batch index set, for example, batch 1 contains task 1, batch 2 contains task 7, and batch 4 contains task 10. This screening logic ensures that only the batch with sufficient computing power is entered into the downstream strategy execution when the computing power fluctuates dynamically, ensuring the stability of system scheduling.

[0106] S513: Based on the adaptation task batch index set, extract the task number and execution unit mapping information, generate an execution list according to the batch number arrangement order, call the mapping task index and execution unit number of each record, and use the formula:

[0107] ;

[0108] Calculate the task batch mapping stability score, determine the sorting weight and position relationship according to the score, and establish a strategy execution sequence list;

[0109] Among them, Indicates the first​ a mapping stability score of a task batch, an average execution load normalization value of a task batch, a task resource contention rate normalization value in a batch, an average waiting time normalization value of a task batch, a task waiting impact adjustment factor.

[0110] Based on the adaptive task batch index set , the corresponding task number and execution unit mapping information are extracted, and an execution list is generated in the batch number sorting order. The mapping task index and its execution unit number of each record are called to further calculate the batch mapping stability score, using the formula: ;

[0111] Parameter description: is the batch mapping stability score, is the average execution load normalization value of the batch, the value range , for example (normalized by the percentage of task to unit computing power), is the batch task resource contention rate normalization value, for example , is the average waiting time normalization value of the batch, for example , and the waiting impact adjustment factor is set according to the strategy optimization experiment, the recommended interval , and the current selection .

[0112] The formula operation process is as follows (taking batch 1 as an example): ;

[0113] The innovation point of the formula is to smooth the high contention task fluctuation through the square root term, and to balance the performance mutation caused by long-term waiting through the logarithmic term, thereby enhancing the stability and response efficiency of the scheduling sequence in a complex load environment. The current batch score result is, for example , the execution priority weight is determined according to the score, and finally the strategy execution sequence list is established.

[0114] ​​​​The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in other forms. Any skilled person in the art can modify or change the disclosed technical content into equivalent embodiments with equivalent changes, and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solution content of the present application, still falls within the protection scope of the present application.

Claims

1. A robot decision-making method based on deep reinforcement learning, characterized in that, The method comprises the following steps: S1: acquiring real-time environment data of a robot in new energy vehicle generation, calculating adjacent frame difference values, screening data frames greater than a change rate threshold, extracting data segment feature mean vector in a corresponding time period, and generating a mutation segment feature mean vector set; S2: based on the mutation segment feature mean vector set, calculating the ratio between the remaining time and the shortest distance of the obstacle, calling the ratio and the mutation segment feature mean vector set for splicing, constructing a state input encoding matrix, and generating a strategy input encoding matrix; S3: based on the strategy input encoding matrix, calling existing strategy branch weight parameter vectors in a deep reinforcement learning strategy network, calculating the inner product between the encoding matrix and the weight parameter vector, selecting a strategy branch smaller than a reference value, adjusting the corresponding weight parameter value, and generating a strategy branch updated weight set; S4: acquiring task dependency completion identification data in a task queue, screening a task group with no conflict, calculating a task group strategy mapping score based on the strategy branch updated weight set, sorting by score, screening a task group with a high ranking, and generating a strategy mapping task batch set; S5: based on the strategy mapping task batch set, calling the remaining computing capacity of an execution unit, calculating the difference between the strategy mapping task batch set and the remaining computing capacity coefficient, judging whether the difference is less than a computing capacity adaptation threshold, screening a task batch meeting the adaptation condition, and generating a strategy execution sequence list; The mutation segment feature mean vector set comprises a time period number index, a change rate statistical vector, a feature segment duration, a feature segment start and end timestamp, and a feature segment weight score. The strategy input encoding matrix comprises a time period input encoding index, a feature splicing vector, a remaining time proportion item, an obstacle distance proportion item, and a state vector length identifier. The strategy branch updated weight set specifically comprises a strategy number index, an adjusted weight value, an original weight difference value, an update trigger identifier, and an update timestamp. The strategy mapping task batch set comprises a batch number, a task index in the batch, a mapping score sorting in the batch, a batch execution priority identifier, and a batch available resource index. The strategy execution sequence list comprises a sequence number, a task mapping index in the sequence, a corresponding execution unit number, a mapping priority score, and an available computing power adaptation record.

2. The deep reinforcement learning based robot decision-making method according to claim 1, characterized in that, The acquisition step of the mutation segment feature mean vector set specifically comprises: S111: acquiring real-time environment data of a robot in new energy vehicle generation, the real-time environment data comprising a position change rate data sequence, an environment light intensity change rate sequence, and an end effector contact force change rate sequence, calculating the difference between adjacent data frames of the data sequence, calling the difference and a set change rate threshold for comparison, judging whether the difference is greater than the threshold, screening data frames greater than the threshold, and generating a threshold overrun frame index set; S112: based on the threshold overrun frame index set, calling the original position change rate data sequence, the environment light intensity change rate data sequence, and the end effector contact force change rate data sequence, screening data frames in a corresponding time period, and calculating the mean value of each segment change rate to obtain a segment change rate mean value set; S113: According to the segment change rate average set, the corresponding time period number index, the change rate statistical value, the segment duration, the segment start and end timestamp are extracted, the segment change rate statistical value is compared with the set segment weight score benchmark value, the segment characteristics meeting the conditions are screened, and the mutation segment characteristic average vector set is generated.

3. The deep reinforcement learning based robot decision-making method according to claim 2, characterized in that, The strategy input coding matrix acquisition step is specifically: S211: Based on the mutation segment characteristic average vector set, the current task remaining execution time recorded by the robot task scheduling unit is called, the time difference value between the task remaining time value corresponding to the current time and the task estimated completion time is collected, the unit progress time matching value is calculated combined with the task actual execution progress data, and the task urgency value is generated; S212: The shortest distance sequence of the obstacle collected by the current environment perception unit is called, the nearest obstacle distance value at the current timestamp is obtained, the ratio calculation is performed based on the task urgency value and the shortest distance value of the obstacle, the ratio parameter between the task urgency and the space obstruction is calculated, and the task environment correlation ratio information is generated; S213: The task environment correlation ratio information and the mutation segment characteristic average vector set are called, and the two are vector spliced and combined, the spliced index mapping is performed according to the frame number dimension, the joint representation structure of each segment state input information is constructed, and the strategy input coding matrix is established.

4. The deep reinforcement learning based robot decision-making method according to claim 3, characterized in that, The strategy branch update weight set acquisition step is specifically: S311: Based on the strategy input coding matrix, the weight parameter vector of each strategy branch in the deep reinforcement learning strategy network is called, the inner product calculation is performed on each row vector in the strategy input coding matrix and the corresponding strategy weight parameter vector according to the index position, the coding response score of each strategy branch is obtained, and the strategy response score sequence is generated; S312: According to the strategy response score sequence, the strategy priority adjustment benchmark value is called, and each strategy response score is compared with the benchmark value, the strategy number smaller than the benchmark value is screened, the strategy identification meeting the condition is counted, and the strategy weight adjustment index set is generated; S313: The strategy weight adjustment index set is called, the original weight value in the weight parameter vector is located, the corresponding response score is called and modified combined with the strategy input matrix average value, the weight variation amplitude is calculated and obtained, the weight value is adjusted and the weight parameter of the strategy branch is reconstructed, and the strategy branch update weight set is obtained.

5. The deep reinforcement learning based robot decision-making method according to claim 4, characterized in that, The strategy mapping task batch set acquisition step is specifically: S411: The task dependency completion identification data in the task queue is acquired, the calculation resource occupation conflict identification data in the current computing node is called, the task number with no conflict identification and completed state is screened, the task number set meeting the condition is counted, and the non-conflict task number set is generated; S412: Based on the non-conflict task number set, the weight parameter value corresponding to each strategy number in the strategy branch update weight set is called, the mapping is performed according to the task number and the strategy number, the strategy mapping score of each task is calculated, and the task mapping score sequence is obtained; S413: According to the task mapping score sequence, ranking from high to low, screening the top task number group, establishing the corresponding batch number mapping relationship according to the order, and establishing the strategy mapping task batch set.

6. The deep reinforcement learning based robot decision-making method according to claim 5, characterized in that, The acquisition step of the strategy execution sequence list is specifically: S511: Based on the strategy mapping task batch set, call the remaining computing capacity data of each execution unit, extract the batch task corresponding allocation unit number, match the execution unit and call the current remaining computing capacity value, compare with the average computing power demand value of the batch task, calculate the numerical difference between the task batch demand and the execution unit capacity, and generate the batch capacity difference value set; S512: According to the batch capacity difference value set, call the computing capacity adaptation threshold, judge whether each difference value is less than the adaptation threshold, screen the task batch number that meets the adaptation condition, integrate the batch information and the task list that meets the condition, and generate the adaptation task batch index set; S513: Based on the adaptation task batch index set, extract the task number and execution unit mapping information, generate the execution list according to the batch number arrangement order, call the mapping task index and execution unit number of each record, calculate the task batch mapping stability score, determine the sorting weight and position relationship according to the score, and establish the strategy execution sequence list.

Images