Robot decision-making method based on deep reinforcement learning

Abstract

The invention relates to the technical field of robot control, in particular to a robot decision-making method based on deep reinforcement learning, which comprises the following steps: acquiring a change rate data sequence, calculating a frame difference value, screening data frames greater than a threshold value, extracting a mean vector, generating a feature vector set, splicing a ratio of remaining time to distance, constructing a coding matrix and adjusting weight parameters. Screening task groups, mapping and sorting, generating a task batch set, screening adaptive tasks, and generating an execution sequence list. According to the method, adjacent frame difference screening is carried out on multiple change rate sequences in the environment perception data, the mutation section features are extracted, the strategy priority is dynamically adjusted by screening specific inner product results in the strategy evaluation process, the strategy adaptability is improved, the task combination with the minimum conflict and the highest priority is screened, and the strategy evaluation efficiency is improved. And task batches with higher matching degree are screened according to the calculation capability difference value, so that the autonomous response capability and execution reliability of the robot system in a complex task scene are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot decision-making method based on deep reinforcement learning. Background Art

[0002] The field of robotic control technology primarily involves motion control, actuator actuation, sensor information processing, path planning, task scheduling, and coordinated control of robotic systems. This field encompasses the core control technologies required for different types of robots, including industrial robots, service robots, and specialized robots, to perform their intended tasks. These technologies typically include sensor data acquisition and fusion, control algorithm design, real-time feedback regulation, kinematic and dynamic modeling, stability control, and error compensation mechanisms. The control goal is typically to achieve high-precision, high-efficiency, and high-reliability robotic motion execution and multi-robot collaboration. Application scenarios include automated production lines, logistics handling, medical assistance, disaster relief, and other fields.

[0003] Robot decision-making refers to the process by which a robot system generates appropriate action or behavior sequences based on environmental perception data, internal state, and preset goals, using specific algorithms and models. Its purpose is to enable robots to make autonomous decisions, dynamically adjust their behavior strategies based on real-time changing environmental conditions and task requirements, and implement functions such as autonomous navigation, path selection, task priority determination, and action selection. These methods are widely used in scenarios such as new energy vehicle production and processing, intelligent manufacturing, and more.

[0004] Although traditional decision-making methods can achieve real-time feedback adjustment through sensor data collection and fusion, they usually process environmental states with static models, making it difficult to effectively distinguish key change segments in dynamic environments, resulting in delayed responses to sudden environmental events. In addition, the coordinated control of traditional methods is mostly driven by fixed priorities or static policy tables, lacking a dynamic adjustment mechanism for task priorities during execution. In the task scheduling process, computing resource conflicts and dependencies are not adequately handled, resulting in unbalanced resource allocation when multiple tasks are concurrent. It is easy for some high-priority tasks to be blocked by low-priority tasks, affecting the overall task completion efficiency and the real-time performance and reliability of the robot system. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art and propose a robot decision-making method based on deep reinforcement learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a robot decision-making method based on deep reinforcement learning, comprising the following steps: S1: Obtain the real-time environmental data of the robot in the generation of new energy vehicles, calculate the difference between adjacent frames, filter the data frames with a rate of change greater than the threshold, extract the feature mean vector of the data segment in the corresponding time period, and generate the feature mean vector set of the mutation segment; S2: Based on the mutation segment feature mean vector set, calculate the ratio between the remaining time and the shortest distance to the obstacle, call the ratio and the mutation segment feature mean vector set for splicing, construct a state input encoding matrix, and generate a strategy input encoding matrix; S3: Based on the policy input encoding matrix, call the existing policy branch weight parameter vector in the deep reinforcement learning policy network, calculate the inner product between the encoding matrix and the weight parameter vector, select the policy branch with a smaller value than the benchmark value, adjust the corresponding weight parameter value, and generate a policy branch update weight set; S4: Obtain task dependency completion identification data in the task queue, filter out task groups with conflict identification as conflict-free, update the weight set based on the strategy branch, calculate the strategy mapping score of the task group, sort by score, filter out the top-ranked task groups, and generate a strategy mapping task batch set.

[0007] The present invention has the following improvements: the mutation segment feature mean vector set includes a time period number index, a change rate statistical vector, a duration within the feature segment, a feature segment start and end timestamps, and a feature segment weight score; the strategy input coding matrix includes a time period input coding index, a feature splicing vector, a remaining time proportion item, an obstacle distance proportion item, and a state vector length identifier; the strategy branch update weight set is specifically a strategy number index, an adjusted weight value, an original weight difference, an update trigger identifier, and an update timestamp; the strategy mapping task batch set includes a batch number, a task index within a batch, a mapping score ranking within a batch, a batch execution priority identifier, and a batch available resource index.

[0008] The present invention is improved in that the steps of obtaining the mutation segment feature mean vector set are specifically as follows: S111: Acquire real-time environmental data of the robot in the generation of new energy vehicles, the real-time environmental data including a position change rate data sequence, an ambient light intensity change rate sequence, and an end-effector contact force change rate sequence, calculate the difference between adjacent data frames of the data sequence, call the difference and compare it with a set change rate threshold, determine whether the difference is greater than the change rate threshold, filter out data frames with a value greater than the threshold, and generate a threshold-exceeding frame index set; S112: Based on the threshold-exceeding frame index set, the original position change rate data sequence, the ambient light intensity change rate data sequence, and the end-effector contact force change rate data sequence are called to filter the data frames within the corresponding time period, and the mean change rate of each segment is calculated to obtain the mean change rate set within the segment; S113: Based on the intra-segment change rate mean value set, extract the corresponding time period number index, change rate statistics, intra-segment duration, and segment start and end timestamps, call the segment change rate statistics and compare them with the set segment weight score benchmark value, screen the segment features that meet the conditions, and generate the mutation segment feature mean vector set.

[0009] The present invention is improved in that the step of obtaining the strategy input coding matrix is ​​specifically as follows: S211: Based on the mutation segment feature mean vector set, the remaining execution time of the current task recorded by the robot task scheduling unit is called, the time difference between the remaining time value of the task corresponding to the current moment and the expected completion time of the task is collected, and the unit progress time ratio value is calculated in combination with the actual execution progress data of the task to generate the task urgency value; S212: Calling the obstacle shortest distance sequence collected by the current environment perception unit, obtaining the nearest obstacle distance value at the current timestamp, performing a ratio calculation based on the task urgency value and the obstacle shortest distance value, calculating a ratio parameter between the task urgency and the spatial obstruction, and generating task environment correlation ratio information; S213: Call the task environment association ratio information and the mutation segment feature mean vector set, perform vector splicing combination on the two, perform splicing index mapping according to the frame number dimension, construct a joint representation structure for each segment of state input information, and establish a strategy input encoding matrix.

[0010] The present invention is improved in that the step of obtaining the strategy branch update weight set is specifically as follows: S311: Based on the policy input encoding matrix, call the weight parameter vector of each policy branch in the deep reinforcement learning policy network, perform inner product calculation on each row vector in the policy input encoding matrix and the corresponding policy weight parameter vector according to the index position, obtain the encoding response score of each policy branch, and generate a policy response score sequence; S312: Based on the policy response score sequence, call the policy priority adjustment benchmark value, compare each policy response score item by item to see if it is less than the benchmark value, filter the policy numbers that are less than the benchmark value, count the policy identifiers that meet the conditions, and generate a policy weight adjustment index set; S313: Call the strategy weight adjustment index set, locate the original weight value in the weight parameter vector, call the corresponding response score and make corrections based on the average value of the strategy input matrix, calculate to obtain the weight change amplitude, adjust the weight value and reconstruct the weight parameters of the strategy branch to obtain the strategy branch update weight set.

[0011] The present invention is improved in that the steps of obtaining the strategy mapping task batch set are specifically as follows: S411: Obtain task dependency completion identification data in the task queue, call computing resource occupation conflict identification data in the current computing node, filter task numbers with conflict identifications of no conflict and dependency completion status of completed, count task numbers that meet the conditions, and generate a set of conflict-free task numbers; S412: Based on the conflict-free task number set, calling the strategy branch to update the weight parameter value corresponding to each strategy number in the weight set, mapping the task number with the strategy number, calculating the strategy mapping score of each task, and obtaining a task mapping score sequence; S413: Sort the task mapping score sequence from high to low by score, filter the top-ranked task number groups, establish corresponding batch number mapping relationships according to the sorting order, and establish a strategy mapping task batch set.

[0012] The present invention is improved in that the method further comprises the following steps: S5: Based on the strategy mapping task batch set, call the remaining computing capacity of the execution unit, calculate the difference between the strategy mapping task batch set and the remaining computing capacity coefficient, determine whether the difference is less than the computing capacity adaptation threshold, screen the task batches that meet the adaptation conditions, and generate a strategy execution sequence list; The strategy execution sequence list includes a sequence number, a task mapping index within the sequence, a corresponding execution unit number, a mapping priority score, and an available computing power adaptation record.

[0013] The present invention is improved in that the steps of obtaining the strategy execution sequence list are specifically as follows: S511: Based on the strategy mapping task batch set, call the remaining computing capacity data of each execution unit, extract the allocation unit number corresponding to the batch task, match the execution unit and call the current remaining computing capacity value, compare it with the average computing capacity requirement value of the batch task, calculate the numerical difference between the task batch requirement and the execution unit capacity, and generate a batch capacity difference value set; S512: Based on the batch capacity difference set, the computing capacity adaptation threshold is called to determine whether each difference is less than the adaptation threshold, the task batch numbers that meet the adaptation conditions are screened, the batch information that meets the conditions is integrated with the corresponding task list, and an index set of adapted task batches is generated; S513: Based on the adaptation task batch index set, extract the task number and execution unit mapping information, generate an execution list in the order of batch number arrangement, call the mapping task index and execution unit number of each record, calculate and obtain the task batch mapping stability score, determine the sorting weight and position relationship according to the score, and establish a strategy execution sequence list.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, by performing adjacent frame difference screening on multiple change rate sequences in environmental perception data and extracting mutation segment features, the ability to capture key change trends in dynamic environments can be enhanced, and the data structure of decision input is optimized by combining the ratio information calculated by the remaining execution time of the task and the obstacle distance. In the strategy evaluation process, the strategy priority is dynamically adjusted by screening specific inner product results, thereby improving the strategy adaptability. In task scheduling, the task dependency completion identifier and the computing resource occupancy conflict identifier are integrated to screen the task combination with the minimum conflict and the highest priority, and the task batches with higher matching degree are screened according to the computing power difference. As a whole, the flexibility of strategy adjustment and the efficiency of task scheduling in dynamic environments are improved, and the autonomous response capability and execution reliability of the robot system in complex task scenarios are enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a flow chart of the present invention for obtaining a feature mean vector set of a mutation segment; Figure 3 A flow chart of the encoding matrix for obtaining the strategy input of the present invention; Figure 4 A flow chart of obtaining a strategy branch update weight set for the present invention; Figure 5 A flowchart of obtaining a strategy mapping task batch set for the present invention; Figure 6 A flowchart of obtaining a strategy execution sequence list for the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0017] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the devices or elements referred to must have a specific direction, be constructed and operate in a specific direction, and therefore should not be understood as limiting the present invention. In addition, in the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0018] See also Figure 1 The present invention provides a technical solution: a robot decision-making method based on deep reinforcement learning, comprising the following steps: S1: Acquire the real-time environmental data of the robot in the generation of new energy vehicles. The real-time environmental data includes a position change rate data sequence, an ambient light intensity change rate sequence, and an end-effector contact force change rate sequence. Calculate the difference between adjacent frames based on the three data items between adjacent data frames, call the difference and compare it with the set change rate threshold, determine whether the difference is greater than the change rate threshold, filter out data frames with a value greater than the threshold, extract the feature mean vector of the data segment in the corresponding time period based on the screening result, and generate a mutation segment feature mean vector set; S2: Based on the feature mean vector set of the mutation segment, the remaining execution time of the robot's current task and the shortest distance sequence of the current obstacle are called, the ratio between the remaining time and the shortest distance to the obstacle is calculated, and the ratio is called and spliced ​​with the feature mean vector set of the mutation segment to construct the state input encoding matrix and generate the strategy input encoding matrix; S3: Based on the policy input encoding matrix, call the existing policy branch weight parameter vector in the deep reinforcement learning policy network, calculate the inner product between the encoding matrix and the weight parameter vector, determine whether the inner product value is less than the set policy priority adjustment benchmark value, filter out policy branches with a value less than the benchmark value, adjust the corresponding weight parameter value based on the filtered policy branches, and generate a policy branch update weight set; S4: Obtain task dependency completion identification data in the task queue, call the computing resource occupation conflict identification, filter the task group with conflict identification as conflict-free, update the weight set based on the policy branch, calculate the policy mapping score of the task group, sort by score, filter the top-ranked task group, and generate a policy mapping task batch set; S5: Based on the strategy mapping task batch set, call the remaining computing capacity of the execution unit, calculate the difference between the strategy mapping task batch set and the remaining computing capacity coefficient, determine whether the difference is less than the computing capacity adaptation threshold, screen the task batches that meet the adaptation conditions, and generate a strategy execution sequence list; The position change rate is the change value of the end effector or robot reference point position per unit time; the ambient light intensity change rate is the rate of change of the brightness collected by the light sensor per unit time; the end effector contact force change rate refers to the rate of change of the end force obtained by the force sensor over time; the remaining execution time is the difference between the current time and the scheduled completion time of the task; the shortest obstacle distance is the minimum distance between the obstacle detected by the robot and the robot path; the weight parameter vector is the trainable weight corresponding to each action in the policy network; the policy priority adjustment baseline value is the weight adjustment threshold set during system design, which is usually a value in the normalized range of 0-1; the task dependency completion flag is a Boolean value that marks whether the task has unfinished dependencies; the computing resource occupation conflict flag is a Boolean value that marks whether the current resources are available for scheduling the task; the remaining computing capacity of the execution unit is a remaining capacity score calculated by CPU usage, memory remaining, I / O load, etc.; the computing capacity adaptation threshold is the minimum schedulable capacity reference value set for the execution unit.

[0019] The mutation segment feature mean vector set includes the time period number index, the change rate statistical vector, the duration within the feature segment, the feature segment start and end timestamps, and the feature segment weight score. The strategy input coding matrix includes the time period input coding index, the feature splicing vector, the remaining time proportion item, the obstacle distance proportion item, and the state vector length identifier. The strategy branch update weight set specifically includes the strategy number index, the adjusted weight value, the original weight difference, the update trigger identifier, and the update timestamp. The strategy mapping task batch set includes the batch number, the task index within the batch, the mapping score ranking within the batch, the batch execution priority identifier, and the batch available resource index. The strategy execution sequence list includes the sequence number, the task mapping index within the sequence, the corresponding execution unit number, the mapping priority score, and the available computing power adaptation record.

[0020] See also Figure 2 , the specific steps for obtaining the mutation segment feature mean vector set are: S111: Acquire real-time environmental data of the robot in the generation of new energy vehicles, the real-time environmental data including a position change rate data sequence, an ambient light intensity change rate sequence, and an end-effector contact force change rate sequence, calculate the difference between adjacent data frames of the data sequence, call the difference and compare it with a set change rate threshold, determine whether the difference is greater than the change rate threshold, filter out data frames with a value greater than the threshold, and generate a threshold-exceeding frame index set; Obtain the position change rate data sequence, ambient light intensity change rate data sequence, and end-effector contact force change rate data sequence in the robot environment perception unit. For the position change rate data sequence, calculate the difference between adjacent data frames frame by frame. The sampling period is , the collected position change rate data sequence is , the result of calculating adjacent differences is Similarly, the data sequence of the ambient light intensity change rate , the calculated difference is , data series of the contact force change rate of the end effector , the calculated difference is , call the corresponding change rate threshold for each set of difference values ​​for comparison, the position change rate threshold Set to , ambient light intensity change rate threshold , contact force change rate threshold , threshold The setting basis is: according to the abnormal value monitoring interval of the position change rate under the high-speed motion state of the robot , take the middle of the interval ; The rate of change of light intensity is based on the actual fluctuation range of illumination mutation scene , take the median ; The contact force change rate is based on the actual measurement of contact impact during movement , select , filter the difference according to the above threshold, the 3rd and 4th items of the position change rate difference meet the requirements, the 2nd item of the light intensity change rate difference meets the requirements, and the 1st and 3rd items of the contact force change rate difference meet the requirements. The filter result corresponds to the frame index , after merging and removing duplicates, it is sorted into , get the threshold-exceeding frame index set.

[0021] S112: Based on the threshold-exceeding frame index set, call the original position change rate data sequence, the ambient light intensity change rate data sequence, and the end-effector contact force change rate data sequence to filter the data frames within the corresponding time period using the formula: ; The mean of the rate of change in each segment is obtained by operation, and the mean set of the rate of change within the segment is obtained; in, Indicates the The mean rate of change of the segment, Indicates the Normalized value of frame position change rate, Indicates the Normalized value of the frame ambient light intensity change rate, Indicates the Normalized value of the contact force change rate of the end effector of the frame, Indicates the The number of data frames in the segment; Based on the threshold exceeding frame index set, call the original position change rate data sequence , data series of ambient light intensity change rate , data series of the contact force change rate of the end effector , filter out the over-limit frame segment, segment division window , normalize the data within the segment, and normalize the position change rate interval , normalized results , normalized interval of light intensity change rate , normalized , normalized range of contact force change rate , normalized , put it into the formula: ; Formula parameter description: For the The mean of the segment change rate, For the Normalized value of frame position change rate, For the Normalized value of the frame ambient light intensity change rate, For the Normalized value of the frame contact force change rate, For the The calculation logic for the number of frames within a segment is to weight the three types of change rate parameters, normalize them, and process spatial changes (position), external perception (lighting), and force feedback (contact) respectively. The square root and logarithm are used to enhance the difference in data distribution and improve the ability to distinguish sudden segments.

[0022] The innovation of the formula is that it uses the normalized values ​​of the three types of sensors and Operation, strengthen the influence of different physical quantities, and enhance the ability to identify abnormal segments.

[0023] Calculation process: ; The calculation process is similar. After calculation, the mean of segment 1 is 0.795, and the mean of segment 2 is 0.934, generating a set of mean values ​​of the intra-segment change rate.

[0024] .

[0025] As shown in Table 1, the normalized parameter results provide basic data support for the intra-segment mean calculation process.

[0026] S113: Based on the intra-segment change rate mean value set, the corresponding time period number index, change rate statistics, intra-segment duration, and segment start and end timestamps are extracted. The segment change rate statistics are compared with the set segment weight score benchmark value, and the segment features that meet the conditions are screened to generate the mutation segment feature mean vector set.

[0027] According to the mean value of the intra-segment change rate , time period number , duration of each segment , segment start and end timestamps segment 1 is , segment 2 is , call the segment weight score benchmark value , Setting basis: The average difference range of the robot's high-dynamic operation scene monitoring , select the median This value fluctuates with the standard deviation of environmental disturbance. When the standard deviation is greater than 0.15, it is recommended to take , the current test standard deviation is 0.12, select , compare the mean change rate within the segment, segment 1 , segment 2 , filter segment 2 to meet the conditions, establish segment feature vector, number 2, change rate statistics value 0.934, duration , start and end timestamps , generate a set of feature mean vectors of mutation segments.

[0028] See also Figure 3 , the steps to obtain the strategy input encoding matrix are as follows: S211: Based on the mutation segment feature mean vector set, the remaining execution time of the current task recorded by the robot task scheduling unit is called, and the time difference between the remaining time value of the task corresponding to the current moment and the expected completion time of the task is collected. Combined with the actual execution progress data of the task, the unit progress time ratio value is calculated to generate the task urgency value; Based on the mutation segment feature mean vector set, call the remaining execution time of the current task recorded in the robot task scheduling module and collect the current timestamp The remaining time of the corresponding task ,calculate Estimated completion time of the task The time difference between , combined with the task completion progress recorded by the task execution module , calculate the unit progress time ratio value , , such as the current timestamp , estimated completion time of the task , the task has been completed ,calculate , unit progress time ratio value Task urgency value Through normalization, ,in It is the maximum unit progress time ratio value in the current batch, in the whole batch Monitoring obtained ,but Urgency normalization coefficient The setting basis is that the task set is collected multiple times in actual operation and the interval is found The most distinguishing, set the reference interval according to the actual operation scene task density adjustment, if the robot high density scheduling scene, it is recommended to take the interval This time In line with the reference interval, the task urgency value is finally generated.

[0029] S212: Calling the obstacle shortest distance sequence collected by the current environment perception unit, obtaining the nearest obstacle distance value at the current timestamp, performing a ratio calculation based on the task urgency value and the shortest obstacle distance value, calculating the ratio parameter between the task urgency and the spatial obstruction, and generating task environment correlation ratio information; Call the shortest distance sequence of obstacles collected in the current environment perception module, the current timestamp Minimum distance to the obstacle The reading value is 1.8m, which is the task urgency value obtained from paragraph 1. , the two parameters are divided to calculate the ratio parameter between task urgency and space obstruction ,but Obstacle distance threshold The setting is based on the robot's safe operating distance standard and is set according to ISO10218-1 standard. ,current , so it enters the normal judgment process. Spatial hindrance ratio The reference interval is summarized after field data collection, and the normalized reference interval is ,current It is valid within the interval and generates task-environment correlation ratio information. As shown in Table 2, the monitoring data of urgency value and environmental obstacle ratio under different batches of tasks are summarized.

[0030] ; As shown in Table 2, each batch 、 They are all within the reference range, effectively reflecting the relationship between the urgency of different tasks and obstacles.

[0031] S213: Calling the task-environment correlation ratio information and the mutation segment feature mean vector set, performing vector splicing and combination on the two, performing splicing index mapping according to the frame number dimension, constructing a joint representation structure for each segment of state input information, and establishing a strategy input encoding matrix; Calling task environment correlation ratio information The vector set of the feature mean of the mutation segment is aligned with the frame number dimension to perform vector splicing and construct a joint representation structure of the state input information. The splicing operation adopts a horizontal splicing method, and each frame corresponds to the splicing field. , such as segment number correspond , splicing results . Splicing mapping weight coefficient use The setting basis is based on the priority of the influence ratio of the sudden segment feature in the task state input feature, and is recommended after field test verification. , currently After concatenation, calculate the state input joint encoding value: ; Bring in data . Splicing joint weight The innovation of this method is that by weighting the mutation segment features and the spatial barrier features, the strategy input encoding matrix is ​​made more sensitive to the spatial mutation situation, thus improving the strategy convergence stability. Higher than the normalized input benchmark of 0.8, this result indicates that the current segment number The state input will focus on strategy encoding and eventually establish a strategy input encoding matrix.

[0032] See also Figure 4 , the steps for obtaining the updated weight set of the strategy branch are as follows: S311: Based on the policy input encoding matrix, call the weight parameter vector of each policy branch in the deep reinforcement learning policy network, calculate the inner product of each row vector in the policy input encoding matrix and the corresponding policy weight parameter vector according to the index position, obtain the encoding response score of each policy branch, and generate a policy response score sequence; Based on the policy input encoding matrix, the weight parameter vector of each policy branch in the deep reinforcement learning policy network is called, and the number of policy branches in the policy network is set to , each branch dimension , and their weight parameter vectors are ,For example , , and so on. The current strategy input encoding matrix is ​​obtained from the previous step, and the matrix row vector corresponds to For each row vector and the corresponding strategy branch weight parameters Perform inner product operation and calculate the formula: ; That is, for The strategy branch has The inner product calculation is also performed on the remaining strategy branches to obtain the encoding response scores corresponding to all strategy branches, and the strategy response score sequence is summarized The purpose of using this calculation logic for the inner product operation is to directly measure the matching degree of the strategy input state on different strategy branches. The weight parameter The interval setting is based on the strategy training phase. The average strategy weight value when the round converges is taken in the interval , ensuring that after each training, the strategy input state will not affect multi-strategy matching due to excessive single bias.

[0033] S312: Based on the policy response score sequence, call the policy priority adjustment benchmark value, compare each policy response score item by item to see if it is less than the benchmark value, filter the policy numbers that are less than the benchmark value, count the policy identifiers that meet the conditions, and generate a policy weight adjustment index set; Score sequence according to policy response , call policy priority adjustment baseline value to filter. The setup logic is as follows: In the initial training phase of the policy network, the normalized mean of the policy response scores in each round is calculated. and standard deviation ,For example , when adjusting The purpose of setting this interval is to prevent the weakening of the screening effect after all strategy response scores approach consistency, and to dynamically adjust and enhance the adjustment probability of low-activity strategies. Compare them in turn, if Then filter the policy number, for example, if ,but Are lower than , screening number , the summary statistics are the strategy weight adjustment index set. The setting basis is derived from the actual operation data interval of multiple rounds of adjustment intervals in the strategy network structure The offset standard deviation of 0.15 is derived from the actual variance mean during 10 rounds of iterations, which reasonably reflects the response activity between strategy branches, adjusts the dynamic convergence of the screening criteria, and improves the robustness of the strategy network.

[0034] S313: Call the policy weight adjustment index set, locate the original weight value in the weight parameter vector, call the corresponding response score and make corrections based on the average value of the policy input matrix using the formula: ; The calculation obtains the weight change amplitude, adjusts the weight value and reconstructs the weight parameters of the strategy branch to obtain the updated weight set of the strategy branch; in, Indicates the The weight change range of each strategy branch, Indicates the The normalized value of the encoding response score corresponding to each strategy branch, represents the mean normalized value of all row vectors of the strategy input encoding matrix, represents the mean normalized value of the variance of all vectors in the policy input encoding matrix, represents the policy response distribution adjustment factor, Indicates the Normalized value of the standard deviation of historical changes of each strategy branch, represents the strategy change inhibition factor, is the normalized change range of the strategy weight, is the base of natural logarithms.

[0035] Call strategy weight adjustment index centralized strategy number , respectively locate their corresponding weight parameters , and call the corresponding response score and the row mean of the strategy input encoding matrix and variance mean , example Strategy response distribution adjustment factor Currently set to 1.0, derived from the sensitivity analysis of the strategy to the input variance, this Through the strategy active status monitoring module setting, when the active strategy volatility is greater than 10%, set , otherwise the default . Historical change standard deviation Example values ​​are , Strategy Change Inhibitor , derived from the policy cumulative error control mechanism, interval . Use the modified calculation formula: ; Formula parameter description: is the change range of strategy weight, is the normalized value of the strategy response score, The mean of the encoding matrix, The mean of the variance of the encoding matrix, Strategy response distribution adjustment factor, Normalized value of historical change standard deviation, Strategy change inhibitor, Take the absolute value to ensure that the change is positive, the denominator Used to smooth low variance states, The update rate of strategies with high historical volatility is automatically lowered. The innovation lies in the composite adjustment mechanism that simultaneously considers response volatility and historical error accumulation, enhancing the convergence of dynamic adjustment of strategy branches.

[0036] Bring the example into calculation, ; ; The result In the adjustment range The numerical meaning indicates that the second branch of the current strategy branch is active but lower than the expected activity level. The branch weight update needs to be moderately improved. The update result will be written into the strategy branch update weight set for the next round of strategy training parameter adjustment.

[0037] See also Figure 5 ,The steps for obtaining the strategy mapping task batch set are as follows: S411: Obtain task dependency completion identification data in the task queue, call computing resource occupation conflict identification data in the current computing node, filter task numbers with conflict identifications of no conflict and dependency completion status of completed, count task numbers that meet the conditions, and generate a set of conflict-free task numbers; Get the task dependency completion identification data in the task queue. The current task queue contains The task dependency completion flag is a binary state, 1 indicates completed dependency, 0 indicates uncompleted dependency, the example task dependency status data is Synchronously call the computing resource occupation conflict identification data in the current computing node. The conflict identification is a binary state, 1 means there is a conflict, 0 means there is no conflict. The example conflict status data is . Filter out the task numbers that meet both the dependency completion status = 1 and the conflict flag = 0. The example filtering process is as follows: Task 1 dependency completed, no conflict; Task 4 dependency completed, no conflict; Task 6 dependency unfinished skipped; Task 7 dependency completed, no conflict; Task 10 dependency completed, no conflict. After summarizing and filtering, we get a set of task numbers that meet the conditions. The conflict determination logic uses binary intersection screening to prevent conflicting tasks from occupying resources and improve scheduling real-time performance. The conflict identification is based on the task load analysis module, and the state synchronization cycle is 1 second. The current screening results are highly stable. Finally, a conflict-free task number set is generated.

[0038] S412: Based on the conflict-free task number set, call the strategy branch to update the weight parameter value corresponding to each strategy number in the weight set, map the task number to the strategy number, calculate the strategy mapping score of each task, and obtain a task mapping score sequence; Based on a set of conflict-free task numbers , call the strategy branch to update the weight parameter value corresponding to each strategy number in the weight set. The current updated weight set contains strategy branches, and the weight examples are The task number and the strategy number are mapped one by one. The mapping rule is indexed by the remainder of the task number modulo the strategy number (the strategy number in this embodiment is 6) + 1. For example, task number 1 is mapped to strategy 1+1=2. , Task number 4 mapping strategy 4+1=5 , Task number 7 mapping strategy 7+1=2 , Task number 10 mapping strategy 10+1=5 Calculate the strategy mapping score of each task by summing and averaging. If task number 1 is mapped , task number 4 mapping , respectively, the mapping score of the example task number 1 is 0.72. If the task is mapped to multiple strategy numbers, the average value is calculated. The mapping process uses the strategy branch to update the latest parameters in the weight set. The weight parameter interval , comes from the latest convergence state after dynamic training of the policy network. The update cycle is adjusted according to the policy training rounds. Generally, it is synchronized once every 5 rounds. After summarizing the mapping, the task mapping score sequence is obtained, for example .

[0039] S413: Sort the task mapping score sequence from high to low by score, filter the top-ranked task number groups, establish corresponding batch number mapping relationships according to the sorting order, and establish a strategy mapping task batch set; Mapping score sequences according to tasks , sort by score from high to low. If there are tasks with the same score, sort them in ascending order by task number. The current sorting results are task 1, task 7 (both 0.72), task 4, task 10 (both 0.66). Filter the top-ranked task number groups, take all sorting results, and establish a batch number mapping relationship. The batch numbers are numbered in ascending order according to the sorting order. Task 1 is numbered batch 1, task 7 is numbered batch 2, task 4 is numbered batch 3, and task 10 is numbered batch 4. The reference standard for sorting mapping is: the standard interval of the average response score of the policy network , batch mapping prioritizes filtering above the median of the interval The tasks with the same strategy are prioritized and placed in the front row, improving the execution response priority. The results of batch number mapping are shown in Table 3. A strategy mapping task batch set is established for the subsequent strategy scheduling stage.

[0040] .

[0041] As shown in Table 3, each task number corresponds to a strategy mapping number and a clear mapping score, and subsequent scheduling can be performed in batch priority order.

[0042] See also Figure 6 , the steps for obtaining the strategy execution sequence list are as follows: S511: Based on the strategy mapping task batch set, call the remaining computing capacity data of each execution unit, extract the allocation unit number corresponding to the batch task, match the execution unit and call the current remaining computing capacity value, compare it with the average computing capacity requirement value of the batch task, calculate the numerical difference between the task batch requirement and the execution unit capacity, and generate a batch capacity difference value set; Based on the strategy mapping task batch set, the remaining computing capacity data of each execution unit is called. The current system deployment has execution units, instance number , the unit of remaining computing power data is , real-time collection value The batch task number is mapped as follows: Batch 1 corresponds to , batch 2 corresponds to , batch 3 corresponds to , batch 4 corresponds to The average computing power requirement of a batch task is calculated based on the historical data of task execution, and the unit is , for example, batch 1 requirements , batch 2 requirements , batch 3 requirements , batch 4 requirements For each batch, call the current remaining computing power value of the corresponding execution unit, compare it directly with the average computing power requirement value of the batch task, and calculate the difference For example, the difference between batch 1 and batch 2 is , batch 2 , batch 3 , batch 4 Computing power difference The positive or negative value of reflects whether the execution unit currently has sufficient capacity to execute the task batch. Indicates surplus. The current capacity is insufficient. Finally, a batch capacity difference set is generated. .

[0043] S512: Based on the batch capacity difference set, the computing capacity adaptation threshold is called to determine whether each difference is less than the adaptation threshold. The task batch numbers that meet the adaptation conditions are screened, and the batch information that meets the conditions is integrated with the corresponding task list to generate an index set of adapted task batches. Based on the batch capability difference set, the computing capability adaptation threshold is called , determine whether each difference meets the adaptation condition. The setting is based on the optimization strategy of the execution unit load management module, combined with the historical task throughput standard, the current setting , the interval is , according to the resource fluctuation statistical standard deviation setting, the current system standard deviation . Execute the difference filter rule: If The screening is passed, the example judgment is as follows: Batch 1 , passed; Batch 2 , passed; batch 3 , failed; Batch 4 , passed. The batch numbers that meet the adaptation conditions are screened out as The filtered results are integrated with the task list corresponding to the batch number to form an index set of adapted task batches. For example, batch 1 contains task 1, batch 2 contains task 7, and batch 4 contains task 10. This filtering logic ensures that when computing power fluctuates dynamically, only batches with sufficient surplus computing power will enter the downstream strategy execution, ensuring system scheduling stability.

[0044] S513: Based on the adapted task batch index set, extract the mapping information between the task number and the execution unit, generate an execution list in the order of the batch numbers, and call the mapping task index and execution unit number of each record using the formula: ; Obtain task batch mapping stability scores through calculation, determine sorting weights and position relationships based on the scores, and establish a strategy execution sequence list; in, Indicates the The mapping stability score of the task batch, Indicates the The normalized value of the average execution load of a batch of tasks, Indicates the The normalized value of the task resource contention rate in a batch, Indicates the The normalized value of the average waiting time of batch tasks, Indicates the task waiting impact adjustment factor.

[0045] Based on the adaptation task batch index set , extract the mapping information of the corresponding task number and execution unit, and generate the execution list in the order of batch number. Call the mapping task index and execution unit number of each record to further calculate the batch mapping stability score using the formula: ; Parameter Description: For the Batch mapping stability score, Normalized value of the average execution load of the batch of tasks, with a value range of ,For example (normalized by the percentage of tasks in the unit computing power), Normalized value for the resource contention rate of tasks within a batch, example , Normalized value for the average waiting time of batch tasks, example , waiting for the impact adjustment factor The setting is based on the strategy tuning experiment, and the recommended interval , currently selected .

[0046] The formula calculation process is as follows (taking batch 1 as an example): ; The innovation of the formula is to smooth the fluctuation of highly contention tasks through the square root term, balance the performance mutation caused by long-term waiting through the logarithmic term, and enhance the stability and response efficiency of the scheduling sequence under complex load environments. , determine the execution priority weight according to the score, and finally establish a strategy execution sequence list.

[0047] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A robot decision-making method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Obtain the real-time environmental data of the robot in the generation of new energy vehicles, calculate the difference between adjacent frames, filter the data frames with a rate of change greater than the threshold, extract the feature mean vector of the data segment in the corresponding time period, and generate the feature mean vector set of the mutation segment; S2: Based on the mutation segment feature mean vector set, calculate the ratio between the remaining time and the shortest distance to the obstacle, call the ratio and the mutation segment feature mean vector set for splicing, construct a state input encoding matrix, and generate a strategy input encoding matrix; S3: Based on the policy input encoding matrix, call the existing policy branch weight parameter vector in the deep reinforcement learning policy network, calculate the inner product between the encoding matrix and the weight parameter vector, select the policy branch with a smaller value than the benchmark value, adjust the corresponding weight parameter value, and generate a policy branch update weight set; S4: Obtain task dependency completion identification data in the task queue, filter out task groups with conflict identification as conflict-free, update the weight set based on the strategy branch, calculate the strategy mapping score of the task group, sort by score, filter out the top-ranked task groups, and generate a strategy mapping task batch set.

2. The robot decision-making method based on deep reinforcement learning according to claim 1, characterized in that: The mutation segment feature mean vector set includes the time period number index, the change rate statistical vector, the duration within the feature segment, the feature segment start and end timestamps and the feature segment weight score; the strategy input coding matrix includes the time period input coding index, the feature splicing vector, the remaining time proportion item, the obstacle distance proportion item and the state vector length identifier; the strategy branch update weight set is specifically the strategy number index, the adjusted weight value, the original weight difference, the update trigger identifier and the update timestamp; the strategy mapping task batch set includes the batch number, the task index within the batch, the mapping score ranking within the batch, the batch execution priority identifier and the batch available resource index.

3. The robot decision-making method based on deep reinforcement learning according to claim 2, characterized in that: The steps for obtaining the mutation segment feature mean vector set are specifically as follows: S111: Acquire real-time environmental data of the robot in the generation of new energy vehicles, the real-time environmental data including a position change rate data sequence, an ambient light intensity change rate sequence, and an end-effector contact force change rate sequence, calculate the difference between adjacent data frames of the data sequence, call the difference and compare it with a set change rate threshold, determine whether the difference is greater than the change rate threshold, filter out data frames with a value greater than the threshold, and generate a threshold-exceeding frame index set; S112: Based on the threshold-exceeding frame index set, the original position change rate data sequence, the ambient light intensity change rate data sequence, and the end-effector contact force change rate data sequence are called to filter the data frames within the corresponding time period, and the mean change rate of each segment is calculated to obtain the mean change rate set within the segment; S113: Based on the intra-segment change rate mean value set, extract the corresponding time period number index, change rate statistics, intra-segment duration, and segment start and end timestamps, call the segment change rate statistics and compare them with the set segment weight score benchmark value, screen the segment features that meet the conditions, and generate the mutation segment feature mean vector set.

4. The robot decision-making method based on deep reinforcement learning according to claim 3, characterized in that: The steps for obtaining the strategy input coding matrix are specifically as follows: S211: Based on the mutation segment feature mean vector set, the remaining execution time of the current task recorded by the robot task scheduling unit is called, the time difference between the remaining time value of the task corresponding to the current moment and the expected completion time of the task is collected, and the unit progress time ratio value is calculated in combination with the actual execution progress data of the task to generate the task urgency value; S212: Calling the obstacle shortest distance sequence collected by the current environment perception unit, obtaining the nearest obstacle distance value at the current timestamp, performing a ratio calculation based on the task urgency value and the obstacle shortest distance value, calculating a ratio parameter between the task urgency and the spatial obstruction, and generating task environment correlation ratio information; S213: Call the task environment association ratio information and the mutation segment feature mean vector set, perform vector splicing combination on the two, perform splicing index mapping according to the frame number dimension, construct a joint representation structure for each segment of state input information, and establish a strategy input encoding matrix.

5. The robot decision-making method based on deep reinforcement learning according to claim 4, characterized in that: The steps for obtaining the strategy branch update weight set are specifically as follows: S311: Based on the policy input encoding matrix, call the weight parameter vector of each policy branch in the deep reinforcement learning policy network, perform inner product calculation on each row vector in the policy input encoding matrix and the corresponding policy weight parameter vector according to the index position, obtain the encoding response score of each policy branch, and generate a policy response score sequence; S312: Based on the policy response score sequence, call the policy priority adjustment benchmark value, compare each policy response score item by item to see if it is less than the benchmark value, filter the policy numbers that are less than the benchmark value, count the policy identifiers that meet the conditions, and generate a policy weight adjustment index set; S313: Call the strategy weight adjustment index set, locate the original weight value in the weight parameter vector, call the corresponding response score and make corrections based on the average value of the strategy input matrix, calculate to obtain the weight change amplitude, adjust the weight value and reconstruct the weight parameters of the strategy branch to obtain the strategy branch update weight set.

6. The robot decision-making method based on deep reinforcement learning according to claim 5, characterized in that: The steps for obtaining the strategy mapping task batch set are specifically as follows: S411: Obtain task dependency completion identification data in the task queue, call computing resource occupation conflict identification data in the current computing node, filter task numbers with conflict identifications of no conflict and dependency completion status of completed, count task numbers that meet the conditions, and generate a set of conflict-free task numbers; S412: Based on the conflict-free task number set, calling the strategy branch to update the weight parameter value corresponding to each strategy number in the weight set, mapping the task number with the strategy number, calculating the strategy mapping score of each task, and obtaining a task mapping score sequence; S413: Sort the task mapping score sequence from high to low by score, filter the top-ranked task number groups, establish corresponding batch number mapping relationships according to the sorting order, and establish a strategy mapping task batch set.

7. The robot decision-making method based on deep reinforcement learning according to claim 6, characterized in that: The method further comprises the following steps: S5: Based on the strategy mapping task batch set, call the remaining computing capacity of the execution unit, calculate the difference between the strategy mapping task batch set and the remaining computing capacity coefficient, determine whether the difference is less than the computing capacity adaptation threshold, screen the task batches that meet the adaptation conditions, and generate a strategy execution sequence list; The strategy execution sequence list includes a sequence number, a task mapping index within the sequence, a corresponding execution unit number, a mapping priority score, and an available computing power adaptation record.

8. The robot decision-making method based on deep reinforcement learning according to claim 7, characterized in that: The steps for obtaining the policy execution sequence list are specifically as follows: S511: Based on the strategy mapping task batch set, call the remaining computing capacity data of each execution unit, extract the allocation unit number corresponding to the batch task, match the execution unit and call the current remaining computing capacity value, compare it with the average computing capacity requirement value of the batch task, calculate the numerical difference between the task batch requirement and the execution unit capacity, and generate a batch capacity difference value set; S512: Based on the batch capacity difference set, the computing capacity adaptation threshold is called to determine whether each difference is less than the adaptation threshold, the task batch numbers that meet the adaptation conditions are screened, the batch information that meets the conditions is integrated with the corresponding task list, and an index set of adapted task batches is generated; S513: Based on the adaptation task batch index set, extract the task number and execution unit mapping information, generate an execution list in the order of batch number arrangement, call the mapping task index and execution unit number of each record, calculate and obtain the task batch mapping stability score, determine the sorting weight and position relationship according to the score, and establish a strategy execution sequence list.

Images