FSM simulation method and system based on virtual factory robot

Abstract

The invention discloses an FSM simulation method and system based on a virtual factory robot, and particularly relates to the technical field of FSM simulation, and the method comprises the steps: employing a Q-Learning algorithm in FSM simulation to optimize a decision process of the robot, extracting a current task and a future task of the robot in the decision process of the robot, and carrying out the optimization of the current task and the future task of the robot, historical reward values of different future tasks of the robot are analyzed, exponential weighted moving average analysis and probability density analysis are carried out on the historical reward values of the future tasks, uncertainty information of the future tasks is determined, and dependency information of the future tasks is obtained through comparative analysis; according to the method, uncertainty information and dependency information of future tasks are comprehensively analyzed, discount factors of a Q-Learning algorithm are dynamically adjusted, the discount factors are selectively used and updated, the robot is helped to more efficiently complete the tasks in a complex and dynamic factory environment through FSM simulation, and therefore intelligent production management is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of FSM simulation, and more specifically, to a FSM simulation method and system based on a virtual factory robot. Background Art

[0002] With the continuous development of automation and artificial intelligence technology, the decision-making ability of robots in complex environments has become increasingly important. Reinforcement learning, as an adaptive learning technology, has gradually gained attention. Among them, the Q-Learning algorithm has become a popular choice for robot decision optimization due to its simple and effective characteristics. In a virtual simulation training system based on industrial robots in the prior art (CN116229792B), data integration analysis and quantitative setting of workstations are used to clearly set the number of industrial robot workstations required for a manufacturing plant, thereby clarifying the simulation planning and layout plan of the industrial robots. However, as the complexity of tasks increases, industrial robots find it difficult to adapt to dynamically changing environments, making it difficult for industrial robots to choose the optimal decision.

[0003] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a FSM simulation method and system based on a virtual factory robot to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A FSM simulation method based on a virtual factory robot, characterized in that it specifically includes the following steps: S1: Use the Q-Learning algorithm in FSM simulation to optimize the robot's decision-making process and extract the robot's current and future tasks during the robot's decision-making process; S2: Analyze the historical reward values ​​of different future tasks of the robot, perform exponential weighted moving average analysis and probability density analysis on the historical reward values ​​of future tasks, determine the uncertainty information of future tasks, and obtain the dependency information of future tasks through comparative analysis; S3: Comprehensively analyze the uncertainty information and dependency information of future tasks, dynamically adjust the discount factor of the Q-Learning algorithm, and determine the updated discount factor and the basic discount factor; S4: Combine the robot’s performance under the updated discount factor and the basic discount factor, analyze the difference information and efficiency information, and selectively use the updated discount factor.

[0006] In a preferred embodiment, the Q-Learning algorithm is used in the FSM simulation to optimize the robot's decision-making process, including: The Q-learning algorithm is used for robots in the virtual factory to help them autonomously learn task priorities based on real-time environment and task changes and optimize their state transition decisions, including: Determine the robot's status information in the virtual factory, including the robot's current position, target position, task status, surrounding environment information, and resource status. The status information will be used as input to the Q-learning algorithm to guide the robot to decide the next action to perform; Determine the robot's action information in the virtual factory, including the robot's movement direction, task selection, and path planning. The action information will serve as the output of the Q-learning algorithm, indicating the actions that the robot should take in the current state.

[0007] In a preferred embodiment, determining uncertainty information of future tasks includes: The uncertainty information is represented by the reward fluctuation coefficient and the probability distribution coercion coefficient; The logic for obtaining the reward fluctuation coefficient is: extract the robot's future tasks, collect the robot's historical data on future tasks, and use exponentially weighted moving average to smooth the reward volatility of future tasks. The expression of exponentially weighted moving average is: ;in, represents the smoothed value at the current moment, represents the smoothing factor, represents the smoothed value at the previous moment, Represents the immediate reward value of the future task; According to the historical data of future tasks, the smoothed value of each task of the future tasks is obtained, the difference between the smoothed value of each task of the future tasks and the actual reward value is calculated, and the difference between the smoothed value of each task of the future tasks and the actual reward value is marked as: ,in, , is the smoothed value of each task in the future, is the actual reward value of each future task, n=1, 2, 3, ..., N, N is a positive integer, and n is the number of the historical data of the future task; Calculate the reward fluctuation coefficient using the following formula: ;in, Reward fluctuation coefficient.

[0008] In a preferred embodiment, the probability distribution stress coefficient includes: The acquisition logic of the probability distribution coercion coefficient is: based on the historical data of the robot in the future task, the Gaussian function is used to represent the probability density function of the future task reward value, and the expression is: ;in, is the standard deviation of future task rewards, is the average value of future task rewards, , ; Calculate the probability distribution stress coefficient, the calculation formula is: ;in, is the minimum reward value of the future task in the historical data, It is the maximum reward value of future tasks in historical data.

[0009] In a preferred embodiment, the dependency information of the future task includes: The dependency information of future tasks is represented by the priority influence coefficient; The logic for obtaining the priority influence coefficient is: according to the historical data of the robot's future tasks, determine the average reward value after the robot completes the current task and selects the future task, and mark the average reward value after the robot completes the current task and selects the future task as: , where i=1, 2, 3, ..., I, I is a positive integer, and i is the number of the future task; Get the maximum and minimum values ​​of the reward value after the robot completes the current task and selects the future task, and mark the maximum and minimum values ​​of the reward value after the robot completes the current task and selects the future task as: and ; Calculate the priority influence coefficient, the calculation formula is: ;in, is the priority influence coefficient.

[0010] In a preferred embodiment, the discount factor of the Q-Learning algorithm is dynamically adjusted, including: Through comprehensive analysis of the uncertainty information and dependency information of the robot's tasks, the reward fluctuation coefficient, probability distribution coercion coefficient and priority influence coefficient are weighted and calculated to construct an adjustment evaluation model and generate an adjustment evaluation coefficient. The expression is: ;in, To adjust the evaluation coefficient, , , is the proportional coefficient of reward fluctuation coefficient, probability distribution coercion coefficient and priority influence coefficient, , , They are both greater than 0; Set an adjustment evaluation coefficient threshold, compare the adjustment evaluation coefficient with the adjustment evaluation coefficient threshold, if the adjustment evaluation coefficient is greater than the adjustment evaluation coefficient threshold, adjust the discount factor with the adjustment evaluation coefficient, if the adjustment evaluation coefficient is less than the adjustment evaluation coefficient threshold, do not adjust the discount factor with the adjustment evaluation coefficient; Set the basic discount factor, adjust the discount factor by adjusting the evaluation coefficient, determine the updated discount factor according to the current robot's working environment, and mark the updated discount factor as: ,in, ,in, is the correction factor.

[0011] In a preferred embodiment, the difference information includes: The difference information is represented by the Q value difference coefficient; The logic for obtaining the Q value difference coefficient is: multiple comparative experiments are conducted on the same robot for the same order of task queues, and experiments are conducted respectively by using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor, and the cumulative sum of the Q values ​​after using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor are obtained respectively, and the average cumulative sum of the Q values ​​of the Q-Learning algorithm using the updated discount factor is marked as: , the average Q value accumulated by the Q-Learning algorithm using the basic discount factor is marked as: ; Calculate the Q value difference coefficient, the calculation formula is: ;in, is the Q value difference coefficient.

[0012] In a preferred embodiment, the efficiency information includes: The efficiency information is expressed by the time deviation coefficient; The logic for obtaining the time deviation coefficient is as follows: multiple comparative experiments are conducted on the same robot for the same order of task queues, respectively by using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor, and the time taken to complete the same number of tasks is compared. The average time taken by the Q-Learning algorithm with an updated discount factor is marked as: , the average time used by the Q-Learning algorithm using the basic discount factor is marked as: ; Calculate the time deviation coefficient, the calculation formula is: ;in, is the time deviation coefficient.

[0013] In a preferred embodiment, the update discount factor is selectively used, including: The Q value difference coefficient and the time deviation coefficient are weighted and calculated to construct an update evaluation model and generate an update evaluation coefficient. The expression is: ;in, To update the evaluation coefficients, and are the proportional coefficients of the Q value difference coefficient and the time deviation coefficient, and They are both greater than 0; Set the update evaluation coefficient threshold, compare the update evaluation coefficient with the update evaluation coefficient threshold, if the update evaluation coefficient is greater than the update evaluation coefficient threshold, continue to use the update discount factor, if the update evaluation coefficient is less than the update evaluation coefficient threshold, use the basic discount factor.

[0014] In a preferred embodiment, a FSM simulation system based on a virtual factory robot includes a Q-Learning algorithm module, a data analysis module, a discount factor dynamic adjustment module, and a selection module, and the modules are signal-connected; Q-Learning algorithm module, which is used to optimize the robot's decision-making process using the Q-Learning algorithm and learn the best strategy for taking different actions in different states; The data analysis module is used to determine the uncertainty information and dependency information of future tasks, and conduct a comprehensive analysis of the uncertainty information and dependency information of future tasks to build an adjustment evaluation model; The discount factor dynamic adjustment module is used to dynamically adjust the discount factor of the Q-Learning algorithm and analyze the robot's performance under the updated discount factor and the basic discount factor; The selection module is used to determine the Q value difference and efficiency deviation of the robot when performing the same work according to the robot's work performance under the use of the updated discount factor and the basic discount factor.

[0015] Technical effects and advantages of the present invention: The present invention uses the Q-Learning algorithm to perform FSM simulation on the robot in a virtual factory, dynamically adjusts the discount factor and reward mechanism to better adapt to the uncertainty and dependency in the factory environment, and selectively uses the dynamically adjusted discount factor by dynamically adjusting the discount factor and comprehensive evaluation mechanism, which helps the robot to complete tasks more efficiently in a complex and dynamic factory environment and realize intelligent production management. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings; Figure 1 A schematic diagram of a flow chart of a FSM simulation method based on a virtual factory robot according to the present invention; Figure 2 The present invention is a structural schematic block diagram of a FSM simulation system based on a virtual factory robot. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0018] like Figure 1 A flow chart of a FSM simulation method based on a virtual factory robot is given, which specifically includes the following steps: S1: Use the Q-Learning algorithm in FSM simulation to optimize the robot's decision-making process and extract the robot's current and future tasks during the robot's decision-making process; S2: Analyze the historical reward values ​​of different future tasks of the robot, perform exponential weighted moving average analysis and probability density analysis on the historical reward values ​​of future tasks, determine the uncertainty information of future tasks, and obtain the dependency information of future tasks through comparative analysis; S3: Comprehensively analyze the uncertainty information and dependency information of future tasks, dynamically adjust the discount factor of the Q-Learning algorithm, and determine the updated discount factor and the basic discount factor; S4: Combine the robot’s performance under the updated discount factor and the basic discount factor, analyze the difference information and efficiency information, and selectively use the updated discount factor.

[0019] Based on the real-time situation of the factory, the robot in the model library is used to simulate the actual operation, and the real-time status of the factory production is realistically reproduced through the simulation environment. Various tasks are performed through the robot model, including: Through virtual simulation, the operation status of various equipment in the factory can be understood, and robots can be dispatched in real time to perform tasks. Before introducing new equipment or optimizing production lines, the feasibility and potential problems of the simulation test scheme can be tested, and the simulation platform can be used to simulate robot behavior, analyze the efficiency of robot task execution, and adjust its workflow; By using modeling tools, the simulation system can automatically generate a virtual environment corresponding to the factory layout, equipment configuration, process flow, etc. based on the real-time data of the factory; The robots in the model library represent the robots used in actual production. The simulation system extracts robots that meet the task requirements from the model library and performs simulation operations of actual factory tasks.

[0020] For virtual scene restoration, it is first necessary to restore the virtual scene of the real production line in the FSM core factory software, which involves two parts of modeling and construction, including: Modeling is done through the Core Factory FSM Model Editor plug-in, which can realize the modeling and configuration of common models such as standard industrial robots, non-standard servo mechanisms, grippers, suction cups, conveyor belts, sensors, stackers, shelves, etc. Users can import the model after modeling into the Core Factory FSM simulation engine, and then use the software's construction function to restore the entire line in a virtual scene.

[0021] Based on the completed virtual factory, the Q-learning algorithm is used on the robots in the virtual factory to help them autonomously learn task priorities according to real-time environment and task changes, and optimize their state transition decisions, including: Determine the robot's status information in the virtual factory, including the robot's current position, target position, task status, surrounding environment information, and resource status. The status information will be used as the input of the Q-learning algorithm to guide the robot to decide the next action to perform; Determine the robot's action information in the virtual factory, including the robot's movement direction, task selection, and path planning. The action information will serve as the output of the Q-learning algorithm, indicating the actions the robot should take in the current state; Define a reward function to guide the robot to obtain feedback through actions. When defining rewards, it is necessary to take into account the goals and requirements in the actual scenario.

[0022] Among them, the expression for updating the Q value of the Q-learning algorithm is: ; where the expression represents the state Take action After that, according to the next state and rewards received , update the current Q value, where, Indicates the current state. Indicates in status The action to be performed is Indicates execution of an action After receiving the instant reward, Indicates execution of an action The next state reached after represents the discount factor, represents the learning rate, Indicates the next state The optimal action value (i.e., the maximum Q value) under the condition represents the expected reward for selecting the best action in the next state.

[0023] Dynamically adjusting the discount factor in the virtual factory helps the robot better adapt to environmental changes and task complexity. Therefore, the uncertainty information and dependency information of the robot's future tasks are collected, and the uncertainty information is represented by the reward fluctuation coefficient and the probability distribution coercion coefficient, and the dependency information is represented by the priority influence coefficient.

[0024] It should be noted that the discount factor is usually a fixed parameter in the reinforcement learning algorithm. When the robot faces multi-objective tasks, especially when the priorities of the multi-objective tasks are the same, the discount factor should be adjusted dynamically in real time according to the robot's tasks, environment, priority and other factors.

[0025] In a factory environment, the volatility of task rewards is affected by many factors, including the production environment, task complexity, human factors, and randomness. The robot's reward mechanism itself has randomness or uncertainty, including: During the robot's work, the task execution environment may be affected by external factors, causing the nature and priority of the task to change at any time. For example, a sudden increase in the production volume of a certain product will cause the production tasks of other products to be postponed, thereby affecting their rewards; During the production process, the quality of the product may fluctuate due to a variety of factors, and these fluctuations will directly affect the rewards for production tasks. For example, improper equipment calibration may lead to reduced product consistency and affect rewards.

[0026] The logic for obtaining the reward fluctuation coefficient is: extract the robot's future tasks, collect the robot's historical data on future tasks, and use exponentially weighted moving average to smooth the reward volatility of future tasks. The expression of exponentially weighted moving average is: ;in, represents the smoothed value at the current moment, represents the smoothing factor, represents the smoothed value at the previous moment, Represents the immediate reward value of the future task; It should be noted that the use of exponentially weighted moving average (EWMA) can be dynamically adjusted based on recent data, especially when the task reward has fluctuations, trends or noise, to more accurately quantify the uncertainty of the task. EWMA assigns different weights to historical reward values, with newer reward values ​​having greater weights and older reward values ​​having gradually reduced weights.

[0027] According to the historical data of future tasks, the smoothed value of each task of the future tasks is obtained, the difference between the smoothed value of each task of the future tasks and the actual reward value is calculated, and the difference between the smoothed value of each task of the future tasks and the actual reward value is marked as: ,in, , is the smoothed value of each task in the future, is the actual reward value of each future task, n=1, 2, 3, ..., N, N is a positive integer, and n is the number of the historical data of the future task; Calculate the reward fluctuation coefficient using the following formula: ;in, is the reward fluctuation coefficient.

[0028] It can be seen from the formula that the larger the reward fluctuation coefficient, the higher the uncertainty of future rewards. The discount factor can be appropriately reduced, so that the robot is more inclined to choose current stable tasks rather than tasks that may fluctuate greatly in the future.

[0029] The historical data obtained by simulating the robot with FSM may be incomplete or noisy, and directly using a single historical reward value to evaluate future rewards becomes unreliable. By constructing a probability density function, the potential variation range and distribution characteristics of the reward can be more comprehensively captured, and the entropy of continuous random variables can be used to quantify the uncertainty of future task rewards.

[0030] The acquisition logic of the probability distribution coercion coefficient is: based on the historical data of the robot in the future task, the Gaussian function is used to represent the probability density function of the future task reward value, and the expression is: ;in, is the standard deviation of future task rewards, is the average value of future task rewards, , ; Calculate the probability distribution stress coefficient, the calculation formula is: ;in, is the minimum reward value of the future task in the historical data, It is the maximum reward value of future tasks in historical data.

[0031] It can be seen from the formula that the larger the probability distribution coercion coefficient, the more dispersed the distribution of future task reward values, and the higher the uncertainty it brings. When choosing tasks, the robot is more inclined to choose current stable tasks rather than tasks that may have greater uncertainty in the future.

[0032] Considering the impact of dependencies on the discount factor is mainly to help the robot more reasonably balance the benefits of current tasks and future tasks. When the future task has a strong dependency on the current task, if the potential benefit of the future task is higher, the robot may need a higher discount factor to value the reward of the future task. This can prevent the robot from investing too many resources in the current task and missing out on high-yield tasks in the future.

[0033] The logic for obtaining the priority influence coefficient is: according to the historical data of the robot's future tasks, determine the average reward value after the robot completes the current task and selects the future task, and mark the average reward value after the robot completes the current task and selects the future task as: , where i=1, 2, 3, ..., I, I is a positive integer, and i is the number of the future task; Get the maximum and minimum values ​​of the reward value after the robot completes the current task and selects the future task, and mark the maximum and minimum values ​​of the reward value after the robot completes the current task and selects the future task as: and ; Calculate the priority influence coefficient, the calculation formula is: ;in, is the priority influence coefficient.

[0034] It can be seen from the formula that the larger the priority influence coefficient is, the higher the average reward value of the future task is than other tasks. The importance of high-reward tasks can be highlighted by increasing the discount factor.

[0035] Through comprehensive analysis of the uncertainty information and dependency information of the robot's tasks, the reward fluctuation coefficient, probability distribution coercion coefficient and priority influence coefficient are weighted and calculated to construct an adjustment evaluation model and generate an adjustment evaluation coefficient. The expression is: ;in, To adjust the evaluation coefficient, , , is the proportional coefficient of reward fluctuation coefficient, probability distribution coercion coefficient and priority influence coefficient, , , Both are greater than 0.

[0036] Set the adjustment evaluation coefficient threshold, and compare the adjustment evaluation coefficient with the adjustment evaluation coefficient threshold. If the adjustment evaluation coefficient is greater than the adjustment evaluation coefficient threshold, it means that the importance of the future task is higher. The discount factor will be adjusted by adjusting the evaluation coefficient. The Q value of the future task can be enhanced by increasing the discount factor. If the adjustment evaluation coefficient is less than the adjustment evaluation coefficient threshold, it means that the importance of the future task is lower. The discount factor does not need to be adjusted by adjusting the evaluation coefficient. The Q value of the future task can be weakened by reducing the discount factor.

[0037] Set the basic discount factor, adjust the discount factor by adjusting the evaluation coefficient, determine the updated discount factor according to the current robot's working environment, and mark the updated discount factor as: ,in, ,in, is the correction factor.

[0038] Based on the simulation of FSM after using the updated discount factor, the robot's working performance under the updated discount factor and the basic discount factor is analyzed. By setting the task queues with the same order, a comparative experiment is carried out to collect difference information and efficiency information. The difference information is represented by the Q value difference coefficient, and the efficiency information is represented by the time deviation coefficient.

[0039] When the same robot conducts comparative experiments on the same order of task queues, experiments are conducted by using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor, respectively. The cumulative sum of the Q values ​​after using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor are obtained, and the final Q values ​​are compared, including: Ensure that the two experiments (using the update discount factor and not using the update discount factor) are consistent in terms of environment, tasks, initial conditions, learning rate, exploration strategy (such as ε-greedy), etc., to exclude external factors from interfering with the results; Conduct enough experiments to ensure the statistical significance of the results. The results of a single experiment may be affected by accidental factors. Increasing the number of experiments will help to obtain more reliable conclusions. During the experiment, monitor the learning curve of the Q value (such as the changing trend of the Q value, cumulative rewards, convergence speed, etc.) to promptly detect abnormal situations in the learning process.

[0040] The logic for obtaining the Q value difference coefficient is: multiple comparative experiments are conducted on the same robot for the same order of task queues, and experiments are conducted respectively by using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor, and the cumulative sum of the Q values ​​after using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor are obtained respectively, and the average cumulative sum of the Q values ​​of the Q-Learning algorithm using the updated discount factor is marked as: , the average Q value accumulated by the Q-Learning algorithm using the basic discount factor is marked as: ; Calculate the Q value difference coefficient, the calculation formula is: ;in, is the Q value difference coefficient.

[0041] From the formula, we can see that the larger the Q-value difference coefficient, the better the performance of the Q-Learning algorithm using the updated discount factor, which means that the higher Q-value accumulation indicates that the robot obtains more total rewards when performing tasks.

[0042] The logic for obtaining the time deviation coefficient is as follows: multiple comparative experiments are conducted on the same robot for the same order of task queues, respectively by using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor, and the time taken to complete the same number of tasks is compared. The average time taken by the Q-Learning algorithm with an updated discount factor is marked as: , the average time used by the Q-Learning algorithm using the basic discount factor is marked as: ; Calculate the time deviation coefficient, the calculation formula is: ;in, is the time deviation coefficient.

[0043] It can be seen from the formula that the smaller the time deviation coefficient is, the better the performance of the Q-Learning algorithm using the updated discount factor is, which means that the robot may complete the task faster using the updated discount factor.

[0044] The Q value difference coefficient and the time deviation coefficient are weighted and calculated to construct an update evaluation model and generate an update evaluation coefficient. The expression is: ;in, To update the evaluation coefficients, and are the proportional coefficients of the Q value difference coefficient and the time deviation coefficient, and Both are greater than 0.

[0045] Set the update evaluation coefficient threshold, compare the update evaluation coefficient with the update evaluation coefficient threshold, if the update evaluation coefficient is greater than the update evaluation coefficient threshold, continue to use the update discount factor, indicating that the robot's work efficiency can be improved by using the update discount factor, if the update evaluation coefficient is less than the update evaluation coefficient threshold, use the basic discount factor.

[0046] The present invention uses the Q-Learning algorithm to perform FSM simulation on the robot in a virtual factory, dynamically adjusts the discount factor and reward mechanism to better adapt to the uncertainty and dependency in the factory environment, and selectively uses the dynamically adjusted discount factor by dynamically adjusting the discount factor and comprehensive evaluation mechanism, which helps the robot to complete tasks more efficiently in a complex and dynamic factory environment and realize intelligent production management. Example 2

[0047] like Figure 2 A structural schematic diagram of a FSM simulation system based on a virtual factory robot is given, which includes a Q-Learning algorithm module, a data analysis module, a discount factor dynamic adjustment module, and a selection module, as well as signal connections between modules. Q-Learning algorithm module, which is used to optimize the robot's decision-making process using the Q-Learning algorithm and learn the best strategy for taking different actions in different states; The data analysis module is used to determine the uncertainty information and dependency information of future tasks, and conduct a comprehensive analysis of the uncertainty information and dependency information of future tasks to build an adjustment evaluation model; The discount factor dynamic adjustment module is used to dynamically adjust the discount factor of the Q-Learning algorithm and analyze the robot's performance under the updated discount factor and the basic discount factor; The selection module is used to determine the Q value difference and efficiency deviation of the robot when performing the same work according to the robot's work performance under the use of the updated discount factor and the basic discount factor.

[0048] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0049] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0050] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0051] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0052] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0053] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0054] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0055] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A FSM simulation method based on a virtual factory robot, characterized in that: The specific steps include: S1: Use the Q-Learning algorithm in FSM simulation to optimize the robot's decision-making process and extract the robot's current and future tasks during the robot's decision-making process; S2: Analyze the historical reward values ​​of different future tasks of the robot, perform exponential weighted moving average analysis and probability density analysis on the historical reward values ​​of future tasks, determine the uncertainty information of future tasks, and obtain the dependency information of future tasks through comparative analysis; S3: Comprehensively analyze the uncertainty information and dependency information of future tasks, dynamically adjust the discount factor of the Q-Learning algorithm, and determine the updated discount factor and the basic discount factor; S4: Combine the robot’s performance under the updated discount factor and the basic discount factor, analyze the difference information and efficiency information, and selectively use the updated discount factor.

2. The FSM simulation method based on a virtual factory robot according to claim 1 is characterized in that: Use the Q-Learning algorithm in FSM simulation to optimize the robot's decision-making process, including: The Q-learning algorithm is used for robots in virtual factories to help them autonomously learn task priorities based on real-time environment and task changes and optimize their state transition decisions, including: Determine the robot's status information in the virtual factory, including the robot's current position, target position, task status, surrounding environment information, and resource status. The status information will be used as input to the Q-learning algorithm to guide the robot to decide the next action to perform; Determine the robot's action information in the virtual factory, including the robot's movement direction, task selection, and path planning. The action information will serve as the output of the Q-learning algorithm, indicating the actions that the robot should take in the current state.

3. The FSM simulation method based on a virtual factory robot according to claim 2 is characterized in that: Determine uncertainty information for future tasks, including: The uncertainty information is represented by the reward fluctuation coefficient and the probability distribution coercion coefficient; The logic for obtaining the reward fluctuation coefficient is: extract the robot's future tasks, collect the robot's historical data on future tasks, and use exponentially weighted moving average to smooth the reward volatility of future tasks. The expression of exponentially weighted moving average is: ;in, represents the smoothed value at the current moment, represents the smoothing factor, represents the smoothed value at the previous moment, Represents the immediate reward value of the future task; According to the historical data of future tasks, the smoothed value of each task of the future tasks is obtained, the difference between the smoothed value of each task of the future tasks and the actual reward value is calculated, and the difference between the smoothed value of each task of the future tasks and the actual reward value is marked as: ,in, , is the smoothed value of each task in the future, is the actual reward value of each future task, n=1, 2, 3, ..., N, N is a positive integer, and n is the number of the historical data of the future task; Calculate the reward fluctuation coefficient using the following formula: ;in, is the reward fluctuation coefficient.

4. The FSM simulation method based on a virtual factory robot according to claim 3 is characterized in that: Probability distribution stress coefficients, including: The acquisition logic of the probability distribution coercion coefficient is: based on the historical data of the robot in the future task, the Gaussian function is used to represent the probability density function of the future task reward value, and the expression is: ;in, is the standard deviation of future task rewards, is the average value of future task rewards, , ; Calculate the probability distribution stress coefficient, the calculation formula is: ;in, is the minimum reward value of the future task in the historical data, It is the maximum reward value of future tasks in historical data.

5. The FSM simulation method based on a virtual factory robot according to claim 4 is characterized in that: Dependency information for future tasks, including: The dependency information of future tasks is represented by the priority influence coefficient; The logic for obtaining the priority influence coefficient is: according to the historical data of the robot's future tasks, determine the average reward value after the robot completes the current task and selects the future task, and mark the average reward value after the robot completes the current task and selects the future task as: , where i=1, 2, 3, ..., I, I is a positive integer, and i is the number of the future task; Get the maximum and minimum values ​​of the reward value after the robot completes the current task and selects the future task, and mark the maximum and minimum values ​​of the reward value after the robot completes the current task and selects the future task as: and ; Calculate the priority influence coefficient, the calculation formula is: ;in, is the priority influence coefficient.

6. The FSM simulation method based on a virtual factory robot according to claim 5 is characterized in that: Dynamic adjustment of the discount factor of the Q-Learning algorithm, including: Through a comprehensive analysis of the uncertainty information and dependency information of the robot's future tasks, the reward fluctuation coefficient, probability distribution coercion coefficient and priority influence coefficient are weighted and calculated to construct an adjustment evaluation model and generate an adjustment evaluation coefficient. The expression is: ;in, To adjust the evaluation coefficient, , , is the proportional coefficient of reward fluctuation coefficient, probability distribution coercion coefficient and priority influence coefficient, , , They are both greater than 0; Set an adjustment evaluation coefficient threshold, compare the adjustment evaluation coefficient with the adjustment evaluation coefficient threshold, if the adjustment evaluation coefficient is greater than the adjustment evaluation coefficient threshold, adjust the discount factor with the adjustment evaluation coefficient, if the adjustment evaluation coefficient is less than the adjustment evaluation coefficient threshold, do not adjust the discount factor with the adjustment evaluation coefficient; Set the basic discount factor, adjust the discount factor by adjusting the evaluation coefficient, determine the updated discount factor according to the current robot's working environment, and mark the updated discount factor as: ,in, ,in, is the correction factor.

7. The FSM simulation method based on a virtual factory robot according to claim 6 is characterized in that: The difference information includes: The difference information is represented by the Q value difference coefficient; The logic for obtaining the Q value difference coefficient is: multiple comparative experiments are conducted on the same robot for the same order of task queues, and experiments are conducted respectively by using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor, and the cumulative sum of the Q values ​​after using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor are obtained respectively, and the average cumulative sum of the Q values ​​of the Q-Learning algorithm using the updated discount factor is marked as: , the average Q value accumulated by the Q-Learning algorithm using the basic discount factor is marked as: ; Calculate the Q value difference coefficient, the calculation formula is: ;in, is the Q value difference coefficient.

8. The FSM simulation method based on a virtual factory robot according to claim 7 is characterized in that: Efficiency information, including: The efficiency information is expressed by the time deviation coefficient; The logic for obtaining the time deviation coefficient is as follows: multiple comparative experiments are conducted on the same robot for the same order of task queues, respectively by using the Q-Learning algorithm with an updated discount factor and the Q-Learning algorithm with a basic discount factor, and the time taken to complete the same number of tasks is compared. The average time taken by the Q-Learning algorithm with an updated discount factor is marked as: , the average time used by the Q-Learning algorithm using the basic discount factor is marked as: ; Calculate the time deviation coefficient, the calculation formula is: ;in, is the time deviation coefficient.

9. The FSM simulation method based on a virtual factory robot according to claim 8, characterized in that: Optionally use update discount factors, including: The Q value difference coefficient and the time deviation coefficient are weighted and calculated to construct an update evaluation model and generate an update evaluation coefficient. The expression is: ;in, To update the evaluation coefficients, and are the proportional coefficients of the Q value difference coefficient and the time deviation coefficient, and They are both greater than 0; Set the update evaluation coefficient threshold, compare the update evaluation coefficient with the update evaluation coefficient threshold, if the update evaluation coefficient is greater than the update evaluation coefficient threshold, continue to use the update discount factor, if the update evaluation coefficient is less than the update evaluation coefficient threshold, use the basic discount factor.

10. A FSM simulation system based on a virtual factory robot, used to implement the FSM simulation method based on a virtual factory robot according to any one of claims 1 to 9, characterized in that: It includes Q-Learning algorithm module, data analysis module, discount factor dynamic adjustment module and selection module, and signal connection between modules; Q-Learning algorithm module, which is used to optimize the robot's decision-making process using the Q-Learning algorithm and learn the best strategy for taking different actions in different states; The data analysis module is used to determine the uncertainty information and dependency information of future tasks, and conduct a comprehensive analysis of the uncertainty information and dependency information of future tasks to build an adjustment evaluation model; The discount factor dynamic adjustment module is used to dynamically adjust the discount factor of the Q-Learning algorithm and analyze the robot's performance under the updated discount factor and the basic discount factor; The selection module is used to determine the Q value difference and efficiency deviation of the robot when performing the same work according to the robot's work performance under the use of the updated discount factor and the basic discount factor.

Images