Robot scheduling method, device, computer readable storage medium and electronic device

Abstract

The present application belongs to the field of robot scheduling technology, and in particular, relates to a robot scheduling method, device, computer-readable storage medium and electronic device. The method comprises: determining a second state obtained after the robot performs a selected action in a first state; when the second state is a target state, determining a first value output of a preset first value model in the second state; wherein the first value model is used to characterize the estimated reward value in the process of returning from the target state to the starting state; updating the preset second value model according to the first value output; wherein the second value model is used to characterize the actual reward value corresponding to various state transitions; after completing the update of the second value model, scheduling the robot according to the second value model. Through the present application, consideration of returning from the target state to the starting state is increased, which effectively improves the global scheduling efficiency.

Description

Technical Field

[0001] The present application belongs to the field of robot scheduling technology, and in particular, relates to a robot scheduling method, device, computer-readable storage medium and electronic device. Background Art

[0002] Digitalization, networking and intelligence are the development needs and trends in the field of industrial manufacturing. Among them, robots used in industrial manufacturing are regarded as the main direction of intelligent development. They play an important role in the digital and intelligent logistics of enterprises. They can replace manual handling, precise distribution, high efficiency and low cost. Many manufacturing companies have introduced robots on their flexible production lines to perform tasks such as material handling.

[0003] In order to enable robots to complete tasks in an orderly and efficient manner, special algorithms are generally required to schedule robots. However, in existing robot scheduling algorithms, only the scheduling from the starting state to the target state of the task is often considered, and the global scheduling efficiency is low. Summary of the invention

[0004] In view of this, embodiments of the present application provide a robot scheduling method, device, computer-readable storage medium, and electronic device to solve the problem of low global scheduling efficiency in existing robot scheduling methods.

[0005] A first aspect of an embodiment of the present application provides a robot scheduling method, which may include:

[0006] Determine a second state obtained by the robot after performing the selected action in the first state;

[0007] When the second state is the target state, determining a first value output of a preset first value model in the second state; wherein the first value model is used to characterize an estimated reward value in the process of returning from the target state to the starting state;

[0008] Determine the basic reward value for transitioning from the first state to the second state; determine the second value output of the preset second value model in the second state; determine the first model update value according to the basic reward value, the first value output and the second value output; update the second value model according to the first model update value; wherein the second value model is used to characterize the actual reward value corresponding to various state transitions;

[0009] After completing the update of the second value model, the robot is scheduled according to the second value model.

[0010] Through the above method, in the process of scheduling the robot, the consideration of returning from the target state to the starting state is added, that is, the robot is scheduled from the global perspective of the entire task closed loop, thereby effectively improving the global scheduling efficiency.

[0011] In a specific implementation of the first aspect, determining a first value output of a preset first value model in the second state may include:

[0012] Traversing each candidate action performed by the robot in the second state, and determining each candidate value output of the first value model;

[0013] The maximum value among the candidate value outputs of the first value model is determined as the first value output of the first value model in the second state.

[0014] Through the above method, the maximum value among the candidate value outputs can be selected by traversing each candidate action. The maximum value can effectively reflect the scheduling trend. On this basis, the second value model is updated to improve the accuracy of the second value model.

[0015] In a specific implementation of the first aspect, determining a second value output of a preset second value model in a second state may include:

[0016] Traversing each candidate action performed by the robot in the second state, and determining each candidate value output of the second value model;

[0017] The maximum value among the candidate value outputs of the second value model is determined as the second value output of the second value model in the second state.

[0018] Through the above method, the maximum value among the candidate value outputs can be selected by traversing each candidate action. The maximum value can effectively reflect the scheduling trend. On this basis, the second value model is updated to improve the accuracy of the second value model.

[0019] In a specific implementation of the first aspect, determining the first model update value according to the basic reward value, the first value output, and the second value output may include:

[0020] Weighting the second value output according to a preset weighting factor to obtain a weighted second value output;

[0021] The basic reward value, the first value output, and the weighted second value output are summed to obtain a first model update value.

[0022] Through the above method, the influence of the second value output on the result can be controlled by the weighted factor, which effectively improves the flexibility of scheduling.

[0023] In a specific implementation of the first aspect, after determining a second state obtained after the robot performs a selected action in the first state, the robot scheduling method may further include:

[0024] When the second state is not the target state, determining a basic reward value for transitioning from the first state to the second state;

[0025] determining a second value output of the second value model in a second state;

[0026] Determine a second model update value according to the base reward value and the second value output;

[0027] The second value model is updated according to the second model update value.

[0028] Through the above method, the second model update value can be determined by comprehensively considering the basic reward value and the second value output, and the second value model can be updated, thereby effectively improving the accuracy of the second value model.

[0029] In a specific implementation of the first aspect, determining the second model update value according to the basic reward value and the second value output may include:

[0030] Weighting the second value output according to a preset weighting factor to obtain a weighted second value output;

[0031] The base reward value and the weighted second value output are summed to obtain a second model update value.

[0032] Through the above method, the influence of the second value output on the result can be controlled by the weighted factor, which effectively improves the flexibility of scheduling.

[0033] In a specific implementation of the first aspect, the robot scheduling method may further include:

[0034] determining an estimated value output of the first value model under the target state;

[0035] Determine the actual reward value of returning from the target state to the starting state;

[0036] Determine the estimated error of the first value model based on the actual reward value and the estimated value output;

[0037] The first value model is updated according to the estimated error.

[0038] Through the above method, the first value model can be updated according to the estimated error between the actual reward value and the estimated value output, thereby effectively improving the accuracy of the first value model.

[0039] A second aspect of an embodiment of the present application provides a robot scheduling device, which may include:

[0040] A state determination module, used to determine a second state obtained after the robot performs a selected action in the first state;

[0041] A first value output determination module, used to determine the first value output of a preset first value model in the second state when the second state is the target state; wherein the first value model is used to represent the estimated reward value in the process of returning from the target state to the starting state;

[0042] The second value model updating module is used to determine the basic reward value for transferring from the first state to the second state; determine the second value output of the preset second value model in the second state; determine the first model update value according to the basic reward value, the first value output and the second value output; update the second value model according to the first model update value; wherein the second value model is used to characterize the actual reward value corresponding to various state transitions;

[0043] The robot scheduling module is used to schedule the robot according to the second value model after completing the update of the second value model.

[0044] In a specific implementation of the second aspect, the first value output determination module may include:

[0045] A first action traversal submodule, used to traverse each candidate action performed by the robot in the second state, and respectively determine each candidate value output of the first value model;

[0046] The first value output determination submodule is used to determine the maximum value among the candidate value outputs of the first value model as the first value output of the first value model in the second state.

[0047] In a specific implementation of the second aspect, the second value model updating module may include:

[0048] A basic reward value determination submodule, used to determine a basic reward value for transitioning from a first state to a second state;

[0049] A second value output determination submodule, used to determine a second value output of the second value model in a second state;

[0050] A first model update value determination submodule, used to determine the first model update value according to the basic reward value, the first value output, and the second value output;

[0051] The second value model updating submodule is used to update the second value model according to the first model update value.

[0052] In a specific implementation of the second aspect, the second value output determination submodule can be specifically used to: traverse each candidate action performed by the robot in the second state, and respectively determine each candidate value output of the second value model; and determine the maximum value among each candidate value output of the second value model as the second value output of the second value model in the second state.

[0053] In a specific implementation of the second aspect, the first model update value determination submodule can be specifically used to: weight the second value output according to a preset weighting factor to obtain a weighted second value output; sum the basic reward value, the first value output and the weighted second value output to obtain the first model update value.

[0054] In a specific implementation of the second aspect, the second value model update module can also be used to: determine the basic reward value for transferring from the first state to the second state when the second state is not the target state; determine the second value output of the second value model in the second state; determine the second model update value based on the basic reward value and the second value output; and update the second value model based on the second model update value.

[0055] In a specific implementation of the second aspect, the robot scheduling device may further include:

[0056] The first value model updating module is used to determine the estimated value output of the first value model in the target state; determine the actual reward value of returning from the target state to the starting state; determine the estimated error of the first value model based on the actual reward value and the estimated value output; and update the first value model based on the estimated error.

[0057] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned robot scheduling methods are implemented.

[0058] A fourth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned robot scheduling methods when executing the computer program.

[0059] A fifth aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned robot scheduling methods when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 A schematic diagram of a specific robot scheduling scenario;

[0062] Figure 2 This is a flow chart of an embodiment of a robot scheduling method in an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of the first value model expressed in matrix form;

[0064] Figure 4 is a schematic flow chart of the training process of the first value model;

[0065] Figure 5 is a schematic diagram of the second value model expressed in matrix form;

[0066] Figure 6 This is a structural diagram of an embodiment of a robot scheduling device in an embodiment of the present application;

[0067] Figure 7 This is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0070] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0071] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0072] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0073] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0074] Digitalization, networking and intelligence are the development needs and trends in the field of industrial manufacturing. Among them, robots used in industrial manufacturing are regarded as the main direction of intelligent development. They play an important role in the digital and intelligent logistics of enterprises. They can replace manual handling, precise distribution, high efficiency and low cost. Many manufacturing companies have introduced robots on their flexible production lines to perform tasks such as material handling.

[0075] In order to enable robots to complete tasks in an orderly and efficient manner, special algorithms are generally required to schedule robots. However, in existing robot scheduling algorithms, only the scheduling from the starting state to the target state of the task is often considered, and the global scheduling efficiency is low.

[0076] In view of this, embodiments of the present application provide a robot scheduling method, device, computer-readable storage medium, and electronic device to solve the problem of low global scheduling efficiency in existing robot scheduling methods.

[0077] In the process of scheduling the robot, the embodiment of the present application adds consideration for returning from the target state to the starting state, that is, the robot is scheduled from the global perspective of the entire task closed loop, thereby effectively improving the global scheduling efficiency.

[0078] The execution subject of the embodiment of the present application may be an electronic device, which may include but is not limited to computing devices such as desktop computers, notebooks, PDAs, and servers. The electronic device may establish a communication connection with the robot and schedule the robot. The robot in the embodiment of the present application may include but is not limited to an automated guided vehicle (AGV) and other types of robots.

[0079] For ease of understanding, Figure 1 A schematic diagram of a specific robot scheduling scenario is shown, in which each node is called a state. For example, state A is the loading port, state B is the heating process, state C is the cooling process, state D is the temporary storage area, and state E is the unloading port. The starting state and target state of the task can be set according to the actual situation. For example, the loading port can be used as the starting state, and the unloading port can be used as the target state. Walking from one node to another is called an action, and one action corresponds to an arrow. For each action, a corresponding action reward value can be set. For example, the action reward value corresponding to the action from state A to state B is 5, the action reward value corresponding to the action from state B to state C is 7, and so on. For each state, a corresponding process reward value can be set. For example, the process reward value corresponding to state B is 60, the process reward value corresponding to state C is 70, and so on. The entire robot scheduling scenario can be divided into several areas according to the actual situation. For example, the state from state A to state C can be divided into one area, and the state from state C to state E can be divided into another area. Each area can be set with a corresponding robot to perform tasks such as material handling.

[0080] See also Figure 2 , an embodiment of a robot scheduling method in an embodiment of the present application may include:

[0081] Step S201, determining a second state obtained after the robot performs a selected action in a first state.

[0082] The first state may be any state, for example, it may be randomly selected from various states to obtain an initial first state. The selected action may be any action performed by the robot in the first state, for example, it may be randomly selected from various actions performed by the robot in the first state to obtain a selected action.

[0083] After the robot performs the selected action in the first state, the state is transferred from the first state to a new state, which is recorded as the second state. For example, if the first state is state B, two actions can be performed in state B, which are recorded as action 1 and action 2, respectively, where action 1 is the action from state B to state A, and action 2 is the action from state B to state C. If the selected action is action 1, the second state obtained by the robot after performing the selected action in the first state is state A, and if the selected action is action 2, the second state obtained by the robot after performing the selected action in the first state is state C.

[0084] Step S202: when the second state is the target state, determine the first value output of the preset first value model in the second state.

[0085] The first value model is used to characterize the estimated reward value in the process of returning from the target state to the starting state.

[0086] The first value model can be written as , in a certain state ( ) to perform an action ( ), the corresponding value output can be obtained. The first value model can be expressed as Figure 3 The matrix form shown in the figure, as shown in the figure, the action from state E to state D is executed in state E, and the corresponding value output is 0. The first value model can also be expressed in the form of a neural network, that is, in a certain state ( ) to perform an action ( ) as the network input and the corresponding value output as the network output.

[0087] In an embodiment of the present application, the candidate actions performed by the robot in the second state can be traversed to determine the candidate value outputs of the first value model respectively, and the maximum value among the candidate value outputs of the first value model can be determined as the first value output of the first value model in the second state.

[0088] The first value model can be obtained through training. Initially, the network parameters in the first value model are not trained, and the obtained value output is not accurate. Through training, the network parameters in the first value model can be continuously updated to obtain a more accurate first value model. The training process of the first value model may include the following: Figure 4 The process shown:

[0089] Step S2021: Determine the estimated value output of the first value model under the target state.

[0090] Assume that When the target state is reached, the state at this time is recorded as , in the state Each candidate action executed under , at this time the network parameters in the first value model are recorded as , the estimated value output of the first value model under the target state can be determined according to the following formula:

[0091]

[0092] in, is the estimated value output of the first value model in the target state. The candidate actions to be performed , respectively determine each candidate value output of the first value model, and determine the maximum value among each candidate value output of the first value model as the estimated value output of the first value model under the target state.

[0093] Step S2022: Determine the actual reward value for returning from the target state to the starting state.

[0094] Assumptions is the number of time steps from the target state to the starting state, the actual reward value from the target state to the starting state can be determined according to the following formula:

[0095]

[0096] in, is the actual reward value of returning from the target state to the starting state, is a preset weighting factor, and its specific value can be flexibly set according to actual conditions. is the time step variable, , For the The reward value obtained by state transition in time steps (including action reward value and process reward value), For the The weighted reward value of each time step is accumulated to get the actual reward value of returning from the target state to the starting state.

[0097] Step S2023: Determine the estimated error of the first value model according to the actual reward value and the estimated value output.

[0098] The specific method for calculating the estimated error can be flexibly set according to the actual situation. As an example, in a specific implementation of the embodiment of the present application, the difference between the actual reward value and the estimated value output can be calculated. , as the estimated error of the first value model.

[0099] Step S2024: Update the first value model according to the estimated error.

[0100] The specific model updating method can be flexibly set according to the actual situation. As an example, in a specific implementation of the embodiment of the present application, the first value model can be updated according to the following formula:

[0101]

[0102] in, It is a preset coefficient, and its specific value can be flexibly set according to actual conditions. It means to derive the network parameters in the first value model. is the network parameter in the updated first value model. Through the above process, the first value model can be updated according to the estimated error between the actual reward value and the estimated value output, thereby effectively improving the accuracy of the first value model.

[0103] Step S203: Update the preset second value model according to the first value output.

[0104] Among them, the second value model is used to characterize the actual reward values ​​corresponding to various state transitions.

[0105] The second value model can be written as , in a certain state ( ) to perform an action ( ), the corresponding value output can be obtained. The second value model can be expressed as Figure 5 The matrix form shown in the figure, as shown in the figure, the action from state A to state B is executed in state A, and the corresponding value output is 5. The second value model can also be expressed in the form of a neural network, that is, in a certain state ( ) to perform an action ( ) as the network input and the corresponding value output as the network output.

[0106] In the embodiment of the present application, the basic reward value for transferring from the first state to the second state can be determined, wherein the basic reward value can be the sum of the action reward value and the process reward value. The second value output of the second value model in the second state can also be determined. Specifically, each candidate action performed by the robot in the second state can be traversed, each candidate value output of the second value model can be determined respectively, and the maximum value among each candidate value output of the second value model can be determined as the second value output of the second value model in the second state.

[0107] After obtaining the basic reward value, the first value output, and the second value output respectively, the first model update value can be determined according to the basic reward value, the first value output, and the second value output. Specifically, the second value output can be weighted according to a preset weighting factor to obtain a weighted second value output, and the basic reward value, the first value output, and the weighted second value output can be summed to obtain the first model update value. Through the above process, the influence of the second value output on the result can be controlled by the weighting factor, which effectively improves the flexibility of scheduling.

[0108] After obtaining the first model update value, the second value model can be updated according to the first model update value, as shown in the following formula:

[0109]

[0110] in, For the first state Transition to the second state The basic reward value is are the candidate value outputs of the first value model, is the first value output of the first value model in the second state, are the candidate value outputs of the second value model, is the second value output of the second value model in the second state, is the weighted second value output, The first model update value is the first model update value. When the selected action is performed in the first state, the corresponding value output is updated to the first model update value. Through the above process, the first model update value can be determined by comprehensively considering the basic reward value, the first value output and the second value output, and the second value model is updated, thereby effectively improving the accuracy of the second value model.

[0111] The above steps S202 to S203 describe the case where the second state is the target state. When the second state is not the target state, there is no need to consider the first value model. At this time, the basic reward value for transferring from the first state to the second state can be determined, and the second value output of the second value model in the second state can be determined.

[0112] After obtaining the basic reward value and the second value output respectively, the second model update value can be determined according to the basic reward value and the second value output. Specifically, the second value output can be weighted according to a preset weighting factor to obtain a weighted second value output, and the basic reward value and the weighted second value output can be summed to obtain the second model update value. Through the above process, the influence of the second value output on the result can be controlled by the weighting factor, which effectively improves the flexibility of scheduling.

[0113] After obtaining the second model update value, the second value model can be updated according to the second model update value, as shown in the following formula:

[0114]

[0115] in, The second model update value is the value output of the second model. The selected action is performed in the first state, and the corresponding value output is updated to the second model update value. Through the above process, the second model update value can be determined by comprehensively considering the basic reward value and the second value output, and the second value model is updated, thereby effectively improving the accuracy of the second value model.

[0116] In the case where the second state is not the target state, after the above-mentioned update of the second value model, the second state can be used as the new first state, and the execution of step S201 and its subsequent steps can be returned until the second state is the target state. In the case where the second state is the target state, after the above-mentioned update of the second value model, the current iteration can be ended, and the next iteration can be restarted, that is, random selection is performed again in each state to obtain the initial first state, and step S201 and its subsequent steps are re-executed until the preset iteration number threshold is reached, wherein each iteration will continuously update the second value model, and gradually improve the accuracy of the second value model.

[0117] Step S204: After completing the update of the second value model, scheduling the robot according to the second value model.

[0118] In the process of scheduling the robot, the corresponding target action can be selected in real time according to the current state based on the second value model. Specifically, the candidate actions executed by the robot in the current state can be traversed, and the candidate value outputs of the second value model can be determined respectively, and the candidate action corresponding to the maximum value can be determined as the target action. The new current state can be obtained by executing the target action, and the corresponding target action can be selected again according to the new current state. The above process is repeated until the robot completes the task.

[0119] In an embodiment of the present application, real-time communication can also be established between the electronic equipment for robot scheduling and the simulation software, and the parallel scheduling algorithm of multiple logistics robots in the actual production process can be simulated and verified in a multi-threaded parallel environment to verify the effect of robot scheduling in real time.

[0120] During the simulation process, a separate thread can be opened for each robot to simulate, so as to perform multi-threaded parallel computing, and each thread shares global data, including but not limited to processing position information and robot information. By sorting the simulation time of task allocation in each thread, the execution time of the thread can be unified with the simulation time of task allocation, and the decision of each robot can be executed in order according to the simulation time.

[0121] In summary, in the process of scheduling the robot, the embodiments of the present application add the consideration of returning from the target state to the starting state, that is, the robot is scheduled from the global perspective of the entire task closed loop, thereby effectively improving the global scheduling efficiency.

[0122] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0123] A robot scheduling method corresponding to the above embodiment, Figure 6 A structural diagram of an embodiment of a robot scheduling device provided in an embodiment of the present application is shown.

[0124] In this embodiment, a robot scheduling device may include:

[0125] A state determination module 601 is used to determine a second state obtained after the robot performs a selected action in a first state;

[0126] A first value output determination module 602 is used to determine the first value output of a preset first value model in the second state when the second state is the target state; wherein the first value model is used to represent the estimated reward value in the process of returning from the target state to the starting state;

[0127] The second value model updating module 603 is used to update the preset second value model according to the first value output; wherein the second value model is used to characterize the actual reward value corresponding to various state transitions;

[0128] The robot scheduling module 604 is used to schedule the robot according to the second value model after completing the update of the second value model.

[0129] In a specific implementation of the embodiment of the present application, the first value output determination module may include:

[0130] A first action traversal submodule, used to traverse each candidate action performed by the robot in the second state, and respectively determine each candidate value output of the first value model;

[0131] The first value output determination submodule is used to determine the maximum value among the candidate value outputs of the first value model as the first value output of the first value model in the second state.

[0132] In a specific implementation of the embodiment of the present application, the second value model updating module may include:

[0133] A basic reward value determination submodule, used to determine a basic reward value for transitioning from a first state to a second state;

[0134] A second value output determination submodule, used to determine a second value output of the second value model in a second state;

[0135] A first model update value determination submodule, used to determine the first model update value according to the basic reward value, the first value output, and the second value output;

[0136] The second value model updating submodule is used to update the second value model according to the first model update value.

[0137] In a specific implementation of an embodiment of the present application, the second value output determination submodule can be specifically used to: traverse each candidate action performed by the robot in the second state, and respectively determine each candidate value output of the second value model; and determine the maximum value among each candidate value output of the second value model as the second value output of the second value model in the second state.

[0138] In a specific implementation of an embodiment of the present application, the first model update value determination submodule can be specifically used to: weight the second value output according to a preset weighting factor to obtain a weighted second value output; sum the basic reward value, the first value output and the weighted second value output to obtain the first model update value.

[0139] In a specific implementation of an embodiment of the present application, the second value model update module can also be used to: determine the basic reward value for transferring from the first state to the second state when the second state is not the target state; determine the second value output of the second value model in the second state; determine the second model update value based on the basic reward value and the second value output; and update the second value model based on the second model update value.

[0140] In a specific implementation of the embodiment of the present application, the robot scheduling device may further include:

[0141] The first value model updating module is used to determine the estimated value output of the first value model in the target state; determine the actual reward value of returning from the target state to the starting state; determine the estimated error of the first value model based on the actual reward value and the estimated value output; and update the first value model based on the estimated error.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0145] like Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned robot scheduling method embodiments are implemented, for example Figure 2 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 6 The functions of modules 601 to 604 are shown.

[0146] Exemplarily, the computer program 72 may be divided into one or more modules / units, one or more modules / units are stored in the memory 71, and are executed by the processor 70 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the electronic device 7.

[0147] The electronic device 7 may include but is not limited to computing devices such as desktop computers, notebooks, PDAs, and servers. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 7 may also include input and output devices, network access devices, buses, etc.

[0148] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0149] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the electronic device 7. The memory 71 is used to store computer programs and other programs and data required by the electronic device 7. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0150] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electric carrier signals and telecommunication signals.

[0157] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A robot scheduling method, characterized in that: include: Determine a second state obtained by the robot after performing the selected action in the first state; In the case where the second state is the target state, determining a first value output of a preset first value model in the second state; wherein the first value model is used to characterize an estimated reward value in the process of returning from the target state to the starting state; Determine a basic reward value for transferring from the first state to the second state; determine a second value output of a preset second value model in the second state; weight the second value output according to a preset weighting factor to obtain a weighted second value output; sum the basic reward value, the first value output and the weighted second value to obtain a first model update value; update the output corresponding to the execution of the selected action by the second value model in the first state to the first model update value; wherein the second value model is used to characterize the actual reward value corresponding to various state transitions; After completing the update of the second value model, the robot is scheduled according to the second value model.

2. The robot scheduling method according to claim 1, characterized in that: The determining a first value output of a preset first value model in the second state includes: Traversing each candidate action executed by the robot in the second state, and respectively determining each candidate value output of the first value model; A maximum value among the candidate value outputs of the first value model is determined as the first value output of the first value model in the second state.

3. The robot scheduling method according to claim 1, characterized in that: The determining of a second value output of the preset second value model in the second state includes: Traversing each candidate action executed by the robot in the second state, and determining each candidate value output of the second value model; A maximum value among the candidate value outputs of the second value model is determined as the second value output of the second value model in the second state.

4. The robot scheduling method according to claim 1, characterized in that: After determining the second state obtained by the robot after performing the selected action in the first state, the method further includes: When the second state is not the target state, determining a basic reward value for transitioning from the first state to the second state; determining a second value output of the second value model in the second state; Determining a second model update value according to the basic reward value and the second value output; The second value model is updated according to the second model update value.

5. The robot scheduling method according to claim 4, characterized in that: The determining a second model update value according to the basic reward value and the second value output includes: Weighting the second value output according to a preset weighting factor to obtain a weighted second value output; The basic reward value and the weighted second value output are summed to obtain the second model update value.

6. The robot scheduling method according to any one of claims 1 to 5, characterized in that: Also includes: determining an estimated value output of the first value model under the target state; Determining an actual reward value for returning from the target state to the starting state; Determining an estimation error of the first value model according to the actual reward value and the estimated value output; The first value model is updated according to the estimated error.

7. A robot scheduling device, characterized in that: include: A state determination module, used to determine a second state obtained after the robot performs a selected action in the first state; A first value output determination module, used to determine the first value output of a preset first value model in the second state when the second state is the target state; wherein the first value model is used to represent the estimated reward value in the process of returning from the target state to the starting state; A second value model updating module is used to determine a basic reward value for transferring from the first state to the second state; determine a second value output of a preset second value model in the second state; weight the second value output according to a preset weighting factor to obtain a weighted second value output; sum the basic reward value, the first value output and the weighted second value output to obtain a first model update value; update the output corresponding to the execution of the selected action by the second value model in the first state to the first model update value; wherein the second value model is used to characterize the actual reward value corresponding to various state transitions; The robot scheduling module is used to schedule the robot according to the second value model after completing the update of the second value model.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the robot scheduling method according to any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the robot scheduling method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the robot scheduling method according to any one of claims 1 to 6 are implemented.

Images