Methods, apparatus, equipment, storage media, and program products for controlling robot systems

Abstract

According to embodiments of this disclosure, a method, apparatus, device, storage medium, and program product for controlling a robot system are provided. The method includes: determining multiple stages associated with a target task to be performed by the robot system, the multiple stages including a perception stage, a planning stage, and a control stage; determining the availability status of computing resources based on historical state information and current state information of a set of computing resources associated with the robot system; determining execution strategies for the multiple stages based on the availability status of the computing resources associated with the robot system, the execution strategies indicating task execution modes for the multiple stages; and controlling the robot system to perform the target task based on the execution strategies. In this manner, embodiments of this disclosure can effectively ensure the safety and stability of the tasks performed by the robot system.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, storage media, and program products for controlling robot systems. Background Technology

[0002] With the development of computer technology, various forms of electronic devices have greatly enriched people's daily lives. For example, people can use electronic devices to control robotic systems to perform target tasks. How to ensure the safety and stability of the tasks performed by robotic systems is currently a key concern. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for controlling a robot system is provided. The method includes: identifying multiple stages associated with a target task to be performed by the robot system, the multiple stages including a perception stage, a planning stage, and a control stage; determining the availability status of computing resources based on historical state information and current state information of a set of computing resources associated with the robot system; determining execution strategies for the multiple stages based on the availability status of the set of computing resources associated with the robot system, the execution strategies indicating task execution modes for the multiple stages; and controlling the robot system to perform the target task based on the execution strategies.

[0004] In a second aspect of this disclosure, an apparatus for controlling a robot system is provided. The apparatus includes: a first determining module configured to determine multiple stages associated with a target task to be performed by the robot system, the multiple stages including a perception stage, a planning stage, and a control stage; a second determining module configured to determine the availability status of computing resources associated with the robot system based on historical state information and current state information of a set of computing resources; a third determining module configured to determine execution strategies for the multiple stages based on the availability status of the set of computing resources associated with the robot system, the execution strategies indicating task execution modes for the multiple stages; and a control module configured to control the robot system to perform the target task based on the execution strategies.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the methods of the first or second aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] In this way, the embodiments of this disclosure can adaptively adjust the execution strategy based on the availability of the computing resource set, which not only ensures the quality of task execution when computing resources are sufficient, but also avoids the problem of unstable task execution when computing resources are insufficient, effectively ensuring the safety and stability of the task execution performed by the robot system.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0012] Figure 2 A flowchart illustrating the process of controlling a robot system according to some embodiments of the present disclosure is shown;

[0013] Figure 3 The diagram shows an example structural diagram of a target task according to some embodiments of the present disclosure;

[0014] Figure 4 A schematic structural block diagram of an apparatus for controlling a robot system according to certain embodiments of the present disclosure is shown;

[0015] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0020] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0021] Traditionally, when determining the execution strategy for controlling a robot system, it is generally assumed that the computing resources associated with the robot are stable and sufficient. However, this lack of awareness of dynamic changes in computing power can easily lead to problems such as decreased perception accuracy, planning delays, unstable control, and even task failure when computing power declines. Consequently, it becomes difficult to meet the requirements for safety, continuity, and robustness in complex application scenarios.

[0022] The embodiments of this disclosure propose a scheme for controlling a robot system. According to this scheme, multiple stages associated with the target task to be performed by the robot system can be determined, including a perception stage, a planning stage, and a control stage. Further, the availability status of computing resources can be determined based on historical and current state information of a set of computing resources, which is associated with the robot system. Further, based on the availability status of the computing resources associated with the robot system, execution strategies for multiple stages can be determined, with the execution strategies indicating the task execution modes for each stage. Further, based on the execution strategies, the robot system can be controlled to perform the target task.

[0023] In this way, the embodiments of this disclosure can adaptively adjust the execution strategy based on the availability of the computing resource set, which not only ensures the quality of task execution when computing resources are sufficient, but also avoids the problem of unstable task execution when computing resources are insufficient, effectively ensuring the safety and stability of the task execution performed by the robot system.

[0024] Example Environment

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include control device 110.

[0026] In this example environment 100, the control device 110 can determine an execution strategy for controlling the robot system 120 to perform a target task, and control the robot system to perform the target task based on the execution strategy.

[0027] In some embodiments, the robot system 120 may correspond to any suitable type. As an example, based on mobility, the robot system 120 may include, but is not limited to, at least one of the following: a mobile robot, a fixed-base robot, or a biomimetic mobile robot. As another example, based on application scenario, the robot system 120 may include, but is not limited to, at least one of the following: a home robot, an industrial robot, etc. As yet another example, based on form and structure, the robot system 120 may include, but is not limited to, at least one of the following: a humanoid robot, a non-humanoid robot, etc.

[0028] In some embodiments, the control device 110 may be any type of mobile terminal, fixed terminal, or portable terminal equipped with a display device, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the control device 110 may also support any type of interface for the target user (such as "wearable" circuitry).

[0029] The control device 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. The control device 110 may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in a cloud environment.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Example process

[0033] Figure 2 A flowchart of a process 200 for controlling a robot system according to some embodiments of the present disclosure is shown. Process 200 can be implemented at a control device 110. Reference is made below. Figure 1 Describe the process 200.

[0034] In step 210, the control device 110 determines multiple phases associated with the target task to be performed by the robot system 120, including a perception phase, a planning phase, and a control phase.

[0035] In some embodiments, the target task to be performed can be any appropriate type of task, such as including but not limited to at least one of the following: grasping task, inspection task, walking task, etc., which will not be elaborated here.

[0036] In some embodiments, the multiple phases associated with the target task can correspond to any appropriate order, such as the execution order from early to late, for example, the perception phase, the planning phase, the control phase, etc.

[0037] In some embodiments, the perception stage can refer to the stage of acquiring, processing, recognizing, and understanding perceived information such as environmental and self-state data through sensors. Sensors can be any suitable device, including, but not limited to, at least one of the following: red-green-blue (RGB) image acquisition devices, depth image acquisition devices, lidar devices, inertial measurement units, etc.

[0038] In some embodiments, the planning phase may indicate a phase that generates a series of subsequent action instructions or trajectory planning based on the perception information acquired or processed in the perception phase.

[0039] In some embodiments, the control phase may indicate the phase of converting the motion instructions or trajectory planning output by the planning phase into specific control signals that can drive the motors of the robot system 120, so as to achieve precise control of the robot system 120.

[0040] As an example, if the target task is a grasping task, then the perception stage corresponding to the target task can be associated with target recognition, the planning stage can be associated with motion trajectory planning, and the control stage can be associated with the joint control of the robot system 120.

[0041] In some embodiments, the various stages corresponding to the target task can be connected via a communication bus or middleware to complete data transmission between at least two stages. For example, if the target task is a grasping task, the output of the perception stage can be the location information of the identified target. The planning stage can perform path planning based on the output of the perception stage, outputting the planned trajectory information. The control stage can determine the joint control commands associated with the robot system 120 based on the planned trajectory information output by the planning stage.

[0042] It should be noted that the perception, planning, and control phases are merely examples. In addition to these phases, the target task can include any other appropriate phases, such as the communication and coordination phase and the status monitoring phase, which will not be elaborated here.

[0043] Step 220: Based on the historical state information and current state information of the computing resource set, determine the availability status of the computing resources, and associate the computing resource set with the robot system 120.

[0044] In some embodiments, the set of computing resources may be a collection of all computing resources available to the robot system 120. In some embodiments, computing resources may be any suitable hardware or virtualized entity capable of performing arithmetic logic operations, data processing, or specific computing tasks.

[0045] In some embodiments, the set of computing resources may include at least one of the following: local computing resources of the robot system 120 and remote computing resources that are communicatively connected to the robot system 120.

[0046] As an example, local computing resources may include, but are not limited to, at least one of the following associated with the robot system 120 itself: a central processing unit (CPU), a graphics processing unit (GPU), or a microcontroller unit (MCU). As an example, remote computing resources may also include, but are not limited to, at least one of the following from the cloud or edge servers: virtual central processing units (vCPUs), virtual graphics processing units (vGPUs), and other virtual computing resources.

[0047] It should be noted that in scenarios where multiple robots collaborate, they may share the computing resources of the same computing node (such as a computing server or accelerator).

[0048] In some embodiments, historical state information may indicate the average state information of the computing resource set within a predetermined time period prior to the current time, or the state information of the computing resource set at a certain historical time prior to the current time. Current state information may indicate the state information of the computing resource set corresponding to the current time.

[0049] In some embodiments, availability status may indicate at least one of the following: the ratio of available computing resources to unavailable computing resources, available computing resources, etc.

[0050] In some embodiments, the control device 110 can determine the fusion status information of the computing resource set based on the historical status information and the current status information of the computing resource set. Further, the control device 110 can determine the availability status of the computing resources based on the fusion status information of the computing resource set.

[0051] In other embodiments, the control device 110 may perform timing smoothing processing based on historical state information and current state information. Timing smoothing processing can be any suitable processing to achieve a smooth transition, such as, but not limited to, trajectory interpolation processing, control parameter gradual change processing, or low-pass filtering, etc., which will not be elaborated here. Further, the control device 110 may determine the available state based on the timing smoothing processing. As an example, the control device 110 may determine the available state based on the smoothed state information determined by the timing smoothing processing.

[0052] Step 230: Based on the availability of the set of computing resources associated with the robot system 120, the control device 110 determines the execution strategy for multiple stages, which indicates the task execution mode for multiple stages.

[0053] In some embodiments, the task execution mode can be any suitable mode, such as including, but not limited to, high computing power mode, low computing power mode, medium computing power mode, etc., and the computing power or computing resources managed by different task execution modes can be different. In some embodiments, the task execution mode can indicate at least one of the following execution configuration information: the global performance target of the stage, the upper limit of computing resources allowed to be used in the stage, the algorithm type or complexity level allowed to be enabled in the stage, etc. The global performance target can include, but is not limited to, at least one of the following: pursuing high accuracy, ensuring low latency, maintaining basic functions, etc. The execution configuration information corresponding to different task execution modes can be different.

[0054] In some embodiments, the control device 110 may determine a first execution mode corresponding to the sensing phase based on the availability state.

[0055] In some embodiments, the first execution mode may include, but is not limited to, indicating at least one of the following: the type of sensor enabled in the perception phase, the perception resolution and perception frequency of the perception phase, the type of perception model in the perception phase, and whether a specific perception task in the perception phase is performed.

[0056] As an example, in response to an availability status indication that available computing resources are less than a first threshold, a first execution mode may indicate that the sensor types activated during the perception phase may include a first type but not a second type. The first type may indicate sensors that are critical to the safety and stability of the robot system 120, while the second type may indicate sensors that do not affect the safety and stability of the robot system 120 but are only intended to improve the task accuracy of the robot system 120.

[0057] For example, the first type may include, but is not limited to, at least one of the following: depth sensor, inertial measurement unit. The second type may include, but is not limited to, at least one of the following: red-green-blue image acquisition device, lidar sensor.

[0058] As another example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, the first execution mode may indicate that the sensor types activated during the perception phase include both a first type and a second type.

[0059] In some embodiments, sensing resolution may indicate the data space or detail density used when processing raw data from sensing sensors. Sensing frequency may indicate the rate at which a complete sensing processing cycle is initiated.

[0060] As an example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, a first execution mode may indicate that the perception resolution of the perception phase is less than a first predetermined threshold and the perception frequency is less than a second predetermined threshold.

[0061] As another example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, the first execution mode may indicate that the perception resolution of the perception phase is greater than or equal to a first predetermined threshold, and the perception frequency is greater than or equal to a second predetermined threshold. For instance, in response to an availability status indicating that available computing resources are greater than or equal to the predetermined threshold, the first execution mode may indicate that high-resolution images and high sampling frequencies are used for environmental perception.

[0062] In some embodiments, the type of perception model in the perception phase may include, but is not limited to, a first model type capable of handling complex perception tasks and a second model type capable of handling simple perception tasks. Generally, the perception model capable of handling complex perception tasks has a more complex model structure, while the second model type capable of handling simple perception tasks has a more lightweight model structure.

[0063] As an example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, the first execution mode can indicate that the type of the perception model in the perception phase is a first model type. For example, a perception model of the first model type can support handling high-complexity perception tasks such as semantic segmentation and instance recognition.

[0064] As another example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, the first execution mode can indicate that the type of the perception model in the perception phase is a second model type. For example, a perception model of the second model type can support models that handle simple perception tasks.

[0065] In addition, in response to an available state indicating that available computing resources are less than a predetermined threshold, the first execution mode can indicate that the perception task in the perception phase can be implemented using a rule-based perception method, rather than using a perception model.

[0066] In some embodiments, a specific perception task can be any appropriate perception task with an importance level less than a threshold, such as a task associated with high-precision semantic understanding or redundant visual acquisition.

[0067] As an example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, a first execution mode can indicate that a specific sensing task in the sensing phase is executed.

[0068] As another example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, a first execution mode can indicate that a specific sensing task in the sensing phase is not executed.

[0069] In some embodiments, the control device 110 may determine a second execution mode corresponding to the planning phase based on the availability status.

[0070] In some embodiments, the second execution mode may include, but is not limited to, indicating at least one of the following: the length of time to be planned in the planning phase, the planning degrees of freedom in the planning phase, the type of parameters to be planned in the robot system 120 indicating the planning degrees of freedom, the number of trajectories to be planned synchronously in the planning phase, and enabling global path planning or local path planning in the planning phase.

[0071] In some embodiments, the length of time to be planned in the planning phase can indicate the length of the future time window considered in the planning phase. Planning with a time length greater than a threshold can be called long-time-domain planning, and planning with a time length less than or equal to the threshold can be called short-time-domain planning. Specifically, long-time-domain planning considers a longer future time window, resulting in better optimization performance but higher computational cost, while short-time-domain planning considers a shorter future time window, resulting in faster response and lower computational load.

[0072] As an example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, a second execution mode may indicate that the length of time to be planned in the planning phase is greater than or equal to a third predetermined threshold.

[0073] As another example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, a first execution mode may indicate that the planning phase has a planning time length that is less than a third predetermined threshold.

[0074] In some embodiments, the planning degrees of freedom in the planning phase can indicate the number of planned independent motion directions of the robot. This number of planned degrees of freedom can be associated with the type of parameter to be planned; for example, if the parameter type indicates the degrees of freedom of a 6-axis robotic arm, then the planning degrees of freedom can be 6.

[0075] As an example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, a second execution mode can indicate that the planning degrees of freedom in the planning phase are greater than or equal to a fourth predetermined threshold. For instance, the second execution mode could indicate that high-degree-of-freedom planning is enabled, allowing all joints of the robot system 120 to participate in the planning calculations.

[0076] As another example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, a second execution mode can indicate that the planning degrees of freedom in the planning phase are less than a fourth predetermined threshold. For instance, the second execution mode could indicate that some degrees of freedom or end poses are frozen, and that planning is performed only for critical degrees of freedom.

[0077] As an example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, a second execution mode could indicate that the number of trajectories planned concurrently during the planning phase is greater than or equal to a fifth predetermined threshold. For instance, the second execution mode could instruct the parallel generation and evaluation of multiple candidate trajectories to improve planning quality and robustness.

[0078] As another example, in response to an availability status indication that available computing resources are less than a predetermined threshold, the second execution could indicate that the number of trajectories planned synchronously during the planning phase is less than a fifth predetermined threshold. For instance, the second execution mode could indicate reducing the number of candidate trajectories or stopping parallel evaluation.

[0079] In some embodiments, the control device 110 may determine a third execution mode corresponding to the control phase based on the availability status.

[0080] In some embodiments, the third execution mode may include, but is not limited to, indicating at least one of the following: the type of control model used in the control phase, the control frequency of the control phase, and the type of node in the robot system 120 to be controlled by the control phase.

[0081] In some embodiments, the type of control model used in the control phase may include, but is not limited to, a full dynamics model, a simplified dynamics model, a pure kinematics control model, etc.

[0082] As an example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, the third execution mode can indicate that the control model used in the control phase is a full dynamic model.

[0083] As another example, in response to an available state indicating that available computing resources are less than a predetermined threshold, the third execution mode can indicate whether the control model used in the control phase is a simplified dynamics model or a pure kinematics control model.

[0084] In some embodiments, the control frequency of the control phase can be the control frequency for controlling the movement of the robot system 120.

[0085] As an example, in response to an availability status indication that available computing resources are greater than or equal to a predetermined threshold, the third execution mode can indicate that the control frequency of the control phase is greater than or equal to a sixth predetermined threshold. For instance, the third execution mode could indicate maintaining high-frequency closed-loop control to achieve fine motion control.

[0086] As another example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, the third execution mode can instruct the control frequency of the control phase to be less than a sixth predetermined threshold. For instance, the third execution mode could instruct a reduction in the control update frequency to alleviate the computational load.

[0087] In some embodiments, the types of nodes to be controlled by the control phase in the robot system 120 may include, but are not limited to, node types corresponding to a first priority and node types corresponding to a second priority, wherein the first priority is higher than the second priority. The node type can be any suitable type, such as, but not limited to, joint nodes, center-of-mass nodes, actuator nodes, etc.

[0088] As an example, in response to an availability status indicating that available computing resources are greater than or equal to a predetermined threshold, a third execution mode may indicate that the type of the node to be controlled by the control phase in the robot system 120 is a node type corresponding to the first priority and a node type corresponding to the second priority.

[0089] As another example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, the third execution mode can instruct the type of nodes in the robot system 120 to be controlled by the control phase to be only the node type corresponding to the first priority. For example, in response to an availability status indicating that available computing resources are less than a predetermined threshold, the third execution mode can prioritize the control of core joints, while the remaining joints maintain their current state or perform predefined holding actions.

[0090] In some embodiments, each of these multiple stages may correspond to multiple subtasks. In some embodiments, the multiple subtasks may be functional units that the robot system 120 can execute directly or indirectly, and the successful execution of a stage is guaranteed only if at least some of the multiple subtasks of a certain stage are successfully executed.

[0091] To improve the efficiency of subtask lookup, the control device 110 can pre-configure a task configuration library, which stores a list of subtasks corresponding to multiple stages of each task. In some embodiments, the control device 110 can search the task configuration library for a list of subtasks corresponding to multiple stages of a target task. This list of subtasks indicates multiple subtasks associated with the corresponding stage.

[0092] In other embodiments, the control device 110 may also utilize a task planning model to determine multiple sub-tasks included in each of the multiple stages corresponding to the target task. For example, the control device 110 may provide the description information of the target task to the task planning model to determine the multiple sub-tasks included in each of the multiple stages corresponding to the target task. The task planning model can be any suitable machine learning model, which will not be elaborated here.

[0093] In some embodiments, each of the multiple stages includes multiple subtasks that may correspond to multiple types, meaning that each of the multiple stages may include multiple types of subtasks. In some embodiments, these multiple types of subtasks may include, but are not limited to, a first type of subtask, a second type of subtask, and a third type of subtask.

[0094] Figure 3 Example diagrams of the structure 300 of a target task according to some embodiments of the present disclosure are shown, now directed to Figure 3 Please provide an explanation.

[0095] like Figure 3 As shown, the target task may include a perception stage 310, a planning stage 320, and a control stage 330. Each stage may include multiple types of subtasks, including at least a first type of subtask, a second type of subtask, a third type of subtask, etc.

[0096] In some embodiments, the first type of subtask can be used to ensure the safety of the robot system 120. In some embodiments, the second type of subtask can be used to improve the task accuracy of the robot system 120. In some embodiments, the third type of subtask does not affect whether the target task is successfully executed. For example, the third type of subtask can be a task for performing visual acquisition, or posture fine-tuning, or decorative movements.

[0097] In some embodiments, these various types of subtasks can correspond to different priority information. Priority information can be associated with the importance of the subtask. In some embodiments, the higher the priority of a subtask indicated by the priority information, the greater the likelihood that the execution strategy indicates that the subtask needs to be executed.

[0098] In some embodiments, the priority of the first type of subtask may be higher than the priority of the second type of subtask, and the priority of the second type of subtask may be higher than the priority of the third type of subtask.

[0099] In some embodiments, for each stage, the execution strategy can indicate which subtasks of various types corresponding to that stage need to be executed and which do not. Specifically, whether these various types of subtasks are executed can be determined based on availability and priority information.

[0100] As one example, in response to the available computing resources indicated by the availability status being less than a threshold, the subtasks corresponding to the first type can be executed in each stage, while the first and third types of subtasks may not be executed. As another example, in response to the available computing resources indicated by the availability status being greater than or equal to a threshold, all subtasks corresponding to the first, third, and fourth types can be executed in each stage.

[0101] Step 240: Control device 110 controls robot system 120 to execute target task based on execution strategy.

[0102] For ease of description, the execution strategy mentioned above can be referred to as the first execution strategy.

[0103] The availability of computing resources can be updated in real time. For example, multiple computing tasks within a robot system 120 may run in parallel, and the start, termination, and load changes of different tasks will cause local computing resources to change continuously over time. Furthermore, remote computing resources may be affected by network latency, link jitter, and cloud scheduling strategies, and therefore also possess uncertainty.

[0104] To ensure the safety and stability of the tasks performed by the robot system 120, in some embodiments, the control device 110 can detect updates to the available state during the execution of the target task. Furthermore, in response to an update to the available state, the control device 110 can determine a second execution strategy based on the updated available state. It should be noted that the process of determining the second execution strategy based on the updated available state is the same as the process of determining the second execution strategy based on the available state before the update, and will not be elaborated upon here.

[0105] Furthermore, the control device 110 can execute at least one subtask of the target task that has not been completed, based on the second execution strategy.

[0106] Since some subtasks may be partially executed, in order to ensure that the partially executed subtasks can be completed stably, as an example, the control device 110 may continue to execute the first subtask based on the first execution strategy in response to the first subtask of the target task being partially executed before its available state is updated.

[0107] Since updating the execution strategy does not affect tasks that have not been executed, and in order to better adapt to changes in the state of the computing resource set, as another example, the control device 110 may execute the second subtask based on the second execution strategy in response to the second subtask of the target task not being executed before its availability state is updated.

[0108] In this way, the embodiments of this disclosure can adaptively adjust the execution strategy based on the availability of the computing resource set, which not only ensures the quality of task execution when computing resources are sufficient, but also avoids the problem of unstable task execution when computing resources are insufficient, effectively ensuring the safety and stability of the task execution performed by the robot system.

[0109] Example devices and equipment

[0110] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an apparatus 400 for controlling a robot system according to certain embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in the control device 110 discussed above. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0111] like Figure 4 As shown, the device 400 includes a first determining module 410 configured to determine multiple stages associated with the target task to be performed by the robot system, the multiple stages including a perception stage, a planning stage, and a control stage; a second determining module 420 configured to determine the availability status of computing resources based on historical state information and current state information of a set of computing resources associated with the robot system; a third determining module 430 configured to determine execution strategies for the multiple stages based on the availability status of the set of computing resources associated with the robot system, the execution strategies indicating the task execution modes of the multiple stages; and a control module 440 configured to control the robot system to perform the target task based on the execution strategies.

[0112] In some embodiments, the set of computing resources includes at least one of the following: local computing resources of the robot system; and remote computing resources that are communicatively connected to the robot system.

[0113] In some embodiments, the second determining module 420 is further configured to: perform time-series smoothing processing based on historical state information and current state information; and determine available states based on the time-series smoothing processing.

[0114] In some embodiments, the third determining module 430 is further configured to: determine a first execution mode corresponding to the sensing phase based on the availability state, the first execution mode indicating at least one of the following: the type of sensor enabled in the sensing phase; the sensing resolution and sensing frequency of the sensing phase; the type of sensing model in the sensing phase; and whether a specific sensing task in the sensing phase is performed.

[0115] In some embodiments, the third determining module 430 is further configured to: determine a second execution mode corresponding to the planning phase based on the available state, the second execution mode indicating at least one of the following: the length of time to be planned in the planning phase; the planning degrees of freedom in the planning phase, the planning degrees of freedom indicating the types of parameters to be planned in the robot system; the number of trajectories to be planned simultaneously in the planning phase; and enabling global path planning or local path planning in the planning phase.

[0116] In some embodiments, the third determining module 430 is further configured to determine a third execution mode corresponding to the control phase based on the availability state. The third execution mode indicates at least one of the following: the type of control model adopted by the control phase; the control frequency of the control phase; and the type of node in the robot system to be controlled by the control phase.

[0117] In some embodiments, each of the multiple stages includes multiple types of subtasks, including: a first type of subtask for ensuring the safety of the robot system; a second type of subtask for improving the task accuracy of the robot system; and a third type of subtask that does not affect whether the target task is successfully executed.

[0118] In some embodiments, whether multiple types of subtasks are executed is determined based on availability and priority information, wherein the priority of a first type of subtask is higher than that of a second type of subtask, and the priority of a second type of subtask is higher than that of a third type of subtask.

[0119] In some embodiments, the execution strategy is a first execution strategy, and the control module 440 is further configured to: detect an update of the available state during the execution of the target task; determine a second execution strategy based on the updated available state in response to the update of the available state; and execute at least one subtask of the target task that has not been completed based on the second execution strategy.

[0120] In some embodiments, the control module 440 is further configured to: continue executing the first subtask based on a first execution strategy in response to the first subtask of the target task being partially executed before its availability state is updated; or execute the second subtask based on a second execution strategy in response to the second subtask of the target task not being executed before its availability state is updated.

[0121] The units included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 400 may be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), etc.

[0122] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 The control device 110 shown.

[0123] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors 510 or processing units, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0124] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 500.

[0125] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0126] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0127] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0128] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0129] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0130] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0131] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0133] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for controlling a robot system, characterized in that, The method includes: Identify multiple phases associated with the target task to be performed by the robotic system, including a perception phase, a planning phase, and a control phase; Based on the historical state information and current state information of the computing resource set, the availability status of the computing resources is determined, and the computing resource set is associated with the robot system; Based on the availability status of the set of computing resources associated with the robot system, an execution strategy for the plurality of stages is determined, the execution strategy indicating the task execution mode of the plurality of stages; and Based on the execution strategy, control the robot system to execute the target task; The determination of the execution strategy for the plurality of stages based on the availability status of the set of computing resources associated with the robot system includes: determining a first execution mode corresponding to the perception stage based on the availability status, wherein the first execution mode indicates at least one of the following: the type of sensor enabled in the perception stage; the perception resolution and perception frequency of the perception stage; the type of perception model in the perception stage; and whether a specific perception task in the perception stage is performed. The determination of the execution strategy for the plurality of stages based on the availability status of the set of computing resources associated with the robot system includes: determining a second execution mode corresponding to the planning stage based on the availability status, wherein the second execution mode indicates at least one of the following: the length of time to be planned in the planning stage; the degrees of freedom of the planning stage, wherein the degrees of freedom indicate the types of parameters to be planned in the robot system; the number of trajectories to be planned simultaneously in the planning stage; and whether global path planning or local path planning is enabled in the planning stage. The determination of the execution strategy for the plurality of stages based on the availability status of the set of computing resources associated with the robot system includes: determining a third execution mode corresponding to the control stage based on the availability status, wherein the third execution mode indicates at least one of the following: the type of control model adopted by the control stage; the control frequency of the control stage; and the type of node in the robot system to be controlled by the control stage.

2. The method according to claim 1, characterized in that, The set of computing resources includes at least one of the following: The robot system's local computing resources; Remote computing resources that are communicatively connected to the robot system.

3. The method according to claim 1, characterized in that, Determining the availability status of the computing resources based on the historical status information and current status information of the computing resource set includes: Based on the historical state information and the current state information, perform time-series smoothing processing; and Based on the aforementioned timing smoothing process, the available state is determined.

4. The method according to claim 1, characterized in that, Each of the multiple stages includes multiple types of subtasks, which include: The first type of sub-task is used to ensure the safety of the robot system; The second type of sub-task is used to improve the task accuracy of the robot system; and The third type of subtask does not affect whether the target task is successfully executed.

5. The method according to claim 4, characterized in that, Whether the various types of subtasks are executed is determined based on the availability status and priority information, wherein the priority of the first type of subtask is higher than the priority of the second type of subtask, and the priority of the second type of subtask is higher than the priority of the third type of subtask.

6. The method according to claim 1, characterized in that, The execution strategy is a first execution strategy, and controlling the robot system to execute the target task based on the execution strategy includes: During the execution of the target task, updates to the available status are detected; In response to the update of the availability state, a second execution strategy is determined based on the updated availability state; and Based on the second execution strategy, at least one subtask of the target task that has not been completed is executed.

7. The method according to claim 6, characterized in that, The step of executing at least one subtask that has not been completed in the target task based on the second execution strategy includes: In response to the first subtask of the target task being partially executed before the available state is updated, the first subtask continues to be executed based on the first execution strategy; or If the second subtask of the target task has not been executed before the availability status is updated, the second subtask is executed based on the second execution strategy.

8. A device for controlling a robot system, characterized in that, The apparatus is used to implement the method of any one of claims 1 to 7, the apparatus comprising: a first determining module configured to determine a plurality of stages associated with a target task to be performed by the robot system, the plurality of stages including a perception stage, a planning stage and a control stage; The second determining module is configured to determine the availability status of the computing resources based on the historical status information of the computing resource set and the current status information of the computing resource set, wherein the computing resource set is associated with the robot system. The third determining module is configured to determine the execution strategy for the plurality of stages based on the availability status of the set of computing resources associated with the robot system, the execution strategy indicating the task execution mode for the plurality of stages; and The control module is configured to control the robot system to perform the target task based on the execution strategy; The third determining module is further configured to determine a first execution mode corresponding to the sensing phase based on the available state, wherein the first execution mode indicates at least one of the following: the type of sensor enabled in the sensing phase; the sensing resolution and sensing frequency of the sensing phase; the type of sensing model in the sensing phase; and whether a specific sensing task in the sensing phase is performed. The third determining module is further configured to determine a second execution mode corresponding to the planning phase based on the available state. The second execution mode indicates at least one of the following: the length of time to be planned in the planning phase; the planning degrees of freedom in the planning phase, which indicate the parameter types to be planned in the robot system; the number of trajectories to be planned simultaneously in the planning phase; and whether global path planning or local path planning is enabled in the planning phase. The third determining module is further configured to determine a third execution mode corresponding to the control phase based on the available state, wherein the third execution mode indicates at least one of the following: the type of control model adopted by the control phase; the control frequency of the control phase; and the type of node in the robot system to be controlled by the control phase.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 7 when executed by the at least one processor.

10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, When the computer-executable instructions are executed by a processor, they implement the method according to any one of claims 1 to 7.

11. A computer program product comprising computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 7.

Images