Robot control system and method, medium, electronic device and program product

Abstract

The invention provides a robot control system and method, a medium, electronic equipment and a program product, and relates to the technical field of robot control. The robot control system comprises a first control system and a second control system which are in cascade connection; wherein the first control system is configured to generate a plurality of interface control signals based on the target task instruction and the environment perception data, the execution sequence of the plurality of interface control signals has an association relationship, and the generation and execution of each interface control signal depend on the completion state of the previous interface control signal; and the second control system is configured to obtain the target interface control signal and generate a control instruction based on the target interface control signal and the environment sensing data, so that the robot executes a corresponding action based on the control instruction. According to the invention, the response speed and the execution efficiency of the robot control system are obviously improved, and the time delay caused by waiting is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to a robot control system and method, a medium, an electronic device, and a program product. BACKGROUND

[0002] In visual closed-loop-based control, a robot first perceives the surrounding environment through vision and a point cloud model, then performs motion planning, and finally executes body control. After this process is completed, the robot will again perform a cycle of perception, planning, and control to achieve closed-loop control. However, this serial execution mode causes the execution time to increase linearly, and there is a strong dependency between the time sequences of the various links. When the previous link is not completed, the subsequent link will be in a waiting state. This close coupling between the front and rear links makes the system lack flexibility in responding to real-time environmental changes. Once the execution of a single link is delayed, the entire control process will be blocked.

[0003] Especially when the closed-loop execution time is too long, it will cause the control of the robot to appear to be stuck, or because the speed of environmental change exceeds the execution speed of the robot control system, the task cannot be completed. Therefore, although the robot control system has made significant progress in perception and control, it still faces challenges in improving response speed and the ability to handle complex tasks. SUMMARY

[0004] In view of this, the embodiments of the present application provide a robot control system and method, a medium, an electronic device, and a program product.

[0005] In a first aspect, an embodiment of the present application provides a robot control system, comprising: a first control system; a second control system cascadedly connected to the first control system; the first control system is configured to generate a plurality of interface control signals based on a target task instruction and environment perception data, the execution order of the plurality of interface control signals has a correlation relationship, and the generation and execution of each interface control signal depend on the completion state of the previous interface control signal; the second control system is configured to obtain a target interface control signal, generate a control instruction based on the target interface control signal and the environment perception data, so that the robot executes a corresponding action based on the control instruction.

[0006] In combination with the first aspect, in some implementations of the first aspect, the first control system is a high-order control system, and is further configured to decompose the target task instruction into sub-tasks of different granularities, and generate interface control signals of the sub-tasks of the corresponding granularities; the second control system is a low-order control system; preferably, the first control system and the second control system are decoupled from each other.

[0007] With reference to the first aspect, in some implementations of the first aspect, the subtasks of different granularities include: a subtask composed of a composite task executed by the second control system, wherein the composite task represents a task that can be executed independently as a whole by the second control system and does not require action decomposition by the first control system; and a subtask composed of an atomic action executed by the second control system.

[0008] With reference to the first aspect, in some implementations of the first aspect, before the first control system generates the current interface control signal, and / or in a case where the robot does not complete an action corresponding to a control instruction of a previous interface control signal corresponding to the current interface control signal, the target interface control signal is the previous interface control signal, the current interface control signal and the previous interface control signal are one of the plurality of interface control signals; after the first control system generates the current interface control signal, and in a case where the robot completes the action corresponding to the control instruction of the previous interface control signal, the target interface control signal is the current interface control signal.

[0009] With reference to the first aspect, in some implementations of the first aspect, the first control system is further configured to: determine the completion state of the previous interface control signal based on the environment perception data; if the determination result is successful completion, allow the second control system to execute a subsequent interface control signal corresponding to the previous interface control signal; if the determination result is an execution result error, control the second control system to re-execute the previous interface control signal.

[0010] With reference to the first aspect, in some implementations of the first aspect, if the determination result is that the previous interface control signal is not completed within the allowable time, control the second control system to continue executing the previous interface control signal; if the determination result is that the previous interface control signal is not completed beyond the allowable time, control the second control system to terminate execution of the previous interface control signal.

[0011] With reference to the first aspect, in some implementations of the first aspect, the second control system is further configured to determine the completion state of the previous interface control signal by itself; the first control system is further configured to correct the second control system if it is determined that the self-determination result of the second control system is incorrect.

[0012] In a possible implementation of the first aspect, the first control system comprises: an encoding module configured to preliminarily parse the current pose of the robot, the target task instruction, and the environment perception data to obtain first target features; a transformer base large model configured to optimize and enhance the first target features to obtain second target features; and a decoding module configured to process the second target features to generate a plurality of interface control signals; wherein the interface control signals comprise at least one of text, image, video, bounding box, hidden feature, and future target observation; and the future target observation is used to show an image of an expected scene after completion of the task, so that the second control system compares actual environment perception data with the future target observation during execution, and further judges a completion state of the task.

[0013] In a possible implementation of the second aspect, the robot control method is applied to a first control system, and the method comprises: obtaining environment perception data; and generating a plurality of interface control signals based on a target task instruction and the environment perception data, wherein an execution order of the plurality of interface control signals has a correlation relationship, and generation and execution of each interface control signal depend on a completion state of a previous interface control signal.

[0014] In a possible implementation of the third aspect, the robot control method is applied to a second control system, and the method comprises: obtaining environment perception data and a target interface control signal, the target interface control signal being one of a plurality of interface control signals generated by the first control system based on a target task instruction and the environment perception data, wherein an execution order of the plurality of interface control signals has a correlation relationship, and generation and execution of each interface control signal depend on a completion state of a previous interface control signal; and generating a control instruction based on the environment perception data and the target interface control signal, so that a robot executes a corresponding action based on the control instruction.

[0015] In a possible implementation of the fourth aspect, a computer readable storage medium stores a computer program for executing the robot control method of the second aspect and / or the third aspect.

[0016] In a possible implementation of the fifth aspect, an electronic device comprises: a processor; a memory for storing processor-executable instructions; and the processor is configured to execute the robot control method of the second aspect and / or the third aspect.

[0017] In a possible implementation of the sixth aspect, a computer program product comprises instructions for causing an electronic device to implement the robot control method of the second aspect and / or the third aspect when the instructions are executed on the electronic device.

[0018] In the present application, the robot control system adopts a cascaded connection mode to realize efficient collaborative work of the first control system and the second control system. Specifically, in this structure, the first control system is responsible for generating a plurality of interface control signals with associated relationships in execution sequence based on target task instructions and environmental perception data, and the second control system is responsible for converting target interface control signals among them into specific control instructions to drive the robot to perform corresponding actions. In addition, when the first control system generates the first interface control signal based on the target task instruction and the environmental perception data and sends it to the second control system, its work does not stop, but immediately carries out the pre-generation work of the subsequent interface control signal based on the association relationship of the task sequence and the latest environmental state in parallel. At the same time, the second control system independently generates the control instructions corresponding to the interface control signal currently received. This design forms an overlapping work mode of planning and execution. Therefore, the overall response time of the robot control system is no longer equal to the linear accumulation of the execution time of each link, but depends on the longer one of the planning flow and the execution flow, thereby significantly compressing the idle waiting time between task chains. It can be seen that the scheme in the present embodiment significantly improves the response speed and execution efficiency of the robot control system, and reduces the time delay caused by waiting. In addition, this parallel processing mechanism enables the robot control system to adapt to environmental changes more quickly and adjust the control strategy in time, thereby reducing the risk of task failure caused by the speed of environmental change exceeding the execution speed of the robot control system. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application, taken in conjunction with the accompanying drawings. The drawings provided in the present application are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 Fig. 1 shows a structural schematic diagram of a robot control system provided by an embodiment of the present application.

[0021] Figure 2 Fig. 2 shows a structural schematic diagram of a robot control system provided by another embodiment of the present application.

[0022] Figure 3 Fig. 3 shows a structural schematic diagram of a first control system provided by an embodiment of the present application.

[0023] Figure 4 Fig. 4 shows a structural schematic diagram of a second control system provided by an embodiment of the present application.

[0024] Figure 5Fig. 1 shows a flowchart of a robot control method according to an embodiment of the present application.

[0025] Figure 6 Fig. 2 shows a flowchart of a robot control method according to another embodiment of the present application.

[0026] Figure 7 Fig. 3 shows a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0028] With the development of technology, the robot control system has been transformed from simply repeating fixed actions to an intelligent system capable of closed-loop control according to environmental changes. This transformation is due to the development of large language models, multi-modal models and embodied intelligence technology, so that the robot control system can be implemented in an end-to-end manner without complex intermediate processing steps.

[0029] In the end-to-end robot control system, the robot first accurately perceives the surrounding environment using vision and point cloud models in order to understand the environment, then performs motion planning, i.e., determines the path for the robot to move from the current position to the target position, and finally performs body control, which is used to convert the above motion planning into actual actions of the robot to ensure that the robot can accurately perform the task.

[0030] However, this closed-loop control process of perception, planning and control faces challenges in actual operation. Specifically, each control cycle of the system is strictly executed in the order of perception-planning-execution, and when the previous link is not completed, the subsequent link will be in a waiting state. This strong dependency in time sequence causes the system execution time to linearly accumulate, and the overall response delay is equal to the sum of the processing time of each link. When there are dynamic obstacles in the environment or the task requires a quick response, the long closed-loop execution cycle will cause serious problems.

[0031] For example, on the one hand, the motion trajectory generated in the planning stage based on outdated environmental information may no longer be applicable in the execution stage, resulting in collisions between the robot and newly appearing obstacles in the environment, or missing the best operation opportunity; on the other hand, from the perspective of user experience, the robot action will appear obvious stuttering, hesitation and other phenomena, and the performance is not smooth and natural.

[0032] Further analysis shows that this serial architecture also has the risk of blocking during execution. For example, the planning system completes the planning of the entire task sequence before the behavior system starts execution. During this process, the planning system cannot pre-plan for subsequent tasks while the behavior system is executing the current task. Once the execution of a single task is time-consuming due to environmental interference, or the planning link is delayed due to complex scenarios, the response performance of the entire robot control system will be severely affected.

[0033] Therefore, although the robot control system has made significant progress in the single ability of perception and control, it is still difficult to balance the needs of high-precision planning and high real-time response when dealing with complex tasks in dynamic environments.

[0034] Therefore, the present application is proposed. Specifically, Figure 1 As shown in the structural schematic diagram of the robot control system provided by an embodiment of the present application. As Figure 1 As shown, the robot control system 10 includes a first control system 110 and a second control system 120, and the second control system 120 is connected in cascade to the first control system 110.

[0035] Specifically, the first control system 110 is configured to generate a plurality of interface control signals based on target task instructions and environmental perception data. The second control system 120 is configured to obtain target interface control signals, generate control instructions based on the target interface control signals and environmental perception data, so that the robot executes corresponding actions based on the control instructions.

[0036] Specifically, the execution order of the plurality of interface control signals has a correlation relationship. That is, the plurality of interface control signals are not isolated, but are correlated with each other in a specific order. These interface control signals represent different stages or steps of the robot executing a task, and the generation and execution of each interface control signal depend on the completion state of the previous interface control signal, ensuring the continuity and coordination of the target task execution. For example, in the task of picking and placing a target object, the first interface control signal is used to instruct the robot to move to the position of the target object, the second interface control signal is used to instruct the robot to pick up the target object, and the third interface control signal is used to instruct the robot to place the target object at a specified position. It can be understood that if the execution order of any step is disturbed, it may cause the robot task execution to fail or the robot action to be uncoordinated. Therefore, in this embodiment, the generation and execution of the plurality of interface control signals are in order, reflecting the logical flow of task execution, so that the robot can accurately complete the task according to the predetermined plan.

[0037] To isolate these interface control signals, all possible interface control signals will be investigated and collected, and then in the training samples, these possible interface control signals will be manually labeled, manually assisted automatic labeling, model automatic labeling, or multi-modal large model scoring Monte Carlo labeling, in order to prepare a large number of supervised signals for training the first control system and the second control system.

[0038] In some embodiments, the first control system 110 receives target task instructions, which define the task that the robot needs to complete and the expected result, for example, the target task instruction is "throw the garbage on the table into the garbage can". At the same time, the first control system collects environmental perception data through sensors (such as cameras), including the position of the robot, the position and state of surrounding objects, and other factors that may affect task execution. The first control system 110 calculates and makes decisions on these information through its internal algorithms, decomposes complex tasks, and generates corresponding interface control signals for each subtask decomposed.

[0039] Exemplarily, each interface control signal includes but is not limited to the following representations: Text: Provide task description of subtask in natural language or instruction code, such as the above-mentioned task of picking and placing target object, where the text representation of one interface control signal is to move to the position of the target object, so as to tell the robot the specific action to be performed.

[0040] Image: Contains identification information of execution target of subtask, such as image containing execution target (such as garbage on the table), which is used to provide visual information to the robot.

[0041] Video: Includes a sequence of continuous multiple frames of images, which provides dynamic prediction and state evolution reference for low-level control system to the task execution process.

[0042] Bounding box: A rectangular box that identifies the position and size of the execution target in the picture, usually containing coordinate and size information of the execution target.

[0043] Functional bounding box: Used to guide the robot how to interact with the execution target, such as grasping, pushing.

[0044] Future target observation: Used to show the image of the expected scene after the completion of the task, in order to guide the robot to reach a specific target state.

[0045] Hidden features: Used to capture and process complex data patterns inside the first control system.

[0046] For example, for the interface control signal in the form of a single image, such as the task of clearing the table, the high-level control system sends the low-level control system the task of clearing the table, but the carrier is not limited to text, but also includes images, such as the picture of the current messy kitchen, and the picture of the clean kitchen with the dishes already arranged as the target image.

[0047] For the interface control signal in the form of a single image, such as the task of clearing the table, the high-level control system sends the low-level control system the task of clearing the table, but the carrier is not limited to text, but also includes images, such as the picture of the current messy kitchen, and the picture of the clean kitchen with the dishes already arranged as the target image. Figure 1 , the robot arm picks up the nearest bowl; Figure 2 , the robot arm moves the picked-up bowl to the left; Figure 3 , the robot arm moves the picked-up bowl down and puts it into the sink; Figure 4 , the robot arm moves to the nearest plate from the sink; Figure 5 , the robot arm picks up the plate.

[0048] As can be seen, the interface control signal contains rich information, which is provided to the second control system in different forms, enabling it to understand the context of the task, the characteristics of the execution target, the expected action results, and how to interact with the environment.

[0049] By adopting a multi-modal interface control signal combining text, images, bounding boxes, functional bounding boxes, future target observations, and hidden features, a complete control system with environment perception, task planning, and state prediction capabilities is constructed. Specifically, the text instruction provides an explicit task semantic framework, the image and bounding box provide accurate visual positioning information, the functional bounding box realizes the binding of interaction intent and spatial operation, and together they form a three-dimensional description of the environment and task. The future target observation provides the expected scene image after the task is completed, enabling the low-level control system to continuously compare real-time perception with predicted targets (such as image similarity calculation), forming a prediction-based closed-loop verification mechanism, giving the robot control system the ability of forward-looking planning and self-verification. At the same time, the hidden features, as high-dimensional abstract semantic information, optimize the processing flow and decision-making efficiency. The output of this multi-modal fusion world model ultimately realizes intelligent control from abstract instructions to concrete actions, from current planning to future prediction, enabling the robot control system to have high reliability and execution efficiency in complex environments.

[0050] In some embodiments, the second control system 120 receives the target interface control signals, and generates specific control instructions in combination with the real-time environmental perception data. Then, the second control system 120 sends the control instructions to the actuators of the robot, such as motors or joints, to control the robot to perform corresponding actions. Specifically, the control instructions include robot control values, including but not limited to at least one of joint rotation angle, travel distance, travel direction, end effector position, head pose, torso lift-pitch value.

[0051] The joint rotation angle refers to the specific angle that each joint in the robot's mechanical arm or dexterous hand and other components should rotate. In the case of a mechanical arm, the joint rotation angle determines the position and direction of the end effector of the mechanical arm. For a dexterous hand, the joint rotation angle determines the opening and closing degree and pose of the fingers, thereby achieving grasping, releasing or other operations.

[0052] The travel distance refers to the straight-line distance that the robot chassis or other moving components need to move, which is a basic parameter in robot path planning and navigation. By precisely controlling the travel distance, the robot can accurately reach the predetermined target position or move along a specific path.

[0053] The travel direction refers to the orientation of the robot when moving. The travel direction can be an angle value indicating the rotation angle of the robot relative to its current orientation or a certain reference direction (such as north), ensuring that the robot can move in the correct direction, avoid collisions, and effectively plan the path.

[0054] The end effector position refers to the three-dimensional coordinates of the robot's end effector (such as the tool or gripper at the end of the mechanical arm) in space, usually including X, Y, Z coordinates, indicating its specific position relative to a certain reference point.

[0055] The head pose refers to the direction and inclination state of the robot's head relative to its body or base, including yaw angle, pitch angle and roll angle.

[0056] The torso lift-pitch value refers to the vertical movement and inclination angle of the robot's torso part. The lift value indicates the position of the torso moving up and down, and the pitch value indicates the angle of the torso tilting forward and backward.

[0057] In the embodiment, the robot control system adopts a cascade connection mode to realize efficient collaborative work of the first control system and the second control system. Specifically, in this structure, the first control system is responsible for generating a plurality of interface control signals with a correlation in execution sequence based on target task instructions and environmental perception data, and the second control system is responsible for converting target interface control signals among them into specific control instructions to drive the robot to perform corresponding actions. In addition, when the first control system generates the first interface control signal based on the target task instruction and the environmental perception data and sends it to the second control system, its work does not stop, but immediately carries out the pre-generation of the subsequent interface control signal based on the correlation of the task sequence and the latest environmental state. At the same time, the second control system independently generates the control instructions corresponding to the currently received interface control signal. This design forms an overlapping work mode of planning and execution. Therefore, the overall response time of the robot control system is no longer equal to the linear accumulation of the execution time of each link, but depends on the longer one of the planning flow and the execution flow, thereby significantly compressing the idle waiting time between task chains. It can be seen that the scheme in the embodiment significantly improves the response speed and execution efficiency of the robot control system, and reduces the time delay caused by waiting. In addition, this parallel processing mechanism enables the robot control system to adapt to environmental changes more quickly and adjust the control strategy in time, thereby reducing the risk of task failure caused by the speed of environmental change exceeding the execution speed of the robot control system.

[0058] In some embodiments, based on the position and function of the first control system in the control hierarchy, the first control system is determined as a high-level control system, which is further configured to decompose the target task instruction into subtasks of different granularities and generate interface control signals of the subtasks of corresponding granularities; and the second control system is determined as a low-level control system. The two systems are designed to be decoupled with each other to improve the efficiency and reliability of the entire robot control system.

[0059] Figure 2 The structure schematic diagram of the robot control system provided by another embodiment of the application is shown. As can be seen, the high-level control system (the first control system) is responsible for understanding the scene where the robot is located, including task planning and state judgment, and generating a series of interface control signals with a correlation in execution sequence. These functions usually do not require a very high update frequency, but involve complex algorithms and a large amount of data processing.

[0060] In the embodiment, the decomposition of the target task instruction into subtasks of different granularities is a key capability of the high-level system. It can understand the target task instruction (for example: help me clean the kitchen) issued by the user, which is usually abstract or macroscopic, and decompose it from top to bottom into a series of specific steps that are logically coherent and sequentially executed.

[0061] Different granularities are the core feature of this decomposition process, embodying the intelligence and flexibility of the first control system. Specifically, the granularity of decomposition can be adaptively adjusted according to the complexity of the task and the execution capability of the second control system (low-level system).

[0062] Optionally, the subtasks of different granularities include subtasks composed of composite tasks executed by the second control system, wherein the composite task represents a task that can be executed independently as a whole by the second control system and does not require action decomposition by the first control system. And subtasks composed of atomic actions executed by the second control system.

[0063] Among them, the composite task can be regarded as a coarse-grained task, for example: first clean the table, then wash the dishes and utensils, and finally pour the garbage. These tasks correspond to the low-level control system having corresponding known skill groups, and after receiving the instruction, they can independently call the internal logic to complete all the details.

[0064] The subtask composed of atomic actions can be regarded as a fine-grained task, for example: pick up the bowl closest to yourself, put the bowl in your hand into the sink, pick up the plate closest to the sink, and put the plate into the sink. This corresponds to the high-level control system needing more thorough and step-by-step guidance when facing unknown tasks.

[0065] The low-level control system (second control system) needs to respond to environmental changes in real time, generate control instructions that can directly drive the robot hardware according to the target interface control signal provided by the high-level control system and the latest environmental perception data. It can be seen that the low-level control system focuses more on real-time high-frequency control and needs to handle the immediate actions and feedback of the robot. This division of labor allows the robot control system to maintain the ability to understand and plan complex tasks while also being able to quickly and accurately execute actions, thereby achieving efficient and coordinated robot control.

[0066] In this embodiment, the high-level control system decomposes the target task instruction into subtasks of different granularities, and through the multi-granularity task decomposition mechanism, the high-level control system can intelligently select the optimal planning granularity according to the complexity of the task and the ability reserve of the low-level control system. Specifically, for composite tasks mastered by the low-level control system, the high-level control system only needs to issue high-level instructions and make full use of existing skill modules. For completely new or complex unknown tasks, precise guidance is carried out through fine-grained atomic actions. This adaptive planning strategy allows the robot control system to efficiently handle routine tasks and respond to unexpected situations and novel scenarios, greatly expanding the application boundary and work reliability of the robot.

[0067] In addition, the high-level control system and the low-level control system are decoupled from each other, that is, they are relatively independent in function, reducing direct dependence between each other. This design allows the high-level control system and the low-level control system to work independently, but can also cooperate. This allows the robot control system to maintain high-level understanding of complex tasks while accurately controlling each action of the robot, resulting in more efficient and flexible robot operation. In addition, this also allows the high-level control system and the low-level control system to be upgraded and optimized independently without affecting the other system.

[0068] In addition, the interface control signal is a standardized, modal-rich intermediate instruction in the robot control system, which serves as an abstract interaction layer connecting the first control system (high-level) and the second control system (low-level), and is the core carrier for decoupling and parallel processing of the high-level control system and the low-level control system. Specifically, the interface control signal encapsulates the task intent through a pre-set standardized format (such as text, image, bounding box, future target observation, etc.), allowing the high-level control system to focus on high-level planning of what to do without needing to understand the specific implementation details of how to do it inside the low-level control system. This information hiding and abstraction is the architectural basis for functional decoupling.

[0069] Secondly, during operation, the high-level control system can pre-generate subsequent interface control signals based on the environment state and task sequence asynchronously after issuing the current interface control signal without waiting for the low-level control system to complete execution. This signal pre-generation mechanism breaks the strict timing dependence of planning and execution in traditional serial architectures, allowing the planning process of the high-level control system and the execution process of the low-level control system to overlap in time, forming a parallel pipeline.

[0070] Finally, through the use of a large-scale supervised signal prepared by a multi-modal labeling method during the training phase, the high-level control system learns to generate accurate and independently interpretable interface control signals. The high-level control system can focus on improving task decomposition and forward planning capabilities, while the low-level control system can focus on strengthening signal interpretation and action execution efficiency. Both work together through a standardized interface to improve the response speed and execution efficiency of the robot control system.

[0071] In some embodiments, the robot control system of the present application also considers the execution order and dynamic update of the interface control signals to adapt to rapidly changing environments and task requirements. Specifically, the following mechanisms are adopted when processing the interface control signals: before the first control system generates the current interface control signal, and / or in the case that the robot has not completed the action corresponding to the control instructions of the previous interface control signal corresponding to the current interface control signal, the target interface control signal is the previous interface control signal; after the first control system generates the current interface control signal, and in the case that the robot has completed the action corresponding to the control instructions of the previous interface control signal, the target interface control signal is the current interface control signal. It can be understood that the current interface control signal and the previous interface control signal are both one of the plurality of interface control signals.

[0072] In this embodiment, the second control system continues to use the previous interface control signal to generate control instructions before the new interface control signal (the current interface control signal) is generated or before the previous action (the action corresponding to the control instructions of the previous interface control signal) is completed. For example, the target task instruction of the robot is to grasp three objects in sequence, the first control system has generated the interface control signal for grasping the first object, and the robot is executing the action corresponding to the control instructions generated based on the interface control signal. Then, before the action is completed, even if the first control system has generated the interface control signal for grasping the second object, the second control system will still use the interface control signal for grasping the first object to guide the action of the robot. Only after the robot completes the action of grasping the first object and the first control system has generated the new interface control signal, i.e., the interface control signal for grasping the second object, the second control system will generate control instructions based on the new interface control signal.

[0073] In this embodiment, allowing the second control system to maintain the use of the previous interface control signal before the new interface control signal is generated or before the previous action is completed ensures the continuity and stability of the robot control system, so that the robot can continuously perform tasks. At the same time, when the new interface control signal is generated and the previous action is completed, the second control system can update the target interface control signal in time, so that the robot can quickly respond to the control instructions corresponding to the updated target interface control signal, and optimize the entire workflow of the robot.

[0074] In order to make the robot control system more intelligent to adapt to dynamic environments and ensure the continuity and reliability of task execution, the present application also provides an optional embodiment, which introduces an environment perception-based state judgment and dynamic error correction mechanism to improve the integrity and reliability of closed-loop control, which is implemented as follows.

[0075] In some implementations, the first control system is further configured to determine a completion state of the previous interface control signal based on the environment perception data, allow the second control system to execute a subsequent interface control signal corresponding to the previous interface control signal if the determination result is successful completion, control the second control system to re-execute the previous interface control signal if the determination result is an execution result error, and control the second control system to continue executing the previous interface control signal if the determination result is that the previous interface control signal is not completed within a permissible time, and control the second control system to terminate execution of the previous interface control signal if the determination result is that the previous interface control signal is not completed beyond the permissible time.

[0076] The workflow of the scheme is specifically described below through a complete task example.

[0077] Suppose that the target task instruction received by the robot is to assemble a part on a workbench into a specified product.

[0078] The first control system parses the task into a series of interface control signals with sequential correlation, for example, signal A, grasp the part on the workbench; signal B, move the part to the product assembly position; and signal C, perform the assembly operation.

[0079] After the first control system issues signal A, the system enters a supervision state. Specifically, when it is determined based on the environment perception data that the previous interface control signal is successfully completed, for example, the vision sensor identifies that the end effector of the robot has stably grasped the target part and the position is accurate, the first control system allows the second control system to start executing the subsequent signal B, thereby pushing the task flow into the next stage.

[0080] When it is determined based on the environment perception data that the execution result is an error, for example, the camera image shows that the grasping action deviates, the part is not grasped or has fallen, the first control system controls the second control system to re-execute the current signal A instead of continuing to execute the subsequent step, thereby correcting the error in time and avoiding the failure of the entire task chain due to the failure of the previous task.

[0081] When it is determined based on the environment perception data that the previous interface control signal is not completed within a permissible time, for example, within a preset 5-second permissible period, the first control system observes that the robot is still performing the grasping alignment, but the process is continuously normal. At this time, the first control system controls the second control system to continue executing the current signal A, maintains the continuity of task execution, and avoids unnecessary interruption.

[0082] When the environment perception data-based judgment exceeds the allowable time, for example, after exceeding the allowable time of 5 seconds, the first control system observes that the robot is still repeating invalid grasping attempts, which may be caused by part jamming. At this time, the first control system will control the second control system to terminate the execution signal A to prevent the second control system from falling into a dead loop and create conditions for triggering higher-level exception handling programs (such as re-planning or requesting manual intervention).

[0083] In this embodiment, by introducing an intelligent state judgment and dynamic error correction mechanism based on environment perception, not only does it break the time sequence dependence bottleneck of the traditional serial architecture, allowing the first control system to judge the state in parallel and prepare subsequent instructions while the second control system is executing the current task, thereby greatly compressing the task connection gap and improving response speed and execution efficiency. More through an independent environment perception verification mechanism, effectively identify and correct execution deviation and errors, combined with the timeout termination function to prevent the second control system from being abnormally blocked, enhancing the reliability and robustness in dynamic environments.

[0084] In other implementations, the second control system is further configured to judge the completion state of the previous interface control signal by itself; and the first control system is further configured to correct the second control system if it determines that the self-judgment result of the second control system is wrong.

[0085] Specifically, the second control system will judge the completion state of the previous interface control signal by itself during execution according to its internal logic (such as sensor feedback, joint position, etc.). At the same time, the first control system makes an independent judgment based on more comprehensive environment perception data (such as visual information) based on the real-world state. When the judgment results of the two systems are inconsistent, the first control system will take its own judgment as the standard and forcibly correct the behavior of the second control system.

[0086] Optionally, the correction of the second control system includes: If the first control system judges that the previous interface control signal is not successfully completed, and the second control system judges that it is completed, the first control system controls the second control system to re-execute the previous interface control signal; If the first control system judges that the previous interface control signal is successfully completed, and the second control system judges that it is not completed, the first control system controls the second control system to terminate the current action, and generates a subsequent interface control signal and sends it to the second control system.

[0087] For example, in the task of putting a bottle into a recycling bin, the second control system may judge that the placing action has been completed due to sensor error, while the bottle is still in the mechanical claw. The first control system observes through the camera that the bottle has not entered the recycling bin, and determines that the second control system has made a mistake. The first control system then controls the second system to re-execute the placing interface control signal to ensure that the task is actually completed.

[0088] In the task of tightening a screw, the screw has actually been tightened to the right position, but the second control system may mistakenly believe that the task is not completed due to fluctuations in the torque sensor reading, causing the actuator to continue to apply torque without stopping. The first control system confirms through visual analysis that the screw cap has been properly attached to the surface, and determines that the second control system has made a mistake. At this time, the first control system controls the second control system to terminate the invalid tightening action, and immediately generates and issues a subsequent interface control signal (such as moving the screwdriver away), thereby avoiding damage to the device and ensuring smooth progress of the task flow.

[0089] In this embodiment, through the independent verification of the first control system based on environmental perception, the two typical error states of false positive completion and false negative completion of the second control system can be accurately identified, and corresponding corrective measures of re-execution and forced advancement can be taken respectively. This scheme not only significantly improves the accuracy and reliability of the robot control system in judging the execution state of the task, prevents the failure of the entire task chain or damage to the device caused by a single point of judgment error, but also avoids invalid waiting or repeated attempts of the system in the error state through intelligent intervention, optimizes the utilization efficiency of computing resources and time resources, and finally makes the robot exhibit stronger adaptability and task completion ability in complex working environments.

[0090] The foregoing embodiments mainly introduce the basic components of the robot control system and the corresponding control process. Next, the internal working mechanism of the first control system is further described in order to more specifically describe how it generates interface control signals.

[0091] Specifically, the first control system includes: a coding module configured to preliminarily analyze the current pose of the robot, the target task instruction, and the environmental perception data to obtain a first target feature; a transformer base large model configured to optimize and enhance the first target feature to obtain a second target feature; and a decoding module configured to process the second target feature to generate a plurality of interface control signals.

[0092] In some embodiments, the encoding module learns and identifies, through its deep network structure, features in the current pose of the robot, the target task instruction, and the environmental perception data that are more important for completing the task. These features are integrated and encoded into first target features. It can be understood that the first target features are a set of high-level abstract representations that capture the essence of the task and the dynamic characteristics of the environment, providing a basis for the decision-making and action of the robot.

[0093] Then, the transformer-based large model weights the first target features through its self-attention layer to identify and emphasize the most critical information for completing the task. Meanwhile, the multi-head attention mechanism of the transformer-based large model can handle the feature relationships in multiple subspaces simultaneously, thereby comprehensively understanding various aspects of the task. Through a deep feedforward network, the first target features are nonlinearly transformed to enhance their expressive power and make them more suitable for the needs of robot control. Finally, the transformer-based large model can extract more rich and useful information from the first target features to form second target features. These second target features not only contain precise guidance for robot actions, but also incorporate adaptability to environmental changes and a deep understanding of the task target.

[0094] Then, the decoding module parses the second target features to understand the task requirements and environmental conditions represented by the second target features. These abstract features are then mapped to specific control parameters to generate multiple interface control signals.

[0095] In this embodiment, the first control system uses a hierarchical processing procedure to make the robot control system more efficient, flexible, and reliable when handling complex tasks. Moreover, the transformer-based large model can handle complex data relationships, reducing unnecessary computational steps and thereby reducing the consumption of computing resources. This improves the robot control system's ability to perceive, understand, and plan the environment, and increases the response speed and execution efficiency of the robot control system.

[0096] In some embodiments, to further enhance the processing capability of the first control system for multi-source information, the encoding module and the decoding module are designed to include multiple dedicated encoders and decoders. Specifically, Figure 3 Fig. 1 shows a structural schematic diagram of a first control system according to an embodiment of the present application. As shown in Fig. 1, the first control system includes an encoding module, a transformer-based large model, and a decoding module. Figure 3 As shown, the encoding module includes a visual encoder, a language encoder, an audio encoder, and a sensor data encoder; the decoding module includes a language decoder and a visual decoder.

[0097] In the encoding stage, the visual encoder is used to process image-based environmental perception data, such as visual information of the surrounding environment obtained through a camera, which enables the robot to understand and respond to changes in the visual scene. The language encoder and the audio encoder are used to process text-based or audio-based target task instructions, so that the robot can receive and understand natural language commands or audio signals from humans, enhancing the naturalness and flexibility of human-robot interaction. The sensor data encoder is used to process the current pose of the robot, ensuring that the robot has an accurate understanding of its position and pose in space.

[0098] In the decoding stage, the language decoder is used to output the task description in the interface control signal, which helps the robot understand the background and target of the task, making the execution process more in line with human instructions. The visual decoder is used to output the recognition information of the execution target in the interface control signal, so that the robot can accurately identify and locate the execution target in a complex environment.

[0099] Correspondingly, the loss function of the first control system includes the classification loss of text prediction, the mean square error loss of image prediction, and the cross-entropy loss. Among them, the classification loss of text prediction is used to optimize the performance of the language decoder. The mean square error loss guides the visual decoder to learn how to more accurately identify and locate the execution target by comparing the difference between the predicted position of the execution target by the visual decoder and the actual position, thereby improving the accuracy of the robot in navigating and operating in the visual scene. The cross-entropy loss function measures the difference between the probability distribution predicted by the visual decoder and the probability distribution of the true label when the visual decoder outputs the recognition information of the execution target in the interface control signal, so that the first control system can more accurately identify the category of the execution target in the environment.

[0100] The first control system of the embodiments of the present application realizes the fundamental difference from the traditional robot control system by constructing a unified multi-modal autoregressive architecture. Specifically, through the multi-modal encoder of vision, language, audio and sensor data, the heterogeneous input information (text, image, speech, pose, camera parameter) is mapped to a unified semantic feature space. Then, through the language decoder and the vision decoder, a multi-modal interface control signal containing text description, target image, bounding box, future observation and hidden feature is generated in a self-recurrent manner. This design breaks the paradigm of independent processing and late fusion of each modality information in traditional systems, and realizes the deep interaction and collaborative reasoning of cross-modal features within the model. Secondly, the autoregressive multi-modal signal generation mechanism ensures the high consistency of the output signal in time sequence and semantics, so that the task planning has the overall and context awareness ability. Finally, the multi-task optimization target composed of text classification loss, image mean square error and cross entropy loss jointly supervises the accuracy of different modal outputs, so that the first control system not only understands the abstract instructions, but also accurately predicts the visual attributes and spatial relationships of the execution target, so that the first control system becomes a robot brain that can deeply understand the environment, make multi-step forward planning and output rich control semantics, significantly improving the intelligence level, generalization ability and naturalness of human-computer interaction of robot task planning.

[0101] Next, the internal working mechanism of the second control system is further described in order to more specifically describe how it generates control instructions. Specifically, the second control system includes an encoding module configured to preliminarily analyze the current pose of the robot, the target interface control signal and the environment perception data to obtain third target features; a transformer configured to optimize and enhance the third target features to obtain fourth target features, so as to output robot control values based on the fourth target features.

[0102] Similarly, the encoding module learns and identifies the feature information important to the robot action execution in the current pose of the robot, the target interface control signal and the environment perception data through its deep network structure, and these features are integrated and encoded into the third target features.

[0103] Then, the transformer analyzes the third target features in depth through its self-attention mechanism to identify the important relationships and interactions between the features. At the same time, the multi-head attention mechanism of the transformer can process multiple feature subspaces simultaneously, so as to more comprehensively understand the task requirements and environmental conditions. After optimizing and enhancing the third target features, the fourth target features are generated. Finally, based on the fourth target features, the transformer outputs specific robot control values for guiding the execution mechanism of the robot, ensuring that the actions of the robot can be executed in the predetermined order.

[0104] In this embodiment, the application of the converter by the second control system improves the performance of the robot control system in feature extraction, optimization and control instruction generation, enabling the robot to perform tasks more flexibly and accurately while maintaining adaptability to complex environments.

[0105] In some embodiments, in order to improve the processing capacity of the second control system for multi-source information, the encoding module contains various dedicated encoders. Specifically, Figure 4 The structure of the second control system provided by an embodiment of the present application is shown. As Figure 4 As shown, the encoding module includes a visual encoder and a language encoder, as well as a sensor data encoder. The visual encoder is used to process image-based environmental perception data, such as visual information of the surrounding environment obtained through a camera, and image information in the interface control signal. The language encoder is used to process text information in the interface control signal. The sensor data encoder is used to process the current pose of the robot, ensuring that the robot has an accurate understanding of its position and attitude in space. In other embodiments, the second control system also includes a language decoder (not shown in the figure) connected in cascade with the transformer, which is used to output a number (such as 1-10) representing the progress of the current action execution.

[0106] Correspondingly, the loss function of the second control system includes the classification loss of text prediction and the mean square error loss of robot control value prediction. Similarly, the classification loss of text prediction is used to optimize the performance of the language decoder. The mean square error loss of robot control value prediction is used to measure the difference between the robot control value output by the second control system and the expected value. By minimizing the mean square error, the second control system can learn how to more accurately adjust the robot's actions to achieve smooth and accurate motion control, ensuring that the robot's action execution meets the task requirements.

[0107] From Figure 3 And Figure 4 It can be seen that the first control system, as a low-frequency control system, includes an audio encoder, a visual encoder, a language encoder and a sensor data encoder in its structure. These encoders are responsible for processing different types of input data to extract key features from complex input data to form the first target features, and then generate the second target features through the optimization of the transformer-based large model. This process improves the robot control system's ability to perceive, understand and plan the environment, and improves the response speed and execution efficiency of the robot control system.

[0108] The second control system, as a high-frequency control system, focuses on outputting robot control values, and thus omits the audio encoder and the visual decoder. The simplified model structure has a lower computational complexity, and can achieve efficient control with a lower computational cost mainly by means of the control commands that have been purified and optimized by the first control system. Meanwhile, the success rate that can be achieved by a larger and more complex model in the past is achieved while maintaining the speed advantage. The structure of the dual control system with division of labor improves the overall performance of the robot control system, so that the robot can better adapt to complex and variable task requirements.

[0109] Figure 5 Fig. 1 shows a flowchart of a robot control method provided by an embodiment of the present application. The method is exemplarily applied to the first control system. As shown in Fig. 1, the method comprises the following steps. Figure 5

[0110] In step S510, environmental perception data is acquired.

[0111] The environmental perception data is exemplarily acquired by a sensor (for example, a camera), and includes the position of the robot, the position and state of surrounding objects, and other factors that can affect task execution.

[0112] In step S520, a plurality of interface control signals are generated based on target task instructions and the environmental perception data.

[0113] The target task instructions define the task to be completed by the robot and the expected result, for example, the target task instructions are to throw the garbage on the table into the garbage can.

[0114] The execution order of the plurality of interface control signals has a correlation relationship, that is, the plurality of interface control signals are not isolated, but are correlated with each other in a specific order. These interface control signals represent different stages or steps of the robot executing the task, and the generation and execution of each interface control signal depend on the completion state of the previous interface control signal, ensuring the continuity and coordination of the target task execution. For example, in the task of picking and placing a target object, the first interface control signal is used to instruct the robot to move to the position of the target object, the second interface control signal is used to instruct the robot to grasp the target object, and the third interface control signal is used to instruct the robot to place the target object at a specified position. It can be understood that if the execution order of any step is disturbed, the robot task execution can fail or the robot action can be uncoordinated. Therefore, in this embodiment, the correlation relationship between the execution order ensures that the generation and execution of the interface control signals are in order, reflecting the logical flow of task execution, so that the robot can accurately complete the task according to the predetermined plan.

[0115] ​In addition, the generation and execution of each interface control signal depends on the completion status of the previous interface control signal. For example, the first control system determines the completion status of the previous interface control signal based on the environmental perception data; if the determination result is successful completion, the second control system is allowed to execute the subsequent interface control signal corresponding to the previous interface control signal; if the determination result is an execution result error, the second control system is controlled to re-execute the previous interface control signal; if the determination result is that the previous interface control signal is not completed within the allowable time, the second control system is controlled to continue executing the previous interface control signal; if the determination result is that the previous interface control signal is not completed beyond the allowable time, the second control system is controlled to terminate the execution of the previous interface control signal. Alternatively, the second control system determines the completion status of the previous interface control signal by itself, and the first control system corrects the second control system if it determines that the self-determination result of the second control system is incorrect.

[0116] Exemplarily, each interface control signal includes but is not limited to the following representation methods: text, picture, bounding box, functional bounding box, future target observation, and implicit feature.

[0117] In some embodiments, the first control system calculates and decides these information through its internal algorithm, decomposes complex tasks, and generates corresponding interface control signals for each decomposed subtask. The interface control signal contains rich information, which is provided to the second control system in different forms, so that the second control system can understand the context of the task, the characteristics of the execution target, the expected action result, and how to interact with the environment.

[0118] In this embodiment, the first control system is responsible for generating a plurality of interface control signals with a correlation relationship in execution sequence based on the target task instruction and the environmental perception data. When the first control system generates the first interface control signal based on the target task instruction and the environmental perception data and sends it to the second control system, its work does not pause, but immediately carries out the pre-generation work of the subsequent interface control signal based on the correlation relationship of the task sequence and the latest environmental state. At the same time, the second control system independently generates the control instruction corresponding to the currently received interface control signal. This design forms a planning and execution overlapping operation mode. Therefore, the overall response time of the robot control system is no longer equal to the linear accumulation of the execution time of each link, but depends on the longer one of the planning flow and the execution flow, thereby significantly compressing the idle waiting time between the task chain. It can be seen that the scheme in this embodiment significantly improves the response speed and execution efficiency of the robot control, reduces the time delay caused by waiting. In addition, this parallel control method enables the robot to adapt to environmental changes more quickly and adjust the control strategy in time, thereby reducing the risk of task failure caused by the speed of environmental change exceeding the execution speed of the robot.

[0119] In some embodiments, the first control system comprises an encoding module, a transformer base large model, and a decoding module; based on the target task instruction and the environment perception data, a plurality of interface control signals are generated, including: the encoding module preliminarily analyzes the current pose of the robot, the target task instruction, and the environment perception data to obtain first target features; the transformer base large model optimizes and enhances the first target features to obtain second target features; and the decoding module processes the second target features to generate a plurality of interface control signals.

[0120] Specifically, the encoding module learns and identifies, through its deep network structure, features in the current pose of the robot, the target task instruction, and the environment perception data that are more important for completing the task. These features are integrated and encoded into first target features. It can be understood that the first target features are a set of high-order abstract representations that capture the essence of the task and the dynamic characteristics of the environment, providing a basis for the decision-making and action of the robot.

[0121] Then, the transformer base large model weights the first target features through its self-attention layer to identify and emphasize the most critical information for completing the task. At the same time, the multi-head attention mechanism of the transformer base large model can simultaneously process the feature relationships in multiple subspaces, thereby more comprehensively understanding various aspects of the task. And through the deep feedforward network, the first target features are nonlinearly transformed to enhance the expression ability of the first features, making them more suitable for the needs of robot control. Finally, the transformer base large model can extract more rich and useful information from the first target features to form second target features. These second target features not only contain precise guidance for robot actions, but also incorporate adaptability to environmental changes and a deep understanding of the task target.

[0122] Then, the decoding module analyzes the second target features to understand the task requirements and environmental conditions represented by the second target features. And these abstract features are mapped to specific control parameters to generate a plurality of interface control signals.

[0123] In this embodiment, through a hierarchical processing procedure, the method is more efficient, flexible, and reliable in handling complex tasks. Moreover, the transformer base large model can handle complex data relationships, reducing unnecessary calculation steps and thereby reducing the consumption of computing resources, improving the perception, understanding, and planning capabilities of the environment, and improving the response speed and execution efficiency of the robot control method.

[0124] In some embodiments, to further enhance the processing capability of the method for multi-source information, the encoding module and the decoding module are designed to include multiple dedicated encoders and decoders. Specifically, the encoding module includes a visual encoder, a language encoder, an audio encoder, and a sensor data encoder; and the decoding module includes a language decoder and a visual decoder.

[0125] In the encoding stage, the visual encoder is used to process the image type of environment perception data, such as the visual information of the surrounding environment obtained through the camera, which enables the robot deploying the method in the Figure 5 understand and respond to changes in the visual scene. The language encoder and the audio encoder are used to process the text type or audio type of target task instructions, so that the robot receives and understands the natural language commands or audio signals of humans, enhancing the naturalness and flexibility of human-robot interaction. The sensor data encoder is used to process the current pose of the robot, ensuring that the robot has an accurate understanding of its own position and pose in space.

[0126] In the decoding stage, the language decoder is used to output the task description in the interface control signal, which helps the robot understand the background and target of the task, making the execution process more in line with human instructions. The visual decoder is used to output the recognition information of the execution target in the interface control signal, so that the robot can accurately identify and locate the execution target in a complex environment.

[0127] The first control system of the embodiments of the present application realizes a fundamental difference from the traditional robot control system by constructing a unified multi-modal autoregressive architecture. Specifically, through the multi-modal encoders of vision, language, audio and sensor data, the heterogeneous input information (text, image, speech, pose, camera parameters) is mapped to a unified semantic feature space; then, through the language decoder and the visual decoder, a multi-modal interface control signal containing text description, target image, bounding box, future observation and hidden features is generated in an autoregressive manner. This design breaks the paradigm of independent processing of each modality information and late fusion in traditional systems, and realizes deep interaction and collaborative reasoning of cross-modal features within the model. Secondly, the autoregressive multi-modal signal generation mechanism ensures the high consistency of the output signal in time sequence and semantics, so that the task planning has the ability of overall and context awareness. Finally, the multi-task optimization target composed of text classification loss, image mean square error and cross entropy loss jointly supervises the accuracy of different modal outputs, so that the first control system not only understands abstract instructions, but also accurately predicts the visual attributes and spatial relationships of the execution target, so that the first control system becomes a robot brain that can deeply understand the environment, make multi-step forward planning and output rich control semantics, significantly improving the intelligence level, generalization ability and naturalness of human-robot interaction of the robot task planning.

[0128] Figure 6 Fig. 1 shows a flowchart of a robot control method provided by another embodiment of the present application. Exemplarily, the method is applied to the second control system. As shown in Figure 6 The method includes the following steps.

[0129] Step S610, obtain environment perception data and target interface control signal.

[0130] The target interface control signal is one of the multiple interface control signals generated by the first control system based on the target task instruction and the environment perception data. The execution order between the multiple interface control signals has a correlation relationship, and the generation and execution of each interface control signal depend on the completion state of the previous interface control signal.

[0131] Exemplarily, the environment perception data is obtained by sensors (e.g., cameras), including the position of the robot, the position and state of surrounding objects, and other factors that may affect task execution.

[0132] Step S620, generate control instructions based on the environment perception data and the target interface control signal.

[0133] The purpose of step S620 is to facilitate the robot to execute corresponding actions based on the control instructions. Specifically, the control instructions include robot control values, which include but are not limited to at least one of joint rotation angle, travel distance, travel direction, end effector position, head pose, and torso pitch value.

[0134] The joint rotation angle refers to the specific angle that each joint in the robot's mechanical arm or dexterous hand and other components should rotate. In the case of a mechanical arm, the joint rotation angle determines the position and direction of the end effector of the mechanical arm. For a dexterous hand, the joint rotation angle determines the opening and closing degree and pose of the fingers, thereby achieving grasping, releasing or other operations.

[0135] The travel distance refers to the straight-line distance that the robot chassis or other moving components need to move, which is a basic parameter in robot path planning and navigation. By accurately controlling the travel distance, the robot can accurately reach the predetermined target position or move along a specific path.

[0136] The travel direction refers to the direction of the robot when moving. The travel direction can be an angle value indicating the rotation angle of the robot relative to its current direction or a certain reference direction (such as north), ensuring that the robot can move in the correct direction, avoid collisions, and effectively plan the path.

[0137] The position of the end effector refers to the three-dimensional coordinates of the robot's end effector (such as the tool or gripper at the end of the mechanical arm) in space, usually including X, Y, Z coordinates, indicating its specific position relative to a certain reference point.

[0138] The head pose refers to the direction and inclination state of the robot's head relative to its body or base, including yaw angle, pitch angle, and roll angle.

[0139] The torso lifting and pitching value refers to the vertical movement and the tilting angle of the robot torso. The lifting value indicates the position of the upward and downward movement of the torso, and the pitching value indicates the angle of the forward and backward tilting of the torso.

[0140] In this embodiment, the second control system is responsible for converting the target interface control signal into specific control instructions to drive the robot to perform corresponding actions. Since the interface control signal is generated independently of the second control system, that is, after the first control system generates an interface control signal, it can start generating the next interface control signal without waiting for the second control system to generate the corresponding instructions and perform the corresponding actions. That is, the first control system and the second control system are dependent and can be executed in parallel. This design significantly improves the response speed and execution efficiency of the robot control, reduces the time delay caused by waiting. In addition, this parallel control method enables the robot to adapt to environmental changes more quickly and adjust the control strategy in a timely manner, thereby reducing the risk of task failure caused by the speed of environmental changes exceeding the execution speed of the robot.

[0141] In some embodiments, the second control system includes an encoding module and a converter; based on the environment perception data and the target interface control signal, the control instructions are generated, including: the encoding module performs preliminary analysis on the current pose of the robot, the target interface control signal and the environment perception data to obtain the third target feature; the converter optimizes and enhances the third target feature to obtain the fourth target feature, so as to output the robot control value based on the fourth target feature.

[0142] Similarly, the encoding module learns and identifies the important feature information in the current pose of the robot, the target interface control signal and the environment perception data through its deep network structure, and these features are integrated and encoded into the third target feature.

[0143] Then, the converter analyzes the third target feature in depth through its self-attention mechanism, and identifies the important relationship and interaction between the features. At the same time, the multi-head attention mechanism of the converter can process multiple feature subspaces at the same time, so as to more comprehensively understand the task requirements and environmental conditions. After optimizing and enhancing the third target feature, the fourth target feature is generated. Finally, based on the fourth target feature, the converter outputs specific robot control values to guide the execution mechanism of the robot, ensuring that the actions of the robot can be executed in the predetermined order.

[0144] In this embodiment, through the application of the converter, the performance of the robot control method in feature extraction, optimization and control instruction generation is improved, so that the robot deployed with the method in the Figure 6 can execute tasks more flexibly and accurately while maintaining the ability to adapt to complex environments.

[0145] In some embodiments, to improve the processing capability of the method in Figure 6 the multi-source information, the encoding module contains multiple dedicated encoders. Specifically, the encoding module includes a visual encoder and a language encoder, and a sensor data encoder. The visual encoder is used to process image type environment perception data, such as visual information of the surrounding environment obtained through a camera, and image information in the interface control signal. The language encoder is used to process text information in the interface control signal. The sensor data encoder is used to process the current pose of the robot, ensuring that the robot has an accurate understanding of its own position and attitude in space. In other embodiments, the second control system further includes a language decoder (not shown in the figure) connected in cascade with the transformer, for outputting a number (such as 1-10) representing the current action execution progress.

[0146] It should be understood that the description of the robot control method embodiments corresponds to the description of the robot control system embodiments, and therefore, the parts of the robot control method not described in detail can be referred to the previous system embodiments.

[0147] In the following, an electronic device according to embodiments of the present application is described with reference to Figure 7 FIG. 1. Figure 7 FIG. 1 shows a structural schematic diagram of an electronic device provided by an exemplary embodiment of the present application.

[0148] As shown in Figure 7 FIG. 1, the electronic device 70 includes one or more processors 701 and a memory 702.

[0149] The processor 701 can be a central processing unit (CPU) or other forms of processing unit having data processing capability and / or instruction execution capability, and can control other components in the electronic device 70 to perform desired functions.

[0150] The memory 702 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM), cache memory, and the like. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor 701 can run the program instructions to implement the robot control method of various embodiments of the present application described above and / or other desired functions. Various contents such as environment perception data, multiple interface control signals, control instructions, and the like can also be stored in the computer readable storage medium.

[0151] In one example, the electronic device 70 can further include an input device 703 and an output device 704, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0152] The input device 703 can include, for example, a keyboard, a mouse, and the like.

[0153] The output device 704 can output various information, including environmental perception data, multiple interface control signals, control instructions, and the like, to the outside. The output device 704 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0154] Of course, in order to simplify, Figure 7 Only some of the components in the electronic device 70 related to the present application are shown in the middle, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 70 can also include any other appropriate components according to specific application cases.

[0155] In addition to the above method and device, the embodiments of the present application can also be a computer program product, which includes computer program instructions that make the processor execute the steps of the robot control method according to various embodiments of the present application described above in the specification when the processor is running.

[0156] The computer program product can be written in any combination of one or more programming languages for executing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and the like, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0157] In addition, the embodiments of the present application can also be a computer readable storage medium, which stores computer program instructions, which make the processor execute the steps of the robot control method according to various embodiments of the present application described above in the specification when the processor is running.

[0158] The computer readable storage medium can be embodied as one or more combinations of a readable medium and a readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] The above description of the application is described with specific embodiments, but it should be pointed out that the advantages, advantages, effects mentioned in the application are only examples and not limitations, and these advantages, advantages, effects cannot be considered as the necessary possession of each embodiment of the application. In addition, the above specific details disclosed are only for the purpose of example and understanding, and not limited to the above specific details, which are not limited to the above specific details.

[0160] The block diagram of the device, apparatus, equipment, system involved in the application is only an illustrative example and is not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatus, equipment, system can be connected, arranged, configured in any way. Words such as "include", "contain", "have" and the like are open-ended words, which mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0161] It should also be noted that in the device, equipment and method of the application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the application.

[0162] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0163] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations, which fall within the scope of the application.

Claims

1. A robot control system, characterized in that, The robot control system comprises: a first control system; a second control system connected in cascade to the first control system; the first control system is configured to generate a plurality of interface control signals based on target task instructions and environment perception data, the execution order of the plurality of interface control signals has a correlation relationship, and the generation and execution of each interface control signal depends on the completion state of the previous interface control signal; the second control system is configured to obtain a target interface control signal, generate a control instruction based on the target interface control signal and the environment perception data, so that the robot performs a corresponding action based on the control instruction.

2. The robot control system of claim 1, wherein, The first control system is a high-order control system, and is further configured to decompose the target task instructions into subtasks of different granularities and generate interface control signals of corresponding granularities; the second control system is a low-order control system; Preferably, the first control system and the second control system are decoupled from each other.

3. The robot control system of claim 2, wherein, The subtasks of different granularities include: subtasks composed of composite tasks executed by the second control system, wherein the composite tasks represent tasks that can be executed independently by the second control system as a whole and do not require action decomposition by the first control system; and subtasks composed of atomic actions executed by the second control system.

4. The robot control system according to claim 1, wherein before the first control system generates a current interface control signal, and / or in the case that the robot has not completed an action corresponding to a control instruction of a previous interface control signal corresponding to the current interface control signal, the target interface control signal is the previous interface control signal, and the current interface control signal and the previous interface control signal are one of the plurality of interface control signals; after the first control system generates the current interface control signal, and in the case that the robot has completed the action corresponding to the control instruction of the previous interface control signal, the target interface control signal is the current interface control signal.

5. The robot control system according to any one of claims 1 to 4, characterized in that, The first control system is further configured to: determine the completion state of the previous interface control signal based on the environment perception data; if the determination result is successful completion, allow the second control system to execute a subsequent interface control signal corresponding to the previous interface control signal; if the determination result is an execution result error, control the second control system to re-execute the previous interface control signal.

6. The robot control system of claim 5, wherein, The first control system is further configured to: if the determination result is that the previous interface control signal has not been completed within a permissible time, control the second control system to continue executing the previous interface control signal; if the determination result is that the previous interface control signal has not been completed beyond the permissible time, control the second control system to terminate execution of the previous interface control signal.

7. The robot control system according to any one of claims 1 to 4, wherein the second control system is further configured to determine the completion state of the previous interface control signal by itself. The first control system is further configured to correct the second control system if it is determined that the self-determination result of the second control system is incorrect.

8. The robot control system according to any one of claims 1 to 4, characterized in that, The first control system comprises: An encoding module configured to preliminarily analyze the current pose of the robot, the target task instruction and the environment perception data to obtain first target features; A transformer base large model configured to optimize and enhance the first target features to obtain second target features; A decoding module configured to process the second target features to generate the plurality of interface control signals; The interface control signals comprise at least one of text, image, video, bounding box, hidden feature and future target observation. The future target observation is used to show the image of the expected scene after the completion of the task, so that the second control system compares the actual environment perception data with the future target observation during the execution process to determine the completion state of the task.

9. A robot control method characterized by, The method applied to the first control system comprises: Obtaining environment perception data; Based on the target task instruction and the environment perception data, a plurality of interface control signals are generated, the execution order of the plurality of interface control signals has a correlation relationship, and the generation and execution of each interface control signal depend on the completion state of the previous interface control signal.

10. A robot control method characterized by, The method applied to the second control system comprises: Obtaining environment perception data and a target interface control signal, the target interface control signal being one of the plurality of interface control signals generated by the first control system based on the target task instruction and the environment perception data, the execution order of the plurality of interface control signals having a correlation relationship, and the generation and execution of each interface control signal depending on the completion state of the previous interface control signal; Based on the environment perception data and the target interface control signal, a control instruction is generated so that the robot performs corresponding actions based on the control instruction.

11. A computer readable storage medium characterized by, The storage medium stores a computer program, and the computer program is used to execute the robot control method in claim 9 and / or 10.

12. An electronic device, comprising: Comprise: A processor; A memory for storing instructions executable by the processor; The processor is used to execute the robot control method in claim 9 and / or 10.

13. A computer program product, characterised in that, The computer program product comprises instructions for executing the robot control method in claim 9 and / or 10 when executed on an electronic device.

Images