Robot control system and method, medium, electronic device and program product

Abstract

The invention provides a robot control system and method, a medium, electronic equipment and a program product, and relates to the technical field of robot control. The robot control system comprises a first control system and a second control system which are in cascade connection; wherein the first control system is configured to generate a plurality of interface control signals based on the target task instruction and the environment perception data, and the execution sequences of the plurality of interface control signals have an association relationship; and the second control system is configured to obtain the target interface control signal and generate a control instruction based on the target interface control signal and the environment sensing data, so that the robot executes a corresponding action based on the control instruction. According to the invention, the response speed and the execution efficiency of the robot control system are obviously improved, and the time delay caused by waiting is reduced.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robot control system and method, medium, electronic equipment and program product. Background Art

[0002] In visual closed-loop control, the robot first perceives the surrounding environment through vision and point cloud models, then performs motion planning, and finally performs body control. After this process is completed, the robot will cycle through perception, planning, and control again to achieve closed-loop control. However, this serial execution method causes the execution time to increase linearly. Especially when the closed-loop execution time is too long, it will cause the robot's control to become stuck, or the task cannot be completed because the speed of environmental changes exceeds the execution speed of the robot control system. Summary of the invention

[0003] In view of this, embodiments of the present application provide a robot control system and method, medium, electronic device and program product.

[0004] In the first aspect, an embodiment of the present application provides a robot control system, comprising: a first control system; a second control system cascadedly connected to the first control system; the first control system is configured to generate multiple interface control signals based on target task instructions and environmental perception data, and the execution order of the multiple interface control signals is associated with each other; the second control system is configured to obtain the target interface control signal, and generate control instructions based on the target interface control signal and environmental perception data, so that the robot performs corresponding actions based on the control instructions.

[0005] In combination with the first aspect, in certain implementations of the first aspect, the first control system is a high-order control system, and the second control system is a low-order control system; preferably, the first control system and the second control system are decoupled from each other.

[0006] In combination with the first aspect, in certain implementations of the first aspect, before the first control system generates a current interface control signal, and / or when the robot has not completed the action corresponding to the current interface control signal and the control instruction of the previous interface control signal, the target interface control signal is the previous interface control signal, and the current interface control signal and the previous interface control signal are one of multiple interface control signals; after the first control system generates the current interface control signal, and when the robot completes the action corresponding to the control instruction of the previous interface control signal, the target interface control signal is the current interface control signal.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the first control system includes: an encoding module, configured to perform preliminary analysis on the robot's current posture, target task instructions and environmental perception data to obtain a first target feature; a converter basic large model, configured to optimize and enhance the first target feature to obtain a second target feature; and a decoding module, configured to process the second target feature and generate multiple interface control signals.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the encoding module includes a visual encoder, a language encoder, an audio encoder and a sensor data encoder; the visual encoder is used to process environmental perception data; the language encoder and the audio encoder are used to process target task instructions; the sensor data encoder is used to process the current posture of the robot; preferably, the decoding module includes a language decoder and a visual decoder; the language decoder is used to output the task description in the interface control signal; and the visual decoder is used to output the identification information of the execution target in the interface control signal.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the second control system includes: an encoding module, configured to perform preliminary analysis on the current posture of the robot, the target interface control signal, and the environmental perception data to obtain a third target feature; a converter, configured to optimize and enhance the third target feature to obtain a fourth target feature, so as to output a robot control value based on the fourth target feature, and the robot control value includes a set of parameters for guiding the robot to perform specific actions.

[0010] In the second aspect, an embodiment of the present application provides a robot control method, which is applied to a first control system, and the method includes: acquiring environmental perception data; generating multiple interface control signals based on target task instructions and environmental perception data, and the execution order between the multiple interface control signals is associated with each other.

[0011] In combination with the second aspect, in certain implementations of the second aspect, the first control system is a high-order control system.

[0012] In combination with the second aspect, in certain implementations of the second aspect, the first control system includes an encoding module, a converter basic large model and a decoding module; based on the target task instructions and environmental perception data, multiple interface control signals are generated, including: the encoding module performs a preliminary analysis of the robot's current posture, target task instructions and environmental perception data to obtain a first target feature; the converter basic large model optimizes and enhances the first target feature to obtain a second target feature; the decoding module processes the second target feature to generate multiple interface control signals.

[0013] In the third aspect, an embodiment of the present application provides a robot control method, which is applied to a second control system, and the method includes: acquiring environmental perception data and a target interface control signal, the target interface control signal being one of multiple interface control signals generated by the first control system based on target task instructions and environmental perception data; generating control instructions based on the environmental perception data and the target interface control signal so that the robot performs corresponding actions based on the control instructions.

[0014] In combination with the third aspect, in certain implementations of the third aspect, the second control system is a low-order control system, and the first control system and the second control system are decoupled from each other.

[0015] In combination with the third aspect, in certain implementations of the third aspect, the second control system includes an encoding module and a converter; based on the environmental perception data and the target interface control signal, a control instruction is generated, including: the encoding module performs a preliminary analysis of the robot's current posture, the target interface control signal and the environmental perception data to obtain a third target feature; the converter optimizes and enhances the third target feature to obtain a fourth target feature, so as to output a robot control value based on the fourth target feature, and the robot control value includes a set of parameters for guiding the robot to perform specific actions.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the robot control method described in the second and third aspects.

[0017] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; the processor is used to execute the robot control method described in the second and third aspects.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes instructions, and when the instructions are executed on an electronic device, the electronic device implements the robot control method described in the first aspect.

[0019] In the present application, the robot control system adopts a cascade connection method to achieve efficient collaborative work between the first control system and the second control system. Specifically, in this structure, the first control system is responsible for generating multiple interface control signals with an associated execution order based on the target task instruction and the environmental perception data, and the second control system is responsible for converting the target interface control signal into a specific control instruction to drive the robot to perform the corresponding action. Since the interface control signal is generated independently of the second control system, that is, after the first control system generates an interface control signal, it can start to generate the next interface control signal without waiting for the second control system to generate the corresponding control instruction and execute the corresponding action. In other words, the first control system and the second control system are both dependent and can be executed in parallel. This design significantly improves the response speed and execution efficiency of the robot control system and reduces the time delay caused by waiting. In addition, this parallel processing mechanism enables the robot control system to adapt to environmental changes more quickly and adjust the control strategy in time, thereby reducing the risk of task failure caused by the speed of environmental changes exceeding the execution speed of the robot control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0021] Figure 1 Shown is a schematic diagram of the structure of a robot control system provided in one embodiment of the present application.

[0022] Figure 2 Shown is a schematic structural diagram of a robot control system provided in another embodiment of the present application.

[0023] Figure 3 Shown is a schematic structural diagram of a first control system provided in an embodiment of the present application.

[0024] Figure 4 Shown is a schematic structural diagram of a second control system provided in one embodiment of the present application.

[0025] Figure 5 Shown is a flow chart of a robot control method provided in one embodiment of the present application.

[0026] Figure 6 Shown is a flow chart of a robot control method provided in another embodiment of the present application.

[0027] Figure 7 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] With the advancement of technology, robot control systems have changed from simple repetitive execution of fixed actions to intelligent systems that can perform closed-loop control according to environmental changes. This transformation is due to the development of large language models, multimodal models, and embodied intelligence technologies, which enable robot control systems to be implemented in an end-to-end manner without complex intermediate processing steps.

[0030] In an end-to-end robot control system, the robot first uses vision and point cloud models to accurately perceive the surrounding environment in order to understand the environment. After that, motion planning is performed, that is, determining the path of how the robot moves from the current position to the target position. Finally, the body control is performed, which is used to convert the above motion planning into the actual action of the robot to ensure that the robot can accurately perform the task.

[0031] However, this closed-loop control process of perception, planning, and control faces challenges in actual operation. Specifically, these steps are executed serially, and the execution time of each stage accumulates, resulting in a linear increase in the overall response time. If the execution time of the closed-loop control is too long, the robot will also experience control jams or be unable to keep up with the rapid changes in the environment, which will lead to the inability to complete the task. In addition, the computational complexity of the robot control system is proportional to its control success rate and control accuracy, but inversely proportional to the control speed. In other words, under the current end-to-end system architecture, it is very difficult to achieve a balance between high-frequency real-time control and high success rate and accuracy. Therefore, although robot control systems have made significant progress in perception and control, they still face challenges in improving response speed and the ability to handle complex tasks.

[0032] In view of this, this application is proposed. Specifically, Figure 1 FIG. 1 is a schematic diagram of the structure of a robot control system provided by an embodiment of the present application. Figure 1 As shown, the robot control system 10 includes a first control system 110 and a second control system 120 , and the second control system 120 is cascade-connected to the first control system 110 .

[0033] Specifically, the first control system 110 is configured to generate a plurality of interface control signals based on the target task instructions and the environmental perception data. The second control system 120 is configured to obtain the target interface control signal, and generate a control instruction based on the target interface control signal and the environmental perception data, so that the robot performs the corresponding action based on the control instruction.

[0034] In some embodiments, the execution order between multiple interface control signals has an associated relationship. That is, multiple interface control signals are not isolated, but are mutually associated in a specific order. These interface control signals represent different stages or steps of the robot's execution of tasks, and the generation and execution of each interface control signal depends on the completion state of the previous interface control signal, ensuring the continuity and coordination of the execution of the target task. For example, in the task of picking and placing the target object, the first interface control signal is used to instruct the robot to move to the position of the target object, the second interface control signal is used to instruct the robot to grab the target object, and the third interface control signal is used to instruct the robot to place the target object at a specified position. It is understandable that if the execution order of any step is disrupted, it may lead to the failure of the robot's task execution or the incoordination of the robot's actions. Therefore, in this embodiment, the generation and execution of multiple interface control signals are orderly, reflecting the logical flow of task execution, so that the robot can accurately complete the task according to the predetermined plan.

[0035] In order to separate these interface control signals, all feasible interface control signals will be investigated and collected. Then, in the training samples, these feasible interface control signals will be manually labeled, manually assisted automatic labeling, model automatic labeling, or Monte Carlo labeling with multimodal large model scoring, so as to prepare a large number of supervision signals for training the first control system and the second control system.

[0036] In some embodiments, the first control system 110 receives a target task instruction, which defines the task that the robot needs to complete and the expected result. For example, the target task instruction is "throw the garbage on the table into the trash can". At the same time, the first control system collects environmental perception data through sensors (e.g., cameras), including the position of the robot, the position and state of surrounding objects, and other factors that may affect the execution of the task. The first control system 110 calculates and makes decisions on this information through its internal algorithm, decomposes complex tasks, and generates corresponding interface control signals for each decomposed subtask.

[0037] For example, each interface control signal includes but is not limited to the following representations:

[0038] Text: Provides a task description of the subtask in the form of natural language or instruction code. For example, in the above task of picking and placing the target object, the text representation of one of the interface control signals is to move to the location of the target object in order to tell the robot the specific action to perform.

[0039] Image: Contains identification information of the execution target of the subtask, such as an image containing the execution target (such as garbage on the table), which is used to provide visual information to the robot.

[0040] Bounding box: A rectangular box that identifies the location and size of the execution target in the image, usually containing the coordinates and size information of the execution target.

[0041] Functional bounding box: used to guide the robot how to interact with the action target, such as grasping, pushing.

[0042] Future target observations: images that show what the robot will see after the task is completed, in order to guide the robot to a specific target state.

[0043] Hidden features: used to capture and process complex data patterns within the first control system.

[0044] It can be seen that the interface control signal contains rich information, which is provided to the second control system in different forms, enabling it to understand the context of the task, the characteristics of the execution target, the expected action results and how to interact with the environment.

[0045] In some embodiments, after receiving the target interface control signal, the second control system 120 combines these target interface control signals and real-time environmental perception data to generate specific control instructions. Then, the second control system 120 sends the control instructions to the robot's actuators, such as motors or joints, to control the robot to perform corresponding actions. Specifically, the control instructions include robot control values, including but not limited to at least one of the joint rotation angle, travel distance, travel direction, end effector position, head posture, and trunk lift and pitch values.

[0046] Joint rotation angle refers to the specific angle that each joint in a robot's mechanical arm or dexterous hand should rotate. In the case of a mechanical arm, the joint rotation angle determines the position and orientation of the end effector of the mechanical arm. For a dexterous hand, the joint rotation angle determines the degree of opening and closing and the posture of the fingers to achieve grasping, releasing or other operations.

[0047] Travel distance refers to the straight-line distance that the robot chassis or other moving parts need to move. This value is a basic parameter in robot path planning and navigation. By precisely controlling the travel distance, the robot can accurately reach a predetermined target position or move along a specific path.

[0048] The direction of travel refers to the orientation of the robot when it moves. The direction of travel can be an angle value that indicates the rotation angle of the robot relative to its current orientation or a reference direction (such as north), ensuring that the robot can move in the right direction, avoid collisions, and plan paths effectively.

[0049] The position of the end effector refers to the three-dimensional coordinates of the robot's end effector (such as the tool or gripper at the end of the robotic arm) in space, usually including X, Y, and Z coordinates, indicating its specific position relative to a reference point.

[0050] Head posture refers to the direction and tilt of the robot's head relative to its body or base, including yaw, pitch, and roll angles.

[0051] The trunk lift and pitch values ​​refer to the vertical movement and tilt angle of the robot's trunk. The lift value indicates the position of the trunk moving up and down, and the pitch value indicates the angle of the trunk tilting forward and backward.

[0052] In this embodiment, the robot control system adopts a cascade connection method to achieve efficient collaborative work between the first control system and the second control system. Specifically, in this structure, the first control system is responsible for generating multiple interface control signals with an associated execution order based on the target task instructions and the environmental perception data, while the second control system is responsible for converting the target interface control signals into specific control instructions to drive the robot to perform corresponding actions. Since the interface control signal is generated independently of the second control system, that is, after the first control system generates an interface control signal, it can start to generate the next interface control signal without waiting for the second control system to generate the corresponding control instructions and execute the corresponding actions. In other words, the first control system and the second control system are both dependent and can be executed in parallel. This design significantly improves the response speed and execution efficiency of the robot control system and reduces the time delay caused by waiting. In addition, this parallel processing mechanism enables the robot control system to adapt to environmental changes more quickly and adjust the control strategy in time, thereby reducing the risk of task failure caused by the speed of environmental changes exceeding the execution speed of the robot control system.

[0053] In some embodiments, based on the position and function of the first control system in the control hierarchy, the first control system is determined as a high-order control system and the second control system is determined as a low-order control system. The two systems are designed to be decoupled from each other to improve the efficiency and reliability of the entire robot control system.

[0054] Figure 2The figure shows a schematic diagram of the structure of a robot control system provided by another embodiment of the present application. It can be seen that the high-level control system (first control system) is responsible for understanding the scene in which the robot is located, including task planning and state judgment, and generating a series of interface control signals with associated execution sequences. These functions usually do not require extremely high update frequencies, but involve complex algorithms and a large amount of data processing.

[0055] The low-level control system (second control system) needs to respond to environmental changes in real time and generate control instructions that can directly drive the robot hardware based on the target interface control signals and the latest environmental perception data provided by the high-level control system.

[0056] It can be seen that low-level control systems focus more on real-time high-frequency control and need to deal with the robot's immediate actions and feedback. This division of labor allows the robot control system to perform actions quickly and accurately while maintaining the ability to understand and plan complex tasks, thereby achieving efficient and coordinated robot control.

[0057] In some embodiments, the high-order control system and the low-order control system are decoupled from each other, that is, they are relatively independent in function, reducing direct dependence on each other. This design allows the high-order control system and the low-order control system to work independently, but can also work together. This allows the robot control system to accurately control every movement of the robot while maintaining a high-level understanding of complex tasks, thereby achieving more efficient and flexible robot operation. In addition, this also allows the high-order control system and the low-order control system to be upgraded and optimized independently without affecting the other system.

[0058] In some embodiments, the robot control system of the present application also considers the execution order and dynamic update of the interface control signal to adapt to the rapidly changing environment and task requirements. Specifically, the following mechanism is adopted when processing the interface control signal: before the first control system generates the current interface control signal, and / or when the robot has not completed the action corresponding to the control instruction of the previous interface control signal corresponding to the current interface control signal, the target interface control signal is the previous interface control signal; after the first control system generates the current interface control signal, and when the robot completes the action corresponding to the control instruction of the previous interface control signal, the target interface control signal is the current interface control signal. It can be understood that the current interface control signal and the previous interface control signal are both one of the multiple interface control signals.

[0059] In this embodiment, before a new interface control signal (current interface control signal) is generated or before the previous action (the action corresponding to the control instruction of the previous interface control signal) is completed, the second control system will continue to use the previous interface control signal to generate control instructions. For example, the target task instruction of the robot is to grab three objects in sequence, the first control system has generated an interface control signal for grabbing the first object, and the robot is executing the action corresponding to the control instruction generated based on the interface control signal. Then, before this action is completed, even if the first control system has generated an interface control signal for grabbing the second object, the second control system will still use the interface control signal for grabbing the first object to guide the robot's action. Only after the robot has completed the action of grabbing the first object and the first control system has generated a new interface control signal, that is, an interface control signal for grabbing the second object, will the second control system generate a control instruction based on the new interface control signal.

[0060] In this embodiment, the second control system is allowed to maintain the use of the previous interface control signal before the new interface control signal is generated or before the previous action is completed, which ensures the continuity and stability of the robot control system and enables the robot to perform tasks uninterruptedly. At the same time, when the new interface control signal is generated and the previous action is completed, the second control system can update the target interface control signal in time, so that the robot can quickly respond to the control instruction corresponding to the updated target interface control signal, optimizing the workflow of the entire robot.

[0061] The aforementioned embodiments mainly introduce the basic composition of the robot control system and its corresponding control process. The following further describes the internal working mechanism of the first control system so as to describe in more detail how it generates the interface control signal.

[0062] Specifically, the first control system includes: an encoding module, configured to perform preliminary analysis on the robot's current posture, target task instructions and environmental perception data to obtain a first target feature; a converter basic large model, configured to optimize and enhance the first target feature to obtain a second target feature; a decoding module, configured to process the second target feature and generate multiple interface control signals.

[0063] In some embodiments, the encoding module learns and identifies the features of the robot's current posture, target task instructions, and environmental perception data that are important for completing the task through its deep network structure, and these features are integrated and encoded into first target features. It can be understood that the first target features are a set of high-level abstract representations that capture the essence of the task and the dynamic characteristics of the environment, and provide a basis for the robot's decision-making and actions.

[0064] Next, the Transformer-based large model weights the first target features through its self-attention layer to identify and emphasize the information that is most critical to completing the task. At the same time, the multi-head attention mechanism of the Transformer-based large model can simultaneously process the feature relationships of multiple subspaces, thereby more comprehensively understanding all aspects of the task. And through a deep feedforward network, the first target features are nonlinearly transformed to enhance the expressive power of the first features, making them more adaptable to the needs of robot control. Finally, the Transformer-based large model can extract richer and more useful information from the first target features to form second target features. These second target features not only contain precise guidance for robot actions, but also incorporate adaptability to environmental changes and a deep understanding of task goals.

[0065] Then, the decoding module analyzes the second target feature to understand the task requirements and environmental conditions represented by the second target feature, and maps these abstract features to specific control parameters, thereby generating multiple interface control signals.

[0066] In this embodiment, the first control system uses a hierarchical processing flow to make the robot control system more efficient, flexible and reliable when processing complex tasks. In addition, the transformer basic large model can process complex data relationships, reduce unnecessary calculation steps, thereby reducing the consumption of computing resources, improving the robot control system's ability to perceive, understand and plan the environment, and improving the robot control system's response speed and execution efficiency.

[0067] In some embodiments, in order to further improve the processing capability of the first control system for multi-source information, the encoding module and the decoding module are designed to include a variety of dedicated encoders and decoders. Specifically, Figure 3 FIG. 1 is a schematic diagram of the structure of a first control system provided by an embodiment of the present application. Figure 3 As shown, the encoding module includes a visual encoder, a language encoder, an audio encoder and a sensor data encoder; the decoding module includes a language decoder and a visual decoder.

[0068] In the encoding stage, the visual encoder is used to process image-based environmental perception data, such as visual information of the surrounding environment obtained through the camera, which enables the robot to understand and respond to changes in the visual scene. The language encoder and audio encoder are used to process text-based or audio-based target task instructions so that the robot can receive and understand human natural language commands or audio signals, enhancing the naturalness and flexibility of human-computer interaction. The sensor data encoder is used to process the robot's current posture to ensure that the robot has an accurate understanding of its position and posture in space.

[0069] In the decoding stage, the language decoder is used to output the task description in the interface control signal, which helps the robot understand the background and goal of the task, making the execution process more consistent with human instructions. The visual decoder is used to output the identification information of the execution target in the interface control signal, so that the robot can accurately identify and locate the execution target in a complex environment.

[0070] Accordingly, the loss function of the first control system includes the classification loss of text prediction, the mean square error loss of image prediction and the cross entropy loss. Among them, the classification loss of text prediction is used to optimize the performance of the language decoder. The mean square error loss guides the visual decoder to learn how to more accurately identify and locate the execution target by comparing the difference between the position of the execution target predicted by the visual decoder and the actual position, thereby improving the accuracy of the robot's navigation and operation in the visual scene. When the cross entropy loss function measures the difference between the probability distribution predicted by the visual decoder and the probability distribution of the true label when the visual decoder outputs the identification information of the execution target in the interface control signal, so that the first control system can more accurately identify the category of the execution target in the environment.

[0071] This design in this embodiment enables the robot control system to process and respond to multi-source information more intelligently, improving the robot's autonomy and accuracy in executing tasks. Through this multi-modal encoding and decoding mechanism, the robot can more comprehensively understand task instructions and environmental conditions, thereby making more reasonable decisions when executing tasks.

[0072] The internal working mechanism of the second control system is further described below to describe in more detail how it generates control instructions. Specifically, the second control system includes: an encoding module configured to perform preliminary analysis on the current posture of the robot, the target interface control signal and the environmental perception data to obtain the third target feature; a transformer configured to optimize and enhance the third target feature to obtain the fourth target feature, so as to output the robot control value based on the fourth target feature.

[0073] Similarly, the encoding module learns and identifies important feature information about the robot's action execution from the robot's current posture, target interface control signals, and environmental perception data through its deep network structure. These features are integrated and encoded into the third target feature.

[0074] Then, the converter uses its self-attention mechanism to conduct an in-depth analysis of the third target feature and identify important relationships and interactions between features. At the same time, the converter's multi-head attention mechanism can process multiple feature subspaces simultaneously, thereby more comprehensively understanding task requirements and environmental conditions. After optimizing and enhancing the third target feature, the fourth target feature is generated. Finally, based on the fourth target feature, the converter outputs specific robot control values ​​to guide the robot's actuators to ensure that the robot's actions can be performed in a predetermined order.

[0075] In this embodiment, the second control system improves the performance of the robot control system in feature extraction, optimization and control instruction generation through the application of the converter, so that the robot can perform tasks more flexibly and accurately while maintaining the ability to adapt to complex environments.

[0076] In some embodiments, in order to improve the processing capability of the second control system for multi-source information, the encoding module includes a variety of dedicated encoders. Specifically, Figure 4 FIG. 1 is a schematic diagram of the structure of a second control system provided by an embodiment of the present application. Figure 4 As shown, the encoding module includes a visual encoder and a language encoder, and a sensor data encoder. The visual encoder is used to process image-based environmental perception data, such as visual information of the surrounding environment obtained by a camera, and image information in the interface control signal. The language encoder is used to process text information in the interface control signal. The sensor data encoder is used to process the current posture of the robot to ensure that the robot has an accurate understanding of its position and posture in space. In other embodiments, the second control system also includes a language decoder (not shown in the figure), which is cascaded with the converter to output a number (such as 1-10) of the current action execution progress.

[0077] Accordingly, the loss function of the second control system includes the classification loss of text prediction and the mean square error loss of robot control numerical prediction. Similarly, the classification loss of text prediction is used to optimize the performance of the language decoder. The mean square error loss of robot control numerical prediction is used to measure the difference between the robot control numerical value output by the second control system and the expected value. By minimizing the mean square error, the second control system can learn how to adjust the robot's movements more accurately to achieve smooth and accurate motion control, ensuring that the robot's movements meet the task requirements.

[0078] from Figure 3 and Figure 4It can be seen that the first control system, as a low-frequency control system, includes an audio encoder, a visual encoder, a language encoder, and a sensor data encoder in its structure. These encoders are responsible for processing different types of input data in order to extract key features from complex input data to form the first target feature, and then generate the second target feature through the optimization of the transformer-based large model. This process improves the robot control system's ability to perceive, understand, and plan the environment, and improves the robot control system's response speed and execution efficiency.

[0079] As a high-frequency control system, the second control system focuses on outputting robot control values, so the audio encoder and visual decoder are omitted. This simplified model structure has a low computational complexity, and mainly relies on the control commands that have been purified and optimized by the first control system, which can achieve efficient control at a low computational cost. In addition, while maintaining the speed advantage, it achieves a success rate that previously required larger and more complex models. This dual control system structure with division of labor and cooperation improves the overall performance of the robot control system, allowing the robot to better adapt to complex and changing task requirements.

[0080] Figure 5 FIG. 1 is a flow chart of a robot control method provided by an embodiment of the present application. By way of example, the method is applied to a first control system. Figure 5 As shown, the method includes the following steps.

[0081] Step S510, obtaining environmental perception data.

[0082] Exemplarily, environmental perception data is acquired through sensors (e.g., cameras), which include the position of the robot, the position and state of surrounding objects, and other factors that may affect task execution.

[0083] Step S520, generating multiple interface control signals based on the target task instructions and the environmental perception data.

[0084] The target task instruction defines the task that the robot needs to complete and the expected results. For example, the target task instruction is to throw the garbage on the table into the trash can.

[0085] The execution order between multiple interface control signals has an associated relationship, that is, multiple interface control signals are not isolated, but are mutually associated in a specific order. These interface control signals represent different stages or steps of the robot's task execution, and the generation and execution of each interface control signal depends on the completion state of the previous interface control signal, ensuring the continuity and coordination of the target task execution. For example, in the task of picking and placing the target object, the first interface control signal is used to instruct the robot to move to the position of the target object, the second interface control signal is used to instruct the robot to grab the target object, and the third interface control signal is used to instruct the robot to place the target object at a specified position. It is understandable that if the execution order of any step is disrupted, it may lead to the failure of the robot's task execution or the incoordination of the robot's actions. Therefore, in this embodiment, the associated relationship between the execution order ensures that the generation and execution of the interface control signal are orderly, reflecting the logical flow of task execution, so that the robot can accurately complete the task according to the predetermined plan.

[0086] Exemplarily, each interface control signal includes but is not limited to the following representations: text, picture, bounding box, functional bounding box, future target observation, and hidden features.

[0087] In some embodiments, the first control system calculates and makes decisions on this information through its internal algorithm, decomposes the complex task, and generates corresponding interface control signals for each decomposed subtask. The interface control signal contains rich information, which is provided to the second control system in different forms, so that it can understand the context of the task, the characteristics of the execution target, the expected action results, and how to interact with the environment.

[0088] In this embodiment, the first control system is responsible for generating a plurality of interface control signals with an associated execution order based on the target task instructions and the environmental perception data. Since the interface control signal is generated independently of the second control system, that is, after the first control system generates an interface control signal, it can start to generate the next interface control signal without having to wait for the second control system to generate the corresponding instructions and execute the corresponding actions before starting. In other words, the first control system and the second control system are both dependent and can be executed in parallel. This design significantly improves the response speed and execution efficiency of the robot control and reduces the time delay caused by waiting. In addition, this parallel control method enables the robot to adapt to environmental changes more quickly and adjust the control strategy in time, thereby reducing the risk of task failure caused by the speed of environmental changes exceeding the execution speed of the robot.

[0089] In some embodiments, the first control system includes an encoding module, a converter basic model and a decoding module; based on the target task instructions and environmental perception data, multiple interface control signals are generated, including: the encoding module performs preliminary analysis on the robot's current posture, target task instructions and environmental perception data to obtain a first target feature; the converter basic model optimizes and enhances the first target feature to obtain a second target feature; the decoding module processes the second target feature to generate multiple interface control signals.

[0090] Specifically, the encoding module learns and identifies the features of the robot's current posture, target task instructions, and environmental perception data that are important for completing the task through its deep network structure. These features are integrated and encoded into the first target features. It can be understood that the first target features are a set of high-level abstract representations that capture the essence of the task and the dynamic characteristics of the environment, providing a basis for the robot's decision-making and actions.

[0091] Next, the transformer-based large model weights the first target features through its self-attention layer to identify and emphasize the information that is most critical to completing the task. At the same time, the multi-head attention mechanism of the transformer-based large model can simultaneously process the feature relationships of multiple subspaces, thereby more comprehensively understanding all aspects of the task. And through a deep feedforward network, the first target feature is nonlinearly transformed to enhance the expressive power of the first feature, making it more adaptable to the needs of robot control. Finally, the transformer-based large model can extract richer and more useful information from the first target feature to form the second target feature. These second target features not only contain precise guidance for the robot's actions, but also incorporate adaptability to environmental changes and a deep understanding of the task objectives.

[0092] Then, the decoding module analyzes the second target feature to understand the task requirements and environmental conditions represented by the second target feature, and maps these abstract features to specific control parameters, thereby generating multiple interface control signals.

[0093] In this embodiment, a hierarchical processing flow is used to make the processing of complex tasks more efficient, flexible and reliable. In addition, the transformer basic large model can handle complex data relationships, reduce unnecessary calculation steps, thereby reducing the consumption of computing resources, improving the perception, understanding and planning capabilities of the environment, and improving the response speed and execution efficiency of the robot control method.

[0094] In some embodiments, in order to further improve the processing capability of the method for multi-source information, the encoding module and the decoding module are designed to include a variety of dedicated encoders and decoders. Specifically, the encoding module includes a visual encoder, a language encoder, an audio encoder, and a sensor data encoder; the decoding module includes a language decoder and a visual decoder.

[0095] In the encoding stage, the visual encoder is used to process image-based environmental perception data, such as visual information of the surrounding environment obtained by the camera, which makes the deployment of Figure 5 The robot with the method in the embodiment can understand and respond to changes in the visual scene. The language encoder and the audio encoder are used to process text or audio target task instructions so that the robot can receive and understand human natural language commands or audio signals, thereby enhancing the naturalness and flexibility of human-computer interaction. The sensor data encoder is used to process the current posture of the robot to ensure that the robot has an accurate understanding of its position and posture in space.

[0096] In the decoding stage, the language decoder is used to output the task description in the interface control signal, which helps the robot understand the background and goal of the task, making the execution process more consistent with human instructions. The visual decoder is used to output the identification information of the execution target in the interface control signal, so that the robot can accurately identify and locate the execution target in a complex environment.

[0097] The method in this embodiment can process and respond to multi-source information more intelligently, improve the robot's autonomy and accuracy in executing tasks, and enable the robot to more comprehensively understand task instructions and environmental conditions, thereby making more reasonable decisions when executing tasks.

[0098] Figure 6 FIG. 2 is a flow chart of a robot control method provided by another embodiment of the present application. For example, the method is applied to the second control system. Figure 6 As shown, the method includes the following steps.

[0099] Step S610, acquiring environmental perception data and target interface control signals.

[0100] The target interface control signal is one of a plurality of interface control signals generated by the first control system based on the target task instruction and the environmental perception data.

[0101] Exemplarily, environmental perception data is acquired through sensors (e.g., cameras), including the position of the robot, the position and state of surrounding objects, and other factors that may affect task execution.

[0102] Step S620: Generate a control instruction based on the environment perception data and the target interface control signal.

[0103] The purpose of step S620 is to facilitate the robot to perform corresponding actions based on the control instructions. Specifically, the control instructions include robot control values, which include but are not limited to at least one of joint rotation angles, travel distances, travel directions, end effector positions, head postures, and trunk elevation and pitch values.

[0104] Joint rotation angle refers to the specific angle that each joint in a robot's mechanical arm or dexterous hand should rotate. In the case of a mechanical arm, the joint rotation angle determines the position and orientation of the end effector of the mechanical arm. For a dexterous hand, the joint rotation angle determines the degree of opening and closing and the posture of the fingers to achieve grasping, releasing or other operations.

[0105] Travel distance refers to the straight-line distance that the robot chassis or other moving parts need to move. This value is a basic parameter in robot path planning and navigation. By precisely controlling the travel distance, the robot can accurately reach a predetermined target position or move along a specific path.

[0106] The direction of travel refers to the orientation of the robot when it moves. The direction of travel can be an angle value that indicates the rotation angle of the robot relative to its current orientation or a reference direction (such as north), ensuring that the robot can move in the right direction, avoid collisions, and plan paths effectively.

[0107] The position of the end effector refers to the three-dimensional coordinates of the robot's end effector (such as the tool or gripper at the end of the robotic arm) in space, usually including X, Y, and Z coordinates, indicating its specific position relative to a reference point.

[0108] Head posture refers to the direction and tilt of the robot's head relative to its body or base, including yaw, pitch, and roll angles.

[0109] The trunk lift and pitch values ​​refer to the vertical movement and tilt angle of the robot's trunk. The lift value indicates the position of the trunk moving up and down, and the pitch value indicates the angle of the trunk tilting forward and backward.

[0110] In this embodiment, the second control system is responsible for converting the target interface control signal into a specific control instruction to drive the robot to perform the corresponding action. Since the interface control signal is generated independently of the second control system, that is, after the first control system generates an interface control signal, it can start to generate the next interface control signal without having to wait for the second control system to generate the corresponding instruction and execute the corresponding action. In other words, the first control system and the second control system are both dependent and can be executed in parallel. This design significantly improves the response speed and execution efficiency of the robot control and reduces the time delay caused by waiting. In addition, this parallel control method enables the robot to adapt to environmental changes more quickly and adjust the control strategy in time, thereby reducing the risk of task failure caused by the speed of environmental changes exceeding the execution speed of the robot.

[0111] In some embodiments, the second control system includes an encoding module and a converter; based on the environmental perception data and the target interface control signal, a control instruction is generated, including: the encoding module performs a preliminary analysis of the robot's current posture, the target interface control signal and the environmental perception data to obtain a third target feature; the converter optimizes and enhances the third target feature to obtain a fourth target feature, so as to output the robot control value based on the fourth target feature.

[0112] Similarly, the encoding module learns and identifies important feature information about the robot's action execution from the robot's current posture, target interface control signals, and environmental perception data through its deep network structure. These features are integrated and encoded into the third target feature.

[0113] Then, the converter uses its self-attention mechanism to conduct an in-depth analysis of the third target feature and identify important relationships and interactions between features. At the same time, the converter's multi-head attention mechanism can process multiple feature subspaces simultaneously, thereby more comprehensively understanding task requirements and environmental conditions. After optimizing and enhancing the third target feature, the fourth target feature is generated. Finally, based on the fourth target feature, the converter outputs specific robot control values ​​to guide the robot's actuators to ensure that the robot's actions can be performed in a predetermined order.

[0114] In this embodiment, the application of the converter improves the performance of the robot control method in feature extraction, optimization and control instruction generation, making it possible to deploy Figure 6 The robot using the method in this paper can perform tasks more flexibly and accurately while maintaining the ability to adapt to complex environments.

[0115] In some embodiments, to improve Figure 6 The method in the method has the ability to process multi-source information, and the encoding module includes a variety of dedicated encoders. Specifically, the encoding module includes a visual encoder and a language encoder, as well as a sensor data encoder. The visual encoder is used to process image-based environmental perception data, such as visual information of the surrounding environment obtained by a camera, and image information in the interface control signal. The language encoder is used to process text information in the interface control signal. The sensor data encoder is used to process the current posture of the robot to ensure that the robot has an accurate understanding of its position and posture in space. In other embodiments, the second control system also includes a language decoder (not shown in the figure), which is cascaded with the converter to output a number (such as 1-10) of the current action execution progress.

[0116] It should be understood that the description of the robot control method embodiment corresponds to the description of the robot control system embodiment. Therefore, the parts of the robot control method that are not described in detail can refer to the previous system embodiment.

[0117] Below, reference Figure 7 To describe an electronic device according to an embodiment of the present application. Figure 7 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0118] like Figure 7 As shown, the electronic device 70 includes one or more processors 701 and a memory 702 .

[0119] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 70 to perform desired functions.

[0120] The memory 702 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may run the program instructions to implement the robot control method of each embodiment of the present application described above and / or other desired functions. Various contents such as environmental perception data, multiple interface control signals, control instructions, etc. may also be stored in the computer-readable storage medium.

[0121] In one example, the electronic device 70 may further include: an input device 703 and an output device 704, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0122] The input device 703 may include, for example, a keyboard, a mouse, etc.

[0123] The output device 704 can output various information to the outside, including environmental perception data, multiple interface control signals, control instructions, etc. The output device 704 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0124] Of course, to simplify, Figure 7 Only some of the components related to the present application in the electronic device 70 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 70 may also include any other appropriate components.

[0125] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the robot control method according to various embodiments of the present application described above in this specification.

[0126] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0127] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the robot control method according to various embodiments of the present application described above in this specification.

[0128] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0130] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0131] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0132] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0133] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A robot control system, characterized in that: include: First control system; a second control system, cascade-connected to the first control system; The first control system is configured to generate a plurality of interface control signals based on the target task instruction and the environmental perception data, and the execution order of the plurality of interface control signals has an associated relationship; The second control system is configured to obtain a target interface control signal, and generate a control instruction based on the target interface control signal and the environmental perception data, so that the robot performs a corresponding action based on the control instruction.

2. The robot control system according to claim 1, characterized in that: The first control system is a high-order control system, and the second control system is a low-order control system; Preferably, the first control system and the second control system are decoupled from each other.

3. The robot control system according to claim 1, characterized in that: Before the first control system generates a current interface control signal, and / or when the robot has not completed an action corresponding to the control instruction of the current interface control signal and the previous interface control signal, the target interface control signal is the previous interface control signal, and the current interface control signal and the previous interface control signal are one of the multiple interface control signals; After the first control system generates the current interface control signal, and when the robot completes the action corresponding to the control instruction of the previous interface control signal, the target interface control signal is the current interface control signal.

4. The robot control system according to any one of claims 1 to 3, characterized in that: The first control system comprises: An encoding module is configured to perform a preliminary analysis on the current posture of the robot, the target task instruction and the environmental perception data to obtain a first target feature; A transformer basic large model is configured to optimize and enhance the first target feature to obtain a second target feature; The decoding module is configured to process the second target feature and generate the plurality of interface control signals.

5. The robot control system according to claim 4, characterized in that: The encoding module includes a visual encoder, a language encoder, an audio encoder and a sensor data encoder; The visual encoder is used to process the environmental perception data; The speech encoder and the audio encoder are used to process the target task instruction; The sensor data encoder is used to process the current posture of the robot; Preferably, the decoding module includes a language decoder and a visual decoder; The language decoder is used to output the task description in the interface control signal; The visual decoder is used to output identification information of the execution target in the interface control signal.

6. The robot control system according to claim 1, characterized in that: The second control system comprises: an encoding module, configured to preliminarily analyze the current posture of the robot, the target interface control signal and the environmental perception data to obtain a third target feature; The converter is configured to optimize and enhance the third target feature to obtain a fourth target feature so as to output a robot control value based on the fourth target feature, wherein the robot control value includes a set of parameters for guiding the robot to perform a specific action.

7. A robot control method, characterized in that: Applied to a first control system, the method comprises: Acquire environmental perception data; Based on the target task instruction and the environmental perception data, a plurality of interface control signals are generated, and the execution order between the plurality of interface control signals is associated with each other.

8. The robot control method according to claim 8, characterized in that: The first control system is a high-order control system.

9. The robot control method according to claim 7 or 8, characterized in that: The first control system includes an encoding module, a converter basic model and a decoding module; the generating a plurality of interface control signals based on the target task instruction and the environmental perception data includes: The encoding module preliminarily analyzes the current posture of the robot, the target task instruction and the environmental perception data to obtain a first target feature; The converter basic large model optimizes and enhances the first target feature to obtain a second target feature; The decoding module processes the second target feature and generates the plurality of interface control signals.

10. A robot control method, characterized in that: Applied to the second control system, the method comprises: Acquire environmental perception data and a target interface control signal, wherein the target interface control signal is one of a plurality of interface control signals generated by the first control system based on the target task instruction and the environmental perception data; Based on the environmental perception data and the target interface control signal, a control instruction is generated so that the robot performs a corresponding action based on the control instruction.

11. The robot control method according to claim 10, characterized in that: The second control system is a low-order control system, and the first control system and the second control system are decoupled from each other.

12. The robot control method according to claim 10 or 11, characterized in that: The second control system includes an encoding module and a converter; the generating of control instructions based on the environmental perception data and the target interface control signal includes: The encoding module preliminarily analyzes the current posture of the robot, the target interface control signal and the environmental perception data to obtain a third target feature; The converter optimizes and enhances the third target feature to obtain a fourth target feature, so as to output a robot control value based on the fourth target feature, wherein the robot control value includes a set of parameters for guiding the robot to perform a specific action.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the robot control method according to any one of claims 7 to 12.

14. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the robot control method described in any one of claims 7 to 12.

15. A computer program product, characterized in that The computer program product comprises instructions, which, when executed on an electronic device, enable the electronic device to implement the robot control method according to any one of claims 7 to 12.

Images