Robot control system, data processing method, electronic device, and program product

Abstract

The invention provides a robot control system, a data processing method, electronic equipment and a program product, and relates to the field of robot control. According to the robot control system, a first system obtains environment sensing data and task description data of a target robot, after feature fusion, a state vector is generated at a first operation frequency and sent to a second system and a third system; and the second system receives the state vector and the environment perception data and generates description data of the target object at a second operation frequency higher than the first operation frequency. And the third system receives the state vector, the environment sensing data and the state data of the target robot, generates a control instruction of the target robot at a third operation frequency higher than the first operation frequency, and drives the target robot to execute corresponding actions. According to the method, the real-time perception of the target robot on the target object in a complex task is remarkably enhanced, and a more efficient and accurate solution is provided for a task with a body operation.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robot control system, a data processing method, an electronic device, and a program product. Background Art

[0002] Current robot control systems struggle to balance environmental understanding, target perception, and the real-time and accurate generation of control commands when processing data. Furthermore, existing system architectures experience processing delays when processing data that requires frequent updates, which in turn impacts the robot's response speed and action execution. Furthermore, the system architecture lacks modularity, hindering functional expansion and system optimization, making it difficult to meet the complex needs of robots in diverse application scenarios. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a robot control system, a data processing method, an electronic device, and a program product.

[0004] In the first aspect, an embodiment of the present application provides a robot control system, which is applied to a target robot, and the robot control system includes: a first system; a second system and a third system, which are respectively connected to the first system; the first system is configured to obtain environmental perception data of the target robot and task description data corresponding to the target robot, and perform feature fusion based on the environmental perception data and the task description data, generate a state vector at a first operating frequency, and send the state vector to the second system and the third system; the second system is configured to receive the state vector and obtain environmental perception data; based on the state vector and the environmental perception data, generate description data for the target object at a second operating frequency, the description data of the target object is used for environmental understanding and task decision-making, and the second operating frequency is higher than the first operating frequency; the third system is configured to receive the state vector and obtain environmental perception data and the target robot's own state data; based on the environmental perception data, the state vector and the target robot's own state data, generate control instructions for the target robot at a third operating frequency, the control instructions are used to drive the target robot to perform corresponding actions, and the third operating frequency is higher than the first operating frequency.

[0005] In combination with the first aspect, in certain implementations of the first aspect, the second system is further configured to: generate a query vector of the current time series based on the environmental perception data of the current time series and the feature vector of the previous time series, the query vector of the current time series representing a feature description of the target object, the target object including a dynamic target object; fuse the query vector and state vector of the current time series to generate a feature vector of the current time series; and generate description data for the target object corresponding to the current time series based on the feature vector of the current time series.

[0006] In combination with the first aspect, in certain implementations of the first aspect, the environmental perception data includes environmental observation images and / or environmental observation point clouds; the task description data includes image description data and / or point cloud description data; in the first system, the image encoder is configured to process environmental observation images and image description data; the point cloud encoder is configured to process environmental observation point clouds and point cloud description data; in the second system and the third system, the image encoders are both configured to process environmental observation images; the point cloud encoders are both configured to process environmental observation point clouds; the image encoder and point cloud encoder of the first system are shared with the image encoder and point cloud encoder of the second system; and / or the image encoder and point cloud encoder of the first system are shared with the image encoder and point cloud decoder of the third system.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the second system is further configured to send the description data of the target object to the third system; the third system is further configured to receive the description data of the target object; and generate control instructions for the target robot at a third operating frequency based on the description data of the target object, environmental perception data, state vector, and the target robot's own state data.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the first system is further configured to determine a target prompt word template adapted to the multimodal task description data based on the modal type of the multimodal task description data, where the target prompt word template is a structured instruction format that matches the modal type of the multimodal task description data; and based on the target prompt word template, generate a state vector that matches the modal type of the multimodal task description data.

[0009] In combination with the first aspect, in some implementations of the first aspect, the robot control system is further configured to: when there is no need to generate description data for the target object, close the second system's processing link for the state vector and environmental perception data.

[0010] In a second aspect, an embodiment of the present application provides a data processing method applied to a second system of a target robot, the target robot also including a first system and a third system. The method comprises: receiving a state vector sent by the first system, and acquiring environmental perception data, wherein the state vector is generated based on the environmental perception data and task description data; generating description data for a target object based on the state vector and the environmental perception data, the description data of the target object being used for environmental understanding and task decision-making; and sending the description data of the target object to the third system, so that the third system uses the description data of the target object to generate control instructions for the target robot, and the control instructions are used to drive the target robot to perform corresponding actions.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for executing the data processing method described in the second aspect.

[0012] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; and the processor is configured to execute the data processing method described in the second aspect.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on an electronic device, the electronic device implements the data processing method described in the second aspect.

[0014] In this application, the first system performs feature fusion on environmental perception data and task description data and generates a state vector. The state vector generated by the first system then serves as input to the second and third systems, avoiding the information loss caused by complex structured expressions. Furthermore, as an efficient intermediate representation, the state vector can be used as an intermediate result for observation and analysis, thereby improving the interpretability of the robot control system. It can also quickly transmit information, allowing the second system to quickly, in real time, and accurately locate and identify the target object. Furthermore, the third system deeply fuses its own state data with the environmental perception data and state vector to form a comprehensive state representation. This fusion process enables the robot control system to generate more accurate and adaptable control instructions, enhancing the stability and safety of the target robot in complex environments and enabling the target robot to more intelligently and flexibly respond to various task challenges. Furthermore, through the modular design of the three systems, this application allows the functions of the first, second, and third systems to be flexibly expanded. At the same time, through the collaborative work of the three systems, a more efficient and accurate solution is provided for embodied manipulation tasks. Finally, the three systems operate at different frequencies. This design better balances long-running tasks such as task reasoning and tasks with higher real-time requirements such as target object description data and control instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 Shown is a schematic structural diagram of a robot control system provided in one embodiment of the present application.

[0017] Figure 2 Shown is a schematic structural diagram of a robot control system provided in another embodiment of the present application.

[0018] Figure 3 FIG2 is a flowchart showing a process of performing dynamic tracking by the second system provided in one embodiment of the present application.

[0019] Figure 4 The figure shows a processing flow chart of the second system provided in one embodiment of the present application when performing static recognition.

[0020] Figure 5 The figure is a flow chart of a data processing method provided in one embodiment of the present application.

[0021] Figure 6 Shown is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] Figure 1 The figure shows a schematic diagram of the structure of a robot control system provided by an embodiment of the present application. Figure 1 As shown, the robot control system 10 includes a first system 110 , a second system 120 and a third system 130 , wherein the second system 120 and the third system 130 are respectively connected to the first system 110 .

[0024] Specifically, the first system 110 is configured to obtain environmental perception data of the target robot and the corresponding task description data of the target robot, and perform feature fusion based on the environmental perception data and the task description data, generate a state vector at a first operating frequency, and send the state vector to the second system and the third system. The second system 120 is configured to receive the state vector and obtain environmental perception data; based on the state vector and the environmental perception data, generate description data for the target object at a second operating frequency, the description data of the target object is used for environmental understanding and task decision-making, and the second operating frequency is higher than the first operating frequency. The third system 130 is configured to receive the state vector and obtain environmental perception data and the target robot's own state data; based on the environmental perception data, the state vector and the target robot's own state data, generate control instructions for the target robot at a third operating frequency, the control instructions are used to drive the target robot to perform corresponding actions, and the third operating frequency is higher than the first operating frequency.

[0025] In some embodiments, environmental perception data refers to information about the surrounding environment obtained by the target robot through sensors (such as cameras, lidar, etc.). Figure 2 FIG. 1 is a schematic diagram of a robot control system according to another embodiment of the present invention. Figure 2 As shown, environmental perception data includes environmental observation images and / or environmental observation point clouds. Environmental observation images are two-dimensional image data of the surrounding environment acquired by a camera or other imaging device; environmental observation point clouds are three-dimensional data of the surrounding environment acquired by a lidar, binocular camera, or other three-dimensional scanning device.

[0026] Task description data refers to data that describes the target task of the target robot. Optionally, the task description data includes multimodal task description data, such as Figure 2 As shown, the multimodal task description data includes text description data, image description data, and / or point cloud description data. Specifically, text description data refers to the description of the task in natural language (such as Chinese, English, etc.); image description data describes the task through images, which can be photos of the target object related to the target task, screenshots of the scene, or schematic diagrams; point cloud description data describes the task through 3D point cloud data, which can be a 3D model of the target object or 3D scan data of the scene where the target task is located.

[0027] The state vector is an implicit state representation used to describe the target robot's task requirements and current environmental characteristics. The descriptive data of the target object is used for environmental understanding and task decision-making. Specifically, it helps the target robot better understand its environment and provides a basis for decision-making when performing subsequent tasks. For example, this descriptive data includes the target object's pose, category, bounding box, and grasping pose. Specifically, the pose of the target object refers to the target object's position and pose in space; the category refers to the type or type of the target object, helping the target robot identify and distinguish different objects and adopt different operation strategies based on different types, such as grasping or carrying methods; the bounding box refers to the boundary used to enclose the target object, allowing for quick determination of the target object's position and general shape. For example, the bounding box is a rectangular box (in a two-dimensional image) or a cuboid box (in three-dimensional space); and the grasping pose refers to the position and pose that the robot's robotic arm or gripper needs to achieve when grasping the target object, typically including the location of the grasping point and the orientation of the gripper.

[0028] In some embodiments, the first system 110 first obtains the environmental perception data of the target robot in order to understand the environmental conditions around the target robot. At the same time, the first system 110 also obtains the task description data corresponding to the target robot in order to clarify the target task that the target robot needs to complete. Then, the first system 110 extracts and aligns the features of these two types of data, and fuses the aligned features through the attention mechanism, and finally outputs the state vector. Optionally, the first system 110 adopts the Transformer architecture, and the text task description data in the task description data is vectorized using Tokenizer. Then, the second system 120 uses the state vector generated by the first system 110 and the environmental perception data collected by the sensor, and after self-attention and cross-attention, decodes and outputs the description data for the target object. Optionally, the second system 120 adopts the structure of Transformer and Diffusion.

[0029] Furthermore, the target robot's own state data includes the target robot's position, posture, speed, battery level, etc. In some embodiments, the third system 130 performs feature encoding on its own state data, and then aligns the encoded features with the features corresponding to the environmental perception data. Next, the aligned features and state vectors are cross-attention fused, and finally, control instructions are generated by decoding and / or denoising. Optionally, these control instructions are used to guide the target robot to perform corresponding actions, so that it can complete the task efficiently and safely. Optionally, the third system 130 uses a structure that combines Transformer and Diffusion.

[0030] For example, a target robot needs to locate and transport a piece of item A in a warehouse. First, the target robot uses cameras and / or lidar to acquire environmental perception data within the warehouse, such as shelf locations and item distribution, and receives a textual task description: "Carry item A numbered 123 to the exit." The first system 110 then fuses the environmental perception data and the textual task description, extracting key information and generating a state vector. Simultaneously, the second system 120 utilizes the state vector generated by the first system 110 and the acquired environmental perception data to generate descriptive data specific to item A. For example, a more accurate 3D model of item A is generated, including its position, size, and posture, as well as the grasping posture for item A. This allows the target robot to more accurately grasp item A and complete the transport task based on this descriptive data. Furthermore, the third system 130 uses internal sensors to acquire the target robot's own state data, as well as environmental perception data such as images and distance information within the warehouse, and a state vector containing the characteristics and location of item C. The third system 130 then fuses this data to generate control instructions. Finally, according to the control instructions, the target robot is guided to move 0.5 meters to the right and grab cargo C. This process ensures that the target robot can complete the task efficiently based on its own status and environmental information.

[0031] In some embodiments, the first system 110, the second system 120, and the third system 130 do not rely on a synchronized clock signal for updates, allowing each system to operate independently at its own frequency. This improves the interpretability of the end-to-end robotic control system model. Furthermore, independent operation and information updates ensure that the second system 120 and the third system 130 receive complete information (i.e., state vectors).

[0032] Specifically, the first system 110 needs to perform in-depth processing on complex environmental perception data and task description data to generate a stable and accurate state vector. Due to the complexity of its tasks and the large amount of calculation, the first operating frequency of the first system 110 is a lower value (such as less than 10Hz) that is lower than the second operating frequency. On the other hand, the generation of the state vector does not require extremely high real-time performance, but focuses more on accuracy and stability. Therefore, a lower operating frequency can also reduce the consumption of computing resources and avoid overloading the first system 110. Optionally, in order to ensure the accuracy of reasoning, the parameter amount of the first system 110 is set to be greater than 0.5B, which enables it to process complex multimodal data and generate high-quality state vectors.

[0033] Second system 120 is responsible for generating description data for the target object. Therefore, the operating frequency of second system 120 is set to a higher value (e.g., greater than 10 Hz) than the first operating frequency to ensure rapid response to environmental changes, thereby enabling second system 120 to promptly capture changes in the target object in a dynamic environment. Optionally, to ensure efficiency and real-time performance, the parameter size of second system 120 is recommended to be less than 0.5B. This allows it to maintain a high operating frequency without consuming excessive computing resources.

[0034] Third system 130 is responsible for generating control commands for the target robot, ensuring that the target robot can make timely decisions and take actions based on the latest perception information and its own state data. Therefore, the operating frequency of the third system is set to be higher than that of the first system. Optionally, to ensure real-time performance and efficiency, the parameter size of third system 130 is also recommended to be less than 0.5B. This allows it to maintain efficient operation without consuming excessive computing resources.

[0035] In this embodiment, the first system performs feature fusion on the environmental perception data and task description data and generates a state vector. The state vector generated by the first system then serves as input to the second and third systems, avoiding the information loss caused by complex structured expressions. Moreover, as an efficient intermediate representation, the state vector can be observed and analyzed as an intermediate result, thereby improving the interpretability of the robot control system; on the other hand, it can quickly transmit information, allowing the second system to quickly, in real time, and accurately locate and identify the target object. Furthermore, the third system deeply fuses its own state data with the environmental perception data and state vector to form a comprehensive state representation. This fusion process enables the robot control system to generate more accurate and adaptable control instructions, enhances the stability and safety of the target robot in complex environments, and enables the target robot to respond to various task challenges more intelligently and flexibly. In addition, through the modular design of the three systems, the functions of the first, second, and third systems can be flexibly expanded. At the same time, through the collaborative work of the three systems, a more efficient and accurate solution is provided for embodied operation tasks. Finally, the three systems operate at different frequencies. This design better balances long-running tasks such as task reasoning and tasks with higher real-time requirements such as target object description data and control instructions.

[0036] In some embodiments, the robot control system is further configured to: when there is no need to generate description data for the target object, close the processing link of the second system 120 for the state vector and the environment perception data.

[0037] In other words, the robot control system has an optimization scheme: when no descriptive data for the target object is required, the processing link for the state vector and environmental perception data by the second system 120 can be shut down. This means that in certain specific situations, when task execution does not rely on a detailed description of the target object, the robot control system can flexibly adjust resource allocation to avoid unnecessary computation and resource consumption. This scheme allows the robot control system to dynamically adjust the operating state of the second system according to task requirements, shutting down related processing flows when additional environmental understanding and task decision-making information is not required, saving computing resources and energy consumption, and improving the overall operational efficiency of the robot control system.

[0038] Continue to refer to Figure 1 and Figure 2 As shown, in some embodiments, information is also exchanged between the second system 120 and the third system 130. Specifically, the second system 120 sends the generated description data of the target object to the third system 130. Then, the third system 130 generates control instructions for the target robot at the third operating frequency based on the description data of the target object, the environmental perception data, the state vector, and the target robot's own state data.

[0039] It is understood that the target object's description data provides third system 130 with detailed information about the target object, enabling it to more comprehensively understand the environment and avoid biased decision-making. Furthermore, the target object's description data helps third system 130 understand the dynamic changes of the target object in real time and adjust control instructions promptly, enabling the target robot to better adapt to environmental changes, enhancing its adaptability and autonomy in complex environments, and ultimately improving the performance and reliability of the entire robot control system.

[0040] In some embodiments, to improve the adaptability and accuracy of first system 110 when processing task description data, a prompt word template mechanism is introduced into first system 110, thereby ensuring more accurate and efficient generation of state vectors. Specifically, first system 110 is further configured to determine a target prompt word template adapted to the task description data based on the modality type of the task description data; and based on the target prompt word template, generate a state vector that matches the modality type of the task description data.

[0041] The target prompt word template represents a structured instruction format that matches the modality type of the task description data, and is used to guide the first system 110 on how to process task description data of a specific modality, and to help the first system 110 understand the specific requirements of the target task, thereby generating an accurate state vector.

[0042] Optionally, the target prompt word template is such as "given the observation images <img1>depth orpoints <pts1>,pick the object <object description>”;"given the observationimages <img1>,depth or points <pts1>,pick the object as <object img>;given theobservation images <img1>,depth or points <pts1>,pick the object as <object pts>". The first target prompt word template indicates that, in the environmental observation image and the environmental observation point cloud received by the first system 110, a target object that conforms to the text description data is identified. The second target prompt word template indicates that, in the environmental observation image and the environmental observation point cloud received by the first system 110, a target object that conforms to the image description data is identified. The third target prompt word template indicates that, in the environmental observation image and the environmental observation point cloud received by the first system 110, a target object that conforms to the point cloud description data is identified.

[0043] This flexible prompt template mechanism clarifies the specific task that the first system needs to perform, such as identifying a specific target object. It also guides the first system in which modality of task description data to apply to complete the task, thereby generating more accurate state vectors and improving the versatility and applicability of the robot control system.

[0044] Based on the aforementioned embodiments, to improve the efficiency and performance of the robot control system, an encoder sharing mechanism is also considered in the design of the robot control system. Specifically, the environmental perception data includes environmental observation images and / or environmental observation point clouds; the task description data includes image description data and / or point cloud description data. In the first system 110, the image encoder is configured to process environmental observation images and image description data; the point cloud encoder is configured to process environmental observation point clouds and point cloud description data; in the second system 120 and the third system 130, the image encoder is configured to process environmental observation images; and the point cloud encoder is configured to process environmental observation point clouds.

[0045] The image encoder and point cloud encoder of the first system 110 are shared with the image encoder and point cloud encoder of the second system 120 ; and / or, the image encoder and point cloud encoder of the first system 110 are shared with the image encoder and point cloud decoder of the third system 130 .

[0046] Specifically, in the architecture of the robot control system, the first system 110, the second system 120, and the third system 130 are equipped with image encoders and point cloud encoders, respectively used to process their respective tasks. In addition, these encoders are shared between different systems, that is, the image encoder and point cloud encoder of the first system 110 are shared with the corresponding encoders of the second system 120 and the third system 130. This sharing means that when processing the same type of environmental perception data, each system can use a unified encoding model and parameters to ensure data consistency and accuracy. Through this sharing model, the robot control system can reduce repeated calculations, improve resource utilization efficiency, while maintaining consistency in data processing between systems, ultimately improving the performance and reliability of the entire robot control system in environmental perception and task execution.

[0047] Based on the aforementioned embodiments, the second system 120 is not only capable of generating descriptive data for the target object, but also possesses the capabilities of dynamic tracking and static recognition. Specifically, in some embodiments, the second system 120 is further configured to: generate a query vector for the current time series based on the environmental perception data of the current time series and the feature vector of the previous time series, wherein the query vector of the current time series represents a feature description of the target object, including dynamic target objects; fuse the query vector of the current time series with the state vector to generate a feature vector for the current time series; and generate descriptive data for the target object corresponding to the current time series based on the feature vector of the current time series.

[0048] The current time series refers to the latest time point currently being processed by the second system 120, while the previous time series refers to a time point before the current time series. This division of time series allows the second system 120 to utilize previously processed information (i.e., the feature vector of the previous time series) to assist in current processing (i.e., generating a query vector for the current time series), thereby enabling tracking of dynamic objects and identification of static objects.

[0049] Figure 3 The figure shows a processing flow chart of the second system for dynamic tracking provided by an embodiment of the present application. Figure 3 As shown, the second system 120 includes a query organization unit, a feature fusion unit and an information decoding unit (the image encoder and the point cloud encoder are shared with the first system 110, Figure 3 (not shown in the figure). Specifically, the second system 120 first processes the environmental observation image and environmental observation point cloud using a shared encoder and decoder to obtain image features and point cloud features. Then, a multi-layer perceptron-like architecture module is used to implement spatial mapping of the features, adapting the image features and point cloud features to the structure of the second system 120, resulting in aligned image features and point cloud features. Next, the query organization unit processes the aligned image features and point cloud features, as well as the feature vector of the previous time series, to obtain a query vector for the current time series. The feature fusion module then fuses the query vector of the current time series with the state vector generated by the first system 110 to obtain a feature vector for the current time series. The feature vector is then input into the information decoding unit, which processes the feature vector of the current time series and outputs descriptive data for the target object. The feature vector of the current time series is then input into the query organization unit, which processes the feature vector and the aligned image features and point cloud features generated based on the updated environmental perception data to obtain a query vector for the next time series. Finally, the information decoding unit generates descriptive data for the target object. Through this iterative update mechanism, the second system 120 can dynamically adapt to changes in the environment, continuously optimize the perception of the target object, and achieve dynamic tracking.

[0050] It should be noted that when generating the feature vector for the first time, the query organization unit obtains the query vector for the current time series based on the aligned image features and point cloud features, as well as the initial query vector. The initial query vector is a pre-existing, randomly initialized vector that represents the initial feature description of the target object.

[0051] For example, in a warehouse transport task, the target robot's task is to identify and transport a specific cargo pallet. In this scenario, there are multiple pallets in the warehouse, and the positions of pallets may change over time (for example, due to being moved by other robots). Therefore, the second system 120 is tasked with sensing the position of the target pallet in real time and dynamically tracking its changes.

[0052] Based on this, the second system 120 first generates a current query vector at time point T1 based on the environmental perception data (such as the activity information of other robots or equipment in the warehouse) and the initial query vector. Next, the second system 120 fuses the current query vector at time point T1 with the state vector to generate a feature vector at time point T1. Based on the feature vector at time point T1, the second system 120 generates description data for the target pallet. For example, the description data includes the shape, color and position of the target pallet, as well as the grasping posture of the target pallet. Subsequently, the second system 120 generates a current query vector at time point T2 based on the feature vector at time point T1 and the feature vector corresponding to the updated environmental perception data (for example, the latest position of the target pallet after being moved by other robots), ensuring that the second system 120 can adapt to changes in the environment and providing a basis for subsequent iterative updates. Finally, based on the current query vector at time point T2, the second system 120 iteratively updates the feature vector corresponding to time point T2 and the description data of the target object. For example, if the target pallet is moved to a new location at time point T2, the second system 120 will update its description data, reposition the target pallet, and adjust the movement path of the target robot to complete the transport task.

[0053] In this embodiment, the second system achieves end-to-end, real-time, accurate, and dynamic perception of the target object by combining environmental perception data, query vectors, and state vectors. This dynamic adjustment and iterative update mechanism also improves the overall performance and adaptability of the robot control system, making it particularly suitable for scenarios requiring real-time perception and dynamic tracking.

[0054] In other embodiments, the second system 120 is further configured to generate a target query vector based on the environmental perception data and the initial query vector; fuse the target query vector with the state vector to generate a target feature vector; and generate description data for the target object based on the target feature vector.

[0055] Figure 4 The figure shows a processing flow chart of the second system for static recognition provided by an embodiment of the present application. Figure 3 Similarly, the second system 120 first processes the environmental observation image and the environmental observation point cloud through a shared encoder and decoder to obtain image features and point cloud features. Then, similar multi-layer perceptron architecture modules are used to implement spatial mapping of features, so that the image features and point cloud features adapt to the structure of the second system 120, and aligned image features and point cloud features are obtained. Next, the query organization unit processes the aligned image features and point cloud features, as well as the initial query vector, to obtain the target query vector. Then, the feature fusion module fuses the target query vector with the state vector generated by the first system 110 to obtain the target feature vector. Finally, the information decoding unit processes the target feature vector and outputs description data for the target object. Figure 3 The difference is that the generated feature vector does not need to be input into the query organization unit for the next time series processing.

[0056] In this embodiment, the second system mainly performs static identification of the target object. Specifically, by combining environmental perception data and the initial query vector, it more accurately generates description data for the target object, improves the accuracy of target identification and the efficiency of task completion, enriches the performance of the robot control system, and expands the application scenarios of the robot control system.

[0057] Figure 5 FIG. 1 is a flow chart of a data processing method provided by an embodiment of the present application. For example, the method is applied to the second system of a target robot, which also includes a first system and a third system. Figure 5 As shown, the method includes the following steps.

[0058] Step S510: receiving a state vector sent by the first system and acquiring environmental perception data.

[0059] The state vector is generated based on environmental perception data and task description data. It is an implicit state representation used to describe the task requirements of the target robot and the current environment characteristics.

[0060] Environmental perception data refers to information about the target robot's surrounding environment acquired through sensors (such as cameras, lidar, etc.). Optionally, environmental perception data includes environmental observation images and / or environmental observation point clouds. Environmental observation images are two-dimensional image data of the surrounding environment acquired by a camera or other imaging device; environmental observation point clouds are three-dimensional data of the surrounding environment acquired by lidar, binocular cameras, or other three-dimensional scanning devices.

[0061] Step S520: Generate description data for the target object based on the state vector and the environment perception data.

[0062] The descriptive data of the target object is used for environmental understanding and task decision-making. That is, on the one hand, it helps the target robot better understand its environment; on the other hand, it provides a basis for the target robot to make decisions when performing the next task. For example, the descriptive data includes the target object's posture, category, envelope, and grasping posture. Specifically, the posture of the target object refers to the position and posture of the target object in space; the category refers to the type or type of the target object, so as to help the target robot identify and distinguish different objects, and then adopt different operation strategies according to different types, such as grasping method or carrying method; the envelope refers to the boundary used to surround the target object, so as to quickly determine the position and approximate shape of the target object. For example, the envelope is a rectangular box (in a two-dimensional image) or a cuboid box (in a three-dimensional space); the grasping posture refers to the position and posture that the robotic arm or gripper needs to reach when the target robot grasps the target object, usually including the position of the grasping point and the orientation of the gripper.

[0063] Optionally, the state vector and the environmental perception data collected by the sensor are used to decode and output description data for the target object after self-attention and cross-attention.

[0064] Step S530: Send the description data of the target object to the third system.

[0065] The purpose of step S530 is to facilitate the third system to generate control instructions for the target robot using the description data of the target object, and the control instructions are used to drive the target robot to perform corresponding actions.

[0066] In this embodiment, the state vector serves as the input of the second system, avoiding the information loss caused by complex structured expressions. Moreover, as an efficient intermediate representation, the state vector can be observed and analyzed as an intermediate result, thereby improving the interpretability of the robot control system; on the other hand, it can quickly transmit information, allowing the second system to quickly, in real time, and accurately locate and identify the target object. In addition, the second system sends the description data of the target object to the third system, which helps the third system to more accurately judge the state of the target object, thereby more accurately generating control instructions and reducing decision-making errors. In addition, it also helps the third system to grasp the dynamic changes of the target object in real time and adjust the control instructions in time, so that the target robot can better adapt to environmental changes, enhance its adaptability and autonomy in complex environments, and thus improve the performance and reliability of the entire robot control system.

[0067] Combined with the above Figure 5 , describes in detail the data processing method embodiment of the present application. It should be understood that the description of the data processing method embodiment corresponds to the description of the robot control system embodiment. Therefore, for parts not described in detail, reference can be made to the previous method embodiment.

[0068] Below, reference Figure 6 To describe the electronic device according to the embodiment of the present application. Figure 6 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0069] like Figure 6 As shown, the electronic device 60 includes one or more processors 601 and a memory 602 .

[0070] The processor 601 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 60 to perform desired functions.

[0071] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the data processing methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as environmental perception data, task description data, state vectors, description data of target objects, etc. may also be stored in the computer-readable storage medium.

[0072] In one example, the electronic device 60 may further include an input device 603 and an output device 604 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0073] The input device 603 may include, for example, a keyboard, a mouse, and the like.

[0074] The output device 604 can output various information to the outside, including environmental perception data, multimodal task description data, state vectors, target object description data, etc. The output device 604 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.

[0075] Of course, to simplify, Figure 6 Only some of the components related to the present application in the electronic device 60 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 60 may further include any other appropriate components according to specific application scenarios.

[0076] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the data processing method according to various embodiments of the present application described above in this specification.

[0077] The computer program product may be written in any combination of one or more programming languages to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0078] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the data processing method according to various embodiments of the present application described above in this specification.

[0079] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0080] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0081] The block diagrams of the devices, devices, equipment, and systems involved in this application are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including but not limited to," and can be used interchangeably therewith. The words "or," "and," and "as used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as," as used herein, refers to the phrase "such as, but not limited to," and can be used interchangeably therewith.

[0082] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0083] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0084] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.< / object> < / object> < / object>

Claims

1. A robot control system, characterized in that: Applied to a target robot, the robot control system includes: First System; The second system and the third system are respectively connected to the first system; The first system is configured to obtain environmental perception data of the target robot and task description data corresponding to the target robot, perform feature fusion based on the environmental perception data and the task description data, generate a state vector at a first operating frequency, and send the state vector to the second system and the third system; The second system is configured to receive the state vector and obtain the environmental perception data; generate description data for the target object based on the state vector and the environmental perception data at a second operating frequency, wherein the description data of the target object is used for environmental understanding and task decision-making, and the second operating frequency is higher than the first operating frequency; The third system is configured to receive the state vector and obtain the environmental perception data and the target robot's own state data; based on the environmental perception data, the state vector and the target robot's own state data, generate control instructions for the target robot at a third operating frequency, and the control instructions are used to drive the target robot to perform corresponding actions, and the third operating frequency is higher than the first operating frequency.

2. The robot control system according to claim 1, characterized in that: The second system is further configured to: Generate a query vector of the current time series based on the environmental perception data of the current time series and the feature vector of the previous time series, wherein the query vector of the current time series represents a feature description of the target object, wherein the target object includes a dynamic target object; Fusing the query vector of the current time series with the state vector to generate a feature vector of the current time series; Based on the feature vector of the current time series, description data for the target object corresponding to the current time series is generated.

3. The robot control system according to claim 1, characterized in that: The environmental perception data includes environmental observation images and / or environmental observation point clouds; The task description data includes image description data and / or point cloud description data; In the first system, the image encoder is configured to process the environment observation image and the image description data; the point cloud encoder is configured to process the environment observation point cloud and the point cloud description data; In the second system and the third system, the image encoder is configured to process the environment observation image; the point cloud encoder is configured to process the environment observation point cloud; The image encoder and point cloud encoder of the first system are shared with the image encoder and point cloud encoder of the second system; and / or, the image encoder and point cloud encoder of the first system are shared with the image encoder and point cloud decoder of the third system.

4. The robot control system according to claim 1, characterized in that: The second system is further configured to send the description data of the target object to the third system; The third system is further configured to receive description data of the target object; and generate control instructions for the target robot at a third operating frequency based on the description data of the target object, the environmental perception data, the state vector, and the target robot's own state data.

5. The robot control system according to claim 1, characterized in that: The first system is further configured to: Determining a target prompt word template adapted to the task description data based on the modality type of the task description data, wherein the target prompt word template is a structured instruction format that matches the modality type of the task description data; Based on the target prompt word template, a state vector matching the modality type of the task description data is generated.

6. The robot control system according to claim 1, characterized in that: The robot control system is further configured as: When there is no need to generate description data for the target object, the processing link of the second system for the state vector and the environment perception data is closed.

7. A data processing method, characterized in that: The second system is applied to a target robot, the target robot also including the first system and the third system, and the method includes: receiving a state vector sent by the first system, and acquiring environmental perception data, wherein the state vector is generated based on the environmental perception data and task description data; generating description data for the target object based on the state vector and the environmental perception data, wherein the description data of the target object is used for environmental understanding and task decision-making; The description data of the target object is sent to the third system, so that the third system generates a control instruction for the target robot using the description data of the target object, and the control instruction is used to drive the target robot to perform a corresponding action.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the data processing method according to claim 7.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the data processing method described in claim 7.

10. A computer program product, characterized in that The computer program product includes instructions, which, when executed on an electronic device, enable the electronic device to implement the data processing method according to claim 7.

Images