Robot control method and related equipment

Abstract

The embodiment of the invention provides a robot control method and related equipment. The robot control method specifically comprises the steps that target action information used for controlling robot movement is generated based on a robot control task; performing security verification on the target action information based on the motion security constraint information of the robot; wherein the motion safety constraint information is used for constraining the motion range of the robot in the space from at least one spatial dimension; under the condition that the security verification is not passed, optimizing the target action information until the optimized target action information passes the security verification; and driving the robot to move by using the target action information passing the safety verification. According to the invention, unsafe robot behaviors can be reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of robot technology, and in particular, to a robot control method and related device. BACKGROUND

[0002] In the field of robot control, a model containing a motion generation strategy can be used to control a robot to move in a physical space to complete a real task. However, the real environment in which the motion robot for model generation is located has higher complexity and uncertainty compared to the training environment, and the robot motion generated by the model is prone to unsafe behavior, such as runaway behavior, due to visual errors and the like. The related technology generally sets an intervention mechanism to shut down the robot in an emergency after an unsafe behavior occurs. However, in this way, the unsafe behavior still affects the environment. SUMMARY

[0003] Therefore, embodiments of the present application are dedicated to providing a robot control method and related device, which can reduce unsafe robot behavior.

[0004] One embodiment of the present application provides a robot control method, the method comprising: generating target motion information for controlling robot motion based on a robot control task; performing safety verification on the target motion information based on motion safety constraint information of the robot; wherein the motion safety constraint information is used to constrain the motion range of the robot in space from at least one spatial dimension; in the case where the safety verification fails, optimizing the target motion information until the optimized target motion information passes the safety verification; and driving the robot motion using the target motion information that passes the safety verification.

[0005] Optionally, the step of performing safety verification on the target motion information based on the motion safety constraint information of the robot comprises: loading the motion safety constraint information corresponding to the robot control task from a constraint information library; wherein the robot control task is generated based on a robot visual information and a language control information calling a VLA model, and the constraint information library comprises a plurality of motion safety constraint information adapted to different task types; and performing safety verification on the target motion information based on the motion safety constraint information.

[0006] Optionally, the method further comprises: controlling the robot to stop moving based on any one of an obstacle detection signal, robot motion compliance, and a task progress of the robot control task during the motion of the robot.

[0007] Optionally, the monitoring frequency of the obstacle detection signal is higher than the monitoring frequency of the robot action compliance; and / or, the monitoring frequency of the robot action compliance is higher than the monitoring frequency of the robot control task progress.

[0008] Optionally, the step of controlling the robot to stop movement based on the obstacle detection signal comprises: in a case where it is detected that the obstacle detection signal indicates to avoid obstacles, sending a signal to a safety control circuit in the robot to make the safety control circuit cut off a driving power supply of the robot to make the robot stop movement.

[0009] Optionally, the step of controlling the robot to stop movement based on the robot action compliance comprises: in a case where it is detected that the robot action does not have compliance, sending an action interruption signal to a movement controller in the robot to make the movement controller control the robot to stop movement.

[0010] Optionally, the step of controlling the robot to stop movement based on the robot control task progress comprises: in a case where the robot control task progress meets a preset task stagnation condition and / or a preset task timeout condition, sending a task termination instruction to a movement controller in the robot to make the movement controller control the robot to stop movement and report an alarm information.

[0011] Optionally, the method further comprises: in a case where the robot control task progress indicates a task exception, triggering detection of the obstacle detection signal and / or the robot action compliance according to a type of the task exception.

[0012] Optionally, the movement safety constraint information comprises at least one of: spatial obstacle coordinate information and an upper limit value of joint torque of the robot.

[0013] Optionally, the task type comprises at least one of: a grasping task, a carrying task, an assembly task and a human-robot collaboration task; wherein, the movement safety constraint information corresponding to the grasping task is at least used to limit a force range of an end effector of the robot, the movement safety constraint information corresponding to the carrying task is at least used to limit inertia and acceleration in path planning, the movement safety constraint information corresponding to the assembly task is at least used to limit position error and torque of the end effector, and the movement safety constraint information corresponding to the human-robot collaboration task is at least used to limit joint speed of the robot.

[0014] One embodiment of the present application further provides a robot control device, comprising: an action generation module, configured to generate target action information for controlling robot movement based on a robot control task; a safety verification module, configured to perform safety verification on the target action information based on movement safety constraint information of the robot; wherein the movement safety constraint information is used to constrain movement range of the robot in space from at least one spatial dimension; an action optimization module, configured to optimize the target action information until the optimized target action information passes the safety verification in case that the safety verification fails; and a driving module, configured to drive the robot to move by using the target action information that passes the safety verification.

[0015] One embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the method as described above.

[0016] One embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores at least one computer program, and the at least one computer program is executed by a processor to implement the method as described above.

[0017] One embodiment of the present application further provides a computer program product, which is used to implement the method as described above.

[0018] In the embodiments provided in the present application, the generated action is verified and optimized based on the movement safety constraint information, which realizes dynamic correction of the action, ensures safety of the generated action, guarantees safety of the robot when performing the action, and reduces the risk caused by the action performed by the robot. Compared with the post-intervention solution in the related art, the embodiments of the present application can ensure continuity of the task execution and reduce the situation of task termination caused by unexpected action. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A system architecture diagram to which the robot control method provided in an embodiment of the present application is applied.

[0020] Figure 2 A flowchart of the robot control method provided in an embodiment of the present application.

[0021] Figure 3 A structure diagram of the robot control device provided in an embodiment of the present application.

[0022] Figure 4 A schematic diagram of the computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The to-be-retrieved information in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application.

[0024] In the description of the embodiments of the present application, it should be understood that the terms "first", "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly and specifically limited.

[0025] In the case of complex task goals, the need for inputting multi-modal information or high environmental uncertainty, it is difficult for robots relying on rules or single perception channel to understand human intent and adapt to changing environment, therefore, it is necessary to deploy VLA model in the robot to improve the ability of decision-making and task adaptation of the robot. According to different user needs and working environment, robots are designed into multiple types, including but not limited to: household service robots controlled by voice instructions, industrial collaborative robots used for performing assembly and detection tasks, warehouse logistics robots based on vision and text input for sorting and handling, and human-robot collaborative teaching robots that learn new tasks through user demonstration and password.

[0026] The VLA model is a multi-modal model for intelligent control of robots, aiming to let the robot understand the environment, understand the instructions and execute actions. The VLA model integrates computer vision capability, natural language processing capability and motion control capability, and can realize human-computer interaction with stronger generalization capability. Specifically, the VLA model is used to fuse robot visual information and natural language instructions in multiple modalities, and then generate specific control tasks (such as grasping sequence, moving path, etc.) and control the robot to gradually complete these tasks.

[0027] The VLA model includes a vision module, a language module, and an action module (Action / Policy). The vision module is configured to receive input camera image data, point cloud data, depth map data, and the like, and combine a residual neural network (Residual Neural Network, ResNet), a contrastive language-image pre-training model (Contrastive Language-Image Pre-training, CLIP), a segment anything model (Segment Anything Model, SAM), and the like to understand the scene in which the robot is located, identify objects, obtain spatial relationship and position information, and the like, as perception information. The language module is configured to receive input natural language instructions (for example, place the cup on the table), and combine a bidirectional encoder representation model (Bidirectional Encoder Representations from Transformers, BERT), a generative pre-trained transformer model (Generative Pre-trained Transformer, GPT), a text-to-text transfer transformer model (Text-To-Text Transfer Transformer, T5), a large language model (Large Language Model, LLM), and the like, to parse task objectives, understand action semantics, and establish task abstract information for converting intent in natural language into structured representation, as language features. The action module is configured to receive fused perception information and language features, and generate action instructions executable by the robot (for example, a navigation path) based on a reinforcement learning-based policy network, a simulated reality transfer policy or a modular controller.

[0028] However, in practical applications, the action generation strategy of the VLA model relies on offline training, and the behavior decision of the VLA model is generally learned in a sample data set and a static scene. However, the perception environment and task semantics faced by the robot in the real environment are uncertain. When such a VLA model is deployed in a robot, it is easy to cause out-of-bound movement, unexpected collision, or execution of dangerous actions due to visual errors (such as occlusion, blur, identification failure, etc.), language instruction ambiguity, or model prediction errors when generating actions. The related art generally reduces the impact range when such problems occur through post-intervention methods (such as emergency stop, action interruption, etc.), and then reduces the occurrence rate of such problems through manual adjustment of the VLA model or retraining. However, this method generally only works after unsafe behavior occurs, and it is difficult to solve the safety hazard problem from the source to ensure the safety and stability of the robot action.

[0029] Therefore, it is necessary to provide a robot control method to solve the robot movement accident caused by the lack of safety constraint ability during the execution of the robot action. Specifically, the generated action is verified and optimized based on the motion safety constraint information, the dynamic correction of the action is realized to ensure the safety of the generated action, the safety of the robot during the execution of the action is ensured, and the danger caused by the robot during the execution of the action is reduced. Compared with the post-intervention solution of the related art, the present application can ensure the continuity of task execution and reduce the task termination caused by unexpected action.

[0030] Please refer to Figure 1 In the multiple embodiments provided in the present application, the robot control method can be applied to a robot control system. The robot control system is a computing system for controlling the robot to execute actions, performing safety control on the robot, and executing logic calculation for completing control tasks.

[0031] Generally, the robot control system includes a software logic layer and a hardware support layer. The software logic layer is used to execute logic calculation such as logic code that can realize a specified function, model reasoning, decision flow, and control flow. The hardware support layer is used to provide a physical computing platform and peripheral interface for running software logic. The physical computing platform can include but is not limited to a main control board, an industrial computer, and an actuator controller, etc. The peripheral interface can be used to connect components such as sensors, motor drivers, and communication modules.

[0032] Further, in order to enhance the safety guarantee of the robot behavior on the basis of the conventional action control, the software logic layer can further include, but is not limited to, a safety control layer and a monitoring execution layer. The safety control layer is configured to perform verification and optimization operations before the action is generated, to ensure that the generated action is safe. The monitoring execution layer is configured to monitor the safety during the execution of the action, and to control the robot to stop moving in a timely manner in the case that an unsafe factor is monitored.

[0033] Specifically, the monitoring execution layer can include, but is not limited to, a redundancy monitoring subsystem, a real-time obstacle avoidance subsystem, and a task monitoring subsystem. Exemplarily, the redundancy monitoring subsystem can include an obstacle sensor array and an emergency stop actuator, the real-time obstacle avoidance subsystem can include an action compliance detector and a task interruption controller, and the task monitoring subsystem can include a task progress tracker and an abnormality alarm module.

[0034] The redundancy monitoring subsystem, the real-time obstacle avoidance subsystem, and the task monitoring subsystem constitute a multi-layer safety guarantee mechanism with different trigger frequencies and response paths, which can adapt the robot control system to diversified risk scenarios and make timely safety responses.

[0035] In some embodiments, the task monitoring subsystem is configured to trigger the redundancy monitoring subsystem or the real-time obstacle avoidance subsystem to recheck obstacles in the environment or recheck compliance of the action, in the case that the task progress indicates a task abnormality, while the monitoring results of the redundancy monitoring subsystem or the real-time obstacle avoidance subsystem do not indicate an abnormality, so as to correct the false obstacle detection signal or the action compliance checking result in a timely manner, thereby improving the fault tolerance of the robot control system.

[0036] In some embodiments, the redundancy monitoring subsystem can be deployed in, for example, an edge processor or an embedded sensor module. The real-time obstacle avoidance subsystem can be deployed on, for example, a robot main control board. The task monitoring subsystem can be deployed in, for example, a central task scheduler or an upper computer. The upper computer can be an electronic device with certain computing power and network access capability. Exemplarily, the electronic device can be a desktop computer, a notebook computer, a tablet computer, or a server. The electronic device can be connected to the server through a network. The server can be a distributed server including multiple processors, memories, network communication modules, and the like, which work cooperatively to realize various functions. Alternatively, the server can also be a server cluster formed by a plurality of servers, which has higher computing and data processing capabilities. With the development of science and technology, the server can also be implemented by using new forms of technical means, such as a new type of “server” based on quantum computing.

[0037] In some embodiments, the redundant monitoring subsystem is used for the barrier sensor array relying on the hardware support layer to collect barrier signals of the space environment where the robot is located at a preset frequency, and in the case that the barrier signals are used to indicate the risk of collision, the emergency stop actuator relying on the hardware support layer stops the robot movement to protect the safety of the robot body and the environment; wherein the barrier sensor array can include but is not limited to: lidar, infrared sensor, ultrasonic sensor or depth camera.

[0038] In some embodiments, the real-time obstacle avoidance subsystem is used to trigger the action compliance detector to detect the compliance of the action being executed by the robot, and in the case that the action is detected to be non-compliant, the task interruption controller is triggered to stop the action execution of the robot in a hardware and software cooperative manner; wherein the action compliance detector and the task interruption controller can be represented in the form of control program, script or embedded algorithm module, etc.

[0039] In some embodiments, the task monitoring subsystem is used to trigger the task progress tracker to record the execution status of the robot for the current control task, and compare the status with the preset task plan, so as to identify abnormal states such as task stagnation and task timeout, and then the abnormal alarm module can be triggered to terminate the execution of the task and generate corresponding alarm information for reporting or prompting. Wherein the task progress tracker and the abnormal alarm module can be represented in the form of control program, script or embedded algorithm module, etc.

[0040] Please refer to Figure 2 One embodiment of the present application provides a robot control method. The robot control method can be applied in a robot control device. The robot control method can include the following steps.

[0041] Step S110: generating target action information for controlling robot movement based on a robot control task.

[0042] Step S120: performing safety verification on the target action information based on movement safety constraint information of the robot; wherein the movement safety constraint information is used to constrain the movement range of the robot in space from at least one spatial dimension;

[0043] Step S130: in the case that the safety verification fails, optimizing the target action information until the optimized target action information passes the safety verification;

[0044] Step S140: driving the robot movement using the target action information that passes the safety verification.

[0045] In this embodiment, the target action information is generated based on robot vision information and language control information calling VLA model or other related models.

[0046] In the embodiment, the robot vision information is vision data related to the external environment obtained by the robot through a perception device. Optionally, the perception device can include, but is not limited to, a Red-Green-Blue Image (RGB) camera, a Red-Green-Blue with Depth (RGB-D) camera, a panoramic camera, a laser radar, a structured light, a Time of Flight (ToF) depth camera, an industrial vision card, and an edge AI module. Optionally, the vision data can include, but is not limited to, an image, a depth map, a point cloud, a video stream, an object detection box, and a semantic segmentation map.

[0047] In some embodiments, the robot vision information can be obtained in the following ways: obtaining an RGB image and a depth map based on an RGB-D camera as the robot vision information; or constructing an environment map based on a binocular camera array connected to an industrial PC as the robot vision information; or constructing a scene reconstruction image based on a biomimetic fisheye panoramic camera and a neural radiance field as the robot vision information.

[0048] In the embodiment, the language control information is a high-level natural language instruction (e.g., "get a cup") input by the user through voice, text, voice-to-text, etc. The language control information is used to indicate the task target of the robot. Optionally, the language control information can be obtained by a microphone array, a speech recognition engine, a text input device (e.g., a remote client), or a language understanding module (e.g., an LLM).

[0049] In some embodiments, the language control information can be obtained in the following ways: performing local speech recognition based on a microphone array to obtain the language control information; or parsing a text instruction sent by a client to obtain the language control information.

[0050] In the embodiment, the robot vision information can describe the spatial environment in which the robot is located, and the language control information can indicate the target expected to be achieved by the user. Therefore, the VLA model can generate a task for achieving the target based on the robot vision information and the language control information, and execute the task to generate corresponding action control instructions. The action control instructions can be parsed into target action information containing various action parameters for driving the actuators, and the target action information can drive the robot to perform the target action to achieve the target expected by the user, for example, to pick up a cup. The task can be a single task or a task sequence, which is not limited in the embodiments of the present application.

[0051] In the embodiment, the target action information refers to a set of motion parameters that the robot needs to perform at a specified time instant / period in order to complete the action execution plan. Optionally, the set of motion parameters can include but is not limited to: an end-effector pose representing the spatial position and orientation of the end-effector, a joint angle that should be reached for one or more joints, a velocity vector, a trajectory curve, path planning information, an operation identifier, an operation execution time, an operation duration, a torque control parameter, and reference coordinate system information.

[0052] In some embodiments, the corresponding set of motion parameters varies in field for different types of robots. For example, for a velocity-controlled robot, the corresponding set of motion parameters includes a velocity vector, a direction, and an operation duration. For a high-degree-of-freedom flexible robot, the corresponding set of motion parameters includes joint compliance parameters and deformation descriptors.

[0053] In the embodiment, since the parameters of the VLA model are fixed in the offline training process, and the environment faced by the robot can be complex and variable, when the VLA model is deployed to the robot entity, it can generate target action information that does not meet the safety specification due to various reasons such as robot vision information error. If the robot is triggered to perform an action based on the target action information, it is likely to cause the robot to exhibit unexpected behavior, thereby causing the robot to be damaged or destroy the space environment. In order to reduce this problem, the present application has motion safety constraint information, and the motion safety constraint information acts on the decision-making link of the VLA model, for checking the target action information generated by the VLA model based on the motion safety constraint information. In the case where the target action information meets the requirements of the motion safety constraint information, the robot can be driven based on the target action information, and in the case where the target action information does not meet the requirements of the motion safety constraint information, the target action information is optimized until it meets the motion safety constraint information.

[0054] In the embodiment, the motion safety constraint information is used to constrain the motion range of the robot in space from at least one spatial dimension, which can ensure the safety of the behavior of the robot and reduce the probability of damage to the robot and adverse effects on the environment. The spatial dimension can include but is not limited to: Cartesian dimension, joint dimension, torque dimension, moment dimension, velocity dimension, power dimension, and semantic space dimension.

[0055] In some embodiments, the manner of determining the motion safety constraint information can include, but is not limited to: responding to static information set by user operation, such as automatically generating the concave polyhedron information of the human body approaching area as the motion safety constraint information; or, according to dynamic information updated by the position of the robot in the space environment, such as updating the buffer zone radius as the motion safety constraint information according to the change of the human-robot distance; or, generating semantic constraints according to natural language information, such as generating camera envelope information based on "do not touch the camera on the left side of the production line" as the motion safety constraint information.

[0056] In some embodiments, the motion safety constraint information includes at least one of: space obstacle coordinate information, joint torque upper limit value of the robot, forbidden area in the space environment, and task constraint (such as completing the task within 5 minutes). Wherein, the space obstacle coordinate information represents the position and shape of the obstacle in the space environment where the robot is located, which can be represented in the form of coordinates and size, and can be used as a constraint for robot motion path planning and action safety check, to reduce the collision between the robot and the obstacle in the environment. Wherein, the joint torque upper limit value represents the maximum torque value that each joint of the robot can withstand, which depends on the motor specification or structural strength. The joint torque upper limit value as the motion safety constraint information can prevent the robot from performing actions that exceed the joint torque upper limit value and cause joint damage and other problems. Wherein, the forbidden area in the space environment refers to the space area that the robot is prohibited to enter (such as equipment maintenance area, employee rest area), and the forbidden area can be dynamically updated. Wherein, the task constraint is used to non-spatially constrain one or more dimensions of the task being performed by the robot, such as the time, path, and path of the task execution. For example, the task constraint is used to constrain the duration of task execution to ensure that the duration of resource occupation during task execution is within the expected range, for example, the material must be put on the assembly line within 5 minutes.

[0057] In this embodiment, the safety verification of the target action information based on the motion safety constraint information of the robot can be understood as verifying whether the target action information meets the motion range of each spatial dimension. If it meets, it can be determined that the safety verification is passed. If it does not meet the motion range of any spatial dimension, it can be determined that the safety verification is not passed, and then the target action information can be optimized until the optimized target action information passes the safety verification.

[0058] In some embodiments, the manner of optimizing the target action information until the optimized target action information passes the safety verification can be: calling a quadratic programming solver to adjust the target action information until the target action information meets the motion safety constraint information; or uploading the target action information to a cloud server to enable the cloud server to solve the target action information that meets the motion safety constraint information through a quadratic optimization algorithm.

[0059] In the embodiment, the target action information verified by the safety verification can be used to drive the robot to move according to the parameters such as speed and joint torque defined in the target action information. Since the target action information of the driving robot is verified by safety or quadratic programming, it can be ensured that the robot performs the expected action, and the probability of unsafe behavior is reduced.

[0060] In summary, in the embodiment, the generated action can be verified and optimized based on the motion safety constraint information, the dynamic correction of the action is realized to ensure the safety of the generated action, the safety of the robot performing the action is ensured, and the danger caused by the robot performing the action is reduced. And compared with the post-intervention solution of the related art, the present application can ensure the continuity of task execution and reduce the termination of task due to unexpected action.

[0061] In some embodiments, the robot control device can perform the following steps. The step of verifying the safety of the target action information based on the motion safety constraint information of the robot includes loading the motion safety constraint information corresponding to the robot control task from the constraint information library; wherein the robot control task is generated based on the robot visual information and the language control information calling the VLA model, and the constraint information library includes a plurality of motion safety constraint information suitable for different task types; and verifying the safety of the target action information based on the motion safety constraint information.

[0062] In the embodiment, in order to perform personalized and targeted motion safety constraints, the motion safety constraint information in the constraint information library can be related to the type of robot control task, so that different types of robot control tasks can be constrained by the related motion safety constraint information.

[0063] In the embodiment, the robot control task can be an operation intention generated by jointly processing the robot visual information and the user language control information based on the VLA model, such as moving an object, avoiding obstacles, and path navigation, etc. The generation method of the robot control task can be: jointly encoding the robot visual information and the language control information through the VLA model, and then generating a structured robot control task according to the joint encoding; or identifying a target object (such as a red cup) in the environment as robot visual information through the vision module in the VLA model, and analyzing the semantics of a natural language instruction (such as "red cup to me") as language control information through the language module, and then integrating the robot visual information and the language control information into a robot control task (such as "grab the red cup - move - release").

[0064] In the embodiment, the motion safety constraint information is loaded from a constraint information library. The constraint information library is a data structure system for storing various motion safety constraint information in categories, which can include but is not limited to the following categories: static parameters (e.g., maximum torque, etc.), environmental constraints (e.g., spatial obstacle coordinate information, etc.), and behavior constraints (e.g., task time window, etc.). Each category corresponds to one or more motion safety constraint information. The constraint information library can be deployed in the robot or in an external server, which is not limited in the embodiment.

[0065] In the embodiment, the motion safety constraint information corresponding to different robot control tasks can be different, so that the target motion constraint corresponding to different robot control tasks is constrained as necessary, and unnecessary constraint information is reduced. Different robot control tasks can refer to different types of robot control tasks, or different robot control tasks with different purposes, which are not limited in the embodiment. For example, for the same type of robot control tasks such as grasping, transporting, and navigation, motion safety constraint information containing speed limit, path boundary, torque threshold, etc. can be loaded.

[0066] Optionally, the task type includes at least one of the following: a grasping task, a carrying task, an assembly task, and a human-robot collaboration task; the motion safety constraint information corresponding to the grasping task is used to at least limit the force range of an end effector of the robot, the motion safety constraint information corresponding to the carrying task is used to at least limit inertia and acceleration in path planning, the motion safety constraint information corresponding to the assembly task is used to at least limit position error and torque of the end effector, and the motion safety constraint information corresponding to the human-robot collaboration task is used to at least limit joint speed of the robot.

[0067] In some embodiments, the same robot control task corresponds to different versions of motion safety constraint information, which can be loaded from the constraint information library based on robot vision information, to improve the adaptability of robot behavior to the spatial environment and reduce unsafe behavior. For example, the motion safety constraint information corresponding to the grasping task has outdoor and indoor versions, and the different versions of motion safety constraint information at least correspond to different obstacle avoidance distances.

[0068] In some embodiments, the method can further comprise the following steps: parsing the intent in the language control information (e.g., bring the cup, be quiet), generating customized motion safety constraint information corresponding to the intent, and updating the customized motion safety constraint information into the constraint information library. The customized motion safety constraint information can be automatically loaded as the basis for optimization of the target action information when the same robot control task is executed for the same user. The customized motion safety constraint information can also be applied to the same robot control task for different users. Optionally, whether the customized motion safety constraint information can be applied to the same robot control task for different users depends on its generality, which can be evaluated by historical experience information or by calling a pre-trained large model.

[0069] In some embodiments, the way to load the motion safety constraint information corresponding to the robot control task from the constraint information library can be: generating a task identifier corresponding to the robot control task, calling the constraint information library, querying the associated motion safety constraint information (e.g., maximum moving speed 0.2 m / s, prohibited edge area) from the constraint information library based on the task identifier, and loading the motion safety constraint information into the action planning module as the basis for generating the target action information. Optionally, the associated motion safety constraint information can belong to different fields respectively.

[0070] In some embodiments, the step of loading the motion safety constraint information corresponding to the robot control task from the constraint information library can further comprise: monitoring dynamic changing robot vision information, and dynamically loading the motion safety constraint information according to the changing robot vision information. For example, when the robot vision information indicates that someone is approaching, the motion safety constraint information related to it in the constraint information library can be loaded, such as reducing the speed by 50% when someone is approaching.

[0071] In some embodiments, the robot control device can perform the following steps. The method further comprises: based on any one of the obstacle detection signal, the robot action compliance, and the task progress of the robot control task, controlling the robot to stop moving during the motion of the robot.

[0072] In this embodiment, in order to perform real-time safety monitoring during action execution, the obstacle detection signal, the robot action compliance, or the task progress of the robot control task can be detected. If any of the information represents an abnormality, it can be used as the basis for execution stop logic, and further multi-level safety monitoring is achieved on the basis of action safety constraints.

[0073] In the embodiment, the obstacle detection signal can be collected in real time by a sensor module (e.g., a laser radar, a depth camera, an ultrasonic sensor, etc.), and analyzing the obstacle detection signal can determine the distance of the obstacle in the spatial environment. When the obstacle detection signal indicates that the distance of the obstacle is less than a preset value, the robot can be controlled to stop moving in time to reduce the occurrence of a collision.

[0074] In the embodiment, the robot action compliance is used to evaluate whether the action currently performed by the robot meets the parameter range of the target action information that has been previously verified for safety. The parameter range can be motion safety constraint information or other information. Monitoring the robot action compliance can control the robot to stop moving in time in response to action execution problems such as joint step loss, motor blockage, or end offset exceeding the limit.

[0075] In the embodiment, the task progress of the robot control task is used to represent the state of the robot control task. When the task progress represents an abnormal state such as stasis, jamming, or timeout, the robot can be controlled to stop moving in time to reduce the occupation of robot resources for a long time.

[0076] In some embodiments, the monitoring frequency of the obstacle detection signal is higher than the monitoring frequency of the robot action compliance; and / or, the monitoring frequency of the robot action compliance is higher than the monitoring frequency of the task progress of the robot control task.

[0077] In the embodiment, in order to configure multi-level safety monitoring with hierarchy, different monitoring frequencies are configured for the obstacle detection signal, the robot action compliance, and the task progress of the robot control task.

[0078] In the embodiment, the monitoring frequency of the obstacle detection signal refers to the frequency of collecting environmental information by a sensor. Since the risk level and urgency of the obstacle detection signal are relatively high, a higher monitoring frequency than that of the robot action compliance can be configured for the obstacle detection signal, for example, triggered once every 20 ns.

[0079] In the embodiment, the monitoring frequency of the robot action compliance is a motion level monitoring, which can be used to monitor the consistency of the trajectory deviation and execution error between the action performed by the robot and the target action information. The risk level and urgency of the robot action compliance are lower than those of the obstacle detection signal. A lower monitoring frequency than that of the obstacle detection signal can be configured for the robot action compliance, for example, triggered once every 200 ms.

[0080] In the embodiment, the monitoring frequency of the task progress is task-level monitoring, which realizes periodic checking of the entire control task state (e.g., whether completed, whether timed out, whether stuck), and task stagnation or timeout is less likely to cause safety problems such as collision. Therefore, the risk level and urgency of the task progress are lower than those of the robot action compliance, and the monitoring frequency configured for the task progress can be lower than that of the robot action compliance, thereby reducing performance degradation caused by frequent polling of the task state, for example, triggered once every 1s.

[0081] In some embodiments, for monitoring of the obstacle detection signal, the robot action compliance, or the task progress of the robot control task, polling at a monitoring frequency can also be event-driven. For example, if a dangerous action occurrence event is detected, the logic for checking the robot action compliance can also be triggered.

[0082] In some embodiments, the robot control device can perform the following steps. The step of controlling the robot to stop moving based on the obstacle detection signal includes: in the case where it is detected that the obstacle detection signal indicates obstacle avoidance, sending a signal to a safety control circuit in the robot to cause the safety control circuit to cut off the driving power supply of the robot to stop the robot from moving.

[0083] In the embodiment, in order to quickly cut off the robot power before physical collision and reduce the risk of robot damage. The obstacle detection signal indicates that there is an obstacle within a minimum safety distance (e.g., 30 cm) or the obstacle is close at a dangerous angle, indicating that obstacle avoidance is needed. In some embodiments, an emergency stop actuator (e.g., a safety relay, a motor power-off module) can be used to send a signal to the safety control circuit corresponding to the robot power system to cut off the driving power supply, thereby controlling the robot to stop moving; or a stop signal can be sent to the driving layer through the obstacle detection chip to cut off the driving power supply, thereby controlling the robot to stop moving.

[0084] In the embodiment, the control of the robot is realized by directly operating the robot bottom layer physical components or control circuits without relying on high-level software logic judgment, which has the characteristics of fast response speed, low execution level, and strong anti-interference ability. Optionally, the hardware control mode relies on the following hardware: embedded control chip, control relay, driving circuit, FPGA, PLC, and safety switch.

[0085] In the embodiment, the hardware control mode can be used to quickly cut off the robot power before physical collision and reduce the risk of robot damage.

[0086] In some embodiments, the robot control device can perform the following steps. The step of controlling the robot to stop motion based on robot action compliance includes: in the case of detecting that the robot action does not have compliance, sending an action interruption signal to a movement controller in the robot to make the movement controller control the robot to stop motion.

[0087] In the present embodiment, the target action information is the parameter information of the verified or optimized safe action, and therefore the action performed by the robot should meet the target action information. In some cases, the robot may, during execution, deviate from the target action information due to sensor errors, external environmental disturbances, or mechanical errors, and the robot can be controlled to interrupt the execution of the target action. In some embodiments, the parameters of the action currently performed by the robot can be obtained through a sensor and compared with the target action information, and if the deviation represented by the comparison result exceeds a preset range, it can be determined that the robot action does not have compliance.

[0088] In the present embodiment, the action interruption signal is sent to the movement controller in the robot, aiming to schedule, stop, or interrupt the robot motion at the logic layer or application layer through program logic (such as thread termination, control signal withdrawal, state flag setting, etc.), relying on the operating system, running environment, communication protocol, and task scheduling system, and having the characteristics of strong scalability, clear logic, and easy integration. The movement controller is implemented as a program module or a combination of a program module and hardware.

[0089] In some embodiments, the action interruption controller can call a software interface to suspend a task thread, and issue a motion deceleration instruction to reduce mechanical impact caused by sudden stopping; or the compliance detector can issue an action interruption signal, and stop a high-level control thread through software, and lock the current joint position through a hardware driving interface to reduce the execution of erroneous actions.

[0090] In the present embodiment, the erroneous action of the robot can be efficiently and stably interrupted, and the problem of continuous execution of dangerous actions of the robot caused by motion drift or model deviation can be reduced.

[0091] In some embodiments, the robot control device can perform the following steps. The step of controlling the robot to stop motion based on the task progress of the robot control task includes: in the case that the task progress of the robot control task meets a preset task stagnation condition and / or a preset task timeout condition, sending a task termination instruction to a movement controller in the robot to make the movement controller control the robot to stop motion and report an alarm information.

[0092] In the embodiment, in order to realize the task-level security control, task stagnation monitoring and task timeout monitoring can be performed. The task progress can indicate the current state of the robot, and the task progress can be used to evaluate whether the robot control task is normally promoted. When the task progress meets the preset task stagnation condition and / or the preset task timeout condition, it can be determined that the task is abnormal, and the robot executing the robot control task is terminated. The preset task stagnation condition is used to indicate a rule for determining a stagnation state based on the task progress, for example, no state change for more than 10s. The preset task timeout condition is used to indicate a rule for determining a timeout state based on the task progress, for example, failing to complete the robot control task within the maximum allowed time (e.g., 300s).

[0093] In the embodiment, the task termination instruction is sent to the movement controller in the robot to make the movement controller control the robot to stop moving and report an alarm information. The manner can be that the task termination instruction is sent to the movement controller in the robot to stop scheduling the robot control task and lock the robot control task, and trigger the execution of the alarm logic. Further optionally, a task stop report for being used as a basis for user analysis of abnormal reasons can also be generated.

[0094] In some embodiments, the robot control device can perform the following steps. The method further comprises: in the case that the task progress of the robot control task indicates a task abnormality, triggering detection of an obstacle detection signal and / or robot action compliance according to the type of the task abnormality.

[0095] In the embodiment, when the task progress of the robot control task indicates a task abnormality such as task stagnation or task timeout, there can be some other problems that can be checked by the underlying control system at the same time, so the detection of the obstacle detection signal or the robot action compliance can be triggered to make it be corrected in time in the case that the obstacle detection signal or the robot action compliance has an error. Optionally, the corrected obstacle detection signal or robot action compliance can be used as a basis for abnormal analysis of the task abnormality.

[0096] In the embodiment, the type of the task abnormality can include but is not limited to: a stagnation type abnormality and a timeout type abnormality. If the type of the task abnormality is the stagnation type abnormality, there can be an obstacle blocking and the like, so the detection of the obstacle detection signal can be triggered. If the type of the task abnormality is the timeout type abnormality, the robot action can fail multiple times, so the detection of the robot action compliance can be triggered. It can also be that the robot stops during the action execution process, causing the timeout, so the detection of the obstacle detection signal and the robot action compliance can be triggered.

[0097] Please refer to Figure 3The embodiment of the present application further provides a robot control device. The robot control device can comprise: an action generation module, configured to generate target action information for controlling robot movement based on a robot control task; a safety verification module, configured to perform safety verification on the target action information based on movement safety constraint information of the robot; wherein the movement safety constraint information is used to constrain movement range of the robot in space from at least one spatial dimension; an action optimization module, configured to optimize the target action information until the optimized target action information passes the safety verification in case that the safety verification fails; and a driving module, configured to drive the robot to move by using the target action information passing the safety verification.

[0098] In the embodiment, the specific functions and effects of the robot control device can be explained in reference to other embodiments of the present application, and will not be repeated here.

[0099] Please refer to Figure 4 The embodiment of the present application further provides a computer device, comprising: a memory and a processor, at least one computer program is stored in the memory, the at least one computer program is loaded and executed by the processor to realize the method as described above.

[0100] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method as described above.

[0101] The embodiment of the present application further provides a computer program product comprising instructions, and the computer program product is executed by a processor to realize the method as described above.

[0102] It can be understood that the specific examples herein are only to help those skilled in the art better understand the embodiments of the present application, and do not limit the scope of the present application.

[0103] It can be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and inherent function, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0104] It can be understood that the various embodiments described in the present application can be implemented alone or in combination, and the embodiments of the present application do not limit this.

[0105] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The use of the terms "and / or" and "at least one of" includes any and all combinations of one or more of the associated listed items. As used herein, including the claims "a," "an," and "the" preceding a term or phrase, including "at least one of" a term or phrase, are intended to be open-ended terms that do not exclude the presence of one or more additional terms or phrases.

[0106] It can be understood that the processor in the embodiments of the present application can be an integrated circuit chip with a signal processing capability. In the implementation process, the steps of the above method embodiments can be completed by hardware integrated circuit in the processor or by software form instructions. The processor mentioned above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable functional devices, discrete gates or transistor functional devices, discrete hardware components. The disclosed methods, steps and functional block diagrams in the embodiments of the present application can be realized or executed. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0107] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM). It should be noted that the memory of the system and method described herein is intended to include but not limited to these and any other suitable type of memory.

[0108] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the information to be retrieved. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0110] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of the units is only a functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0111] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0112] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0113] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the information to be searched of the present application or the part of the information to be searched which essentially contributes to the prior art or the part of the information to be searched can be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0114] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A robot control method, characterized in that: The method comprises: Generate target action information for controlling robot motion based on the robot control task; Performing safety verification on the target motion information based on motion safety constraint information of the robot; wherein the motion safety constraint information is used to constrain the range of motion of the robot in space from at least one spatial dimension; If the security verification fails, optimizing the target action information until the optimized target action information passes the security verification; The robot is driven to move using target motion information that has passed safety verification.

2. The method according to claim 1, characterized in that The step of performing safety verification on the target motion information based on the motion safety constraint information of the robot comprises: Loading motion safety constraint information corresponding to a robot control task from a constraint information library; wherein the robot control task is generated by calling a VLA model based on robot vision information and language control information, and the constraint information library includes a variety of motion safety constraint information suitable for different task types; The target motion information is safety verified based on the motion safety constraint information.

3. The method according to claim 1, characterized in that The method further comprises: During the movement of the robot, the robot is controlled to stop moving based on any one of an obstacle detection signal, robot motion compliance, and task progress of the robot control task.

4. The method according to claim 3, characterized in that The monitoring frequency of obstacle detection signals is higher than the monitoring frequency of robot movement compliance; And / or, the frequency of monitoring the compliance of the robot's actions is higher than the frequency of monitoring the task progress of the robot's control task.

5. The method according to claim 3, characterized in that The step of controlling the robot to stop moving based on the obstacle detection signal comprises: When it is detected that the obstacle detection signal indicates obstacle avoidance, a signal is sent to the safety control circuit in the robot, so that the safety control circuit cuts off the driving power of the robot to stop the robot from moving.

6. The method according to claim 3, characterized in that The step of controlling the robot to stop moving based on the robot motion compliance comprises: When it is detected that the robot's action does not meet the compliance, a motion interruption signal is sent to the mobile controller in the robot, so that the mobile controller controls the robot to stop moving.

7. The method according to claim 3, characterized in that The step of controlling the robot to stop moving based on the task progress of the robot control task includes: When the task progress of the robot control task meets the preset task stagnation condition and / or the preset task timeout condition, a task termination instruction is sent to the mobile controller in the robot so that the mobile controller controls the robot to stop moving and report alarm information.

8. The method according to claim 3, characterized in that The method further comprises: In the case where the task progress of the robot control task indicates a task abnormality, an obstacle detection signal and / or robot action compliance is triggered according to the type of the task abnormality.

9. The method according to any one of claims 1 to 8, characterized in that The motion safety constraint information includes at least one of the following: spatial obstacle coordinate information and an upper limit value of the joint torque of the robot.

10. The method according to claim 2, characterized in that The task types include at least one of the following: grasping tasks, handling tasks, assembly tasks and human-machine collaboration tasks; wherein, the motion safety constraint information corresponding to the grasping task is at least used to limit the force range of the end effector of the robot, the motion safety constraint information corresponding to the handling task is at least used to limit the inertia and acceleration in path planning, the motion safety constraint information corresponding to the assembly task is at least used to limit the position error and torque of the end effector, and the motion safety constraint information corresponding to the human-machine collaboration task is at least used to limit the joint speed of the robot.

11. A computer program product, characterized in that When the computer program product is executed by a processor, the robot control method according to any one of claims 1 to 10 is implemented.

Images