Body-equipped robot safety action execution method and body-equipped robot system

Abstract

The application relates to a kind of embodied robot safety action execution method and embodied robot system.The method comprises: receiving task instruction, executing the task indicated by the task instruction;In the process of executing the task, if collision event is detected by the collision switch of the embodied robot, record the event information of collision event, multiple event information is arranged in time sequence to form event information sequence, the event information includes collision position and corresponding collision time;According to the first sub-sequence in the first sliding time window in the event information sequence, the collision density information corresponding to the grid map of the embodied robot is generated, the collision density information includes the collision density corresponding to each grid in the grid map;In the case where the collision density information represents that the first safety action execution condition is met, safety action is executed.Using the method can improve safety.

Description

Technical Field

[0001] This application relates to the field of embodied robot technology, and in particular to a method for performing safe actions with an embodied robot and an embodied robot system. Background Technology

[0002] With the development of artificial intelligence and robotics, embodied robots are gradually being integrated into the home environment. For example, in the home environment, embodied robots can perform corresponding tasks according to the user's instructions.

[0003] In existing technologies, force sensors installed at the joints of androids are typically used to detect the contact forces between the android and its environment in order to determine collision events and control contact behavior. Force sensors can provide continuous measurement information on contact force and torque, which helps androids identify minute contacts, determine contact intensity, and ensure safe human-robot interaction.

[0004] However, force sensors are expensive, complex in structure, and have high requirements for installation location, calibration accuracy, and system bandwidth, making them difficult to configure in many cost- or structurally limited embodied robots. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for performing safe actions of an embodied robot and an embodied robot system that can improve safety in response to the above-mentioned technical problems.

[0006] On one hand, this application provides a method for performing safe actions of an embodied robot, comprising: receiving a task instruction and executing the task indicated by the task instruction; during the execution of the task, if a collision event is detected by a collision switch of the embodied robot, recording the event information of the collision event, wherein multiple event information are arranged in chronological order to form an event information sequence, the event information including the collision location and the corresponding collision time; generating collision density information corresponding to a grid map of the embodied robot based on a first subsequence within a first sliding time window in the event information sequence, the collision density information including the collision density corresponding to each cell in the grid map; and performing a safe action when the collision density information indicates that a first safe action execution condition is met.

[0007] In some embodiments, generating collision density information corresponding to the grid map of the embodied robot based on a first subsequence within a first sliding time window in the event information sequence includes: determining a first subsequence within the first sliding time window from the event information sequence; determining the collision density corresponding to each cell in the grid map based on the collision position in the first subsequence and the cell position of the cell; and obtaining the collision density information corresponding to the grid map based on the collision density corresponding to each cell in the grid map.

[0008] In some embodiments, performing a safety action when the collision density information characterizes a first safety action execution condition includes: determining a reference collision density within the first sliding time window based on the collision density contained in the collision density information; and performing the safety action when the reference collision density reaches a collision density threshold.

[0009] In some embodiments, determining the reference collision density within the first sliding time window based on the collision density contained in the collision density information includes: determining the maximum collision density from a plurality of target collision densities contained in the collision density information, and using the maximum collision density as the reference collision density; wherein the plurality of target collision densities are all the collision densities in the collision density information, or the plurality of target collision densities are the collision densities corresponding to a plurality of grids in the target area contained in the grid map.

[0010] In some embodiments, after receiving a task instruction, the method further includes: determining a second sub-sequence within a second sliding time window from the event information sequence, wherein the second duration of the second sliding time window is greater than the first duration of the first sliding time window; and executing the safety action if the second sub-sequence indicates that a second safety action execution condition is met.

[0011] In some embodiments, the step of performing a safety action when the second subsequence characterization satisfies the second safety action execution condition includes: determining the number of collisions and at least one collision time interval within the second sliding time window based on the second subsequence; wherein the collision time interval is the time interval between two adjacent collision events; if the number of collisions is greater than a threshold or any of the collision time intervals is less than a preset interval threshold, then the safety action is triggered.

[0012] In some embodiments, executing the task indicated by the task instruction includes: generating an action sequence based on the task instruction and current first scene information; and executing the corresponding actions in the order of the action sequence to execute the task indicated by the task instruction.

[0013] In some embodiments, before executing the corresponding actions in the order of the action sequence to perform the task indicated by the task instruction, the method further includes: determining the collision restriction type of each action in the action sequence, wherein the collision restriction type includes a restricted collision type or a prohibited collision type, wherein the restricted collision type represents that the collision density is not allowed to reach a collision density threshold during the execution of the action, and the prohibited collision type represents that collision events are prohibited during the execution of the action; generating collision density information corresponding to the grid map of the embodied robot based on the first sub-sequence within the first sliding time window in the event information sequence includes: for the currently executed action, if the collision restriction type of the current action is a restricted collision type, determining from the event information sequence the first sub-sequence generated during the execution of the current action and within the first sliding time window; generating collision density information corresponding to the grid map of the embodied robot based on the first sub-sequence.

[0014] In some embodiments, during the process of executing the corresponding actions in the order of the action sequence to perform the task indicated by the task instruction, the method further includes: if a collision event is detected by the collision switch of the android, and the collision restriction type of the action executed when the collision event occurs is a collision prohibition type, then a safety action is immediately executed.

[0015] In some embodiments, after performing the safety action, the method further includes: obtaining current second scene information, generating a risk record based on the recorded event information and the second scene information, the risk record being used to assist the embodied robot in task planning.

[0016] On the other hand, this application also provides an embodied robot system, including a terminal and an embodied robot. The terminal is used to send task instructions to the embodied robot, and the embodied robot is used to perform the steps in the above-described embodied robot safety action execution method.

[0017] On the other hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method for performing safe actions of a hymenoidae robot.

[0018] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method for performing safe actions of a hybrid robot.

[0019] On the other hand, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method for performing safe actions of an embodied robot.

[0020] The aforementioned embodied robot safety action execution method, embodied robot system, computer equipment, computer-readable storage medium, and computer program product record event information from multiple collision events detected by a collision switch to form an event information sequence. The event information in this sequence is arranged chronologically and includes collision time and location. Based on a first subsequence within a first sliding time window in the event information sequence, collision density information corresponding to the embodied robot's grid map is generated. When the collision density information indicates that a first safety action execution condition is met, a safety action is executed. Thus, by combining the first sliding time window, the event information sequence, and the collision density, collision density information is generated to reflect the frequency and location distribution of collisions. Based on this collision density information, a safety action is triggered. This allows the embodied robot to identify and respond to abnormal contact using a very low-cost but information-limited collision switch, even in the absence of force sensors, thereby reducing the hardware cost of the embodied robot. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an application environment diagram of a method for performing safe actions on a holographic robot in one embodiment;

[0023] Figure 2 This is a flowchart illustrating a method for performing safety actions by an embodied robot in one embodiment;

[0024] Figure 3 This is a flowchart illustrating a method for performing safe actions by a holographic robot in another embodiment;

[0025] Figure 4 This is a structural block diagram of a safety action execution device for a holographic robot in one embodiment;

[0026] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0029] The method for performing safe actions on a hymenoidae robot provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown is as follows. This application scenario includes an embodied robot 102 and a terminal 104, with the embodied robot 102 situated within the current environment. The embodied robot 102 possesses sensing, movement, and interaction capabilities, and can interact with its environment in real time. It can capture information about its surroundings through sensory organs such as cameras or collision switches installed on the embodied robot 102. The embodied robot 102 can be a home service robot, including but not limited to companion robots or humanoid robots. The current environment can be any environment suitable for the application of the embodied robot 102, including but not limited to home environments, industrial manufacturing environments, medical environments, warehousing and logistics environments, agricultural production environments, public service environments, educational and research environments, or entertainment performance environments. The terminal 104 can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The embodied robot 102 and the terminal 104 communicate via a network. The terminal 104 can send task instructions to the embodied robot 102.

[0030] In one exemplary embodiment, such as Figure 2 As shown, a method for performing safe actions of an embodied robot is provided, which can be applied to... Figure 1 Taking the embodied robot 102 as an example, the following steps are included:

[0031] Step 202: Receive the task instruction and execute the task indicated by the task instruction.

[0032] The task instructions can be user-triggered or triggered by the large model based on multimodal input information. Task instructions can be in voice or text form; for example, they can be voice instructions triggered by the user's voice. Task instructions can also be instructions triggered by the user through the app controlling the embodied robot, or button instructions triggered by buttons on the embodied robot. The task indicated by the task instruction can be any task, including but not limited to "washing clothes," "folding clothes," or "tidying the room."

[0033] In an exemplary embodiment, executing the task indicated by the task instruction includes: generating an action sequence based on the task instruction and current first scene information; and executing the corresponding actions in the order of the action sequence to execute the task indicated by the task instruction.

[0034] An action sequence includes one or more action elements corresponding to each action; "multiple" means at least two. The action elements for each action in the action sequence are arranged according to the execution order of the actions. An action element may contain an action identifier (e.g., action number), action description information, and action implementation instructions. The action description information describes the action. The action implementation instructions are used to implement the action; for example, they may be skill function call instructions. The action identifier of an action element can be a sequentially increasing numerical value. The action description information can be information that conforms to the rules of natural language.

[0035] The first scene information may include at least one of the following: three-dimensional image data and two-dimensional image data of the current environment in which the embodied robot is located. The first scene information may be collected by the embodied robot's visual perception module or by independent cameras deployed in the scene.

[0036] For example, an embodied robot can execute a sequence of actions, performing each action in the sequence sequentially.

[0037] For example, the embodied robot may include a visual perception module and a task planning module. The first scene information can be obtained through the visual perception module. The action sequence can be generated through the task planning module. For example, if the task instruction is "drink water," the robot obtains the current first scene information through the visual perception module, and the task planning module outputs the action sequence based on the task instruction "drink water" and the current first scene information: navigate to the table → grab the cup → deliver it. The task planning module can be a Large Language Model.

[0038] In existing technologies, embodied robots typically use large language models to plan tasks. However, large language models are prone to inferences that are inconsistent with the real environment during task planning, resulting in inaccurate task planning. In this embodiment, by combining task instructions and currently collected first scene information to generate action sequences for task planning, the accuracy of task planning can be improved.

[0039] Step 204: If a collision event is detected by the collision switch of the embodied robot during the execution of the task, the event information of the collision event is recorded.

[0040] If multiple collision events are detected by the embodied robot's collision switch, the event information corresponding to each collision event is recorded. These events are arranged chronologically to form an event information sequence. Each event information includes the collision location and its corresponding collision time, with each collision location corresponding to one collision time. Each event information corresponds to one collision event. The collision time refers to the time when the collision event occurred. The collision location is the position of the embodied robot at the time of the collision event, which may include the robot's position in a local coordinate system, such as (x, y), where x is the coordinate along the X-axis and y is the coordinate along the Y-axis. The robot's position in the local coordinate system can be determined based on data output from the robot's wheel odometer. Recording the robot's position in the local coordinate system keeps computation lightweight and eliminates the need for global map maintenance. Alternatively, the collision location can include the robot's position in the global coordinate system at the time of the collision event. The global coordinate system is the coordinate system corresponding to the global map pre-constructed from data collected by the embodied robot using cameras or LiDAR.

[0041] Collision events can be any collision with the outside world that the embodied robot detects through a collision switch. The collision switch is mainly a binary trigger sensor composed of a triggering mechanism (such as a micro switch, spring switch, or Hall switch) used to detect whether a collision has occurred.

[0042] The local coordinate system refers to the android's own coordinate system. The origin of the local coordinate system is the android's geometric center or the center of its wheel axle. The positive X-axis of the local coordinate system points in the android's forward direction, and the positive Y-axis points to the android's left. The android's rotation angle can also be recorded. , This is the angle by which the embodied robot rotates relative to the initial local coordinate system. Since the position of the embodied robot is dynamically changing, the local coordinate system is also dynamically changing.

[0043] For example, the event information may also include a collision restriction type. The collision restriction type in the event information is the collision restriction type of the action performed when a collision event occurs. For example, if a collision event occurs during the execution of action A, event information is generated based on the collision time, collision location, and collision restriction type of action A, and the generated event information is added to the event information sequence.

[0044] For example, during the execution of an action sequence, an event information sequence is maintained, that is, collision events that occur during the execution of each action will be added to the same event information sequence.

[0045] For example, before recording event information, an event information sequence can be initialized, and the initialized event information sequence can be in the form of an array. For example, the initialized event information sequence can be an empty array.

[0046] For example, before executing a sequence of actions, the collision restriction type for each action in the sequence is determined. Collision restriction types include allowed collision types, restricted collision types, and prohibited collision types. Allowed collision types do not limit the upper limit of collision density. For example, when an embodied robot uses an end effector (such as a cleaning cloth, sponge, or brush) to wipe and clean a surface, this action inherently requires maintaining continuous contact and will generate multiple localized pressure or friction triggers; therefore, no upper limit is set on collision density. Restricted collision types do not allow collision density to reach a collision density threshold. This is suitable for actions where a single minor collision may not immediately cause a serious malfunction, but repeated collisions within a short period can lead to performance degradation, localized jamming, or increased environmental risk. For example, when an embodied robot performs a "slow movement along the edge" action in a narrow space, obstacles may occasionally lightly touch the edge of the embodied robot, but continuous pressure or jerky multiple touches within a short period are not allowed. Therefore, this type of high-density collision represents robot instability or deviation from the movement path. Prohibited collision types mean that collision events are prohibited during the execution of the action. For example, in the action of "navigating to a table," collision events are not allowed.

[0047] For example, the action identifiers of actions with allowed collision types, restricted collision types, or prohibited collision types can be recorded in the action set.

[0048] Step 206: Generate collision density information corresponding to the grid map of the embodied robot based on the first subsequence within the first sliding time window in the event information sequence.

[0049] The collision density information includes the collision density corresponding to each cell in the grid map. The grid map is a grid-based map created in the local coordinate system of the embodied robot; therefore, it can also be called a local grid map. The cell position is the position of the cell's center point, which can be represented as (x, y). For example, the actual physical coordinate range of a cell in the grid map can be: The unit is meters (m). Since the position of the embodied robot is dynamically changing, the local coordinate system is also dynamically changing, and consequently, the local grid map is also dynamically changing. Of course, the grid map can also be a global grid map built in a global coordinate system.

[0050] It should be noted that the grid map and the collision locations recorded by the embodied robot use the same coordinate system.

[0051] The collision density corresponding to a grid cell reflects the frequency of collisions occurring at that cell. Collision density is positively correlated with frequency; a higher collision density indicates a higher frequency of collisions. Collision density information can be in the form of a two-dimensional matrix or displayed as a heatmap.

[0052] The first sliding time window has a duration of one fixed time interval. The sliding time window is a time interval with a fixed (or variable) duration that continuously moves forward on the timeline. The first duration can be set according to actual needs; for example, the first duration can be 10 seconds. The first sliding time window can be represented as […]. , ],in, Represents the current time. This represents the first duration. The first sliding time window changes as the current time changes.

[0053] For example, the start time of the first sliding time window (denoted as the first start time) and the end time of the first sliding time window (denoted as the first end time) can be determined, and the sequence of event information whose collision time falls between the first start time and the first end time in the event information sequence can be used as the first sub-sequence.

[0054] For example, during the execution of an action sequence, if a collision event occurs during the execution of the current action, event information is generated and added to the event information sequence. Furthermore, if the collision restriction type of the current action is restricted collision, collision density information corresponding to the embodied robot's grid map is generated based on the first subsequence within the first sliding time window in the event information sequence. If the collision restriction type of the current action is prohibited collision, a safety action is directly triggered if a collision event occurs.

[0055] For example, if the collision restriction type of the current action is a restricted collision type, then the collision density information corresponding to the grid map of the embodied robot is generated according to the first subsequence within the first sliding time window in the event information sequence. If the collision restriction type of the current action is a permitted collision type, then it is determined whether a safety action needs to be triggered according to the second subsequence within the second sliding time window in the event information sequence.

[0056] In an exemplary embodiment, before executing the corresponding actions in the order of the action sequence to perform the task indicated by the task instruction, the method further includes: determining the collision restriction type of each action in the action sequence, wherein the collision restriction type includes a restricted collision type or a prohibited collision type, wherein a restricted collision type means that the collision density is not allowed to reach a collision density threshold during the execution of the action, and a prohibited collision type means that collision events are prohibited during the execution of the action; generating collision density information corresponding to the grid map of the embodied robot based on the first subsequence within the first sliding time window in the event information sequence includes: for the currently executed action, if the collision restriction type of the current action is a restricted collision type, determining the first subsequence generated during the execution of the current action and within the first sliding time window from the event information sequence; and generating collision density information corresponding to the grid map of the embodied robot based on the first subsequence.

[0057] For example, “determining the first subsequence generated during the execution of the current action and within the first sliding time window from the event information sequence” includes: taking the sequence of event information in the event information sequence whose collision time falls between the first start time and the first end time, and whose collision restriction type of the current action being executed is a restricted collision type, as the first subsequence.

[0058] For example, generating collision density information corresponding to the grid map of the embodied robot based on the first sub-sequence includes: for each cell in the grid map, determining the collision density corresponding to the cell based on the collision position in the first sub-sequence and the cell position of the cell; and obtaining the collision density information corresponding to the grid map based on the collision density corresponding to each cell in the grid map.

[0059] In this embodiment, since the contact characteristics of different actions vary greatly, the accuracy of collision judgment is improved by determining the collision restriction type of each action in the action sequence. Furthermore, since the first sub-sequence is generated during the execution of the current action and within the first sliding time window, the collision density information corresponding to the grid map of the embodied robot is generated based on the first sub-sequence. This eliminates interference from event information generated in other actions, thereby improving the accuracy of the collision density information.

[0060] Step 208: If the collision density information characterization meets the conditions for executing the first safety action, execute the safety action.

[0061] The conditions for executing the first safety action may include at least one of the following: the highest collision density in the collision density information is greater than a collision density threshold, or the average collision density in the collision density information is greater than an average density threshold. The collision density threshold and the average density threshold can be set according to actual needs.

[0062] A safety action can be a safety reversal action. For example, a safety action could be: stop moving → reverse a fixed distance → rotate to a safe direction → send a local replanning request to the robot's task planning module. The local replanning request is used to request replanning for the current local path or the current action execution process. If local replanning cannot eliminate collision conflicts, a higher-level task replanning, i.e., global replanning, can be initiated to replan the currently executing task.

[0063] In an exemplary embodiment, if the collision density information characterization meets the first safety action execution condition, a safety action is executed, including: determining a reference collision density under a first sliding time window based on the collision density contained in the collision density information; and executing a safety action when the reference collision density reaches a collision density threshold.

[0064] The collision density threshold can be set according to actual needs or determined experimentally. For example, collision experiments can be conducted at fixed locations, and the collision density can be calculated and updated in real time. When the robot malfunctions or experiences an anomaly due to a collision, the collision is stopped, and the collision density recorded at the time of cessation is taken as the critical collision density. The collision density threshold is determined based on the critical collision density and a preset coefficient. For example, the product of the critical collision density and the preset coefficient can be used as the collision density threshold. The preset coefficient ranges from 0 to 1 and can be a value close to 1, such as 0.8, 0.9, or 0.95.

[0065] For example, the highest collision density in the collision density information can be used as the reference collision density. Alternatively, the average value of the collision densities in the collision density information can be used as the reference collision density.

[0066] In this embodiment, when the reference collision density reaches the collision density threshold, a safety action is performed, thereby preventing risks caused by frequent collisions and ensuring the safety of the embodied robot.

[0067] In an exemplary embodiment, determining a reference collision density within a first sliding time window based on the collision density contained in the collision density information includes: determining the maximum collision density from a plurality of target collision densities contained in the collision density information, and using the maximum collision density as the reference collision density.

[0068] Among them, the collision density of multiple targets is the total collision density in the collision density information, or the collision density of multiple targets is the collision density of multiple grids in the target area contained in the raster map.

[0069] For example, the target area is a portion of a grid map, and the embodied robot is currently located within the target area. For instance, the target area could be an area with a length equal to a preset length, a width equal to a preset width, and centered on the embodied robot's current position. The preset length and preset width can be set according to actual needs.

[0070] In this embodiment, the maximum collision density is used as the reference collision density, so as to avoid performing safety actions when the collision density is low, thus avoiding unnecessary safety actions and only performing safety actions when necessary, thereby improving task execution efficiency.

[0071] In an exemplary embodiment, during the execution of the task indicated by the task instruction by performing corresponding actions in the order of the action sequence, the method further includes: if a collision event is detected by the collision switch of the embodied robot, and the collision restriction type of the action performed when the collision event occurs is a prohibited collision type, then a safety action is immediately executed. For example, if the collision restriction type of the current action is a prohibited collision type, and a collision event occurs during the execution of the current action, a safety action is executed. Wherein, if the collision restriction type of the current action is a prohibited collision type, a safety action is executed as soon as a collision event occurs. For example, when performing the action of "navigating to the table," this action corresponds to a prohibited collision type; therefore, as the embodied robot moves towards the table, a safety action is immediately executed as soon as a collision event occurs. In this embodiment, for actions of the prohibited collision type, a safety action is executed if a collision event occurs, thereby reducing risk and ensuring safety.

[0072] In the aforementioned method for executing safe actions on the embodied robot, event information from multiple collision events detected by a collision switch is recorded to form an event information sequence. The event information in this sequence is arranged chronologically and includes the collision time and location. Based on the first subsequence within the first sliding time window of the event information sequence, collision density information corresponding to the embodied robot's grid map is generated. When the collision density information indicates that the first safe action execution condition is met, a safe action is executed. Thus, by combining the first sliding time window, the event information sequence, and the collision density, collision density information is generated to reflect the frequency and location distribution of collisions. Based on this collision density information, a safe action is triggered. This allows the embodied robot to identify and respond to abnormal contact using a low-cost but information-limited collision switch, even in the absence of force sensors, thereby reducing the hardware cost of the embodied robot.

[0073] In an exemplary embodiment, generating collision density information corresponding to the grid map of the embodied robot based on a first subsequence in the event information sequence within a first sliding time window includes: determining the first subsequence within the first sliding time window from the event information sequence; determining the collision density corresponding to each cell in the grid map based on the collision position in the first subsequence and the cell position of the cell; and obtaining the collision density information corresponding to the grid map based on the collision density corresponding to each cell in the grid map.

[0074] For example, the collision density of a cell can be determined based on the collision locations and cell positions contained in the event information of the first subsequence. The cell position represents the location of the cell's center point in the raster map, and also its location in the local coordinate system.

[0075] For example, the collision density of a lattice can be determined using the KDE (Kernel Density Estimation) algorithm. For instance, if all collision locations in the first subsequence are... Select a KDE bandwidth ( The collision density of each cell in a raster map can be estimated using the following formula:

[0076]

[0077] in, Let (x, y) be the collision position in the first subsequence, and exp be the lattice position. Represents an exponential function. This represents the standard deviation (also the bandwidth). The above formula uses an unnormalized Gaussian kernel function, the purpose of which is to measure the relative collision density of each grid cell. The normalization constant does not affect the determination of the collision density distribution.

[0078] For example, determining the first subsequence within the first sliding time window from the event information sequence includes: for the current action being performed, if the collision restriction type of the current action is a collision-allowed type, determining the first subsequence generated during the execution of the current action and within the first sliding time window from the event information sequence.

[0079] In this embodiment, the collision density corresponding to the cell is determined based on the event information and cell position in the first sub-sequence. The collision density information corresponding to the grid map is obtained according to the collision density corresponding to each cell in the grid map. The process of determining the collision density information has low complexity, thereby improving the efficiency of obtaining the collision density information.

[0080] In an exemplary embodiment, after receiving the task instruction, the method further includes: determining a second subsequence within a second sliding time window from the event information sequence, wherein the second duration of the second sliding time window is greater than the first duration of the first sliding time window; and performing a safety action if the second subsequence indicates that the second safety action execution condition is met.

[0081] Wherein, the duration of the second sliding time window is the second duration, and the second sliding time window is a sliding time window with a duration of the second duration. The second sliding time window can be represented as [ , ],in, Represents the current time. This represents the second duration. The second sliding time window changes as the current time changes. .

[0082] For example, the start time (denoted as the second start time) and the end time (denoted as the second end time) of the second sliding time window can be determined, and the sequence of event information whose collision time falls between the second start time and the second end time in the event information sequence can be used as the second subsequence.

[0083] For example, when the collision restriction type of the current action is a restricted collision type, collision density information corresponding to the grid map of the embodied robot is generated based on the first subsequence within the first sliding time window in the event information sequence. A second subsequence within the second sliding time window is determined from the event information sequence. If the collision density information indicates that the first safe action execution condition is not met, but the second subsequence indicates that the second safe action execution condition is met, then a safe action is executed. When the collision restriction type of the current action is a permitted collision type, a second subsequence within the second sliding time window is determined from the event information sequence. If the second subsequence indicates that the second safe action execution condition is met, then a safe action is executed. Furthermore, the process of generating the collision density is unnecessary.

[0084] In this embodiment, since the second duration of the second sliding time window is longer than the first duration of the first sliding time window, the first sliding time window is more suitable for collision cluster detection in a shorter time and a more local area, while the second sliding time window is more suitable for monitoring the overall collision frequency over a longer period of time to detect systemic degradation. By supporting the determination of whether to perform safety actions through windows of different durations, the determination of whether to perform safety actions is more comprehensive.

[0085] In an exemplary embodiment, when the second subsequence characterization satisfies the second safety action execution condition, a safety action is executed, including: determining the number of collisions and at least one collision time interval within the second sliding time window according to the second subsequence; wherein, the collision time interval is the time interval between two adjacent collision events; if the number of collisions is greater than a threshold or any collision time interval is less than a preset interval threshold, a safety action is triggered.

[0086] The number of times threshold and the preset interval threshold can be set according to actual needs.

[0087] For example, since each event in the event information sequence corresponds to a collision event, each event in the second subsequence also corresponds to a collision event. Therefore, the length of the second subsequence represents the number of collision events, and also the number of collision events occurring within the second sliding time window. Thus, the number of collisions corresponding to the second sliding time window can be determined based on the length of the second subsequence.

[0088] For example, since the event information includes the collision time, the collision time interval between adjacent collision events can be calculated based on the collision time in the adjacent event information.

[0089] For example, the collision time interval between any two adjacent collision events within the second sliding time window can be determined. If any collision time interval is less than a preset interval threshold, a safety action is triggered. Here, a collision time interval less than the preset interval threshold can be understood as continuous collisions.

[0090] In this embodiment, the system monitors the overall collision frequency from a time dimension to identify whether the embodied robot has entered a state of overall performance degradation or continuous failure. Even if the current action is not a prohibited action and the collision has not formed a local hotspot (i.e., the reference collision density has not reached the collision density threshold), as long as the collision event is too frequent within the second sliding time window, a safety action will still be triggered, further improving safety.

[0091] In an exemplary embodiment, after performing the safety action, the method further includes: obtaining current second scene information, generating a risk record based on the recorded event information and the second scene information, the risk record being used to assist the embodied robot in task planning.

[0092] The second scene information can be collected by the embodied robot's visual perception module or by independent cameras deployed in the scene. Task planning can be local replanning (i.e., replanning the current action) or global replanning (i.e., replanning the task to be performed).

[0093] For example, action identifiers, collision events, and collision locations can be extracted from the event information sequence, and visual scene summaries can be extracted from the scene information collected by the visual perception module. The action identifiers, collision events, collision locations, and visual scene summaries can then be combined to form a risk record.

[0094] For example, in addition to generating risk records and supporting local or global replanning, the event information sequence can also be used to determine whether collisions are spatially clustered, i.e., whether they repeatedly occur in similar local areas. It can be used to distinguish between "occasional single collisions" and "repeated collisions in local danger zones". It can be written into a structured fault log, supporting offline playback, reproduction and subsequent analysis, and supporting auditable fault closure after execution errors.

[0095] In this embodiment, risk records are used to assist the embodied robot in task planning, thereby improving the accuracy of task planning.

[0096] In one exemplary embodiment, such as Figure 3 As shown, a flowchart of a method for performing safe actions by an embodied robot is provided, including the following steps:

[0097] Step 302: Receive task instructions, and generate an action sequence based on task instructions and the current first scene information.

[0098] Step 304: Determine the collision constraint type for each action in the action sequence.

[0099] Step 306: Execute the actions in the action sequence in order.

[0100] Step 308: Determine whether a collision event has occurred during the execution of the current action. If yes, proceed to step 310; otherwise, return to step 308.

[0101] The current action is the action currently being performed.

[0102] Step 310: Record the event information of the collision event into the event information sequence.

[0103] Step 312: Determine whether the collision restriction type of the current action is prohibited, allowed, or restricted. If it is prohibited, proceed to step 322; if it is restricted, proceed to step 314; if it is allowed, proceed to step 318.

[0104] Step 314: Generate collision density information corresponding to the grid map of the embodied robot based on the first subsequence within the first sliding time window in the event information sequence.

[0105] Step 316: Determine whether the collision density in the collision density information indicates that the first safety action execution condition is met. If yes, proceed to step 322; otherwise, proceed to step 318.

[0106] Step 318: Determine the second subsequence within the second sliding time window from the event information sequence.

[0107] Step 320: Determine whether the second subsequence represents the fulfillment of the second safety action execution condition. If yes, proceed to step 322; otherwise, return to step 318.

[0108] Step 322: Perform safety actions.

[0109] In an exemplary embodiment, a method for executing safe actions by an embodied robot is provided. The embodied robot includes a task planning module, a skill execution module, a visual perception module, and a deterministic safety loop module. The task planning module, which can be based on an LLM (Large Language Model) or an agent, receives user instructions and a summary of the current scene (generated by the visual perception module) and outputs an action sequence. The skill execution module maps actions to low-level controls (navigation, grasping, opening doors, etc.) and outputs control commands. The visual perception module provides continuous time-series visual frames and can be implemented using a monocular RGB camera or an RGBD depth camera. The collision switch event module outputs collision events (touch events reported by switches, bumpers, or controllers). The deterministic safety loop module periodically reads visual frames and collision events, calculates risk indicators and health scores, triggers safety actions, and overrides control commands with hard priority. Specifically:

[0110] The task planning module receives the user's task instructions and the current scene summary output by the visual perception module. It outputs an action sequence, using action IDs as identifiers, and selects actions where collision events are not allowed, or where the density of unrestricted collision events exceeds a threshold. The selected action IDs are then added to the action set (e.g., ...). ).

[0111] The current scene summary refers to the information of the current first scene.

[0112] The skill execution module outputs and executes control commands based on the action sequence. If a collision event occurs during the execution of the action sequence, the deterministic safety loop module records the event information of the collision event in the event information sequence.

[0113] For the currently executed action, if the deterministic safety loop module determines that the current action is not allowed to cause a collision event, then if a collision event occurs during the execution of the current action, a safety action will be triggered.

[0114] The statement "Collision events are not allowed" indicates that the collision restriction type for the current action is the prohibited collision type.

[0115] For the currently executed action, if the deterministic safety loop module determines that the density of unrestricted collision events for the current action exceeds a threshold, then a collision heatmap is accumulated in the local coordinate system of the embodied robot. The collision density is calculated using kernel density estimation. If the local hotspot density exceeds the collision density threshold within the first sliding time window (e.g., ...), ... If this happens, a safety action will be triggered.

[0116] Here, "the density of unrestricted collision events for the current action exceeds the threshold" means that the collision restriction type for the current action is a restricted collision type. "Local hotspot density" refers to the reference collision density. "Accumulating a collision heatmap in the local coordinate system of the embodied robot and calculating the collision density through kernel density estimation" means generating collision density information.

[0117] Risk records are extracted from the local collision heatmap in the local coordinate system of the embodied robot, and these risk records are matched with the scene currently collected by the visual perception module and saved so that historical collision experience can be used to improve the safety of subsequent navigation.

[0118] The "local collision heatmap in the local coordinate system of the embodied robot" refers to the collision density information generated based on the first subsequence within the first sliding time window. Instead of saving the local collision map (i.e., the local grid map), risk records can be saved to reduce storage usage and minimize the use of outdated local grid maps, thus saving storage space.

[0119] Count the total number of collision events within the second sliding time window. ,like Exceeding the threshold or the interval between adjacent collisions is less than If the action ID is not in the F_steps set, then the safety action will be triggered.

[0120] The total number of collision events refers to the number of collisions. This represents a preset interval threshold. The adjacent collision interval refers to the time interval between adjacent collision events.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0122] In an exemplary embodiment, an embodied robot system is provided. The embodied robot system is used to: receive a task instruction and execute the task indicated by the task instruction; during the execution of the task, if a collision event is detected by the collision switch of the embodied robot, the event information of the collision event is recorded, and multiple event information are arranged in chronological order to form an event information sequence, the event information including the collision location and the corresponding collision time; based on the first subsequence within the first sliding time window in the event information sequence, collision density information corresponding to the grid map of the embodied robot is generated, the collision density information including the collision density corresponding to each cell in the grid map; and if the collision density information indicates that a first safety action execution condition is met, a safety action is executed.

[0123] Based on the same inventive concept, this application also provides a hymenoid robot safety action execution device for implementing the aforementioned hymenoid robot safety action execution method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the hymenoid robot safety action execution device provided below can be found in the limitations of the hymenoid robot safety action execution method described above, and will not be repeated here.

[0124] In one exemplary embodiment, such as Figure 4 As shown, a safety action execution device for an embodied robot is provided, comprising: a task execution module 402, an event information recording module 404, a collision density information generation module 406, and a safety action execution module 408, wherein:

[0125] The task execution module 402 is used to receive task instructions and execute the tasks indicated by the task instructions.

[0126] The event information recording module 404 is used to record the event information of the collision event if a collision event is detected by the collision switch of the embodied robot during the execution of the task. Multiple event information are arranged in chronological order to form an event information sequence. The event information includes the collision location and the corresponding collision time.

[0127] The collision density information generation module 406 is used to generate collision density information corresponding to the grid map of the embodied robot based on the first subsequence within the first sliding time window in the event information sequence. The collision density information includes the collision density corresponding to each cell in the grid map.

[0128] The safety action execution module 408 is used to execute a safety action when the collision density information characterization meets the first safety action execution condition.

[0129] Each module in the aforementioned embodied robot safety action execution device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0130] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. The display unit is used to create a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0131] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0133] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0134] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0135] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for performing safe actions of an embodied robot, characterized in that, The method includes: Receive task instructions and execute the tasks indicated by the task instructions; During the execution of the task, if a collision event is detected by the collision switch of the embodied robot, the event information of the collision event is recorded. Multiple event information are arranged in chronological order to form an event information sequence. The event information includes the collision location and the corresponding collision time. Based on the first subsequence within the first sliding time window in the event information sequence, collision density information corresponding to the grid map of the embodied robot is generated, and the collision density information includes the collision density corresponding to each cell in the grid map; If the collision density information characterization satisfies the first safety action execution condition, a safety action is executed.

2. The method according to claim 1, characterized in that, The step of generating collision density information corresponding to the grid map of the embodied robot based on the first sub-sequence within the first sliding time window in the event information sequence includes: Determine the first sub-sequence within the first sliding time window from the event information sequence; For each cell in the grid map, the collision density corresponding to the cell is determined based on the collision positions in the first sub-sequence and the cell position of the cell. Based on the collision density of each cell in the grid map, the collision density information corresponding to the grid map is obtained.

3. The method according to claim 1, characterized in that, The step of executing a safety action when the collision density information characterization satisfies the first safety action execution condition includes: Based on the collision density contained in the collision density information, a reference collision density is determined within the first sliding time window; The safety action is performed when the reference collision density reaches the collision density threshold.

4. The method according to claim 3, characterized in that, Determining the reference collision density within the first sliding time window based on the collision density information includes: The maximum collision density is determined from the multiple target collision densities included in the collision density information, and the maximum collision density is used as the reference collision density. Wherein, the multiple target collision densities are all the collision densities in the collision density information, or the multiple target collision densities are the collision densities corresponding to multiple grids in the target area contained in the grid map.

5. The method according to any one of claims 1 to 4, characterized in that, After receiving the task instruction, the method further includes: A second subsequence within a second sliding time window is determined from the event information sequence, wherein the second duration of the second sliding time window is greater than the first duration of the first sliding time window; If the second subsequence characterization satisfies the second safety action execution condition, the safety action is executed.

6. The method according to claim 5, characterized in that, The step of executing a safety action when the second sub-sequence representation satisfies the second safety action execution condition includes: Based on the second sub-sequence, determine the number of collisions and at least one collision time interval within the second sliding time window; wherein, the collision time interval is the time interval between two adjacent collision events; If the number of collisions exceeds a threshold or any of the collision time intervals is less than a preset interval threshold, the safety action is triggered.

7. The method according to any one of claims 1 to 4, characterized in that, The task to be executed as indicated by the task instruction includes: Based on the task instructions and the current first scene information, task planning is performed to generate an action sequence; The corresponding actions are executed in the order of the action sequence to perform the task indicated by the task instruction.

8. The method according to claim 7, characterized in that, Before executing the corresponding actions in the order of the action sequence to perform the task indicated by the task instruction, the method further includes: Determine the collision restriction type for each action in the action sequence. The collision restriction type includes a restricted collision type or a prohibited collision type. The restricted collision type means that the collision density is not allowed to reach a collision density threshold during the execution of the action. The prohibited collision type means that collision events are prohibited during the execution of the action. The step of generating collision density information corresponding to the grid map of the embodied robot based on the first sub-sequence within the first sliding time window in the event information sequence includes: For the current action being executed, if the collision restriction type of the current action is a restricted collision type, determine from the event information sequence the first sub-sequence generated during the execution of the current action and within the first sliding time window; Based on the first sub-sequence, collision density information corresponding to the grid map of the embodied robot is generated.

9. The method according to claim 8, characterized in that, In the process of executing the corresponding actions in the order of the action sequence to perform the task indicated by the task instruction, the method further includes: If a collision event is detected by the collision switch of the embodied robot, and the collision restriction type of the action to be performed when the collision event occurs is the collision prohibition type, then a safety action is immediately executed.

10. The method according to any one of claims 1 to 4, characterized in that, After performing the safety action, the method further includes: The system acquires current second scene information and generates a risk record based on the recorded event information and the second scene information. The risk record is used to assist the embodied robot in task planning.

11. A hymenoidae robot system, characterized in that, The device includes a terminal and an embodied robot, the terminal being used to send task instructions to the embodied robot, and the embodied robot being used to perform the method of any one of claims 1 to 10.

Images