An agent-based industrial XR inverse control system

Abstract

The application belongs to the technical field of virtual reality, and particularly relates to an industrial XR reverse control system based on an intelligent agent, which comprises: a sensor network for collecting multi-source perception data of field equipment; a time synchronization unit for attaching a unified timestamp to the multi-source perception data and outputting time-aligned data streams; a state fusioner for performing fusion processing on the time-aligned data streams and outputting an equipment state vector; a digital twin engine for refreshing the attributes of a virtual equipment model according to the equipment state vector and completing state synchronization between the physical equipment and the virtual equipment model; and an extended reality head-mounted device for collecting interactive operation data of the virtual equipment model by an operator. The application realizes reverse control from extended reality virtual operation to actual execution of the physical equipment, and has beneficial effects of natural and intuitive human-computer interaction, deep fusion of task decomposition and safety constraints, robust and reliable trajectory optimization, and modular and expandable system architecture.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality technology, specifically relating to an industrial XR reverse control system based on intelligent agents. Background Technology

[0002] Remote control technology for industrial robots is an important research direction in the field of intelligent manufacturing. Its core objective is to enable operators to complete complex control tasks remotely from hazardous working environments. With the rapid development of extended reality technology, combining extended reality with industrial robot control has become an effective way to improve human-machine interaction efficiency and intuitive operation. Extended reality technology presents a virtual environment to the operator through head-mounted displays or overlays virtual information onto the real environment, allowing the operator to interact with virtual objects in a natural way, thereby achieving indirect control of physical equipment.

[0003] Digital twin technology provides crucial technical support for extended reality applications in industry. A digital twin is a virtual mapping of a physical entity in digital space. By collecting real-time status data of physical equipment and synchronously updating the virtual model, the virtual model accurately reflects the current state of the physical equipment. In the field of industrial robotics, digital twin technology has been widely applied in scenarios such as equipment status monitoring, fault prediction, and offline programming. However, most existing digital twin applications focus on the forward mapping from physical space to virtual space, that is, using sensor data to drive the status updates of the virtual model, while the reverse control link from virtual space to physical space has not been fully studied.

[0004] In terms of reverse control technology, existing solutions typically employ direct mapping to convert operator actions in a virtual environment into robot control commands. While this direct mapping method is simple to implement, it has several inherent drawbacks. First, the operator's hand movements are often not smooth enough, containing jitter and discontinuities. Direct mapping can cause the robot to exhibit jerky movements, affecting execution accuracy and equipment lifespan. Second, operators struggle to accurately perceive and control the robot's speed, acceleration, and contact force during interaction, easily leading to movement exceeding limits or collisions. Third, the direct mapping method lacks an understanding of task semantics and cannot automatically optimize motion trajectories or handle abnormal situations based on task objectives. Summary of the Invention

[0005] Therefore, the main objective of this invention is to provide an industrial XR reverse control system based on intelligent agents, which realizes reverse control from extended reality virtual operation to actual execution by physical equipment. It has the beneficial effects of natural and intuitive human-computer interaction, deep integration of task decomposition and safety constraints, robust and reliable trajectory optimization, and modular and scalable system architecture.

[0006] The technical solution adopted in this invention is as follows:

[0007] An agent-based industrial XR reverse control system includes:

[0008] Sensor networks are used to collect multi-source sensing data from field devices;

[0009] The time synchronization unit is used to attach a unified timestamp to multi-source sensing data and output a time-aligned data stream;

[0010] The state fusion unit is used to fuse time-aligned data streams and output a device state vector.

[0011] The digital twin engine is used to refresh the attributes of the virtual device model based on the device state vector, thereby synchronizing the state between the physical device and the virtual device model.

[0012] Extended reality head-mounted display devices are used to collect data on the operator's interaction with virtual device models;

[0013] The specification generates an intelligent agent, which is used to construct a high-level task description structure based on the interaction operation data, and generate signal timing logic specifications based on the high-level task description structure.

[0014] The task decomposition agent is used to perform hierarchical task network decomposition on the high-level task description structure, and inherit the signal timing logic reduction to the subtasks during the decomposition process, and output the atomic task sequence and the constraints associated with each atomic task.

[0015] The trajectory evaluation agent is used to generate a candidate trajectory set based on the atomic task sequence, select the execution trajectory from the candidate trajectory set based on the signal timing logic robustness evaluation, and discretize the execution trajectory into a control instruction sequence for output.

[0016] Furthermore, the multi-source sensing data includes joint angle data output by the joint encoder, end contact force data output by the six-dimensional force sensor, and workpiece pose data output by the vision sensor.

[0017] Furthermore, the state fusion unit obtains the end effector pose estimate by solving the joint angle data through forward kinematics, and performs coordinate registration between the end effector pose estimate and the workpiece pose data; the device state vector includes the current angle of each joint, the current angular velocity of each joint, the current position of the end effector, the current attitude of the end effector, and the current contact force of the end effector.

[0018] Furthermore, the interactive operation data includes the operator's hand movement trajectory, gesture trigger events, and voice command text; the hand movement trajectory is segmented, pause points are identified, and the movement is divided into segments with adjacent pause points as boundaries; for each movement segment, the starting position, ending position, starting posture, ending posture, and movement duration are extracted; gesture trigger events are mapped to operation type identifiers; keyword matching is performed on the voice command text to extract time constraint phrases, allowed workspace phrases, prohibited area phrases, and force constraint phrases; the high-level task description structure includes task objective fields, task type fields, operation sequence fields, time constraint fields, allowed workspace fields, prohibited area fields, and force constraint fields.

[0019] Furthermore, the specification-generating agent performs the following processes: For the allowed workspace field, a workspace predicate is constructed, defined as each component of the end effector position coordinates being within the corresponding allowed boundary coordinate range; for the prohibited region field, a prohibited region predicate is constructed for each prohibited region, defined as the end effector position not belonging to the internal set of points of the corresponding prohibited region and the shortest distance between the end effector position and the boundary of the corresponding prohibited region being greater than a preset safety gap value; the workspace predicate and all prohibited region predicates are combined using a conjunction operation to form a spatial safety predicate; for the force constraint field, a force safety predicate is constructed, defined as the absolute value of each component of the end effector contact force being below the upper limit of stress; a velocity safety predicate is constructed, defined as the absolute value of each joint angular velocity being below the corresponding upper limit of velocity; the spatial safety predicate, force safety predicate, and velocity safety predicate are combined using a conjunction operation to form a safety holding predicate; a safety holding expression is constructed, using a bounded holding operator to specify that the safety holding predicate remains true throughout the entire task execution time interval.

[0020] Furthermore, the specification-generating agent also performs the following processes: constructing a target achievement predicate based on the task type; if the task type is pose arrival, the target achievement predicate is defined as the Euclidean distance between the end effector position and the target position being less than a preset position tolerance value and the angular deviation between the end effector attitude and the target attitude being less than a preset attitude tolerance value; if the task type is contact operation, the target achievement predicate is defined as the end contact force being within a preset target force range; constructing a target termination expression for the task completion time limit in the time constraint field; the target termination expression uses a bounded termination operator to specify that the target achievement predicate is true at least once within the time interval from the current time to the time limit time; and combining the safety timing expression and the target termination expression through conjunction operation to form a top-level signal timing logic reduction.

[0021] Furthermore, the task decomposition agent performs hierarchical task network-signal timing logic collaborative decomposition, including: creating the root node of the task-constraint tree; storing the task target field of the high-level task description structure into the task slot of the root node; storing the top-level signal timing logic reduction into the constraint slot of the root node; creating a record of attempted methods and initializing it as an empty set; marking the root node as a state to be decomposed; selecting the node in the state to be decomposed from the task-constraint tree as the current node; retrieving decomposition methods with the same task type as the current task from the pre-set method library; eliminating the decomposition methods in the record of attempted methods to form a candidate method set; each decomposition method in the method library includes a list of preconditions, a sequence of subtasks, the temporal relationship between subtasks, and a target achievement subtask marker.

[0022] Furthermore, precondition checks and constraint compatibility checks are performed on each candidate decomposition method in the candidate method set. The constraint compatibility check includes: calculating the relative start and end times of each subtask based on the temporal relationship between subtasks; dividing the time interval of the safety endpoint expression into sub-intervals according to the subtasks and generating safety inheritance constraints; assigning the target endpoint expression to the subtasks indicated by the target achievement subtask markers or defaulting to the last subtask as target inheritance constraints; and checking whether the time interval of the safety inheritance constraints for each subtask is a non-empty interval. If the candidate method set is empty and the current node is the root node, the task is output as infeasible; if the current node is not the root node, backtracking is performed. Redirect to the parent node; if the candidate method set is not empty, select the candidate decomposition method with the fewest subtasks as the selected decomposition method and add the selected decomposition method to the tried method record; create a child node for each subtask in the subtask sequence of the selected decomposition method, store the subtask in the task slot of the child node, store the safe inheritance constraint in the constraint slot of the child node, and for subtasks assigned target inheritance constraints, also store the target inheritance constraints; determine whether the task of each child node is an atomic task and mark it as a leaf node state or a state to be decomposed accordingly; if there is a node in a state to be decomposed, continue iterative decomposition; otherwise, traverse all leaf nodes in depth-first order and output the atomic task sequence and the constraints associated with each atomic task.

[0023] Furthermore, the process of generating a candidate trajectory set by the trajectory evaluation agent includes: retrieving motion primitive templates from a pre-set motion primitive library according to the task type of the current atomic task; instantiating a reference trajectory by taking the current end effector position and attitude as the motion start point and the target position and attitude as the motion end point; the reference trajectory consists of a sequence of sampling times and the corresponding target position and attitude sequences; establishing a local coordinate system with the motion start point as the origin and the first axis pointing from the motion start point to the motion end point; defining the second and third axes as lateral offset axes; offsetting each sampling point of the reference trajectory along the lateral offset axis by a distance value from a pre-set offset distance set; generating the corresponding attitude for the offset sampling points through spherical linear interpolation; generating an offset trajectory for each set of offset sampling points through cubic spline interpolation; and combining the reference trajectory with all offset trajectories to form an initial candidate trajectory set; performing inverse kinematics solution and joint constraint check on each candidate trajectory in the initial candidate trajectory set; and eliminating candidate trajectories that cannot be solved or violate joint constraints to obtain a feasible candidate trajectory set.

[0024] By adopting the above technical solution, the present invention has produced the following beneficial effects: The present invention collects the operator's hand movement trajectory, gesture trigger events and voice command text by using an extended reality head-mounted display device, and transforms natural and intuitive human movements and voice into a structured high-level task description structure, which reduces the operator's dependence on professional programming skills, enabling non-professionals to conveniently operate industrial equipment, and significantly improves the operational efficiency and human-machine collaboration in industrial settings.

[0025] This invention innovatively combines hierarchical task network decomposition with signal temporal logic reduction inheritance. By having the task decomposition agent synchronously transmit the safety endpoint expression and the target endpoint expression to each subtask during the recursive decomposition process, it ensures that the decomposed atomic task sequence strictly adheres to safety constraints while satisfying the task objectives. This fundamentally avoids the constraint mismatch problem caused by the separation of task decomposition and safety verification in traditional methods.

[0026] This invention utilizes a quantitative evaluation method based on signal timing logic robustness for trajectory evaluation agents. This method systematically screens and optimizes candidate trajectory sets, not only determining whether a trajectory meets the specification but also quantifying the degree of satisfaction and selecting the execution trajectory with the largest safety margin. This enhances the system's robustness to execution errors and external disturbances, thereby improving the reliability of industrial field operations. Attached Figure Description

[0027] Figure 1 A schematic diagram illustrating the principle of hierarchical task network and signal timing logic collaborative decomposition provided in this embodiment of the invention;

[0028] Figure 2 A schematic diagram of spatial constraint geometry provided for an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram illustrating the principle of signed distance calculation provided in an embodiment of the present invention. Detailed Implementation

[0030] An agent-based industrial XR reverse control system includes:

[0031] Sensor networks are used to collect multi-source sensing data from field devices;

[0032] The time synchronization unit is used to attach a unified timestamp to multi-source sensing data and output a time-aligned data stream;

[0033] The state fusion unit is used to fuse time-aligned data streams and output a device state vector.

[0034] The digital twin engine is used to refresh the attributes of the virtual device model based on the device state vector, thereby synchronizing the state between the physical device and the virtual device model.

[0035] Extended reality head-mounted display devices are used to collect data on the operator's interaction with virtual device models;

[0036] The specification generates an intelligent agent, which is used to construct a high-level task description structure based on the interaction operation data, and generate signal timing logic specifications based on the high-level task description structure.

[0037] The task decomposition agent is used to perform hierarchical task network decomposition on the high-level task description structure, and inherit the signal timing logic reduction to the subtasks during the decomposition process, and output the atomic task sequence and the constraints associated with each atomic task.

[0038] The trajectory evaluation agent is used to generate a candidate trajectory set based on the atomic task sequence, select the execution trajectory from the candidate trajectory set based on the signal timing logic robustness evaluation, and discretize the execution trajectory into a control instruction sequence for output.

[0039] This embodiment provides a specific implementation scheme for an agent-based industrial XR reverse control system, using a remote control scenario of a six-degree-of-freedom industrial robotic arm as an example for detailed explanation. In this scenario, the operator wears an extended reality head-mounted display and interacts with a virtual device model presented in a digital twin engine to achieve precise control of the real robotic arm in physical space. A sensor network is deployed on the robotic arm itself and in its working environment to acquire real-time equipment operating status and environmental information. In this embodiment, the sensor network includes three types of sensors: joint encoders, six-dimensional force sensors, and vision sensors.

[0040] Joint encoders are installed at each joint of the robotic arm. These are absolute photoelectric encoders with a single-turn resolution of 17 bits, corresponding to 131,072 counting positions, and an angular resolution of approximately 0.00275 degrees. Each joint encoder outputs joint angle data at a sampling frequency of 4000 Hz, directly reflecting the real-time rotation angle of each joint. Absolute encoders were chosen instead of incremental encoders because they can directly acquire the current angle value after system power-on without zero-point calibration. This is crucial for the rapid startup and fault recovery of industrial XR reverse control systems.

[0041] The six-dimensional force sensor is mounted at the connection between the robotic arm's end effector and the flange. Its measurement range is set to ±500 Nm for the force component and ±50 Nm for the torque component, with a resolution of 0.1 Nm for the force component and 0.01 Nm for the torque component. The sensor outputs end-effector contact force data at a sampling frequency of 1000 Hz, including force and torque components in three directions. The reason for mounting the six-dimensional force sensor at the end flange rather than the tool end is that flange mounting avoids recalibrating the sensor when changing the end tool, and the rigid connection at the flange reduces measurement hysteresis caused by tool flexibility.

[0042] The vision sensor employs an industrial-grade binocular stereo camera, mounted above the robotic arm's workspace at a baseline distance of 120 mm. Each eye has a resolution of 1920 x 1080 pixels and a frame rate of 60 frames per second. The binocular camera outputs workpiece pose data, including the workpiece's 3D position and orientation quaternions in the world coordinate system, through a stereo matching algorithm. The reason for using binocular stereo vision instead of monocular vision or structured light is that the binocular system has better robustness in strong light environments and does not require active light source projection, thus avoiding interference with the infrared tracking system of the extended reality head-mounted display.

[0043] In an alternative implementation, the sensor network may also include laser rangefinders for monitoring the distance between the robotic arm and surrounding obstacles, or temperature sensors for monitoring the temperature rise of the joint motors. This additional sensor data can be used as an extended field of the device state vector.

[0044] Because the three types of sensors have different sampling frequencies—4000 Hz for the joint encoder, 1000 Hz for the six-dimensional force sensor, and 60 Hz for the vision sensor—directly using this asynchronous data would lead to inconsistencies in state estimation. The role of the time synchronization unit is to attach a unified time reference to all sensor data, enabling subsequent processing to correctly correlate multi-source data at the same physical moment.

[0045] The time synchronization unit uses the IEEE 1588 precision time protocol to synchronize the clocks of each sensor node, achieving a synchronization accuracy better than 1 microsecond. The specific implementation process is as follows: First, an industrial computer is designated as the master clock source in the sensor network. This computer is equipped with a temperature-controlled crystal oscillator with a frequency stability of ±0.1 ppm. Then, each sensor node acts as a slave clock, periodically exchanging timestamps with the master clock and calculating and compensating for the deviation between the local clock and the master clock. Finally, each sensor appends its current synchronization timestamp when acquiring data, outputting a time-aligned data stream.

[0046] The specific format of the time-aligned data stream is as follows: each frame of data contains a 64-bit timestamp field with nanosecond precision, along with the corresponding sensor measurement value. Since the sampling times of different sensors are not strictly aligned, the subsequent state fusion processor uses linear interpolation to unify the data from each sensor onto the same time base. For example, when it is necessary to obtain joint angle data at a certain moment, and that moment happens to fall between two adjacent sampling points, the state fusion processor performs linear interpolation based on the timestamps and angle values ​​of the two sampling points to calculate the estimated angle value at the target moment.

[0047] The state fusion unit receives time-aligned data streams, performs multi-sensor data fusion processing, and outputs a unified device state vector. The core steps of the fusion processing include forward kinematics calculation and coordinate registration.

[0048] The purpose of forward kinematics calculation is to calculate the position and orientation of the end effector in the base coordinate system based on the current angles of each joint. For a six-DOF robot, a modified DH parameter method is used to establish the kinematic model. Let the... The joint angles of each joint are The length of the connecting rod is The connecting rod torsion angle is The connecting rod offset is Then the first The joint relative to the first The homogeneous transformation matrix of each joint is: ;in Indicates joint rotation angle, Indicates the link Length, Indicates the link The angle of twist, Indicates the link The offset. Multiply the transformation matrices of the six joints sequentially to obtain the total transformation matrix of the end effector relative to the base coordinate system. The end pose estimate is extracted from it, which includes the three-dimensional position vector and the pose quaternion.

[0049] The purpose of coordinate registration is to transform the workpiece pose data measured by the vision sensor from the camera coordinate system to a base coordinate system that matches the results of the forward kinematics calculation. The registration process requires pre-calibrating the rigid body transformation relationship between the camera coordinate system and the robot arm's base coordinate system. This calibration is performed once during system deployment, and the calibration results are stored in the state fusion unit. Let the transformation matrix from the camera coordinate system to the base coordinate system be... The workpiece pose measured by the vision sensor is represented in the camera coordinate system as follows: Then the pose of the workpiece in the base coordinate system is: ;in This indicates the workpiece's pose in the base coordinate system. This represents the transformation matrix from the camera coordinate system to the base coordinate system. This indicates the pose of the workpiece in the camera coordinate system.

[0050] The device state vector output by the state fusion unit includes the following fields: current angle of each joint, a 6-dimensional vector in radians; current angular velocity of each joint, calculated by subtracting the angle data from two adjacent frames and dividing by the time interval, in radians per second; current position of the end effector, a 3-dimensional vector in millimeters; current attitude of the end effector, represented as a quaternion; and current contact force of the end effector, containing three force components and three torque components. The device state vector is updated at a frequency of 250 Hz, i.e., output once every 4 milliseconds.

[0051] The digital twin engine runs on an edge computing server and maintains a virtual device model of the physical robotic arm. This virtual device model uses a unified robot description format for geometric modeling and includes a 3D mesh model of each link of the robotic arm, joint motion constraints, and mass inertia parameters.

[0052] Once the digital twin engine receives the device state vector, it refreshes the joint configuration of the virtual device model based on the current angle of each joint, refreshes the pose display of the virtual model's end effector based on its current position and orientation, and renders the visualization of the driving force based on the current contact force of the end effector. The refresh process employs an incremental update strategy, that is, only updating the changed attribute fields to reduce rendering overhead.

[0053] The latency performance of state synchronization directly affects the operator's immersion and control precision. In this embodiment, the end-to-end latency from physical sensor sampling to the virtual model completing the refresh is controlled within 20 milliseconds, a latency level that meets the real-time perception requirements of the human visual system.

[0054] The operator wears an extended reality headset, which uses a video perspective mixed reality solution and features a 4K resolution display with a 110-degree horizontal x 90-degree vertical field of view and a 90Hz refresh rate. The headset has built-in hand-tracking cameras and eye-tracking sensors, supporting natural gesture interaction and gaze detection.

[0055] Extended reality head-mounted displays collect data on the operator's interaction with the virtual device model. This data consists of three components: the operator's hand movement trajectory, gesture trigger events, and voice command text.

[0056] The operator's hand movement trajectory is output by the hand tracking system at a frequency of 90 Hz. Each frame of data includes the three-dimensional position and posture of the palm center in the head-mounted display coordinate system. As the operator moves their hand in virtual space to guide the movement of the virtual robotic arm's end effector, the system continuously records the hand pose sequence. The method for segmenting the hand movement trajectory is as follows: the Euclidean distance between the hand positions of two adjacent frames is calculated. When the displacement distance of five consecutive frames is less than 2 mm, the current position is determined as a pause point. The continuous movement trajectory is divided into movement segments using adjacent pause points as boundaries. The reason for choosing 2 mm as the pause determination threshold is that even when the human hand is intentionally kept still, there is still about 1 to 2 mm of physiological tremor. This threshold can filter out tremor interference without missing meaningful micro-movements.

[0057] For each motion segment, the following features are extracted: starting position, i.e., the hand position in the first frame of the segment; ending position, i.e., the hand position in the last frame of the segment; starting pose and ending pose, represented in quaternion form; motion duration, i.e., the difference between the timestamps of the last frame and the first frame.

[0058] Gesture trigger events are generated by the hand tracking system recognizing specific gestures. In this embodiment, four operation type identifiers are defined for each gesture trigger event: a pinch gesture with the index finger and thumb is mapped to a grasping operation type identifier; releasing a pinch gesture is mapped to a release operation type identifier; pressing down with a clenched fist is mapped to a pressing operation type identifier; and rotating the wrist with a clenched fist is mapped to a rotation operation type identifier. Gesture recognition employs a keypoint detection-based method, extracting the three-dimensional coordinates of 21 key points on the hand, calculating specific finger-to-finger distances and joint angle features, and determining the gesture type through rule matching.

[0059] The voice command text is captured by the microphone array of the head-mounted display and transcribed by a speech recognition engine. The operator can describe task constraints verbally, such as "complete within 5 seconds," "do not enter the red area on the left," and "contact force not exceeding 10 Newtons." Keyword matching is performed on the voice command text to extract the following constraint phrases:

[0060] Time constraint phrases are extracted by matching keywords such as "within seconds", "before", and "time limit" and their preceding values. For example, the time constraint of 5 seconds is extracted from "complete within 5 seconds".

[0061] The phrase "allow workspace" can be extracted by matching keywords such as "within range", "within area", and "within space". For example, from "operating within a range of 200 mm above the desktop", the vertical lower limit of the allowed workspace can be extracted as the desktop height plus 200 mm.

[0062] The prohibited area phrase is extracted by matching keywords such as "do not enter", "avoid", and "prohibit" and their subsequent positional descriptions. For example, the semantic label of the prohibited area is extracted as "left red area" from "do not enter the left red area". Then, the geometric boundary coordinates corresponding to this label are queried in the digital twin engine.

[0063] Force constraint phrases are extracted by matching keywords such as "force not exceeding", "force less than", and "maximum force" and their associated values. For example, the upper limit of force is extracted as 10 Newtons from "contact force not exceeding 10 Newtons".

[0064] Based on the above information, a high-level task description structure is constructed, which includes the following fields and example values ​​in this embodiment: The task target field stores the endpoint position and endpoint posture; the task type field is determined according to the gesture trigger event. If the last gesture is a grab, the task type is a contact operation type; otherwise, it is a pose arrival type; the operation sequence field stores the start and end point information of each motion segment; the time constraint field stores the task completion time limit of 5 seconds; the allowed work space field stores the geometric boundary coordinates of the work space; the restricted area field stores the geometric boundary coordinates of each restricted area; and the force constraint field stores the upper limit of the force, 10 Newtons.

[0065] The specification-generating agent receives a high-level task description structure, reads the current state of the virtual device model in the digital twin engine, and constructs a formal signal-temporal logic specification. Signal-temporal logic is a formal specification language that can describe the behavior of continuous signals over bounded time intervals. Its advantage lies in its ability to simultaneously express security constraints (certain conditions must always be maintained) and activity constraints (certain goals must eventually be achieved), and it supports quantitative robustness calculation.

[0066] The specification construction process first generates spatial constraint predicates for the allowed job space field and the prohibited region field. The job space predicate is defined as ensuring that each component of the end effector's position coordinates lies within the corresponding allowed boundary coordinate range. Let the allowed job space be an axis-aligned bounding box, whose boundary in the base coordinate system is... direction , direction , direction The current position of the end effector is The workspace predicate holds if and only if: ;in , , , , , These represent the allowed workspace in , , Minimum and maximum boundary coordinates in three directions, The three coordinate components represent the current position of the end effector. This indicates a logical conjunction operation.

[0067] refer to Figure 2 , Figure 2 This diagram illustrates the spatial constraints and their interrelationships involved in the movement of the end effector in an agent-based industrial XR reverse control system. The diagram presents a typical robotic arm working scenario in a two-dimensional planar view, facilitating understanding of the construction principles of spatial safety predicates. Figure 2 The outermost thick solid-line rectangle represents the permissible workspace boundary, which defines the maximum range that the end effector can reach. The permissible workspace boundary is determined by preset geometric boundary coordinates, typically represented as an axis-aligned bounding box in three-dimensional space. The workspace predicate requires that every component of the end effector's position coordinates be within the corresponding permissible boundary coordinate range; that is, the end effector must not exceed this boundary at any time.

[0068] Figure 2 The two rectangular regions marked as No-Entry Zone 1 and No-Entry Zone 2, each with a diagonal fill pattern, represent obstacles or hazardous areas within the workspace. The geometric boundary coordinates of the no-entry zones are extracted from the no-entry zone field of the high-level task description structure. The no-entry predicate constructed for each no-entry zone requires that the end effector position not belong to the internal point set of the corresponding no-entry zone, meaning the end effector must not enter the interior of any no-entry zone. The dashed rectangle drawn around each no-entry zone represents the safety clearance boundary. The distance between the safety clearance boundary and the no-entry zone boundary is a preset safety clearance value. The gap is marked with a double-headed arrow in the diagram. The no-entry predicate not only requires that the end effector position not belong to the set of points inside the no-entry zone, but also requires that the shortest distance between the end effector position and the boundary of the no-entry zone be greater than the preset safety gap value. The purpose of setting a safety clearance is to provide additional buffer space for the movement of the end effector. Taking into account the geometry of the end tool and the tracking error of the position control, the safety clearance ensures that no part of the end tool will come into contact with the restricted area under the worst-case scenario.

[0069] Figure 2 The curve marked with a triangle represents the end effector trajectory, with the direction of the triangle indicating the direction of movement of the end effector. The trajectory starts from the starting point, bypasses two restricted areas, and finally reaches the ending point. The starting point is indicated by a hollow circle, and the ending point is indicated by a star. It can be observed that the end effector trajectory remains within the permitted working space boundary throughout the entire movement, while maintaining a certain distance from the safety clearance boundaries of each restricted area. Figure 2 The document also highlights two key distances: no-entry margin and workspace margin. The no-entry margin, represented by a dashed line, is the distance between the current position of the end effector and the nearest no-entry zone safety clearance boundary. The workspace margin, also represented by a dashed line, is the distance between the current position of the end effector and the permitted workspace boundary. These two margin values ​​play a crucial role in subsequent signal timing logic robustness calculations, collectively constituting a quantitative metric for the space safety predicate.

[0070] The forbidden predicate is constructed separately for each forbidden region. Let a forbidden region also be an axis-aligned bounding box with boundaries defined as follows: The preset safety gap value is The prohibition predicate holds if and only if the end effector position does not belong to the internal point set of the corresponding prohibited region and the shortest distance between the end effector position and the boundary of the corresponding prohibited region is greater than the preset safety gap value. The method for determining whether the end effector position belongs to the internal point set of the prohibited region is to check whether the three coordinate components simultaneously fall within the corresponding boundary interval. The method for calculating the shortest distance between the end effector position and the boundary of the prohibited region is as follows: first, project the end effector position along each coordinate axis onto the corresponding boundary interval to obtain the point on the prohibited region closest to the end effector position; then calculate the Euclidean distance between the end effector position and this closest point; the symbol "×" represents the Cartesian product of the interval, used to characterize the point set in three-dimensional space. Specifically, This represents the set of all three-dimensional coordinate points that satisfy the following condition: the first coordinate component takes values ​​within the interval [range]. Within, the values ​​of the second coordinate component lie in the interval Within, the third coordinate component takes values ​​within the interval In other words, this expression represents a three-dimensional spatial region defined by three coordinate intervals.

[0071] In this embodiment, the preset safety clearance value is 50 mm. This value is selected considering the maximum outer envelope radius of the end effector tool and the tracking error margin of the position control. If the outer envelope radius of the end effector tool is 30 mm and the upper limit of the steady-state error of the position control is 10 mm, then the 50 mm safety clearance can ensure that the tool surface will not come into contact with the restricted area even under the most unfavorable conditions.

[0072] The job space predicate is combined with all forbidden predicates through conjunction to form a space safety predicate. That is, a space safety predicate is true if and only if the job space predicate is true and each forbidden predicate is true.

[0073] The force safety predicate is defined as the absolute values ​​of all components of the end-contact force being below the upper limit of the stress. Let the upper limit of the force specified in the force constraint field be... The three force components of the current contact force at the end are: The force safety predicate is valid if and only if: ;in , , These represent the end contact forces at the points of contact. , , The absolute values ​​of the force components in the three directions. This indicates the preset upper limit of force. In this embodiment... Take 10 Newtons.

[0074] The speed safety predicate is defined as the absolute value of the angular velocity of each joint being below the corresponding upper speed limit. Let the first joint be... The maximum speed of each joint is The current angular velocity is The speed safety predicate holds if and only if for all 6 joints it holds. The upper speed limit of each joint is determined based on the rated speed of the motor and the reduction ratio. In this embodiment, the upper speed limit of each joint is set as follows: 180 degrees per second for joint 1, 180 degrees per second for joints 2 to 4, and 250 degrees per second for joints 5 and 6.

[0075] The space safety predicate, force safety predicate, and velocity safety predicate are combined into a safety-maintaining predicate using the conjunction operation. The safety-always expression uses the bounded-always operator to specify that the safety-maintaining predicate remains true throughout the entire task execution time interval. The semantics of the bounded-always operator is: for the time interval... Every moment within The safety-maintaining predicate takes the value of true at any given moment. This indicates the task completion time limit, which is 5 seconds in this embodiment.

[0076] The construction of the target achievement predicate depends on the task type field. If the task type is pose arrival, the target achievement predicate is defined as follows: the Euclidean distance between the end effector position and the target position is less than a preset position tolerance value, and the angular deviation between the end effector attitude and the target attitude is less than a preset attitude tolerance value. Let the target position be... The current position of the end effector is The preset position tolerance value is The conditions for achieving the position are: ;in The three coordinate components represent the target position. The three coordinate components represent the current position of the end effector. This represents the preset position tolerance value. In this embodiment... Take 5 millimeters.

[0077] Attitude deviation is measured by the rotation angle of the quaternion difference. Let the target attitude quaternion be... The current attitude quaternion of the end effector is Then the attitude error quaternion is ,in To represent quaternion multiplication, express The conjugate of . The rotation angle corresponding to the attitude error quaternion is . ,in This represents the scalar part of the attitude error quaternion. The preset attitude tolerance value is... In this embodiment, the angle is 2. The attitude achievement condition is: .

[0078] If the task type is a contact operation, then the target achievement predicate is defined as the end-effector contact force being within a preset target force range. Let the preset target force range be... The normal component of the end contact force is The conditions for achieving contact are: In this embodiment, if the task is a gripping operation, the preset target force range can be set to 5 Newtons to 15 Newtons, which means that the gripping force needs to reach a certain level to stably grip the workpiece, but it cannot be too large to avoid damaging the workpiece.

[0079] The goal-end expression uses a bounded endpoint operator to specify that the goal-end predicate must be true at least once within the time interval from the current time to the time limit. The semantics of the bounded endpoint operator is: within the time interval... Memory at at least one moment This ensures that the predicate for achieving the goal is true at that moment.

[0080] The safety-maintaining predicate and the goal-achieving predicate are combined using a conjunction operation to form a top-level signal-timing logic reduction. The overall semantics of this reduction are: the safety-maintaining predicate must always be true throughout the entire task execution, and the goal-achieving predicate must be true at least once within the task time limit. The reduction-generating agent outputs the top-level signal-timing logic reduction and passes it to the task decomposition agent.

[0081] The task decomposition agent receives the high-level task description structure and the top-level signal-temporal logic reduction, and performs hierarchical task network-signal-temporal logic collaborative decomposition. Hierarchical task networks are a planning method that recursively decomposes high-level complex tasks into low-level simple subtasks; its core data structure is a task-constraint tree. The innovation of collaborative decomposition lies in synchronously inheriting the constraints of the signal-temporal logic reduction to each subtask, ensuring that the decomposed atomic task sequence still satisfies the top-level reduction.

[0082] The first stage of the decomposition process is initialization. A root node of the task-constraint tree is created, containing three storage slots: a task slot stores the task target field of the high-level task description structure, a constraint slot stores the top-level signal timing logic reduction, and the attempted method record is initialized to an empty set. The root node is then marked as being in a state to be decomposed.

[0083] The second stage is iterative decomposition. A node in the task-constraint tree that is in a state to be decomposed is selected as the current node. In this embodiment, the current node is the root node during the initial iteration. The task slot content of the current node is read as the current task, the constraint slot content is read as the current constraint, and the tried method record is read.

[0084] The decomposition method with the same task type as the current task is retrieved from a pre-defined method library. The method library is a predefined set of decomposition rules, where each decomposition method describes how to decompose a certain type of task into a sequence of subtasks. In this embodiment, the method library contains the following decomposition methods:

[0085] For pose arrival tasks, the method library offers two methods: "linear motion decomposition" and "segmented motion decomposition". The linear motion decomposition method breaks down the pose arrival task into individual linear motion atomic tasks; the segmented motion decomposition method breaks down the pose arrival task into a sequence of linear motion subtasks that pass through intermediate points, and is suitable for situations where there are obstacles that need to be avoided.

[0086] For grasping tasks in contact operation tasks, the method library has the "approach-contact-grip decomposition method", which decomposes the grasping task into a sequence of three subtasks: approach subtask, contact subtask, and gripping subtask.

[0087] Each decomposition method in the method library contains the following fields: a list of preconditions, describing the preconditions under which the method applies; a sequence of subtasks, describing the list of subtasks generated by the decomposition; a temporal relationship between subtasks, describing whether the subtasks are executed sequentially or in parallel; and a goal-achievement subtask flag, indicating which subtask is responsible for achieving the goal-achievement predicate of the original task.

[0088] Methods already existing in the attempted method record are removed from the retrieved decomposition methods to form a candidate method set. Precondition checks and constraint compatibility checks are then performed on each candidate decomposition method in the candidate method set.

[0089] The precondition check compares each precondition in the list of candidate decomposition methods with the current digital twin state. For example, one of the preconditions for the "segmented motion decomposition method" is that the straight path between the start and end points intersects with any restricted areas. If the current digital twin state shows that the straight path does not pass through any restricted areas, then this precondition is not met, and the method is removed from the candidate set.

[0090] refer to Figure 1 , Figure 1 The system is organized using a hierarchical structure, divided into five layers from top to bottom: task layer, subtask layer, atomic task layer, constraint inheritance layer, and time axis layer. In the task layer, the rectangle at the top of the diagram represents the root node of the task constraint tree. The task slot of this root node stores the task objective field of the higher-level task description structure. The diagram uses a workpiece gripping task as an example. The root node points to three child nodes in the subtask layer via three directed arrows, indicating that the higher-level task is decomposed into three sequentially executed subtasks. The subtask layer contains three rectangles, representing the approach subtask, contact subtask, and gripping subtask from left to right. The goal of the approach subtask is to move the end effector from its current position to a preset distance above the workpiece. The goal of the contact subtask is to control the end effector to descend vertically until it makes contact with the workpiece surface. The goal of the gripping subtask is to drive the grippers to close and stably grip the workpiece. The three subtasks are executed in a strictly sequential manner; that is, the next subtask can only be started after the previous one is completed.

[0091] The atomic task layer also contains three rectangles, distinguished from the composite task nodes by their fill style. From left to right, they represent linear motion, vertical descent, and gripper closure—three atomic tasks. An atomic task is defined as one that can be directly mapped to a single motion primitive template in the motion primitive library without further decomposition. Each node in the subtask layer points to a corresponding node in the atomic task layer via a directed arrow, indicating that the subtask has been further decomposed into atomic tasks. In this embodiment, since each subtask is a simple task, each subtask is decomposed into only a single atomic task.

[0092] The constraint inheritance layer is one of the core innovations of this invention. This layer uses three dashed rectangles to represent the constraint slots associated with each atomic task. The constraint slots store sub-constraints inherited from the top-level signal-timing logic reduction. Each node in the atomic task layer points to its corresponding constraint slot in the constraint inheritance layer via a dashed directed arrow, indicating the constraint inheritance relationship. The left constraint slot stores the security inheritance constraints of the first atomic task, using a bounded start operator. Specify in time range Internal security maintenance predicate It continues to hold true, and its formal expression is as follows: The intermediate constraint slot stores the security inheritance constraints and target inheritance constraints of the second atomic task. The security inheritance constraints use the bounded start operator. Specify in time range Internal security maintains the validity of predicates, and the target inheritance constraint uses a bounded finality operator. Specify in time range Internal goal achievement predicate If true at least once, the two are combined through conjunction, and their formal expression is: The right-hand constraint slot stores the security inheritance constraints of the third atomic task, using a bounded constant operator. Specify in time range The internal security predicate continues to hold, and its formal expression is as follows: .

[0093] The timeline layer is located at the bottom of the diagram, represented by a horizontal straight line with arrows pointing in the direction of time progression. Four key moments are marked on the timeline, namely the task start time. End time of the first subtask End time of the second sub-task and task completion time limit Adjacent time points are marked with bidirectional arrows indicating the corresponding execution stage names and time intervals. Corresponding to the approach phase, time interval Corresponding contact phase, time interval This corresponds to the clamping stage. The names of each level are labeled on the left edge of the diagram to help readers understand its organizational structure.

[0094] The purpose of constraint compatibility testing is to ensure that the decomposed subtasks inherit the original constraints without causing contradictions. The specific process is as follows: Calculate the relative start and relative end times of each subtask based on their temporal relationships. For a sequentially executed sequence of subtasks, assume there are a total of... There are 10 sub-tasks, and the estimated execution time for each sub-task is 1000. Then the first The relative start times of each sub-task are The relative end time is .

[0095] For the safety time expression in the current constraint, its time interval Divide into sub-sections according to sub-tasks For each subtask, a security inheritance constraint is generated. The semantics of the security inheritance constraint is: the predicate must remain true for safety throughout the execution time interval of the subtask.

[0096] For the target endpoint expression in the current constraint, the target achievement subtask markers of the candidate decomposition methods are read. If a target achievement subtask is marked in the method, the target endpoint expression is assigned to the marked subtask as a target inheritance constraint; if no target achievement subtask is marked in the method, it is assigned to the last subtask in the subtask sequence by default. The rationale for this design is that, in most cases, the achievement of the task target occurs at the end of the entire task execution.

[0097] Check if the time interval of the safety inheritance constraint for each subtask is a non-empty interval. If the time interval of a subtask is empty, it means that the subtask has no available execution time, the constraint cannot be satisfied, and the candidate decomposition method is removed from the candidate set.

[0098] If the candidate method set is empty and the current node is the root node, the decomposition fails, outputting an infeasible task result, and prompting the operator to adjust task constraints or objectives. If the candidate method set is empty and the current node is not the root node, the process backtracks to the parent node, re-marks the parent node as pending decomposition, and returns to the second stage for execution. The backtracking mechanism ensures that when a decomposition path is infeasible, the system can try other decomposition schemes.

[0099] If the candidate method set is not empty, the candidate decomposition method with the fewest subtasks is selected as the chosen decomposition method. Selecting the method with the fewest subtasks is a greedy strategy aimed at reducing the depth of the task hierarchy and lowering the complexity of subsequent execution. The selected decomposition method is added to the tried method record of the current node to avoid repeatedly trying the same method during backtracking.

[0100] Phase 3 is node expansion. A child node is created for each subtask in the sequence of subtasks for the selected decomposition method. Subtasks are stored in the task slots of the child nodes, and safe inheritance constraints are stored in the constraint slots of the child nodes. For subtasks assigned target inheritance constraints, the target inheritance constraints are also stored in the constraint slots. A record of attempted methods is created for each child node and initialized to an empty set.

[0101] Determine whether the task of each child node is an atomic task. The criterion for an atomic task is that the task can be directly mapped to a single motion primitive template without further decomposition. In this embodiment, linear motion tasks, contact tasks, and clamping tasks are all atomic tasks. If the task of a child node is an atomic task, it is marked as a leaf node state; if it is a composite task, it is marked as a state to be decomposed.

[0102] Phase 4 is the termination check. If there are nodes in the task-constraint tree that are yet to be decomposed, return to Phase 2 to continue iterative decomposition. If all nodes are in leaf node state, the decomposition is complete. Traverse all leaf nodes in depth-first order and output the atomic task sequence and the constraints associated with each atomic task.

[0103] In this embodiment, the high-level task is assumed to be "moving the end effector to the target position to grip the workpiece," and the task type is a contact operation. The resulting atomic task sequence is as follows: the first atomic task is a proximity task, the goal of which is to move the end effector to a position 50 mm above the workpiece, and the associated constraint is a safety inheritance constraint; the second atomic task is a contact task, the goal of which is for the end effector to descend vertically to contact the workpiece surface, and the associated constraints are a safety inheritance constraint and a target inheritance constraint; the third atomic task is a gripping task, the goal of which is for the grippers to close and grip the workpiece, and the associated constraint is a safety inheritance constraint.

[0104] The trajectory evaluation agent sequentially extracts the current atomic task and its associated constraints from the atomic task sequence, reads the current state of the virtual device model in the digital twin engine, and executes the control command generation process.

[0105] First, based on the task type of the current atomic task, a motion primitive template is retrieved from a pre-defined motion primitive library. The motion primitive library stores various standardized motion trajectory templates, including linear motion primitives, circular motion primitives, spline motion primitives, etc. Each primitive template defines the parameterized trajectory form from the starting point to the ending point, as well as the velocity curve type.

[0106] The current end effector position and orientation are used as the starting point of motion, and the target position and orientation of the current atomic task are used as the ending point of motion. The motion primitive template is instantiated as a reference trajectory. The reference trajectory consists of a sequence of sampled time steps and the corresponding target position and orientation sequences. The interval of the sampled time step sequence is 4 milliseconds, consistent with the update period of the device state vector.

[0107] The process of generating a reference trajectory is illustrated using a linear motion primitive as an example. Let the starting position of the motion be... The finish line is The task duration is A trapezoidal velocity curve is adopted, with an acceleration segment accounting for 20%, a constant velocity segment for 60%, and a deceleration segment for 20%. At time... Position interpolation parameters The value is calculated based on the trapezoidal velocity curve and ranges from 0 to 1. The target position is: ;in Represents the starting position vector of the motion. Represents the position vector of the endpoint of the motion. Indicates at time Position interpolation parameters, Indicates at time The target position vector.

[0108] The target attitude is generated using quaternion spherical linear interpolation. Let the initial attitude quaternion be... The final attitude quaternion is Then at time The target pose quaternion is: ;in The quaternion angle between the starting and ending poses. This represents the dot product of two quaternions. Spherical linear interpolation is used instead of simple linear interpolation because quaternions must maintain a unit modulus constraint, and spherical linear interpolation ensures that the interpolation result is always a valid unit quaternion.

[0109] After generating the reference trajectory, a candidate trajectory set is further generated to explore the possible optimization space. The method for generating the candidate trajectory set is to apply a lateral offset based on the reference trajectory. A local coordinate system is established with the origin at the motion start point and the first axis pointing from the motion start point to the motion end point. The second and third axes are defined as lateral offset axes. The direction vector of the first axis is... The second and third axes are constructed in a manner orthogonal to the first axis.

[0110] For each sampling point on the reference trajectory, offset distances from a preset offset distance set are calculated along the positive direction of the second axis, the negative direction of the second axis, the positive direction of the third axis, and the negative direction of the third axis. In this embodiment, the preset offset distance set is set to {10 mm, 20 mm, 30 mm}. The offset sampling points are then used to generate corresponding postures through spherical linear interpolation, maintaining the same posture change pattern as the original reference trajectory. For each set of offset sampling points, an offset trajectory is generated through cubic spline interpolation. Cubic spline interpolation ensures the second-order continuity of the trajectory, resulting in smooth and uninterrupted robotic arm movement.

[0111] The reference trajectory and all offset trajectories together form the initial candidate trajectory set. In this embodiment, the four offset directions are multiplied by three offset distances to obtain 12 offset trajectories, which, together with the original reference trajectory, result in a total of 13 candidate trajectories.

[0112] For each candidate trajectory in the initial candidate trajectory set, inverse kinematics (IK) and joint constraint checks are performed. IK converts the end-effector pose in Cartesian space into joint angle configurations in joint space. For a six-DOF industrial robot, multiple IK solutions are typically available; this system selects the solution closest to the current joint configuration as the target solution. If an IK solution cannot be obtained for a sample point of a candidate trajectory (e.g., the end-effector pose exceeds the reachable workspace of the robot), the candidate trajectory is removed from the set.

[0113] The joint limit test checks whether the joint angles obtained from inverse kinematics are between the upper and lower limits of each joint's angle. In this embodiment, the angle limits for each joint are: ±170 degrees for joint 1, ±120 degrees for joint 2, ±170 degrees for joint 3, ±120 degrees for joint 4, ±170 degrees for joint 5, and ±360 degrees for joint 6. If a sampling point of a candidate trajectory violates the angle limit of any joint, the candidate trajectory is removed from the set.

[0114] After inverse kinematics solution and joint constraint test, a set of feasible candidate trajectories is obtained.

[0115] For each candidate trajectory in the feasible candidate trajectory set, calculate the safety robustness value and the target robustness value. Robustness is a quantitative measure of signal temporal logic reduction; its value reflects the degree to which the system trajectory satisfies or violates the reduction. A positive robustness value indicates that the reduction is satisfied, and a negative robustness value indicates that the reduction is violated. The larger the absolute value of the robustness, the more significant the degree of satisfaction or violation.

[0116] The candidate trajectory is input into the digital twin engine for kinematic and dynamic simulation, outputting the end-effector position, end-effector attitude, angular velocities of each joint, and predicted contact force at each sampling time. The dynamic simulation is based on the rigid body dynamics model of the robotic arm, calculating the joint angular velocities and joint accelerations according to the joint angle trajectory, and then calculating the required driving torque of each joint and the end-effector contact force using the Newton-Euler method.

[0117] The process of calculating the safety robustness value is as follows:

[0118] The inward signed distance between the end position and the boundary of the allowable workspace is calculated as the workspace margin value. The inward signed distance is defined as follows: when the end position is inside the allowable workspace, it is positive and equal to the shortest distance from the end position to the boundary; when the end position is outside the allowable workspace, it is negative and equal to the inverse of the shortest distance from the end position to the boundary. Let the allowable workspace be an axis-aligned bounding box, with the boundary being... The end position is Then the boundary distances in each direction are: ; ; ;in , , These represent the positions of the ends. , , The distances to the allowable workspace boundaries in three directions; positive values ​​indicate inside the boundary, and negative values ​​indicate outside the boundary. The workspace margin value is the minimum of the three. .

[0119] refer to Figure 3 , Figure 3 This diagram illustrates the principle of signed distance calculation, divided into two parts, left and right, which respectively explain the definitions of inward signed distance and outward signed distance. Signed distance is the foundation for calculating the robustness of signal timing logic, using positive and negative signs to distinguish between states that satisfy and violate constraints.

[0120] Figure 3 The left side illustrates the definition of inward signed distance, applicable to constraint determination of the allowable workspace boundary. The thick solid-line rectangles in the figure represent the boundary of the allowable workspace. Two test points are marked in the figure. and , representing two typical scenarios respectively.

[0121] test points Located inside the permitted work space, indicated by a hollow circle marker. From The shortest distance to the boundary of the permitted work space is represented by a dashed line segment, and this distance is denoted as . .because Within the allowed workspace, the inward signed distance is defined as a positive value, i.e. A positive inward signed distance indicates that the end effector position satisfies the job space constraints, and the magnitude of the value reflects the degree of satisfaction; the farther the distance from the boundary, the higher the robustness.

[0122] test points Located outside the permitted work space, indicated by a solid circle mark. From The shortest distance to the boundary of the permitted work space is also represented by a dashed line segment, and the absolute value of this distance is denoted as . .because For operations outside the permitted workspace, the inward signed distance is defined as a negative value, i.e., the opposite of the shortest distance, denoted as: A negative inward signed distance indicates that the end effector position violates the jobspace constraints, and the absolute value of the value reflects the degree of violation; the farther away from the boundary, the more severe the violation.

[0123] Figure 3The right side illustrates the definition of outward signed distance, applicable to constraint determination of restricted areas. The rectangular area with a diagonal fill pattern represents the restricted area, and the dashed rectangular frame surrounding the restricted area represents the safety clearance boundary. The distance between them is the preset safety clearance value. The diagram shows three test points. , and , representing three typical scenarios respectively.

[0124] test points Located outside the safety clearance boundary, it is indicated by a hollow circular mark. From The shortest distance to the boundary of the restricted area is ,because Not only outside the restricted area, but also at a distance greater than the preset safety gap value from the boundary of the restricted area, the outward signed distance is defined as... A positive value indicates that the no-entry constraint is satisfied.

[0125] test points The area located between the safety clearance boundary and the restricted area boundary is indicated by a hollow square mark. Although Outside the restricted area, but at a distance less than the preset safety gap value from the boundary of the restricted area. Therefore, the outward signed distance is negative, indicating a violation of the safety clearance requirement in the no-entry constraint.

[0126] test points Located inside the restricted area, indicated by a solid circle. From The shortest distance to the boundary of the restricted area is denoted as The outward signed distance is defined as the negative of the distance, i.e. This indicates a serious violation of the no-entry constraint. This definition of signed distance allows for a unified approach to robustness calculations across three scenarios: inside the no-entry zone, within the safe gap, and outside the safe zone.

[0127] Calculate the outward signed distance between the end position and each restricted area, and take the minimum value as the restricted area margin value. The outward signed distance is defined as follows: when the end position is outside the restricted area, it is a positive value equal to the shortest distance from the end position to the boundary minus the preset safety gap value; when the end position is inside the restricted area, it is a negative value equal to the opposite of the shortest distance from the end position to the boundary. Let the boundary of a certain restricted area be... The preset safety gap value is The end position is First, determine whether the end position is inside the restricted area. The determination condition is... and and If inside the restricted area, the outward signed distance is negative, and the value is the opposite of the shortest distance from the endpoint to the boundary of the restricted area. If outside the restricted area, the Euclidean distance from the endpoint to the nearest point on the surface of the restricted area is calculated. The outward signed distance is Calculate the outward signed distance for all restricted areas and take the minimum value as the restricted area margin value. .

[0128] Calculate the difference between the angular velocity of each joint and the corresponding upper speed limit, and take the minimum value as the speed margin value. Let the first joint be... The current angular velocity of each joint is The maximum speed is Then the velocity margin of the joint is The speed margin value is the minimum of the speed margins of all six joints: Calculate the difference between each component of the predicted contact force and the upper limit of the stress, and take the minimum value as the force margin. Let the three components of the predicted contact force at the end be... The upper limit of force is Then the force margin value is: .

[0129] The minimum value of the workspace margin, prohibition margin, velocity margin, and force margin at all sampling moments within the time interval of the safety inheritance constraint is taken as the safety robustness value of the candidate trajectory. The safety inheritance constraint corresponds to a bounded start-stop operator, and its robustness semantics are based on the minimum robustness value at all moments within the time interval, reflecting the safety margin under the most unfavorable condition. ;in Indicates the safety robustness value. This indicates the time interval for specifying safe inheritance constraints. , , , They represent the times at time 1 and 2 respectively. The operating space margin, no-entry margin, speed margin, and force margin.

[0130] The process of calculating the target robustness value is as follows:

[0131] If the current atomic task is pose arrival, then for each sampling time, the position achievement value is obtained by subtracting the Euclidean distance between the end position and the target position from the preset position tolerance value, and the attitude achievement value is obtained by subtracting the angular deviation between the end position and the target attitude from the preset attitude tolerance value. The smaller of the position achievement value and the attitude achievement value is taken as the comprehensive achievement value. The position achievement value is: ;in Indicates at time Position achieved value, This indicates the preset position tolerance value. Indicates at time The end position vector, Represents the target position vector. This represents the Euclidean norm.

[0132] Attitude achievement value ,in Indicates at time The attitude achievement value, This indicates the preset attitude tolerance value. Indicates at time The attitude angle deviation. The overall achievement value is .

[0133] The maximum value of the comprehensive achievement at all sampling moments within the time interval of the target inheritance constraint is taken as the target robustness value of the candidate trajectory. The target inheritance constraint corresponds to a bounded termination operator, whose robustness semantics are to take the maximum robustness value at all moments within the time interval, reflecting the degree of target achievement under the most favorable condition: ;in Indicates the target robustness value. This indicates the time interval for the target inheritance constraint.

[0134] If the current atomic task is a contact operation, then for each sampling time, the minimum margin between the predicted contact force and the preset target force interval is calculated as the force achievement value. The minimum margin is defined as follows: when the predicted contact force is within the preset target force interval, it takes a positive value and is equal to the shortest distance from the predicted contact force to the interval boundary; when the predicted contact force is outside the preset target force interval, it takes a negative value and is equal to the inverse of the shortest distance from the predicted contact force to the interval boundary. Let the preset target force interval be... The normal component of the predicted contact force at the end is Then the force achieved is: ;in Indicates at time The force achieved, and These represent the lower and upper bounds of the preset target force range, respectively. This represents the normal component of the predicted contact force at the end point. The maximum force value achieved at all sampling times within the time interval of the target inherited constraints is taken as the target robustness value of the candidate trajectory.

[0135] The smaller of the safety robustness value and the target robustness value is used as the comprehensive robustness value: The method of taking the smaller value is adopted because the signal timing logic specification is composed of the safety start expression and the target end expression through conjunction operation. The robustness semantics of conjunction operation is to take the minimum robustness of each subexpression.

[0136] Candidate trajectories with positive overall robustness values ​​are selected, meaning these trajectories simultaneously satisfy both safety and objective constraints. The trajectory with the highest overall robustness value among these satisfying candidate trajectories is chosen as the execution trajectory. Selecting the trajectory with the highest robustness aims to preserve the maximum safety margin while satisfying the specification, thereby enhancing the system's robustness to execution errors and external disturbances.

[0137] If the overall robustness value of all candidate trajectories is negative, it means that the current atomic task cannot be completed under the given constraints. The trajectory evaluation agent feeds back the failure information to the task decomposition agent, triggering a re-decomposition or reporting that the task is infeasible to the operator.

[0138] The execution trajectory is discretized into a sequence of control commands, each command containing the target joint angle and execution time. The trajectory evaluation agent outputs the control command sequence and sends it to the field device controller for execution.

[0139] In an alternative implementation, the candidate trajectory set can also be generated using a random sampling strategy, randomly generating several candidate trajectories around the reference trajectory to more fully explore the feasible solution space. Another alternative approach is to use a gradient optimization method, with the overall robustness value as the objective function, iteratively optimizing the trajectory parameters through gradient ascent to directly search for the local optimum trajectory.

[0140] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result according to substantially the same method falls within the scope of the present invention. Therefore, the scope of the present invention is defined only by the appended claims.

Claims

1. An industrial XR reverse control system based on intelligent agents, characterized in that, include: Sensor networks are used to collect multi-source sensing data from field devices; The time synchronization unit is used to attach a unified timestamp to multi-source sensing data and output a time-aligned data stream; The state fusion unit is used to fuse time-aligned data streams and output a device state vector. The device state vector includes the current angle of each joint, the current angular velocity of each joint, the current position of the end effector, the current attitude of the end effector, and the current contact force of the end effector. The digital twin engine is used to refresh the attributes of the virtual device model based on the device state vector, thereby synchronizing the state between the physical device and the virtual device model. Extended reality head-mounted display devices are used to collect data on the operator's interaction with virtual device models; The specification generation agent is used to construct a high-level task description structure based on the interactive operation data, and to generate signal timing logic reduction based on the high-level task description structure. The high-level task description structure includes a task target field, a task type field, an operation sequence field, a time constraint field, an allowed job space field, a forbidden area field, and a force constraint field. The task decomposition agent is used to perform hierarchical task network decomposition on the high-level task description structure, and inherit the signal timing logic reduction to the subtasks during the decomposition process, and output the atomic task sequence and the constraints associated with each atomic task. The trajectory evaluation agent is used to generate a candidate trajectory set based on the atomic task sequence, select the execution trajectory from the candidate trajectory set based on the signal timing logic robustness evaluation, and discretize the execution trajectory into a control command sequence for output. The task decomposition agent executes a hierarchical task network-signal-temporal logic collaborative decomposition, including: creating the root node of the task-constraint tree; storing the task target field of the high-level task description structure into the task slot of the root node; storing the top-level signal-temporal logic reduction into the constraint slot of the root node; creating a record of attempted methods and initializing it as an empty set; marking the root node as a state to be decomposed; selecting the node in the state to be decomposed from the task-constraint tree as the current node; retrieving decomposition methods with the same task type as the current task from the pre-set method library; eliminating the decomposition methods in the record of attempted methods to form a candidate method set; each decomposition method in the method library includes a list of preconditions, a sequence of subtasks, temporal relationships between subtasks, and a target achievement subtask marker; For each candidate decomposition method in the candidate method set, perform precondition checks and constraint compatibility checks. The constraint compatibility check includes: calculating the relative start and end times of each subtask based on their temporal relationships; dividing the time interval of the safety endpoint expression into sub-intervals according to the subtasks and generating safety inheritance constraints; assigning the target endpoint expression to the subtasks indicated by the target achievement subtask markers or defaulting to the last subtask as target inheritance constraints; and checking whether the time interval of each subtask's safety inheritance constraint is a non-empty interval. If the candidate method set is empty and the current node is the root node, output the task is infeasible; if the current node is not the root node, backtrack to the parent node. If the candidate method set is not empty, select the candidate decomposition method with the fewest subtasks as the selected decomposition method and add it to the tried method record; create a child node for each subtask in the subtask sequence of the selected decomposition method, store the subtask in the task slot of the child node, store the safe inheritance constraint in the constraint slot of the child node, and for subtasks assigned target inheritance constraints, also store the target inheritance constraints; determine whether the task of each child node is an atomic task and mark it as a leaf node state or a state to be decomposed accordingly; if there is a node in a state to be decomposed, continue iterative decomposition; otherwise, traverse all leaf nodes in depth-first order and output the atomic task sequence and the constraints associated with each atomic task.

2. The system according to claim 1, characterized in that, The multi-source sensing data includes joint angle data output by the joint encoder, end contact force data output by the six-dimensional force sensor, and workpiece pose data output by the vision sensor.

3. The system according to claim 2, characterized in that, The state fusion unit obtains the end-effector pose estimate by solving the joint angle data through forward kinematics, and then performs coordinate registration between the end-effector pose estimate and the workpiece pose data.

4. The system according to claim 1, characterized in that, Interactive operation data includes the operator's hand movement trajectory, gesture trigger events, and voice command text; the hand movement trajectory is segmented, pause points are identified, and the movement is divided into segments with adjacent pause points as boundaries; for each movement segment, the starting position, ending position, starting posture, ending posture, and movement duration are extracted; gesture trigger events are mapped to operation type identifiers; keyword matching is performed on the voice command text to extract time constraint phrases, allowed work space phrases, prohibited area phrases, and force constraint phrases.

5. The system according to claim 4, characterized in that, The specification-generated agent performs the following processes: For the allowed workspace field, a workspace predicate is constructed, defined as each component of the end effector position coordinates falling within the corresponding allowed boundary coordinate range; for the prohibited region field, a prohibited region predicate is constructed for each prohibited region, defined as the end effector position not belonging to the internal set of points within the corresponding prohibited region and the shortest distance between the end effector position and the boundary of the corresponding prohibited region being greater than a preset safety gap value; the workspace predicate and all prohibited region predicates are combined using a conjunction operation to form a spatial safety predicate; for the force constraint field, a force safety predicate is constructed, defined as the absolute value of each component of the end effector contact force falling below the upper limit of stress; a velocity safety predicate is constructed, defined as the absolute value of each joint angular velocity falling below the corresponding upper limit of velocity; the spatial safety predicate, force safety predicate, and velocity safety predicate are combined using a conjunction operation to form a safety hold predicate; a safety hold expression is constructed, using a bounded hold operator to specify that the safety hold predicate remains true throughout the entire task execution time interval.

6. The system according to claim 5, characterized in that, The specification-generated agent also performs the following process: constructing a target achievement predicate based on the task type. If the task type is pose arrival, the target achievement predicate is defined as the Euclidean distance between the end effector position and the target position being less than a preset position tolerance value and the angular deviation between the end effector attitude and the target attitude being less than a preset attitude tolerance value. If the task type is contact operation, the target achievement predicate is defined as the end contact force being within a preset target force range. Construct a target endpoint expression for the task completion deadline in the time constraint field. The target endpoint expression uses a bounded endpoint operator to specify that the target achievement predicate must be true at least once within the time interval from the current time to the deadline. The safety-always expression and the target-endpoint expression are combined using a conjunction operation to form a top-level signal timing logic reduction.

7. The system according to claim 1, characterized in that, The process of generating a candidate trajectory set by the trajectory evaluation agent includes: retrieving motion primitive templates from a pre-set motion primitive library based on the task type of the current atomic task; instantiating a reference trajectory by taking the current end effector position and attitude as the motion start point and the target position and attitude as the motion end point; the reference trajectory consists of a sequence of sampled time points and the corresponding target position and attitude sequences; establishing a local coordinate system with the motion start point as the origin and the first axis pointing from the motion start point to the motion end point; defining the second and third axes as lateral offset axes; offsetting each sample point of the reference trajectory along the lateral offset axis by a distance value from a pre-set offset distance set; generating the corresponding attitude for the offset sample points through spherical linear interpolation; generating an offset trajectory for each set of offset sample points through cubic spline interpolation; and combining the reference trajectory with all offset trajectories to form an initial candidate trajectory set; performing inverse kinematics solution and joint constraint check on each candidate trajectory in the initial candidate trajectory set; and eliminating candidate trajectories that cannot be solved or violate joint constraints to obtain a feasible candidate trajectory set.

Images