Human-computer interaction method and system based on intelligent sensor

Abstract

The invention discloses a man-machine interaction method and system based on an intelligent sensor, belongs to the technical field of robots, solves the problem of misoperation caused by abnormal sensor data to a certain extent, and enables the system to still run stably through quality credibility evaluation and a dynamic weight distribution mechanism when part of sensors fail. Meanwhile, based on a dynamic safety boundary strategy of scene semantic understanding, the robot can automatically adjust motion constraints according to object attributes and spatial relations, and collaborative optimization of safety and operation efficiency is achieved in complex tasks.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a human-computer interaction method and system based on intelligent sensors. Background Technology

[0002] Collaborative robots are key equipment in fields such as intelligent manufacturing and rehabilitation medicine, with their core being the realization of safe, natural, and efficient physical interaction. Currently, interaction methods based on multimodal intelligent sensors (such as tactile and electromyographic signals) have become the mainstream research approach. These methods typically identify interaction intentions and generate control commands by collecting data such as the operator's tactile pressure and electromyographic signals.

[0003] However, existing technologies have significant limitations. First, most methods lack real-time evaluation and processing mechanisms for sensor data quality. When sensors are interfered with or partially malfunction, the system still uses unreliable data for decision-making, leading to misinterpretation of intent and loss of control. Second, the perception, decision-making, and control processes are relatively isolated, failing to form a closed-loop system that dynamically adjusts based on real-time interaction states (such as data reliability and environmental risks). This makes it difficult to guarantee the robot's adaptability, safety, and reliability when facing unstructured environments or complex tasks.

[0004] Therefore, how to provide a safe and reliable human-computer interaction method has become a research hotspot in this field. Summary of the Invention

[0005] This application provides a human-computer interaction method and system based on intelligent sensors, which can improve the reliability of human-computer interaction. The technical solution is as follows: On the one hand, a human-computer interaction method based on intelligent sensors is provided, the method comprising: In response to the operator's human-robot interaction with the target robot, the system acquires raw interaction data between the operator and the target robot, as well as body state sensor data of the target robot, through intelligent sensors. The raw interaction data includes tactile pressure data and electromyographic (EMG) signal sequences. Tactile pressure data represents the pressure distribution pattern of the contact surface during human-robot interaction, while EMG signal sequences represent the spatiotemporal pattern of the operator's muscle activity. Based on the raw interaction data, the robot's dynamics model, and the body state sensor data, the system determines the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot. The semantic map of the interaction scene refers to the semantic understanding representation of the environmental scene, including object recognition results, spatial relationships between objects, and scene functional attributes. Based on the multimodal interaction features, the semantic map of the interaction scene, and the real-time quality reliability of the raw interaction data, the system generates target joint torque commands and dynamic safety boundary strategies for the target robot. The dynamic safety boundary strategies refer to safety constraints that adaptively adjust with the interaction state, including but not limited to speed limits, acceleration limits, force limits, and obstacle avoidance constraints. Based on the target joint torque commands and the dynamic safety boundary strategies, the system controls the target robot to enable it to complete the tasks instructed by the human-robot interaction.

[0006] As can be seen from the above, the human-computer interaction method and system based on intelligent sensors provided in this application solves the problems of low data reliability and rigid security strategies in the prior art by evaluating the quality of sensor data in real time, fusing multimodal features, and generating control commands in combination with dynamic security boundary strategies. It has the advantages of improving interaction security and enhancing environmental adaptability.

[0007] On the one hand, a human-computer interaction system based on intelligent sensors is provided, the system comprising: The acquisition module is used to respond to the human-computer interaction operation of the operator on the target robot, and to acquire the original interaction data between the operator and the target robot and the body state sensor data of the target robot through intelligent sensors. The original interaction data includes tactile pressure data and electromyographic signal sequence. The tactile pressure data is used to represent the pressure distribution pattern of the contact surface of the human-computer interaction, and the electromyographic signal sequence is used to represent the spatiotemporal pattern of the operator's muscle activity. The determination module is used to determine the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot based on the original interaction data, the robot dynamics model of the target robot, and the body state sensor data. The generation module is used to generate the target joint torque command and dynamic safety boundary strategy of the target robot based on the multimodal interaction features, the semantic graph of the interaction scene, and the real-time quality reliability of the original interaction data. The control module is used to control the target robot based on the target joint torque command and the dynamic safety boundary strategy, so that the target robot can complete the task instructed by the human-machine interaction operation.

[0008] On one hand, a robot controller is provided, the robot controller including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the human-computer interaction method based on intelligent sensors.

[0009] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the human-computer interaction method based on a smart sensor.

[0010] On the one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of the robot controller reads the program code from the computer-readable storage medium and executes the program code, causing the robot controller to perform the aforementioned human-computer interaction method based on intelligent sensors. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the implementation environment of a human-computer interaction method based on intelligent sensors provided in an embodiment of this application; Figure 2 This is a flowchart of a human-computer interaction method based on intelligent sensors provided in an embodiment of this application; Figure 3 This is a partial flowchart of a human-computer interaction method based on intelligent sensors provided in an embodiment of this application; Figure 4 This is a partial flowchart of another human-computer interaction method based on intelligent sensors provided in an embodiment of this application; Figure 5 This is a partial flowchart of another human-computer interaction method based on intelligent sensors provided in the embodiments of this application; Figure 6This is a schematic diagram of the structure of a human-computer interaction system based on intelligent sensors provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a robot controller provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0014] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0015] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain better results.

[0016] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0017] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0018] Intelligent sensors are advanced sensor devices that not only possess physical signal sensing capabilities but also integrate microprocessors and specific algorithms. They are capable of performing on-site preprocessing, feature extraction, internal state diagnosis, and even preliminary decision-making on raw sensed signals. Unlike traditional sensors that only output raw signals, intelligent sensors output pre-processed information with higher-level semantics, and typically possess self-calibration, self-compensation, self-diagnosis, and standardized digital interface communication capabilities.

[0019] Tactile pressure data: refers to the data set collected by a distributed tactile sensor array that characterizes the magnitude and distribution of pressure on the contact surface between the robot and the manipulated object or environment.

[0020] Electromyographic signal sequence: refers to the sequence of bioelectrical signals collected in chronological order that characterizes the muscle activity of the operator and is used to infer the operator's movement intention.

[0021] Robot dynamics model: refers to the physical model that describes the mathematical relationship between the robot's motion and its joint driving forces / torques, including parameters such as mass, inertia, Coriolis force, and gravity.

[0022] Interactive Scene Semantic Graph: A structured representation of environmental knowledge, consisting of semantic nodes (such as objects and regions) and relational edges (such as spatial relationships and functional attributes), used by machines to understand the physical constraints and task context of interactive scenes.

[0023] Target joint torque command: refers to the desired torque value calculated by the control algorithm and directly sent to the actuators of each joint of the robot for precise control of robot movement.

[0024] Dynamic safety boundary strategy: refers to a series of safety constraints (such as speed, acceleration, and upper limit of force) that are dynamically adjusted based on the real-time assessment of the interaction risk level, in order to optimize robot performance while ensuring safety.

[0025] Spatiotemporal registration: The process of unifying data collected from different sensors and in different time and spatial coordinate systems to the same time reference and spatial coordinate system.

[0026] Tactile reliability factor: A quantitative indicator used to assess the reliability of tactile pressure data, typically calculated based on characteristics such as spatial consistency and transient anomalies of the signal.

[0027] Electromyographic reliability factor: a quantitative indicator used to evaluate the quality of electromyographic signal sequences, usually calculated based on characteristics such as signal-to-noise ratio and physiological plausibility.

[0028] Spatial distribution characteristics: Features extracted from tactile pressure data that describe the distribution pattern of pressure on the contact surface, such as pressure center, distribution gradient, symmetry, etc.

[0029] Time series features: Features extracted from sensor data sequences that describe how they change over time, such as mean, variance, trend, and stability.

[0030] Frequency domain power spectrum characteristics: The characteristics of the signal energy distribution in the frequency domain after the time domain signal (such as electromyography signal) is subjected to Fourier transform.

[0031] Time-domain amplitude statistical characteristics: statistical characteristics extracted directly from signal waveforms, such as root mean square value, mean absolute value, peak value, etc.

[0032] Spatiotemporal synchronization: enables multi-source sensor data to have a unified timestamp and coordinate reference.

[0033] Basic motion feature set: A set of low-level features extracted from robot body state sensor data (such as joint angles and velocities) that describes the robot's own motion state.

[0034] Basic tactile feature set: A set of low-level features directly extracted from raw tactile pressure data, such as pressure values ​​and rates of change of each sensing unit.

[0035] Basic electromyographic feature set: A set of low-level features directly extracted from the original electromyographic signal sequence, such as signal amplitude and frequency.

[0036] Dynamic weight allocation: During feature fusion, a strategy is used to dynamically adjust the contribution of different data sources based on the real-time reliability factor of the data.

[0037] Spatiotemporal alignment and splicing: the process of aligning and merging feature vectors from different modalities in the temporal and spatial dimensions into a unified feature vector.

[0038] Semantic parsing: The process of using algorithms or models to identify and extract high-level semantic information (such as object categories and interaction relationships) from low-level perceptual data.

[0039] Object attributes: The inherent characteristics of objects in an interactive scene, such as category, material, size, weight, etc.

[0040] Spatial relationships: The relative positions, orientations, and distances between different objects or entities in a scene.

[0041] Task context information: Background information related to the currently executing task, such as task objectives, operation steps, and historical actions.

[0042] Intent probability distribution: a probability vector in which each element represents the likelihood that the operator currently has a certain preset interaction intent.

[0043] Interaction Risk Level: A quantitative assessment of the likelihood and severity of dangerous events that may occur during human-computer interaction in the current or future period.

[0044] Confidence boundary: An area of ​​uncertainty added around the predicted trajectory or parameters to indicate the reliability of the prediction result.

[0045] Standard motion accuracy requirements: The accuracy standards that a robot end effector needs to achieve in terms of position, attitude, or force control to complete a specific type of task.

[0046] Task criticality level: An indicator that grades the severity of the consequences of task failure.

[0047] Trajectory deviation tolerance threshold: The maximum deviation allowed between the robot's actual trajectory and the expected trajectory during task execution.

[0048] Risk impact weighting coefficient: In a risk assessment model, this is a coefficient used to adjust the proportion of different risk factors contributing to the overall risk.

[0049] Environmental complexity index: A quantitative indicator used to comprehensively characterize the degree of difficulty caused to robot movement by the number, distribution, and dynamics of obstacles in the interactive environment.

[0050] Environment-trajectory interaction risk value: The local collision risk quantification value is calculated by comprehensively analyzing the interaction between the predicted motion trajectory and static / dynamic obstacles in the environment.

[0051] Confidence calibration: The process of correcting the credibility of previous calculation results (such as risk probability) using subsequent information (such as data credibility).

[0052] Security boundary relaxation factor: A factor used to scale the default security boundary range. Its value is negatively correlated with the interaction risk level, achieving a trade-off between security and efficiency.

[0053] Feasibility correction: The process of adjusting the initial control commands generated by the optimization algorithm to meet physical constraints such as robot dynamics, kinematics, and safety boundaries.

[0054] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0055] Figure 1 This is a schematic diagram illustrating the implementation environment of a human-computer interaction method based on intelligent sensors provided in this application embodiment. See also... Figure 1 The implementation environment may include a target robot 100, which includes a robot controller 110.

[0056] The robot controller 110 connects to the smart sensors of the target robot, acquires and processes the data collected by the smart sensors, and thus controls the target robot 100. The robot controller 110 has strong computing power and can quickly process complex data.

[0057] In related technologies, human-robot collaborative robots are widely used in intelligent manufacturing and rehabilitation medicine. Their core relies on multimodal intelligent sensors to collect data such as tactile pressure and electromyographic signals from the operator to achieve interactive control. However, existing methods suffer from a lack of sensor data quality assessment mechanisms. When sensors are interfered with, unreliable data is still used for decision-making, leading to the risk of misinterpretation of intent. Furthermore, traditional safety constraints use fixed threshold modes, which cannot be dynamically adjusted according to environmental complexity and interaction status. In unstructured scenarios, this can easily result in delayed actions or excessive safety redundancy. For example, in assembly operations, when sweat on the operator's hands causes the tactile sensors to malfunction, the system cannot recognize the data anomaly and still executes the grasping action according to the erroneous signal, potentially causing workpiece damage or personnel injury.

[0058] To address the aforementioned issues, the inventors observed shortcomings in the linkage between data quality monitoring and security strategies in existing systems. By analyzing typical working conditions in industrial scenarios, such as the sudden appearance of obstacles and abrupt changes in operator intent, they found that traditional methods lack a data reliability assessment mechanism when sensors fail, and that rigid security constraints result in insufficient system flexibility. Based on this, they proposed establishing a multimodal real-time data quality assessment system and dynamically linking security boundary strategies with interaction risk levels. Furthermore, they introduced scene semantic graphs to enhance environmental context understanding, enabling the robot to autonomously adjust its behavior patterns based on object spatial relationships and task attributes.

[0059] Therefore, this application proposes a human-computer interaction method based on intelligent sensors. Taking the robot controller as the execution subject as an example, see [link to relevant documentation]. Figure 2 This includes the following steps.

[0060] 201. In response to the human-machine interaction operation of the operator on the target robot, the original interaction data between the operator and the target robot and the body state sensor data of the target robot are acquired through intelligent sensors. 202. Based on the original interaction data, the robot dynamics model of the target robot, and the body state sensor data, determine the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot; 203. Based on multimodal interaction features, semantic graph of interaction scene, and real-time quality reliability of raw interaction data, generate target joint torque commands and dynamic safety boundary strategies for the target robot. 204. Based on the target joint torque command and dynamic safety boundary strategy, control the target robot so that the target robot can complete the task of human-machine interaction operation instruction.

[0061] The original interaction data includes tactile pressure data and electromyographic (EMG) signal sequences. Tactile pressure data represents the pressure distribution pattern of the contact surface during human-computer interaction, while EMG signal sequences represent the spatiotemporal patterns of the operator's muscle activity. Tactile pressure data refers to the pressure distribution pattern of the contact surface collected by a flexible pressure sensor array, specifically capacitive or piezoresistive sensors, used to characterize the magnitude and distribution characteristics of the contact force applied by the operator. EMG signal sequences refer to the time-series data of muscle electrical activity collected by surface electromyography (SEMG) sensors, specifically implemented using differential electrodes and signal amplifiers, used to reflect the intensity and temporal patterns of the operator's muscle contractions. The robot dynamics model is a mathematical model describing the relationship between robot motion and forces, specifically constructed using Lagrange equations or Newton-Euler equations, used to achieve spatiotemporal registration and physical constraint verification of multi-source data. Multimodal interaction features are composite feature vectors that integrate tactile, EMG, and proprioceptive motion information, specifically generated through spatiotemporal alignment and weighting algorithms, used to characterize the mechanical properties and motion intentions of human-computer interaction. Dynamic safety boundary strategy refers to a set of constraints that are automatically adjusted according to the level of interaction risk. Specifically, it can be implemented using fuzzy logic or model predictive control methods to balance system response speed and safety redundancy.

[0062] Specifically, when the operator performs assembly operations, a flexible pressure sensor array collects the hand contact pressure distribution in real time, while a surface electromyography (EMG) sensor simultaneously acquires forearm muscle electrical signals. The robot's joint encoder and inertial measurement unit provide body motion state data. The tactile data and EMG signals are timestamped using a robot dynamics model, and both tactile and EMG reliability factors are calculated. When an abnormal offset of a pressure sensor in a certain area is detected, the tactile reliability factor decreases, and the system automatically reduces the weight of that area's data in feature fusion. The fused multimodal interaction features are input into a scene analysis model to identify semantic information such as the workpiece position on the workbench and the fixture orientation, constructing a scene map containing spatial relationships between objects. Based on real-time quality reliability assessment results, when the EMG signal-to-noise ratio is below a threshold, the system increases the weight of environmental semantic information in decision-making. The joint torque limit threshold is dynamically adjusted based on risk level, tightening acceleration constraints when avoiding obstacles and relaxing speed constraints during free movement, achieving a balance between safety and efficiency.

[0063] Compared to related technologies, traditional methods employ an architecture that processes data from each sensor independently, failing to establish a mechanism linking data quality with decision-making logic. This solution dynamically adjusts feature fusion weights through a credibility factor, maintaining intent recognition accuracy even when some sensors fail. Existing safety strategies use fixed parameter settings, while this solution uses scene semantic graph parsing to resolve environmental constraints, allowing safety boundaries to adaptively change with object attributes and spatial relationships. For example, when a fragile workpiece is detected, the system automatically increases the force control accuracy requirement of the end effector, a situation that traditional methods cannot achieve.

[0064] Through the above technical solutions, this application addresses the problem of malfunctions caused by abnormal sensor data to a certain extent. By employing a quality reliability assessment and dynamic weight allocation mechanism, the system can still operate stably even when some sensors fail. Furthermore, the dynamic safety boundary strategy based on scene semantic understanding enables the robot to autonomously adjust motion constraints according to object attributes and spatial relationships, achieving synergistic optimization of safety and operational efficiency in complex tasks.

[0065] This application further proposes the following technical solutions, taking a robot controller as the execution subject as an example, see [link to relevant documentation]. Figure 3 This includes the following steps.

[0066] 301. Based on the robot dynamics model, the quality assessment and spatiotemporal registration of the original interaction data are performed to obtain a spatiotemporally synchronized multimodal data stream and the real-time quality reliability of the original interaction data. 302. Based on multimodal data streams, real-time quality reliability, and ontological state sensor data, determine the multimodal interaction features and semantic map of the interaction scenario between the operator and the target robot.

[0067] Real-time quality reliability includes tactile reliability factors and electromyography (EMG) reliability factors. The robot dynamics model, a mathematical model describing the relationship between robot motion and forces, can be constructed using Lagrange equations or Newton-Euler equations to verify whether sensor data conforms to physical laws. Quality assessment and spatiotemporal registration involve reliability discrimination and time synchronization of multi-source data, achieved using Kalman filtering combined with timestamp alignment algorithms to eliminate sensor noise and transmission delays. The tactile reliability factor quantifies the reliability of tactile data, calculated by the matching degree between pressure distribution patterns and dynamic model predictions, used to dynamically adjust the weights of tactile features. The EMG reliability factor assesses the quality of EMG signals, obtained through correlation analysis between signal-to-noise ratio and muscle activation state, used to suppress the influence of abnormal EMG data. The multimodal data stream refers to a spatiotemporally aligned collection of heterogeneous sensor data, generated through interpolation fusion under a unified time reference to ensure temporal consistency in feature extraction.

[0068] The following section explains the method used in the above description to achieve quality assessment and spatiotemporal registration by combining Kalman filtering with a timestamp alignment algorithm.

[0069] This process is a collaborative computational workflow that integrates physical model verification and signal processing, aiming to elevate raw interactive data from different sensors and existing on different spatiotemporal references into a spatiotemporally unified, quality-controllable multimodal data stream. Its core process can be divided into the following two stages: Phase 1: Quality assessment and credibility calculation based on dynamic model.

[0070] The goal at this stage is to assign a quantitative "quality credibility" label to each piece of raw data.

[0071] Physical consistency check: Process: Using a known robot dynamics model, the sensor readings are physically and logically verified. For example, the motor current reading of a joint, the joint angle sensor reading, and the body inertial measurement unit (IMU) reading are input into the dynamics model together.

[0072] Calculation: The model calculates a theoretical joint torque or body acceleration and compares it with the actual readings from the torque sensor or IMU. If the difference exceeds a reasonable threshold preset based on model accuracy and sensor noise levels, the data is considered physically inconsistent.

[0073] Output: Generates a physical consistency error score.

[0074] Signal quality analysis: Process: Perform signal-level quality assessments on the data streams of each individual sensor in parallel.

[0075] Calculation: Kalman filter-based residual analysis: Kalman filtering is applied to data (such as the positions of visual markers) for prediction, and the residual between the predicted and observed values ​​is used as a quality indicator. A sudden increase in residual usually indicates that the data has been affected by sudden noise interference, occlusion, or transmission errors.

[0076] Statistical characteristic check: Check whether the detection signal shows saturation (exceeding the range), abrupt change (jump that does not conform to the kinematic law), or prolonged stagnation (sensor failure).

[0077] Output: Generates a set of signal quality metrics (such as residual size, abrupt change flag, and saturation flag).

[0078] Real-time quality reliability fusion: Process: The above "physical consistency error score" and "signal quality index" are merged.

[0079] Calculation: Using methods such as weighted fusion or fuzzy logic, output a comprehensive, normalized real-time quality confidence score (typically a value between 0 and 1). For example, when the physical consistency error is large and the Kalman filter residual is also large, the confidence score of the data point will be significantly reduced.

[0080] Phase 2: Spatiotemporal registration of multimodal data.

[0081] The goal of this stage is to unify all data into the same time and space coordinate system, solving the questions of "when" and "where".

[0082] Time registration: Process: Solve the problem of "time misalignment" of data caused by different sensors having different acquisition and transmission delays.

[0083] Calculation: Hardware clock synchronization and software timestamp alignment: Synchronize the clocks of each sensor at the hardware level as much as possible, and add a high-precision timestamp to each data packet.

[0084] Interpolation based on a dynamic model: For data that cannot be strictly synchronized, the high-quality data stream calculated in the previous step (such as joint trajectories smoothed by Kalman filtering) is used as the time reference to perform spatiotemporal interpolation on other data with large delays or low sampling rates (such as asynchronously triggered camera images). That is, based on the dynamic model, the theoretical pose of the robot at the precise moment of camera exposure is calculated, thereby associating the image with the correct robot state.

[0085] Spatial registration: Process: Solving the problem of inconsistent coordinate systems among different sensors.

[0086] Calculation: Calibration parameter application: Using the pre-calibrated sensor transformation matrix, visual data, force data, etc., are all converted to the robot base coordinate system or world coordinate system.

[0087] Model-assisted localization: Combining the robot's own joint angle readings (calculated through positive kinematics) and dynamic model, it provides the robot's accurate body state in a unified coordinate system in real time, serving as a spatial reference for multimodal data fusion.

[0088] Final output After the above two stages of processing, the system outputs: a spatiotemporally synchronized multimodal data stream. All data have a unified timestamp and spatial coordinate system, allowing for meaningful fusion and subsequent analysis. For example, it can be known precisely that "at t=1.235 seconds, the position of the robot's end effector in the coordinate system (X,Y,Z) corresponds strictly to the image of the object seen by the camera at that time, and the contact force (Fx, Fy, Fz, Mx, My, Mz) sensed by the six-dimensional force sensor."

[0089] The real-time quality credibility of raw interaction data: Each raw data point or data frame is accompanied by a credibility score. Subsequent algorithms (such as fusion, recognition, and decision-making) can use this score to determine the degree of trust in the data, thereby achieving quality-driven robust interaction.

[0090] Specifically, a robot dynamics model is used to establish physical constraints. By comparing the difference between tactile pressure data and the contact force distribution predicted by the model, a tactile confidence factor is calculated. Simultaneously, correlation analysis is performed between the frequency domain characteristics of the electromyography (EMG) signal sequence and the muscle activation level inferred by the dynamics model to generate an EMG confidence factor. In the spatiotemporal registration stage, data from the tactile sensor, EMG sensor, and proprioception sensor are synchronized through timestamp alignment and motion compensation algorithms, forming a multimodal data stream with a unified time reference. Based on the confidence factor, tactile and EMG features are dynamically weighted and fused, combined with proprioception motion features to generate multimodal interaction features. The scene analysis model constructs a graph structure containing semantic nodes and relational edges by analyzing object contact patterns, motion trajectories, and spatial relationships in the interaction features.

[0091] Compared to related technologies, existing methods typically use raw sensor data directly without a quality assessment mechanism, making them prone to generating erroneous features when sensors are interfered with. This solution verifies data validity and generates a credibility factor through a dynamic model, automatically suppressing the influence of low-quality data during the feature fusion stage. Furthermore, while many related technologies employ fixed-weight multimodal fusion, this solution dynamically adjusts feature weights based on real-time credibility, improving robustness under complex conditions. Moreover, existing methods treat environmental perception and interaction feature processing independently; this solution achieves joint representation of physical interaction features and environmental semantics through semantic graph construction.

[0092] Through the above technical solutions, this application addresses to some extent the technical challenges of reliability assessment and spatiotemporal synchronization of multimodal sensor data, achieving robust feature extraction under low-quality data interference. A reliability-weighted dynamic fusion mechanism ensures that high-reliability data plays a dominant role in decision-making, avoiding misjudgments caused by sensor failure. Spatiotemporal registration technology eliminates the temporal misalignment problem of multi-source data, providing temporal correlation features for subsequent intent recognition. The construction of a semantic graph transforms physical interaction features into structured environmental semantic information, providing multi-dimensional scene understanding support for security policy generation.

[0093] This application further proposes a technical solution for quality assessment and spatiotemporal registration of raw interactive data based on robot dynamics models. This includes extracting spatial distribution features and time series features of tactile pressure data to determine tactile confidence factors, extracting frequency domain power spectrum features and time domain amplitude statistical features of electromyography (EMG) signal sequences to determine EMG confidence factors, and performing spatiotemporal synchronization of tactile pressure data, EMG signal sequences, and proprioceptive state sensor data to obtain a multimodal data stream.

[0094] Among them, the tactile reliability factor is a quantitative indicator that assesses the quality of tactile data through the rationality of pressure distribution patterns. Specifically, it can be achieved by combining spatial pressure gradient distribution characteristics with time-series stability analysis, and is used to detect sensor contact anomalies or signal distortion. The electromyography (EMG) reliability factor is a quantitative indicator that assesses the validity of EMG data through muscle activity signal characteristics. Specifically, it can be achieved by combining frequency band energy proportion analysis with time-domain signal variance calculation, and is used to eliminate noise interference in EMG signal acquisition. Spatiotemporal synchronization refers to the operation of eliminating time delays and spatial coordinate differences in multi-source sensor data. Specifically, it can be achieved by combining timestamp alignment algorithms with inverse kinematics of robots, and is used to ensure the consistency of multimodal data in the spatiotemporal dimensions.

[0095] Specifically, this scheme first performs morphological analysis of the two-dimensional pressure matrix on the tactile pressure data, extracting the spatial gradient features of the pressure distribution to identify contact surface anomalies. Simultaneously, it calculates the autocorrelation coefficient of the time series to determine signal stability, and then comprehensively generates a tactile confidence factor. For electromyography (EMG) signals, it extracts the energy proportion features of specific frequency bands through Fast Fourier Transform, and combines this with sliding window statistics of the time-domain amplitude variance to double-verify signal validity before generating an EMG confidence factor. Finally, it uses the motion state data of the robot's body sensors as a benchmark, eliminates sensor sampling rate differences through timestamp calibration, and uses the robot's kinematic model to unify the data from all sensors into the same coordinate system, forming a spatiotemporally consistent multimodal data stream.

[0096] Compared to related technologies, traditional methods rely solely on a single threshold to determine data validity, failing to differentiate between different failure modes of tactile and electromyographic signals. Existing spatiotemporal synchronization technologies often employ fixed delay compensation, which is ill-suited to coordinate transformations during robot dynamic motion. This solution addresses a key bottleneck in multi-sensor data collaborative processing by employing a modal feature extraction mechanism to design differentiated quality assessment methods based on the physical characteristics of tactile and electromyographic signals. Furthermore, it integrates with the robot's dynamics model to achieve dynamic spatiotemporal registration.

[0097] Through the above technical solutions, this application has, to a certain extent, solved the technical problems of real-time evaluation and spatiotemporal synchronization of sensor data quality, realized dynamic monitoring of tactile and electromyographic signal quality, and avoided system misjudgments caused by the failure of a single sensor. The spatiotemporal registration mechanism ensures the physical consistency of multimodal data during robot motion, providing a reliable data foundation for subsequent interactive decision-making.

[0098] This application further proposes a method for determining multimodal interaction features and semantic graphs of interaction scenarios between an operator and a target robot based on multimodal data streams, real-time quality reliability, and ontological state sensor data. This includes parallel feature extraction from ontological state sensor data and multimodal data streams to obtain basic motion feature sets, basic tactile feature sets, and basic electromyographic feature sets; dynamic weight allocation and feature fusion of the basic tactile feature sets and basic electromyographic feature sets based on tactile reliability factors and electromyographic reliability factors to obtain tactile feature vectors and electromyographic feature vectors; spatiotemporal alignment and concatenation of the tactile feature vectors, electromyographic feature vectors, and basic motion feature sets to obtain multimodal interaction features; inputting the multimodal interaction features into a scene analysis model for semantic parsing to obtain target structured data; and constructing an interaction scenario semantic graph including semantic nodes and relational edges based on the target structured data.

[0099] Parallel feature extraction refers to the process of independently extracting basic features from different modalities of data. Specifically, convolutional neural networks and long short-term memory networks can be used to process spatial distribution features and time-series features respectively, avoiding interference between different modalities. Dynamic weight allocation and feature fusion refers to the process of dynamically adjusting feature fusion weights based on sensor data quality. This can be achieved through a confidence factor weighted summation algorithm, improving the robustness of features under low-quality data. Spatiotemporal alignment and stitching refers to the process of unifying the spatiotemporal reference of multimodal features. This can be achieved using timestamp matching and coordinate system transformation techniques, eliminating feature misalignment caused by sensor acquisition delays. Scene analysis models are algorithmic models that transform multimodal features into structured semantic data. This can be achieved using graph neural networks combined with knowledge graph embedding techniques, enabling deep analysis of environmental semantics. Semantic nodes and relation edges refer to graph elements describing object attributes and spatial relationships. These can be constructed using entity recognition and relation extraction algorithms to form interpretable scene knowledge representations.

[0100] Specifically, the body state sensor data and multimodal data streams are input into three independent feature extraction modules. The motion feature extraction module extracts the robot's body motion state through Kalman filtering and covariance analysis. The tactile feature extraction module obtains the mechanical properties of the contact surface through pressure distribution matrix decomposition and gradient calculation. The electromyography (EMG) feature extraction module analyzes muscle activity patterns through wavelet transform and envelope detection. Tactile and EMG confidence factors are input into an adaptive weighted fusion unit. When the confidence of tactile data is lower than a preset threshold, its weight coefficient is automatically reduced. For example, when the tactile sensor is affected by temperature interference, the confidence factor drops to 0.3, while the EMG feature weight increases to 0.7. The spatiotemporal alignment module uses a dynamic time warping algorithm to align the sampling time axes of different sensors and maps each feature to a unified workspace coordinate system through a coordinate transformation matrix. The graph convolutional network in the scene analysis model embeds nodes into the stitched multimodal features, uses an attention mechanism to identify key objects and their spatial topological relationships, and finally generates a semantic map containing object material, position coordinates, and functional attributes.

[0101] Compared to related technologies, traditional methods employ fixed-weight feature fusion strategies, which cannot guarantee the reliability of fusion results when sensor data quality fluctuates, and environmental semantic modeling is mostly limited to two-dimensional planar relationships. This solution, through a credibility-driven dynamic weight adjustment mechanism, addresses the feature degradation problem to some extent under sensor failure or interference conditions, such as automatically enhancing tactile feature weights when electromyographic signals are interfered with by motion artifacts. Simultaneously, the construction of a three-dimensional semantic map breaks through the flattened representation of spatial relationships in traditional methods, accurately expressing the three-dimensional orientation relationships and functional constraints between objects.

[0102] Through the above technical solutions, this application achieves quality adaptive fusion of multimodal sensor data, significantly improving the robustness of the feature extraction process to noise and interference. Simultaneously, through hierarchical semantic parsing and knowledge graph construction, it effectively enhances the robot's cognitive understanding of complex scenes, providing environmental semantic support for subsequent safety decisions.

[0103] This application further proposes the following technical solutions, taking a robot controller as the execution subject as an example, see [link to relevant documentation]. Figure 4 This includes the following steps.

[0104] 401. Based on multimodal interaction features, semantic graphs of interaction scenarios, and real-time quality reliability of raw interaction data, determine the probability distribution of the operator's intent, the envelope of the robot's expected motion trajectory, and the level of interaction risk. 402. Based on the intent probability distribution, the robot's expected motion trajectory envelope, the interaction risk level, and the semantic graph of the interaction scene, generate the target joint torque command and dynamic safety boundary strategy for the target robot.

[0105] Among these, the intent probability distribution refers to the diversified probability estimation of the operator's intent; multimodal interaction features refer to high-dimensional representation vectors that integrate tactile, electromyographic, and kinematic features, specifically implemented by using a spatiotemporal attention neural network to extract spatiotemporally correlated feature data, which reflects the coupling relationship between the operator's action patterns and the robot's motion state. The interaction scene semantic graph refers to graph-structured data containing object attributes and spatial relationships, specifically implemented by using a scene analysis model to parse semantic information in multimodal data, providing environmental constraints for trajectory planning. Real-time quality reliability refers to a quantitative evaluation index of the reliability of sensor data, specifically implemented by weighted calculation of tactile reliability factors and electromyographic reliability factors, used to dynamically adjust the weight allocation of different modalities during the decision-making process. The intent probability distribution refers to the probabilistic description of the operator's potential intent, specifically implemented by using a spatiotemporal attention mechanism combined with the probability set output by a classifier, its role being to preserve the possibility of multiple intents to cope with data uncertainty. The expected trajectory envelope of a robot refers to a geometric description that includes the feasible region and confidence boundary of the trajectory. Specifically, it can be calculated by superimposing the prediction of the dynamic model with the spatial constraints of obstacles, and is used to limit the physical feasible range of robot motion.

[0106] Specifically, when an operator performs a human-machine collaborative task, tactile sensors and electromyography (EMG) sensors synchronously collect interaction data. A quality assessment module calculates tactile and EMG confidence factors; for example, when high-frequency noise interference occurs in the EMG signal, the confidence factor's weight can be dynamically reduced. Multimodal interaction features are extracted by a spatiotemporal attention network, and the input intent recognition module generates probability distributions including operation categories such as "grab," "translate," and "release." Simultaneously, a scene semantic graph analyzes the distribution of obstacles and the boundaries of the operation area, such as identifying the coordinates of the workbench edge and the workpiece stacking position. The trajectory planning module combines the intent probability distribution and semantic constraints to generate the expected trajectory envelope; for example, when the probability of the "grab" intent reaches a threshold, a feasible region including the gripper opening and closing trajectory and obstacle avoidance path is generated. The safety policy generation module dynamically adjusts constraints based on the trajectory envelope and risk level; for example, it automatically reduces the speed limit threshold when approaching a precision instrument area.

[0107] Compared to related technologies, traditional methods employ fixed-threshold safety boundary strategies, which are prone to over-conservatism or constraint failure when the environment changes dynamically. This solution uses semantic graphs to analyze spatial relationships in real time. For example, when a moving obstacle is detected approaching, the safety margin of the trajectory envelope can be reduced in advance. Existing intent recognition systems typically output a single judgment result, which is prone to misjudgment when sensor data quality fluctuates. This solution, however, retains a multivariate probability distribution. For instance, even when electromyographic signals are interfered with, it can still maintain alternative probabilities of the "grabbing" intent through tactile features, avoiding abrupt changes in control commands.

[0108] Through the above technical solutions, this application solves the problem of misjudgment of intent caused by fluctuations in sensor data quality in traditional human-computer interaction. For example, when electromyographic signals are affected by muscle fatigue, reducing their confidence weight can maintain a stable accuracy rate in intent recognition. Simultaneously, it achieves dynamic adaptation of safety constraints. For instance, it automatically strengthens force restrictions when handling fragile items, while appropriately relaxing acceleration restrictions when performing rapid movements in open areas, thereby improving operational efficiency while ensuring safety. Furthermore, by fusing scene semantic information, it can effectively avoid the risk of collisions between the robotic arm and unknown obstacles. For example, when a temporarily placed toolbox is detected in the semantic graph, the obstacle avoidance constraints are updated promptly.

[0109] This application further proposes a method for determining the operator's intention probability distribution, the robot's expected trajectory envelope, and the interaction risk level based on multimodal interaction features, an interaction scene semantic graph, and real-time quality reliability. This method includes inputting multimodal interaction features into a spatiotemporal attention neural network to extract high-dimensional interaction features with spatiotemporal correlation; dynamically adjusting the attention weights of the spatiotemporal attention neural network based on real-time quality reliability; determining the probability of multiple preset intention categories based on the high-dimensional interaction features and the interaction scene semantic graph to obtain the intention probability distribution; determining the future trajectory of the target robot based on the intention probability distribution and the high-dimensional interaction features; determining the robot's expected trajectory envelope based on the spatial constraints in the interaction scene semantic graph and the future trajectory; determining the risk probability of different interaction strategies based on the intention probability distribution, the robot's expected trajectory envelope, and the interaction scene semantic graph; and determining the interaction risk level based on the risk probabilities of different interaction strategies.

[0110] Among them, the spatiotemporal attention neural network refers to a deep learning model that captures the spatiotemporal correlation of multimodal data through an attention mechanism. Specifically, it can be implemented using a convolutional recurrent neural network with a multi-head self-attention structure. Its attention weights are dynamically adjusted based on real-time quality confidence, reducing the contribution of low-confidence features in feature extraction. The intent probability distribution refers to a diversified probability estimate of the operator's intent. Specifically, it can be achieved by using a classifier based on the softmax function to calculate the multi-class probability of high-dimensional interaction features, expressing the likelihood of different intents through a probability set. The robot's expected motion trajectory envelope refers to the feasible boundary of the robot's future motion trajectory. Specifically, it can be generated by combining trajectory optimization algorithms with dynamic constraints and environmental spatial relationships, such as using model predictive control algorithms to calculate the trajectory confidence interval. The interaction risk level refers to the quantitative assessment result of the safety of the interaction strategy. Specifically, it can be calculated using a risk-weighted assessment model combined with environmental obstacle distribution, trajectory speed indicators, and task criticality level.

[0111] Specifically, after receiving multimodal interaction features, the spatiotemporal attention neural network adjusts the attention weights of each modality feature based on tactile and electromyographic (EMG) confidence factors. For example, when the confidence of the EMG signal decreases, the weight coefficient of the EMG feature in the attention calculation is reduced. High-dimensional interaction features and the spatial relationships of objects in the interaction scene semantic graph are input into the intent classification module, which outputs a confidence ranking of the operator's possible intents based on probability distribution. During the generation of future motion trajectories, candidate trajectories corresponding to high-probability intents are selected based on the intent probability distribution, and the physical feasibility of the trajectory is verified through a robot dynamics model, such as verifying whether the joint torque exceeds the motor output limit. The robot's expected motion trajectory envelope is obtained by spatially superimposing the verified trajectory with the obstacle positions in the scene semantic graph, calculating the minimum safe distance between each point on the trajectory and the obstacle, and forming a trajectory boundary including velocity and acceleration constraints. The determination of the interaction risk level is achieved by analyzing the dynamic relationship between the trajectory envelope and environmental obstacles. For example, when the trajectory envelope overlaps with the predicted path of a dynamic obstacle, the risk probability value is increased, and a weighted comprehensive evaluation is performed based on the task criticality level.

[0112] Compared to related technologies, existing methods typically employ fixed-weight feature fusion to process multimodal data, failing to dynamically adjust feature importance when sensor data quality fluctuates, leading to accumulated intent recognition errors. Our proposed solution dynamically adjusts attention weights based on real-time quality reliability, automatically reducing the impact of interference with electromyographic signals and improving the robustness of feature extraction. Related technologies often neglect environmental semantic constraints in motion trajectory planning, such as failing to consider the impact of obstacle movement trends on trajectory feasibility. Our solution, however, provides prior knowledge of spatial relationships through an interactive scene semantic graph, enabling the trajectory envelope to adapt to environmental changes. Existing risk assessments are mostly based on single indicators, such as considering only the speed of the robotic arm's end effector. Our solution integrates trajectory dynamics, environmental obstacle distribution, and task criticality to achieve multi-dimensional risk coupling assessment.

[0113] Through the above technical solutions, this application reduces the impact of low-quality data on intent recognition by dynamically adjusting feature weights when sensor data quality fluctuates, thereby improving the accuracy of intent probability estimation; it generates trajectory envelopes that conform to physical laws and spatial constraints by fusing environmental semantic information, thus avoiding collisions or boundary crossings during robot movement; and it dynamically evaluates the safety of interaction strategies through multi-dimensional risk indicators, providing a refined basis for subsequent safety strategy adjustments.

[0114] This application further proposes a method for determining the future motion trajectory of a target robot based on the intent probability distribution and high-dimensional interaction features, including: determining multiple candidate task targets and task target weights based on the intent probability distribution; generating multiple predicted motion trajectories based on high-dimensional interaction features and task target weights; verifying and optimizing the feasibility of the predicted motion trajectories based on dynamic constraints and kinematic constraints to obtain a set of motion trajectories; selecting the target trajectory from the set and adding confidence boundaries to obtain the future motion trajectory.

[0115] The intent probability distribution refers to the diversified probability estimation of the operator's intent. Specifically, a spatiotemporal attention neural network can be used to analyze multimodal interaction features and output probability sets for different intent categories to quantify the uncertainty of intent recognition. Task target weights refer to the priority coefficients of candidate task targets in trajectory generation. These can be calculated by combining probability normalization with task criticality assessment to balance the competition among multiple targets. Dynamic constraints refer to physical limitations such as robot joint torques and inertial parameters. These can be verified by establishing a dynamic model using the Lagrange equations to ensure the trajectory conforms to actual driving capabilities. Kinematic constraints refer to geometric limitations such as joint angles and velocities. These can be analyzed using the Jacobian matrix to define the workspace boundaries and prevent the trajectory from exceeding mechanical structural limits. Confidence boundaries refer to the probability envelope range around the target trajectory. These can be generated using a Gaussian process regression model based on the standard deviation of the intent probability distribution to characterize the confidence interval of the trajectory prediction.

[0116] Specifically, when the intent probability distribution indicates the existence of multiple candidate task targets, a task target weighting mechanism is used to associate the operator behavior patterns implicit in the high-dimensional interaction features with each task target, generating predicted motion trajectories with different target orientations. Subsequently, dynamic constraints are used to verify and eliminate trajectory segments exceeding the joint torque limit, and kinematic constraints are used to optimize and adjust the spatial position of trajectory points to avoid mechanical interference. From the trajectory set after dual-constraint filtering, the target trajectory with the highest matching degree to the current interaction features is selected, and the confidence boundary width is calculated based on the difference between the highest and second-highest probability values ​​in the intent probability distribution. For example, when there is high uncertainty in intent recognition, the confidence boundary automatically expands to increase the safety margin; conversely, it shrinks to improve tracking accuracy.

[0117] Compared to related technologies, traditional methods typically generate fixed trajectories based on a single intent assumption, which can easily lead to trajectory failure when intent recognition is flawed. Our proposed solution, however, utilizes multimodal expansion of candidate task objectives and dynamic weight allocation to accommodate multiple possible intents during trajectory generation. It then employs physical constraint verification to eliminate infeasible solutions and finally superimposes confidence boundaries to form a safe trajectory. While related technologies often focus on geometric path planning for trajectory optimization, our solution combines intent uncertainty quantification with physical constraint verification to achieve end-to-end reliability assurance from intent understanding to motion execution.

[0118] Through the above technical solution, this application can effectively avoid the trajectory planning failure problem caused by misjudgment of intent due to sensor noise or environmental interference. The feasibility and safety of the trajectory are improved by the dual mechanism of multi-candidate target generation and physical constraint verification. At the same time, the tightness of trajectory tracking is dynamically adjusted by using confidence boundaries to improve the smoothness of operation while ensuring the safety of human-machine collaboration.

[0119] This application further proposes a method for determining the risk probability of different interaction strategies based on intent probability distribution, robot expected motion trajectory envelope, and interaction scene semantic graph. The method includes: querying the standard action precision requirement and task criticality level corresponding to the candidate intent with the highest probability from the task knowledge base based on the intent probability distribution; extracting the spatial distribution of static obstacles, the motion trajectory of dynamic obstacles, and the boundary of the safe operating area in the environment as basic environmental risk factors based on the interaction scene semantic graph; determining the minimum distance between each point on the trajectory and obstacles, the peak velocity at key points on the trajectory, and the rate of change of acceleration based on the robot's expected motion trajectory envelope to obtain motion trajectory risk indicators; and determining the risk probability of different interaction strategies through a risk-weighted assessment model based on standard action precision requirements, task criticality level, basic environmental risk factors, and motion trajectory risk indicators.

[0120] The intent probability distribution refers to a diversified probability estimate of the operator's intent. Specifically, a spatiotemporal attention neural network can be used to weight and fuse interaction features from different modalities to output a probability set, reflecting the uncertainty of the operator's intent. The task knowledge base is a database storing prior knowledge of the task. Specifically, a graph database can be used to store the mapping relationship between task types and corresponding action precision and criticality levels, providing benchmark parameters for risk assessment. Standard action precision requirements refer to the required operational precision threshold for a specific task type. This can be quantified using millimeter or micrometer-level numerical ranges to set the tolerance for trajectory deviation. Basic environmental risk factors refer to the set of environmental elements affecting safety. Specifically, 3D point cloud data can be used to represent the distribution of static obstacles, and a Kalman filter can be used to predict the trajectory of dynamic obstacles, used to construct an environmental risk model. The motion trajectory risk index refers to a quantitative parameter reflecting the risk of trajectory execution. Specifically, Euclidean distance can be used to calculate the minimum distance between trajectory points and obstacles, and a sliding window can be used to statistically analyze velocity extremes, used to assess motion control risk.

[0121] Specifically, this method matches operational intentions with preset task standards through a task knowledge base to establish a benchmark parameter system for risk assessment. For example, when an assembly task intention is detected, the corresponding high-precision action requirements are automatically invoked as the assessment benchmark. Obstacle spatial distribution data is extracted through an interactive scene semantic graph; for example, a semantic segmentation network is used to identify the coordinates of fixed equipment and the trajectories of moving objects within the work area to construct an environmental risk topology. Kinematic parameters are calculated based on the robot's expected trajectory envelope; for example, the maximum centripetal acceleration is detected at points of abrupt change in trajectory curvature to determine if it exceeds a safety threshold. A risk-weighted assessment model integrates task criticality, environmental complexity, and motion risk indicators; for example, the criticality weight of the assembly task is set to 0.7, and that of the handling task to 0.3, and the comprehensive risk value is calculated by combining obstacle density and peak velocity. A fuzzy inference system is used to handle imprecise parameters; for example, when trajectory deviation is at a critical state, the membership level of the risk probability is calculated using a membership function.

[0122] Compared to related technologies, traditional methods typically assess collision risk based solely on single sensor data, such as calculating only the straight-line distance between the robotic arm's end effector and an obstacle. This solution, however, achieves multi-dimensional risk assessment by integrating prior standards from a task knowledge base with dynamic environmental characteristics. For example, it considers both task accuracy requirements and the predicted trajectory of moving obstacles simultaneously during risk assessment. Related technologies use fixed thresholds to determine risk levels, such as setting a uniform safety distance threshold. This solution, however, improves assessment adaptability through a dynamic weighting mechanism, such as automatically adjusting risk tolerance based on task criticality. Traditional risk assessment models lack handling for the uncertainty of operational intentions. This solution introduces an intention probability distribution; for example, when multiple intentions are detected simultaneously, a weighted average is used to calculate the comprehensive risk value, enhancing system robustness.

[0123] Through the above technical solutions, this application addresses to some extent the problem of inaccurate safety strategies caused by considering only one risk factor in related technologies. By integrating the mission's criticality level with dynamic environmental characteristics, such as automatically increasing sensitivity to trajectory deviations in precision assembly scenarios, the matching between risk assessment and mission objectives is improved. By quantifying motion trajectory risk indicators, such as detecting abrupt changes in trajectory acceleration and correlating them with the distribution of environmental obstacles, proactive prediction of potential collision risks is achieved. By processing multi-source heterogeneous data through fuzzy reasoning, such as converting non-peak velocity data into risk membership, the reliability of decision-making under complex working conditions is improved. The resulting multi-dimensional risk assessment system provides a quantitative basis for dynamically adjusting safety boundaries.

[0124] This application further proposes to determine the trajectory deviation tolerance threshold based on the standard motion accuracy requirements, determine the risk impact weight coefficient based on the task criticality level, determine the environmental complexity index based on the static obstacle spatial distribution and dynamic obstacle motion trajectory, and determine the environment-trajectory interaction risk value based on the minimum distance, peak velocity, and acceleration change rate among the environmental complexity index and motion trajectory risk index. The trajectory deviation tolerance threshold, risk impact weight coefficient, and environment-trajectory interaction risk value are input into the fuzzy inference system, and the initial risk probability is output through a comprehensive risk assessment using a preset fuzzy rule base. The initial risk probability is then calibrated based on the confidence level of the real-time quality credibility to obtain the risk probability.

[0125] The trajectory deviation tolerance threshold refers to the maximum error range allowed for the trajectory to deviate from the preset path. This can be achieved by weighting the task accuracy requirements with the robot's joint positioning errors, quantifying the tolerance for trajectory deviation during task execution. The risk impact weight coefficient is an amplification factor that adjusts the risk assessment results based on task importance. This can be obtained by matching the task's criticality level with a preset weight mapping table; for example, medical operation tasks are assigned a higher weight coefficient. The environmental complexity index is a comprehensive evaluation index of obstacle distribution density and the uncertainty of dynamic obstacle movement. This can be achieved by fusing the number of obstacles, the variance of movement speed, and the probability of path intersection. The environment-trajectory interaction risk value is a quantitative representation of the probability of collision between the robot's trajectory and environmental elements. This can be calculated using a nonlinear combination of the minimum distance between trajectory key points and obstacles, peak velocity, and rate of acceleration change. The fuzzy rule base is a set of logical judgments describing the coupling relationships of multiple risk factors. This can be defined using IF-THEN rules to define the risk level output corresponding to different combinations of input variables.

[0126] Specifically, when the robot performs handling operations, the trajectory deviation tolerance threshold is first calculated to be ±5mm based on the gripping accuracy requirements. Simultaneously, a high-risk impact weight coefficient of 1.8 is set based on the fragile nature of the handled items. The environmental complexity index is calculated to be 0.75 using obstacle distribution data obtained from LiDAR. Combined with the minimum obstacle distance of 0.3m, peak velocity of 1.2m / s, and acceleration of 2m / s² in the trajectory output by the trajectory planning module, the environment-trajectory interaction risk value is calculated to be 0.68. After inputting the above parameters into the fuzzy inference system, the fuzzy rule of "high-complexity environment + medium-risk value" is triggered, outputting an initial risk probability of 65%. At this point, the electromyography signal reliability factor detects sensor noise interference, and the real-time quality reliability drops to 0.7. The risk probability is then calibrated to 78% using a Bayesian probability correction model, ultimately generating a more conservative safety boundary strategy.

[0127] In some specific implementations, the calculation of the environmental complexity index can be decomposed into two dimensions: static complexity and dynamic complexity. For example, the number of static obstacles accounts for 60% of the weight, while the variance of the dynamic obstacle velocity accounts for 40%. The fuzzy rule base can contain nine core rules. For instance, when the trajectory deviation tolerance threshold is lower than the first threshold and the environment-trajectory interaction risk value is higher than the second threshold, a high-risk level determination is directly triggered. The confidence calibration process can use linear interpolation. For example, for every 0.1 decrease in real-time quality confidence, the risk probability increases by 5 percentage points.

[0128] Compared to related technologies, traditional risk assessment methods typically consider only the distance to a single obstacle or a preset safe speed threshold, failing to establish an assessment model that couples multiple factors. This solution significantly improves the accuracy of assessments in complex dynamic scenarios by integrating a three-dimensional risk assessment system that considers task attributes, environmental conditions, and motion characteristics, combined with the ability of fuzzy logic to handle nonlinear relationships. Simultaneously, a real-time quality reliability feedback calibration mechanism is introduced to effectively overcome the interference of sensor data fluctuations on the assessment results.

[0129] Through the above technical solution, this application can automatically adjust the conservatism of risk assessment when sensor data is interfered with, accurately identify motion trajectory segments with high conflict risk, and provide a quantitative basis for dynamic safety boundary strategies. Especially in scenarios with multiple obstacles and high task accuracy requirements, by accurately calculating the risk value of environment-trajectory interaction, potential collision risks can be predicted in advance and robot motion parameters can be adjusted, avoiding misjudgment problems caused by local sensor failure.

[0130] This application further proposes a technical solution for generating target joint torque commands and dynamic safety boundary strategies for a target robot based on intent probability distribution, robot expected motion trajectory envelope, interaction risk level, and semantic graph of the interaction scene. The solution includes: determining a safety boundary relaxation coefficient negatively correlated with the interaction risk level; constructing a dynamic constraint set based on spatial constraint relationships and object attribute information in the semantic graph of the interaction scene, and generating a dynamic safety boundary strategy by combining the safety boundary relaxation coefficient; generating a preliminary joint torque command sequence through a model predictive controller, the optimization objectives of which include trajectory tracking accuracy, motion smoothness, and energy consumption; and using the dynamic safety boundary strategy to perform feasibility correction on the preliminary commands, selecting the first command of the corrected sequence as the target joint torque command.

[0131] The safety boundary relaxation coefficient is an adjustment parameter characterizing the degree of relaxation of safety constraints. Specifically, it can be implemented using an exponential decay function to map risk levels to the relaxation coefficient. This parameter is used to automatically reduce the constraint threshold in high-risk scenarios to enhance safety. The dynamic constraint set refers to variable physical constraints constructed based on environmental semantic understanding. Specifically, it can be implemented using a 3D spatial grid method combined with object motion attributes, used to transform the distribution of obstacles in the scene into no-go zones for robot movement. The model predictive controller refers to a control algorithm based on rolling time-domain optimization. Specifically, it can be implemented using a quadratic programming solver to achieve multi-objective optimization, used to balance trajectory tracking accuracy and energy efficiency. Feasibility correction refers to the constraint compliance verification process of control commands. Specifically, it can be implemented using the projected gradient descent method, used to ensure that the corrected command sequence meets the constraints of the dynamic safety boundary.

[0132] Specifically, when generating control commands, the strictness of safety constraints is first adjusted based on the real-time calculated interaction risk level. For example, when ambiguous operator intent or dense environmental obstacles are detected, the system automatically increases the safety level and tightens the speed limit threshold. Subsequently, based on semantic graphs, the system extracts the spatial coordinates and motion states of the object, constructing a set of safety constraints that includes static obstacle no-go zones and predicted trajectories for dynamic obstacles. After generating preliminary torque commands that balance trajectory tracking and energy consumption optimization through a model predictive controller, a constraint projection algorithm is used to map the command sequence to the feasible region defined by the dynamic safety boundary. Finally, the first command in the corrected sequence is selected as the execution target, forming a rolling optimization mechanism to achieve real-time updates of control commands.

[0133] Compared to related technologies, traditional methods using fixed safety thresholds lead to constraint redundancy in high-risk scenarios or insufficient protection in low-risk scenarios. This solution, however, achieves dynamic constraint adjustment through a risk-level adaptive relaxation coefficient. In related technologies, environmental constraints are often based on preset rules and lack semantic understanding; this solution, however, intelligently constructs constraint conditions by parsing environmental elements using semantic graphs. Traditional control methods treat safety policies as independent modules, resulting in a separation between instruction generation and constraint verification; this solution, through feasibility correction, achieves closed-loop integration of control instructions and safety policies.

[0134] Through the above technical solutions, this application addresses to some extent the problem of uncontrolled actions caused by the disconnect between control commands and safety strategies. When operator intent is ambiguous or environmental complexity suddenly increases, the system can automatically tighten safety constraints to prevent collisions caused by erroneous commands. In low-risk scenarios, constraints can be appropriately relaxed to improve the robot's motion efficiency and flexibility. Through semantically driven dynamic constraint construction, safety strategies can accurately match environmental characteristics; for example, automatically increasing force limits when identifying fragile objects. The combination of model predictive control and real-time correction ensures both forward-looking optimization of command generation and immediate compliance verification of safety boundaries.

[0135] This application further proposes the following technical solutions, taking a robot controller as the execution subject as an example, see [link to relevant documentation]. Figure 5 This includes the following steps.

[0136] 501. Send the target joint torque command to the joint-level controller of the target robot to drive each joint motor to generate the corresponding output torque; 502. During the motion execution process, the actual joint state data and end effector pose data of the robot are collected in real time and compared with the preset safety boundary conditions in the dynamic safety boundary strategy in real time. 503. When a trend of violating safety boundary conditions is detected in the actual motion state, the target joint torque command is corrected online based on the preset safety constraint optimization algorithm to generate a corrected torque command that satisfies all safety boundary conditions. 504. The corrected torque command is sent to the joint-level controller in real time to realize the safe closed-loop control of the robot. After the human-machine interaction task is completed, the process data of this interaction is saved for system parameter optimization.

[0137] Specifically, a distributed joint control system architecture is adopted, with each joint equipped with an independent PID controller or impedance controller. This controller receives torque commands from the upper-level decision-making system and controls the joint motors to output precise torque via a high-precision current loop. In practice, the actual joint angles and angular velocities are acquired through joint encoders, and contact force information is obtained through a six-dimensional force / torque sensor installed at the end effector. Simultaneously, the robot's forward kinematics is used to calculate the actual pose of the end effector. A real-time optimization algorithm based on quadratic programming is employed to transform safety boundary conditions into inequality constraints. Under all constraints, the algorithm finds the corrected torque closest to the original command. For example, when the end effector velocity approaches a preset threshold, the optimization algorithm automatically reduces the torque commands for each joint to ensure the velocity does not exceed the safety limit. Millisecond-level updates of commands are achieved through a real-time Ethernet communication protocol (such as EtherCAT), ensuring the real-time performance of the control system. After the human-machine interaction task is completed, the process data is saved for system parameter optimization, including storing complete motion trajectories, torque command sequences, and safety boundary trigger records. Offline analysis is used to optimize the dynamic model parameters and safety boundary thresholds.

[0138] In precision assembly tasks, when the robot controller detects that the end effector is about to contact a precision workpiece, the dynamic safety boundary strategy narrows the position tolerance and the maximum contact force threshold. If real-time monitoring detects that the actual contact force tends to exceed the threshold, the safety constraint optimization algorithm immediately corrects the torque command to control the contact force within a safe range. Simultaneously, the system records the force control data of this contact process to optimize the impedance control parameters for subsequent similar tasks.

[0139] Compared to traditional position control or fixed-gain force control, this implementation scheme achieves adaptive force control in dynamic environments through real-time safety monitoring and online optimization correction. Especially in unstructured task scenarios, it can effectively avoid excessive force collisions caused by model errors or environmental uncertainties, significantly improving the safety and accuracy of human-machine collaboration.

[0140] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0141] Figure 6 This is a schematic diagram of the structure of a human-computer interaction system based on intelligent sensors provided in an embodiment of this application. See also... Figure 6 The system includes: The acquisition module 601 is used to respond to the human-computer interaction operation of the operator on the target robot. It acquires the original interaction data between the operator and the target robot and the body state sensor data of the target robot through intelligent sensors. The original interaction data includes tactile pressure data and electromyographic signal sequence. The tactile pressure data is used to represent the pressure distribution pattern of the contact surface of the human-computer interaction, and the electromyographic signal sequence is used to represent the spatiotemporal pattern of the operator's muscle activity. The determination module 602 is used to determine the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot based on the original interaction data, the robot dynamics model of the target robot and the body state sensor data. The generation module 603 is used to generate target joint torque commands and dynamic safety boundary strategies for the target robot based on multimodal interaction features, semantic graphs of interaction scenarios, and real-time quality reliability of raw interaction data. The control module 604 is used to control the target robot based on the target joint torque command and the dynamic safety boundary strategy, so that the target robot can complete the task of human-machine interaction operation instructions.

[0142] It should be noted that the above embodiments of the human-computer interaction system based on intelligent sensors, when performing model early warning, are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the robot controller can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the human-computer interaction system based on intelligent sensors provided in the above embodiments and the human-computer interaction method based on intelligent sensors belong to the same concept; the specific implementation process is detailed in the method embodiments and will not be repeated here.

[0143] Figure 7 This is a schematic diagram of a robot controller 700 provided in an embodiment of this application. The robot controller 700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 701 and one or more memories 702. The one or more memories 702 store at least one computer program, which is loaded and executed by the one or more processors 701 to implement the methods provided in the various method embodiments described above. Of course, the robot controller 700 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The robot controller 700 may also include other components for implementing device functions, which will not be elaborated upon here.

[0144] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform a human-computer interaction method based on a smart sensor as described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0145] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of the robot controller reads the program code from the computer-readable storage medium and executes the program code, causing the robot controller to perform the aforementioned human-computer interaction method based on intelligent sensors.

[0146] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a robot controller, or on multiple robot controllers located in one location, or on multiple robot controllers distributed in multiple locations and interconnected through a communication network. Multiple robot controllers distributed in multiple locations and interconnected through a communication network may form a blockchain system.

[0147] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0148] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A human-computer interaction method based on intelligent sensors, characterized in that, The method includes: In response to the human-computer interaction operation of the operator on the target robot, the original interaction data between the operator and the target robot and the body state sensor data of the target robot are acquired through intelligent sensors. The original interaction data includes tactile pressure data and electromyographic signal sequence. The tactile pressure data is used to represent the pressure distribution pattern of the contact surface of the human-computer interaction, and the electromyographic signal sequence is used to represent the spatiotemporal pattern of the operator's muscle activity. Based on the original interaction data, the robot dynamics model of the target robot, and the body state sensor data, the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot are determined. Based on the multimodal interaction features, the semantic graph of the interaction scene, and the real-time quality reliability of the original interaction data, the target joint torque command and dynamic safety boundary strategy of the target robot are generated. Based on the target joint torque command and the dynamic safety boundary strategy, the target robot is controlled to enable the target robot to complete the task instructed by the human-machine interaction operation.

2. The method according to claim 1, characterized in that, The process of determining the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot based on the original interaction data, the robot dynamics model of the target robot, and the body state sensor data includes: Based on the robot dynamics model, the original interaction data is subjected to quality assessment and spatiotemporal registration to obtain a spatiotemporally synchronized multimodal data stream and the real-time quality reliability of the original interaction data. The real-time quality reliability includes a tactile reliability factor and an electromyographic reliability factor. Based on the multimodal data stream, the real-time quality reliability, and the ontological state sensor data, the multimodal interaction features and semantic graph of the interaction scene between the operator and the target robot are determined.

3. The method according to claim 2, characterized in that, The process of performing quality assessment and spatiotemporal registration on the original interaction data based on the robot dynamics model to obtain a spatiotemporally synchronized multimodal data stream and the real-time quality reliability of the original interaction data includes: Extract the spatial distribution features and time series features of the tactile pressure data; determine the tactile confidence factor based on the spatial distribution features and the time series features; Extract the frequency domain power spectrum features and time domain amplitude statistical features of the electromyography signal sequence; determine the electromyography reliability factor based on the frequency domain power spectrum features and the time domain amplitude statistical features; The tactile pressure data, the electromyographic signal sequence, and the proprioceptive state sensor data are spatiotemporally synchronized to obtain the multimodal data stream.

4. The method according to claim 2, characterized in that, The step of determining the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot based on the multimodal data stream, the real-time quality reliability, and the ontological state sensor data includes: Parallel feature extraction is performed on the body state sensor data and the multimodal data stream to obtain a basic motion feature set, a basic tactile feature set, and a basic electromyographic feature set; Based on the tactile confidence factor and the electromyographic confidence factor, dynamic weight allocation and feature fusion are performed on the basic tactile feature set and the basic electromyographic feature set to obtain tactile feature vector and electromyographic feature vector; The tactile feature vector, the electromyographic feature vector, and the basic motion feature set are spatiotemporally aligned and concatenated to obtain the multimodal interaction features; The multimodal interaction features are input into the scene analysis model, and the scene analysis model performs semantic parsing on the multimodal interaction features to obtain target structured data, which includes object attributes, spatial relationships and task context information. Based on the target structured data, a semantic graph of the interaction scenario, including semantic nodes and relation edges, is constructed.

5. The method according to claim 1, characterized in that, The process of generating target joint torque commands and dynamic safety boundary strategies for the target robot based on the multimodal interaction features, the semantic graph of the interaction scene, and the real-time quality reliability of the original interaction data includes: Based on the multimodal interaction features, the semantic graph of the interaction scene, and the real-time quality reliability, the operator's intention probability distribution, the robot's expected motion trajectory envelope, and the interaction risk level are determined. The intention probability distribution refers to the diversified probability estimation of the operator's intention. Based on the intent probability distribution, the robot's expected motion trajectory envelope, the interaction risk level, and the interaction scene semantic graph, the target joint torque command and dynamic safety boundary strategy of the target robot are generated.

6. The method according to claim 5, characterized in that, The determination of the operator's intent probability distribution, the robot's expected motion trajectory envelope, and the interaction risk level based on the multimodal interaction features, the semantic graph of the interaction scene, and the real-time quality reliability includes: The multimodal interaction features are input into a spatiotemporal attention neural network to extract high-dimensional interaction features with spatiotemporal correlation. The attention weights of the spatiotemporal attention neural network are dynamically adjusted according to the real-time quality reliability. Based on the high-dimensional interaction features and the semantic graph of the interaction scene, the probabilities of multiple preset intent categories are determined, and the intent probability distribution is obtained. Based on the intent probability distribution and the high-dimensional interaction features, the future motion trajectory of the target robot is determined; based on the spatial constraint relationships in the semantic graph of the interaction scene and the future motion trajectory, the expected motion trajectory envelope of the robot is determined. Based on the intent probability distribution, the robot's expected motion trajectory envelope, and the semantic graph of the interaction scene, the risk probability of different interaction strategies is determined, and based on the risk probability of different interaction strategies, the interaction risk level is determined.

7. The method according to claim 6, characterized in that, Determining the future trajectory of the target robot based on the intent probability distribution and the high-dimensional interaction features includes: Based on the intent probability distribution, multiple candidate task objectives that the target robot needs to execute and the task objective weight of each candidate task objective are determined. Based on the high-dimensional interaction features and the task target weights, multiple predicted motion trajectories of the target robot are generated; Based on the dynamic and kinematic constraints of the target robot, the feasibility of the multiple predicted motion trajectories is verified and optimized to obtain a set of motion trajectories that conform to physical laws. Select a target trajectory from the set of motion trajectories, and add a confidence boundary to the target trajectory based on the uncertainty of the intention probability distribution to obtain the future motion trajectory.

8. The method according to claim 6, characterized in that, The step of determining the risk probability of different interaction strategies based on the intent probability distribution, the robot's expected motion trajectory envelope, and the semantic graph of the interaction scene includes: Based on the aforementioned intent probability distribution, the standard action precision requirements and task criticality level corresponding to the candidate intent with the highest probability are queried from the task knowledge base. Based on the semantic graph of the interaction scene, the spatial distribution of static obstacles, the movement trajectory of dynamic obstacles, and the boundary of the safe operation area in the environment are extracted as basic environmental risk factors. Based on the expected motion trajectory envelope of the robot, the minimum distance between each point on the trajectory and the obstacle, the peak velocity at key points on the trajectory, and the rate of change of acceleration are determined to obtain the motion trajectory risk index. Based on the standard motion accuracy requirements, the task criticality level, the basic environmental risk factors, and the motion trajectory risk indicators, the risk probability of different interaction strategies is determined through a risk-weighted assessment model.

9. The method according to claim 8, characterized in that, The method for determining the risk probability of different interaction strategies based on the standard motion accuracy requirements, the task criticality level, the basic environmental risk factors, and the motion trajectory risk indicators, through a risk-weighted assessment model, includes: Based on the aforementioned standard motion accuracy requirements, determine the trajectory deviation tolerance threshold; Based on the aforementioned task criticality level, determine the risk impact weighting coefficient; Based on the spatial distribution of static obstacles and the movement trajectory of dynamic obstacles in the basic environmental risk factors, an environmental complexity index is determined; based on the environmental complexity index and the minimum distance, peak velocity, and rate of change of acceleration in the movement trajectory risk index, an environment-trajectory interaction risk value is determined. The trajectory deviation tolerance threshold, risk impact weight coefficient, and environment-trajectory interaction risk value are input into the fuzzy inference system. A comprehensive risk assessment is performed through a preset fuzzy rule base, and the initial risk probability of different interaction strategies is output. The initial risk probability is calibrated based on the real-time quality confidence level to obtain the risk probability of different interaction strategies.

10. A human-computer interaction system based on intelligent sensors, characterized in that, The system includes: The acquisition module is used to respond to the human-computer interaction operation of the operator on the target robot, and to acquire the original interaction data between the operator and the target robot and the body state sensor data of the target robot through intelligent sensors. The original interaction data includes tactile pressure data and electromyographic signal sequence. The tactile pressure data is used to represent the pressure distribution pattern of the contact surface of the human-computer interaction, and the electromyographic signal sequence is used to represent the spatiotemporal pattern of the operator's muscle activity. The determination module is used to determine the multimodal interaction features and semantic map of the interaction scene between the operator and the target robot based on the original interaction data, the robot dynamics model of the target robot, and the body state sensor data. The generation module is used to generate the target joint torque command and dynamic safety boundary strategy of the target robot based on the multimodal interaction features, the semantic graph of the interaction scene, and the real-time quality reliability of the original interaction data. The control module is used to control the target robot based on the target joint torque command and the dynamic safety boundary strategy, so that the target robot can complete the task instructed by the human-machine interaction operation.

Images