Robot man-machine interaction method based on visual gesture recognition

Abstract

The invention discloses a robot man-machine interaction method based on visual gesture recognition. According to the method, the real-time response speed and the long-term adaptive capacity of gesture recognition are remarkably improved by dynamically updating the model weight and a confusion matrix optimization mechanism. And when the two-channel network structure receives the feature data in the step S3, static spatial features and dynamic time sequence features can be synchronously processed, and a classification penalty strategy is adjusted in combination with the user error correction data in the step S8, so that the model continuously optimizes the capability of distinguishing complex gestures in the interaction process. When an intention correction signal is returned, the online distillation mechanism quickly updates the weight of the LSTM layer, so that the problem of lag caused by traditional batch training is avoided, a robot can gradually adapt to the action habits and expression styles of a user in continuous interaction, the intention judgment accuracy and environmental adaptability are effectively enhanced, a decision instruction better meets the actual operation requirement, and the user experience is improved. And the misjudgment probability caused by environmental interference or action deformation is reduced.

Description

Technical Field

[0001] This invention belongs to the field of robot interaction technology, specifically a robot human-computer interaction method based on visual gesture recognition. Background Technology

[0002] Robot-human interaction refers to the two-way information exchange and collaboration between humans and robots in a natural and efficient manner, integrating various technologies such as speech recognition, visual perception, tactile feedback, and affective computing. Robot gesture recognition interaction is a human-computer interaction method based on computer vision and deep learning technologies. It captures the user's hand movements through cameras or sensors, and uses image processing and pattern recognition algorithms to analyze posture, trajectory, and gesture meaning in real time, thereby enabling command control of the robot. This interaction method is natural, intuitive, and contactless, suitable for various scenarios such as smart homes, service robots, education and training, and medical rehabilitation. Gesture recognition not only improves the convenience and flexibility of human-computer interaction but also effectively reduces the hygiene risks associated with physical contact. With the advancement of artificial intelligence technology and the upgrading of hardware devices, gesture recognition interaction is developing towards higher precision, stronger robustness, and multimodal fusion, providing important support for building intelligent and human-centered human-computer collaborative systems.

[0003] However, existing technologies for gesture recognition in robot interaction often suffer from insufficient adaptability due to fixed model parameters, making it difficult to optimize the recognition accuracy of complex gestures in real time. When fusing multimodal data, simple splicing methods are often used without dynamic weight allocation, making them susceptible to interference from a single data source. User feedback mechanisms are mostly processed offline, failing to form a real-time closed loop, resulting in a lack of personalized interactive experience and difficulty in effectively reducing the misjudgment rate over long-term use. Summary of the Invention

[0004] The purpose of this invention is to provide a robot human-computer interaction method based on visual gesture recognition in order to solve the problems mentioned above.

[0005] The technical solution adopted in this invention is as follows: a robot human-computer interaction method based on visual gesture recognition, the method comprising the following steps: S1: Synchronously acquires dynamic sequences of user gestures and depth information of the surrounding environment through an RGB-D camera and an infrared sensor, and receives system optimization parameters output by S9 in real time to adjust the acquisition frame rate and resolution to ensure stable image quality under complex lighting conditions.

[0006] S2: Perform background subtraction and motion blur correction on the raw data collected by S1, adaptively adjust the edge detection threshold according to the feature extraction requirements of S3, and transmit the denoised image data to S3 for feature parsing.

[0007] S3: Extract the coordinates of key hand points and micro-features of muscle movement from the image output by S2, construct the spatiotemporal trajectory matrix of dynamic gestures, and simultaneously push the feature vectors to the S4 model training module and the S5 intention prediction unit.

[0008] S4: Train a dual-channel CNN-LSTM network using the feature data from S3, optimize the confusion matrix by combining the user error correction data fed back from S8, output the gesture category probability to S5 in real time, and receive the intent correction signal returned from S5 to update the model weights.

[0009] S5: Integrates the gesture recognition results of S4, the environmental semantic information of S1, and the historical interaction records stored in S8. It focuses on key action segments through an attention mechanism, predicts the user's potential operation intentions, generates high-level decision instructions and sends them to S6, and feeds back the intention confidence to S4 to optimize the recognition accuracy.

[0010] S6: Based on the decision instructions of S5, call the kinematic model to generate joint trajectories, receive real-time force feedback data from S7 to adjust the motion flexibility parameters, and package the optimized execution instructions to transmit to the S7 execution unit.

[0011] S7: Executes the action command issued by S6, senses the change in contact force through the torque sensor, and dynamically corrects the movement path by combining the environmental obstacle data of S1, and synchronously transmits the trajectory adjustment result back to the S8 feedback system.

[0012] S8: Capture user satisfaction with the execution of the S7 action through facial expression recognition and voice emotion analysis, and push the quantitative feedback data to the S4 model optimization module and the S9 system parameter library to establish a user preference profile.

[0013] S9: Calculate the performance indicators of each module based on the feedback data of S8, dynamically adjust the acquisition parameters of S1, the recognition threshold of S4 and the action planning algorithm of S6, generate a system optimization report and send the updated parameters to S1 to execute a new round of interactive loop.

[0014] In a preferred embodiment, in step S1, gesture and environmental information are acquired using a depth camera and a six-axis inertial sensor. The depth camera captures RGB-D images of the hand area at a frequency of 30 frames per second, simultaneously outputting a 1920×1080 pixel color image and a 640×480 pixel depth map; the inertial sensor records wrist acceleration and angular velocity data in real time, with a sampling frequency set to 100Hz. The environmental information acquisition module integrates a temperature sensor and an ultrasonic rangefinder to monitor temperature changes in the operating area and obstacle distances, respectively. Upon system startup, the initial parameter configuration issued by S9 is loaded, including sensor sampling thresholds and data filtering coefficients. During operation, the system continuously receives parameter adjustments dynamically from S9; for example, when the ambient light intensity exceeds 500 lux, the camera exposure compensation is automatically increased to +0.7EV, while the depth map noise filtering threshold is reduced to 85% of the original set value. The acquired raw data is transmitted to the preprocessing unit via a USB 3.0 interface, with the transmission delay controlled within 20ms.

[0015] In a preferred embodiment, in step S2, the image data preprocessing process sequentially includes Gaussian filtering for noise reduction, hand region segmentation based on a skin color model, and scale normalization to 224×224 pixels. The filter kernel size is dynamically adjusted according to the feature extraction requirements fed back in S3; a 3×3 kernel is used when S3 needs to enhance edge features, and a 5×5 kernel is used when focusing on texture details. Inertial data undergoes motion noise elimination using a Kalman filter. The process noise covariance matrix of the state equation during filtering is determined by the trajectory modeling accuracy requirements of S3; when the allowable range of trajectory prediction error increases, the diagonal element values ​​of the matrix increase by 20%. The preprocessed data is encapsulated into feature tensors according to the format specified in S3. Image data is converted to floating-point arrays and normalized to the [0,1] interval, while inertial data retains three decimal places and uses timestamps as indices to construct a time-series matrix. Data transmission uses the TCP protocol, and the size of each batch of data packets is controlled within 1MB to ensure real-time performance.

[0016] In a preferred embodiment, in step S3, the spatial feature extraction branch uses an improved MobileNetV2 network to extract the coordinates of 21 key points of the hand from the normalized image through depthwise separable convolution, including the three-dimensional coordinates of the fingertip position, knuckle angle, and palm center. The temporal feature modeling branch uses a bidirectional GRU network to encode the temporal matrix constructed from inertial data, outputting a 128-dimensional feature vector containing the rate of change of gesture speed and the duration of the action. In the feature fusion stage, the weight distribution of spatial and temporal features is calculated through a self-attention mechanism. When the hand movement speed exceeds 0.5 m / s, the weight of the temporal feature is increased to 0.6, and the weight of the spatial feature is 0.7 for static gestures. After modeling, two data products are generated: a structured feature table and a visual heatmap. The structured feature table is pushed to S4 in JSON format, containing 18 feature indicators such as the mean of key point coordinates and trajectory curvature; the heatmap is sent to S5 in PNG format for visual reference to assist in intent judgment. The feature extraction time is controlled within 30ms to meet the requirements of real-time interaction.

[0017] In a preferred embodiment, step S4 employs a dual-channel spatiotemporal fusion CNN-LSTM network algorithm, comprising a spatial feature extraction channel and a temporal dynamic modeling channel: the spatial channel uses three convolutional layers (Conv2D+BN+ReLU) to extract static features such as relative joint positions and palm orientation from the hand keypoint heatmap output in S3; the temporal channel uses two LSTM layers to temporally encode the gesture trajectory matrix generated in S3, capturing the rate of change of gesture speed and the pause pattern of the action. The features from the two channels are weighted and fused through an attention mechanism and then input into a fully connected layer, outputting a 128-dimensional gesture feature vector.

[0018] During the training phase, the feature data of S3 is divided into training and validation sets in an 8:2 ratio, and the cross-entropy loss function is used to optimize the network parameters. Simultaneously, user error correction data from S8 is received, and the loss calculation is optimized by dynamically adjusting the penalty coefficients for the corresponding categories in the confusion matrix (increasing the weight of misclassified samples by 1.5 times). During inference, the network outputs in real time the probability distribution of the current gesture belonging to 16 predefined categories (e.g., "rotate" 0.87, "translate" 0.11) and sends it to S5. When S5 returns an intent correction signal (e.g., "the most probable 'rotate' gesture conflicts with the environmental semantics; the actual intent is 'press'"), the system triggers an online distillation mechanism. A pseudo-sample is constructed using the corrected intent label ("press") and the original feature vector, and the LSTM layer weights are updated using momentum gradient descent (temporarily increasing the learning rate to 1.2 times the base value), achieving dynamic adaptation of the model during the interaction process.

[0019] The formula for dynamically updating model weights is: In the formula: Wt+1 represents the updated model weight matrix; Wt represents the weight matrix at the current time step; η represents the base learning rate; ∇Lt represents the gradient of the cross-entropy loss function at time t; α represents the feedback adjustment coefficient (ranging from 0.05 to 0.2, dynamically determined by the S8 feedback satisfaction level). sign(∇Lt) represents the gradient direction sign function; C S8 This represents the S8 user error correction confidence vector (elements are 0-1, quantized based on the intensity of user feedback). I S5 This indicates the S5 intent correction indicator vector (the correct intent category corresponds to 1, and the rest are 0); ⊙ represents the Hadamard product (element-by-element multiplication).

[0020] In a preferred embodiment, in step S5, the Transformer model is used to integrate the gesture recognition results of S4, the environmental semantic information of S1, and the historical interaction records stored in S8. The model comprises three parallel encoding branches and a cross-attention fusion layer: the gesture encoding branch converts the 16-dimensional gesture category probability vector output by S4 (e.g., "rotate" 0.87) into a feature matrix through an embedding layer; the environment encoding branch performs word vector mapping and position encoding on the environmental semantic data collected by S1 (e.g., "a glass on the table" "light intensity 300 lux"); and the history encoding branch extracts user preference features (e.g., "right-handed" "frequently performs grasping actions") from the interaction records stored in S8. The features from the three branches are normalized by the layers and then input into the cross-attention layer, which dynamically calculates attention weights (e.g., gesture feature weight 0.6, environment feature weight 0.3, history feature weight 0.1) to focus on key action segments (e.g., the "pinch" gesture when the hand approaches the glass). The fused features are input into the bidirectional Transformer decoder, and the output of the softmax layer is the probability distribution of 8 potential intentions (e.g., "pouring water" 0.72, "moving" 0.21). The intention with the highest probability is selected to generate a high-level decision instruction (e.g., "execute the sequence of grabbing and pouring a glass") and sent to S6. At the same time, the intention confidence score is calculated (e.g., 0.72). If the score is lower than the threshold (0.65), the low-confidence intention label (e.g., "pouring water") and the corresponding gesture features are fed back to S4, triggering the S4 model to adjust the confusion matrix penalty coefficient for that gesture category. The formula for calculating multimodal attention weights is: ; In the formula: Am represents the attention weight of the m-th modality feature. Here, m=1 represents the gesture modality, m=2 represents the context modality, and m=3 represents the history modality.

[0021] Sim(Fm,Q) represents the cosine similarity between the m-th modal feature matrix F_mFm and the query vector QQ.

[0022] λm represents the modality adjustment factor. The gesture modality is fixed at 1.2, the environment modality is dynamically set from 0.8 to 1.5 by the S1 environment complexity, and the history modality is set from 0.5 to 1.0 by the S8 interaction frequency.

[0023] In a preferred embodiment, in step S6, the system's built-in action primitive library contains 8 categories of basic operations, each category of operation being associated with 5-8 sub-action sequences. The planning process employs a hierarchical optimization strategy: firstly, through A... The algorithm plans the global path, and the path cost function comprehensively considers obstacle distance, joint motion energy consumption, and execution time. The energy consumption weight is dynamically adjusted based on motor temperature data fed back from S7; when the temperature exceeds 60℃, the weight increases to 1.3 times the original set value. Local paths are corrected in real-time using a model predictive control algorithm with a control cycle of 10ms. During the motion parameter generation stage, the execution precision is adjusted based on the confidence level of the S5 intent. When the confidence level is higher than 0.8, a high-precision mode is activated, reducing the allowable joint angle control error range to ±0.5°; when the confidence level is lower than 0.6, a robust mode is switched, increasing the allowable error range to ±1.2° by adding a safety margin. The generated execution instructions are encapsulated in XML format, including joint angle sequences, motion speed curves, and force feedback thresholds, and are sent to the execution unit via the CAN bus.

[0024] In a preferred embodiment, in step S7, the execution unit consists of a 6-DOF robotic arm and an end effector, with each joint equipped with an absolute encoder and a torque sensor. Motion control employs a position-force hybrid control strategy, with a position loop control frequency of 1kHz and a force loop sampling frequency of 2kHz. During execution, three key indicators are monitored in real time: the deviation between the actual joint position and the commanded position, the end effector contact force, and the smoothness of the motion trajectory. When the position deviation exceeds a threshold or the contact force suddenly increases by more than 15%, the system triggers an emergency deceleration mechanism and simultaneously initiates path replanning. Path adjustment uses a Bezier curve interpolation algorithm, with the new path generation time controlled within 50ms. The execution result includes two parts: a motion completion score and an anomaly event log. The completion score is calculated based on trajectory tracking error and force control accuracy, with a maximum score of 10 points; the anomaly event log records the time of exceeding the tolerance, sensor values, and adjustment measures. The result data is packaged and transmitted back to S8 every 200ms, using the UDP protocol to reduce latency.

[0025] In a preferred embodiment, in step S8, the interface supports three input methods: voice commands, touch operation, and facial expression recognition. Voice feedback uses an end-to-end recognition model based on Transformer, with an accuracy rate set above 95%. A secondary confirmation mechanism is triggered when the recognition confidence level falls below 80%. Touch feedback is implemented through virtual buttons on the interface, with six standard feedback options such as "correct execution," "excessive force," and "trajectory deviation." After the user clicks, the system records the feedback type and trigger timestamp. Facial expression recognition uses a lightweight CNN model to extract emotional features from facial images captured by the camera, classifying the user's emotional state during feedback into three categories: "satisfied," "neutral," and "dissatisfied." The classification results are stored in association with the feedback content. After preprocessing, the feedback data generates two outputs: an error correction sample set and an emotion score. The error correction sample set contains feature data of incorrectly executed actions and correct labels, and is updated daily to the S4 model training library. The emotion score uses a 5-point scale, with the average score calculated daily and pushed to S9 as one of the bases for parameter optimization.

[0026] In a preferred embodiment, step S9 includes three dimensions for the optimization objective: recognition accuracy, action execution efficiency, and user satisfaction, with weights set to 0.4, 0.3, and 0.3, respectively. The system parameter library is divided into static and dynamic parameters. Static parameters, such as sensor sampling frequency, are evaluated and adjusted quarterly, while dynamic parameters, such as model learning rate, are updated daily. The optimization process is divided into offline and online phases: the offline phase uses historical data from the past 7 days for batch optimization, employing a genetic algorithm to search for the optimal parameter combination, with a population size of 50 and 50 iterations; the online phase fine-tunes based on the sentiment score pushed in real-time in S8. When the satisfaction score is below 3.5 for three consecutive hours, a rapid optimization process is initiated, temporarily increasing the learning rate by 20% and adjusting the confusion matrix penalty coefficient. The optimized parameters are sent to S1 via an encrypted channel, using an incremental update mechanism that transmits only changed parameter items to reduce data volume. The system maintains normal operation during parameter updates, and a smooth transition strategy is used when switching to new parameters to avoid abrupt changes in action execution.

[0027] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. In this invention, the real-time response speed and long-term adaptive capability of gesture recognition are significantly improved through a dynamic update mechanism for model weights and a confusion matrix optimization mechanism. The dual-channel network structure of S4, when receiving feature data from S3, can simultaneously process static spatial features and dynamic temporal features. Combined with user error correction data from S8, it adjusts the classification penalty strategy, enabling the model to continuously optimize its ability to distinguish complex gestures during interaction. When S5 returns the intent correction signal, the online distillation mechanism quickly updates the LSTM layer weights, avoiding the lag problem caused by traditional batch training, allowing the robot to gradually adapt to the user's movement habits and expression style during continuous interaction.

[0028] 2. In this invention, the introduction of multimodal data fusion and attention mechanisms effectively enhances the accuracy and environmental adaptability of intent judgment. The S5 Transformer model integrates gesture probability vectors, environmental semantic information, and historical interaction records. By dynamically adjusting modal weights, it focuses on key action segments, reducing the limitations of single data sources. For example, in complex environments, the system can prioritize referencing environmental semantic data to correct gesture ambiguities. When the user frequently performs specific actions, it automatically increases the weight of historical interaction features, making decision-making instructions more aligned with actual operational needs and reducing the probability of misjudgment due to environmental interference or action distortion.

[0029] 3. In this invention, the user feedback closed loop and dynamic parameter adjustment mechanism further enhance the smoothness and personalized experience of human-computer interaction. Error correction data and sentiment scores collected in S8 are continuously fed back into the model training process. S4 optimizes the focus on frequently corrected gesture categories through a confusion matrix, while S5 dynamically adjusts the decision threshold based on intent confidence. These two elements work together to form a complete closed loop of "recognition-feedback-optimization." This design allows the system to maintain stable interactive performance under different users and scenarios, while gradually accumulating preference models for specific users, making the robot's response more aligned with individual usage habits and improving user satisfaction in long-term interactions. Attached Figure Description

[0030] Figure 1 This is a schematic diagram illustrating the process principle of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] Reference Figure 1 A robot human-computer interaction method based on visual gesture recognition, the method includes the following steps: S1: Synchronously collects dynamic sequences of user gestures and depth information of the surrounding environment through an RGB-D camera and an infrared sensor, and receives system optimization parameters output by S9 in real time to adjust the acquisition frame rate and resolution to ensure stable image quality under complex lighting conditions.

[0033] S2: Perform background subtraction and motion blur correction on the raw data collected by S1, adaptively adjust the edge detection threshold according to the feature extraction requirements of S3, and transmit the denoised image data to S3 for feature parsing.

[0034] S3: Extract the coordinates of key hand points and micro-features of muscle movement from the image output by S2, construct the spatiotemporal trajectory matrix of dynamic gestures, and simultaneously push the feature vectors to the S4 model training module and the S5 intention prediction unit.

[0035] S4: Train a dual-channel CNN-LSTM network using the feature data from S3, optimize the confusion matrix by combining the user error correction data fed back from S8, output the gesture category probability to S5 in real time, and receive the intent correction signal returned from S5 to update the model weights.

[0036] S5: Integrates the gesture recognition results of S4, the environmental semantic information of S1, and the historical interaction records stored in S8. It focuses on key action segments through an attention mechanism, predicts the user's potential operation intentions, generates high-level decision instructions and sends them to S6, and feeds back the intention confidence to S4 to optimize the recognition accuracy.

[0037] S6: Based on the decision instructions of S5, call the kinematic model to generate joint trajectories, receive real-time force feedback data from S7 to adjust the motion flexibility parameters, and package the optimized execution instructions to transmit them to the S7 execution unit.

[0038] S7: Executes the action command issued by S6, senses the change in contact force through the torque sensor, and dynamically corrects the movement path by combining the environmental obstacle data of S1, and synchronously transmits the trajectory adjustment result back to the S8 feedback system.

[0039] S8: Capture user satisfaction with the execution of the S7 action through facial expression recognition and voice emotion analysis, and push the quantitative feedback data to the S4 model optimization module and the S9 system parameter library to establish a user preference profile.

[0040] S9: Calculate the performance indicators of each module based on the feedback data of S8, dynamically adjust the acquisition parameters of S1, the recognition threshold of S4 and the action planning algorithm of S6, generate a system optimization report and send the updated parameters to S1 to execute a new round of interactive loop.

[0041] In step S1, gesture and environmental information are acquired using a depth camera and a six-axis inertial sensor. The depth camera captures RGB-D images of the hand area at 30 frames per second, simultaneously outputting a 1920×1080 pixel color image and a 640×480 pixel depth map; the inertial sensor records wrist acceleration and angular velocity data in real time, with a sampling frequency of 100Hz. The environmental information acquisition module integrates a temperature sensor and an ultrasonic rangefinder to monitor temperature changes in the operating area and obstacle distances, respectively. Upon system startup, the initial parameter configuration issued by S9 is loaded, including sensor sampling thresholds and data filtering coefficients. During operation, it continuously receives parameter adjustments dynamically from S9; for example, when the ambient light intensity exceeds 500 lux, the camera exposure compensation is automatically increased to +0.7EV, while the depth map noise filtering threshold is reduced to 85% of the original setting. The acquired raw data is transmitted to the preprocessing unit via a USB 3.0 interface, with a transmission delay controlled within 20ms.

[0042] In step S2, the image data preprocessing process consists of Gaussian filtering for noise reduction, hand region segmentation based on a skin color model, and scale normalization to 224×224 pixels. The filter kernel size is dynamically adjusted based on the feature extraction requirements fed back in S3; a 3×3 kernel is used when edge features need to be enhanced, and a 5×5 kernel is used when texture details are emphasized. Inertial data undergoes motion noise elimination using a Kalman filter. The process noise covariance matrix of the state equation during filtering is determined by the trajectory modeling accuracy requirements of S3; when the allowable range of trajectory prediction error increases, the diagonal elements of the matrix increase by 20%. The preprocessed data is encapsulated into feature tensors according to the format specified in S3. Image data is converted to floating-point arrays and normalized to the [0,1] interval, while inertial data retains three decimal places and uses timestamps as indices to construct a time-series matrix. Data transmission uses the TCP protocol, with each batch of data packets kept under 1MB to ensure real-time performance.

[0043] In step S3, the spatial feature extraction branch uses an improved MobileNetV2 network to extract the coordinates of 21 key points of the hand from the normalized image through depthwise separable convolutions, including the three-dimensional coordinates of fingertip position, knuckle angle, and palm center. The temporal feature modeling branch uses a bidirectional GRU network to encode the temporal matrix constructed from inertial data, outputting a 128-dimensional feature vector containing the rate of change of gesture speed and the duration of the action. In the feature fusion stage, the weight distribution of spatial and temporal features is calculated through a self-attention mechanism. When the hand movement speed exceeds 0.5 m / s, the weight of the temporal feature is increased to 0.6, and the weight of the spatial feature is 0.7 for static gestures. After modeling, two data products are generated: a structured feature table and a visual heatmap. The structured feature table is pushed to S4 in JSON format, containing 18 feature indicators such as the mean of keypoint coordinates and trajectory curvature; the heatmap is sent to S5 in PNG format for visual reference to assist in intent judgment. The feature extraction time is controlled within 30ms to meet the requirements of real-time interaction.

[0044] In step S4, a dual-channel spatiotemporal fusion CNN-LSTM network algorithm is used, comprising a spatial feature extraction channel and a temporal dynamic modeling channel: the spatial channel uses three convolutional layers (Conv2D+BN+ReLU) to extract static features such as relative joint positions and palm orientation from the hand keypoint heatmap output in S3; the temporal channel uses two LSTM layers to temporally encode the gesture trajectory matrix generated in S3, capturing the rate of change of gesture speed and the pause pattern of the action. The features from the two channels are weighted and fused through an attention mechanism and then input into a fully connected layer, outputting a 128-dimensional gesture feature vector.

[0045] During the training phase, the feature data of S3 is divided into training and validation sets in an 8:2 ratio, and the cross-entropy loss function is used to optimize the network parameters. Simultaneously, user error correction data from S8 is received, and the loss calculation is optimized by dynamically adjusting the penalty coefficients for the corresponding categories in the confusion matrix (increasing the weight of misclassified samples by 1.5 times). During inference, the network outputs in real time the probability distribution of the current gesture belonging to 16 predefined categories (e.g., "rotate" 0.87, "translate" 0.11) and sends it to S5. When S5 returns an intent correction signal (e.g., "the most probable 'rotate' gesture conflicts with the environmental semantics; the actual intent is 'press'"), the system triggers an online distillation mechanism. A pseudo-sample is constructed using the corrected intent label ("press") and the original feature vector, and the LSTM layer weights are updated using momentum gradient descent (temporarily increasing the learning rate to 1.2 times the base value), achieving dynamic adaptation of the model during the interaction process.

[0046] The formula for dynamically updating model weights is: In the formula: Wt+1 represents the updated model weight matrix; Wt represents the weight matrix at the current time step; η represents the base learning rate; ∇Lt represents the gradient of the cross-entropy loss function at time t; α represents the feedback adjustment coefficient (ranging from 0.05 to 0.2, dynamically determined by the S8 feedback satisfaction level). sign(∇Lt) represents the gradient direction sign function; C S8 This represents the S8 user error correction confidence vector (elements are 0-1, quantized based on the intensity of user feedback). I S5 This indicates the S5 intent correction indicator vector (the correct intent category corresponds to 1, and the rest are 0); ⊙ represents the Hadamard product (element-by-element multiplication); In step S5, the Transformer model is used to integrate the gesture recognition results from S4, the environmental semantic information from S1, and the historical interaction records stored in S8. The model comprises three parallel encoding branches and a cross-attention fusion layer: the gesture encoding branch converts the 16-dimensional gesture category probability vector output by S4 (e.g., "rotate" 0.87) into a feature matrix through an embedding layer; the environment encoding branch performs word vector mapping and position encoding on the environmental semantic data collected by S1 (e.g., "a glass on the table" "light intensity 300 lux"); and the history encoding branch extracts user preference features (e.g., "right-handed" "frequently performs grasping actions") from the interaction records stored in S8. The features from the three branches are normalized by the layers and then input into the cross-attention layer, which dynamically calculates attention weights (e.g., gesture feature weight 0.6, environment feature weight 0.3, history feature weight 0.1) to focus on key action segments (e.g., the "pinch" gesture when the hand approaches the glass). The fused features are input into the bidirectional Transformer decoder, and the output of the softmax layer is the probability distribution of 8 potential intentions (e.g., "pouring water" 0.72, "moving" 0.21). The intention with the highest probability is selected to generate a high-level decision instruction (e.g., "execute the sequence of grabbing and pouring a glass") and sent to S6. At the same time, the intention confidence score is calculated (e.g., 0.72). If the score is lower than the threshold (0.65), the low-confidence intention label (e.g., "pouring water") and the corresponding gesture features are fed back to S4, triggering the S4 model to adjust the confusion matrix penalty coefficient for that gesture category. The formula for calculating multimodal attention weights is: ; In the formula: Am represents the attention weight of the m-th modality feature. Here, m=1 represents the gesture modality, m=2 represents the context modality, and m=3 represents the history modality.

[0047] Sim(Fm,Q) represents the cosine similarity between the m-th modal feature matrix F_mFm and the query vector QQ.

[0048] λm represents the modality adjustment factor. The gesture modality is fixed at 1.2, the environment modality is dynamically set from 0.8 to 1.5 by the S1 environment complexity, and the history modality is set from 0.5 to 1.0 by the S8 interaction frequency.

[0049] In step S6, the system's built-in action primitive library contains 8 basic operation categories, each associated with 5-8 sub-action sequences. The planning process employs a hierarchical optimization strategy: firstly, through A... The algorithm plans the global path, and the path cost function comprehensively considers obstacle distance, joint motion energy consumption, and execution time. The energy consumption weight is dynamically adjusted based on motor temperature data fed back from S7; when the temperature exceeds 60℃, the weight increases to 1.3 times the original set value. Local paths are corrected in real-time using a model predictive control algorithm with a control cycle of 10ms. During the motion parameter generation stage, the execution precision is adjusted based on the confidence level of the S5 intent. When the confidence level is higher than 0.8, a high-precision mode is activated, reducing the allowable joint angle control error range to ±0.5°; when the confidence level is lower than 0.6, a robust mode is switched, increasing the allowable error range to ±1.2° by adding a safety margin. The generated execution instructions are encapsulated in XML format, including joint angle sequences, motion speed curves, and force feedback thresholds, and are sent to the execution unit via the CAN bus.

[0050] In step S7, the execution unit consists of a 6-DOF robotic arm and an end effector, with each joint equipped with an absolute encoder and a torque sensor. Motion control employs a position-force hybrid control strategy, with a position loop control frequency of 1kHz and a force loop sampling frequency of 2kHz. During execution, three key indicators are monitored in real time: the deviation between the actual joint position and the commanded position, the end effector contact force, and the smoothness of the motion trajectory. When the position deviation exceeds a threshold or the contact force suddenly increases by more than 15%, the system triggers an emergency deceleration mechanism and initiates path replanning. Path adjustment uses a Bezier curve interpolation algorithm, with the new path generation time controlled within 50ms. The execution results include two parts: a motion completion score and an anomaly event log. The completion score is calculated based on trajectory tracking error and force control accuracy, with a maximum score of 10 points; the anomaly event log records the time of exceeding the tolerance, sensor values, and adjustment measures. The result data is packaged and transmitted back to S8 every 200ms using the UDP protocol to reduce latency.

[0051] In step S8, the interface supports three input methods: voice commands, touch operation, and facial expression recognition. Voice feedback uses an end-to-end recognition model based on Transformer, with an accuracy rate set above 95%. A secondary confirmation mechanism is triggered when the recognition confidence level falls below 80%. Touch feedback is achieved through virtual buttons on the interface, with six standard feedback options: "Correct Execution," "Excessive Force," and "Trajectory Deviation." The system records the feedback type and trigger timestamp after the user clicks. Facial expression recognition uses a lightweight CNN model to extract emotional features from facial images captured by the camera, categorizing the user's emotional state into "Satisfied," "Neutral," and "Dissatisfied." The classification results are stored in association with the feedback content. After preprocessing, the feedback data generates two outputs: an error correction sample set and an emotion score. The error correction sample set contains feature data of incorrectly executed actions and correct labels, and is updated daily to the S4 model training library. The emotion score uses a 5-point scale, with the average score calculated daily and pushed to S9 as one of the bases for parameter optimization.

[0052] In step S9, the optimization objectives include three dimensions: recognition accuracy, action execution efficiency, and user satisfaction, with weights set to 0.4, 0.3, and 0.3 respectively. The system parameter library is divided into static and dynamic parameters. Static parameters, such as sensor sampling frequency, are evaluated and adjusted quarterly, while dynamic parameters, such as model learning rate, are updated daily. The optimization process consists of two phases: offline and online. In the offline phase, historical data from the past 7 days is used for batch optimization, employing a genetic algorithm to search for the optimal parameter combination. The population size is set to 50, and the number of iterations is 50 generations. In the online phase, fine-tuning is performed based on the sentiment scores pushed in real-time by S8. When the satisfaction score is below 3.5 for three consecutive hours, a rapid optimization process is initiated, temporarily increasing the learning rate by 20% and adjusting the confusion matrix penalty coefficient. The optimized parameters are sent to S1 via an encrypted channel, using an incremental update mechanism that transmits only changed parameter items to reduce data volume. The system maintains normal operation during parameter updates, and a smooth transition strategy is used when switching to new parameters to avoid abrupt changes in action execution.

[0053] From the above, we can conclude that: In this invention, the real-time response speed and long-term adaptive capability of gesture recognition are significantly improved through a dynamic update mechanism for model weights and a confusion matrix optimization mechanism. The dual-channel network structure of S4, while receiving feature data from S3, can simultaneously process static spatial features and dynamic temporal features. Combined with user error correction data from S8, it adjusts the classification penalty strategy, enabling the model to continuously optimize its ability to distinguish complex gestures during interaction. When S5 returns the intent correction signal, the online distillation mechanism quickly updates the LSTM layer weights, avoiding the lag problem caused by traditional batch training, allowing the robot to gradually adapt to the user's movement habits and expression style during continuous interaction.

[0054] In this invention, the introduction of multimodal data fusion and attention mechanisms effectively enhances the accuracy and environmental adaptability of intent judgment. The S5 Transformer model integrates gesture probability vectors, environmental semantic information, and historical interaction records. By dynamically adjusting modal weights, it focuses on key action segments, reducing the limitations of single data sources. For example, in complex environments, the system can prioritize referencing environmental semantic data to correct gesture ambiguities. When the user frequently performs specific actions, it automatically increases the weight of historical interaction features, making decision-making instructions more aligned with actual operational needs and reducing the probability of misjudgment due to environmental interference or action distortion.

[0055] In this invention, the user feedback closed loop and dynamic parameter adjustment mechanism further enhance the smoothness and personalized experience of human-computer interaction. Error correction data and sentiment scores collected in S8 are continuously fed back into the model training process. S4 optimizes the focus on frequently corrected gesture categories through a confusion matrix, while S5 dynamically adjusts the decision threshold based on intent confidence. These two elements work together to form a complete closed loop of "recognition-feedback-optimization." This design allows the system to maintain stable interactive performance under different users and scenarios, while gradually accumulating preference models for specific users, making the robot's response more aligned with individual usage habits and improving user satisfaction in long-term interactions.

[0056] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0057] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot human-computer interaction method based on visual gesture recognition, characterized in that: The method includes the following steps: S1: Synchronously collects dynamic sequences of user gestures and depth information of the surrounding environment through an RGB-D camera and an infrared sensor, and receives system optimization parameters output by S9 in real time to adjust the acquisition frame rate and resolution to ensure stable image quality under complex lighting conditions. S2: Perform background subtraction and motion blur correction on the raw data collected by S1, adaptively adjust the edge detection threshold according to the feature extraction requirements of S3, and transmit the denoised image data to S3 for feature parsing. S3: Extract the coordinates of key hand points and muscle movement micro-features from the image output by S2, construct the spatiotemporal trajectory matrix of dynamic gestures, and push the feature vectors synchronously to the S4 model training module and the S5 intention prediction unit. S4: Train a dual-channel CNN-LSTM network using the feature data from S3, optimize the confusion matrix by combining the user error correction data fed back from S8, output the gesture category probability to S5 in real time, and receive the intent correction signal returned from S5 to update the model weights. S5: Integrates the gesture recognition results of S4, the environmental semantic information of S1, and the historical interaction records stored in S8. It focuses on key action segments through an attention mechanism, predicts the user's potential operation intentions, generates high-level decision instructions and sends them to S6, and feeds back the intention confidence to S4 to optimize the recognition accuracy. S6: Based on the decision instructions of S5, call the kinematic model to generate joint trajectories, receive real-time force feedback data from S7 to adjust the motion flexibility parameters, and package the optimized execution instructions to transmit them to the S7 execution unit. S7: Executes the action command issued by S6, senses the change in contact force through the torque sensor, and dynamically corrects the movement path by combining the environmental obstacle data of S1, and synchronously transmits the trajectory adjustment result back to the S8 feedback system. S8: Capture user satisfaction with S7 action execution through facial expression recognition and voice emotion analysis, and push the quantitative feedback data to the S4 model optimization module and the S9 system parameter library respectively to establish user preference profiles; S9: Calculate the performance indicators of each module based on the feedback data of S8, dynamically adjust the acquisition parameters of S1, the recognition threshold of S4 and the action planning algorithm of S6, generate a system optimization report and send the updated parameters to S1 to execute a new round of interactive loop.

2. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S1, gesture and environmental information are acquired using a depth camera and a six-axis inertial sensor. The depth camera captures RGB-D images of the hand area at a frequency of 30 frames per second, simultaneously outputting a 1920×1080 pixel color image and a 640×480 pixel depth map. The inertial sensor records wrist acceleration and angular velocity data in real time, with a sampling frequency of 100Hz. The environmental information acquisition module integrates a temperature sensor and an ultrasonic rangefinder to monitor temperature changes in the operating area and distance to obstacles, respectively. When the system starts, it loads the initial parameter configuration issued by S9, including sensor sampling thresholds and data filtering coefficients. During operation, it continuously receives parameter commands dynamically adjusted by S9. For example, when the ambient light intensity exceeds 500 lux, it automatically increases the camera exposure compensation to +0.7EV and simultaneously reduces the depth map noise filtering threshold to 85% of the original set value. The acquired raw data is transmitted to the preprocessing unit via a USB 3.0 interface, with the transmission delay controlled within 20ms.

3. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S2, the image data preprocessing process consists of Gaussian filtering for noise reduction, hand region segmentation based on skin color model, and scale normalization to 224×224 pixels. The filter kernel size is dynamically adjusted according to the feature extraction requirements fed back by S3. When S3 needs to enhance edge features, a 3×3 kernel is used, and when focusing on texture details, a 5×5 kernel is switched. Inertial data is filtered by a Kalman filter to eliminate motion noise. The process noise covariance matrix of the state equation during the filtering process is determined by the trajectory modeling accuracy requirements of S3.

4. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S3, the spatial feature extraction branch uses an improved MobileNetV2 network to extract the coordinates of 21 key points of the hand from the normalized image through depthwise separable convolution, including the three-dimensional coordinates of the fingertip position, knuckle angle and palm center; the temporal feature modeling branch uses a bidirectional GRU network to encode the temporal matrix constructed from the inertial data and output a 128-dimensional feature vector containing the rate of change of gesture speed and the duration of the action.

5. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S4, a dual-channel spatiotemporal fusion CNN-LSTM network algorithm is used, which includes a spatial feature extraction channel and a temporal dynamic modeling channel: the spatial channel uses 3 convolutional layers to extract static features such as the relative position of joints and the orientation of the palm from the heat map of key hand points output by S3. The temporal channel uses a 2-layer LSTM to temporally encode the gesture trajectory matrix generated by S3, capturing the rate of change of gesture speed and the motion pause pattern; the features of the two channels are weighted and fused by an attention mechanism and then input into a fully connected layer, outputting a 128-dimensional gesture feature vector. The formula for dynamically updating model weights is: In the formula: Wt+1 represents the updated model weight matrix; Wt represents the weight matrix at the current time step; η represents the base learning rate; ∇Lt represents the gradient of the cross-entropy loss function at time t; α represents the feedback adjustment coefficient; sign(∇Lt) represents the gradient direction sign function; C S8 This represents the error correction confidence vector for S8 users; I S5 This indicates that S5 intends to modify the instruction vector; ⊙ represents the Hadamard.

6. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S5, the Transformer model is used to integrate the gesture recognition results of S4, the environmental semantic information of S1, and the historical interaction records stored in S8. The model consists of three parallel encoding branches and a cross-attention fusion layer: the gesture encoding branch converts the 16-dimensional gesture category probability vector output by S4 into a feature matrix through the embedding layer; the environment encoding branch performs word vector mapping and position encoding on the environmental semantic data collected by S1; and the history encoding branch extracts user preference features from the interaction records stored in S8. The three-branch features are normalized by layers and then input into the cross-attention layer, where attention weights are dynamically calculated to focus on key action segments. The fusion feature input is a bidirectional Transformer decoder, which outputs the probability distribution of 8 potential intentions through a softmax layer. The intention with the highest probability is selected to generate a high-level decision instruction and sent to S6. At the same time, the intention confidence score is calculated. If the score is lower than the threshold, the low confidence intention label and the corresponding gesture feature are fed back to S4, triggering the S4 model to adjust the confusion matrix penalty coefficient for that gesture category. The formula for calculating multimodal attention weights is: ； In the formula: Am represents the attention weight of the m-th modality feature; where m=1 represents the gesture modality, m=2 represents the context modality, and m=3 represents the history modality; Sim(Fm,Q) represents the cosine similarity between the m-th modal feature matrix F_mFm and the query vector QQ; λm represents the modality adjustment factor; the gesture modality is fixed at 1.2, the environment modality is dynamically set from 0.8 to 1.5 by the S1 environment complexity, and the history modality is set from 0.5 to 1.0 by the S8 interaction frequency.

7. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S6, the system's built-in action primitive library contains 8 basic operations, each associated with 5-8 sub-action sequences; the planning process adopts a hierarchical optimization strategy: firstly, through A... The algorithm plans the global path, and the path cost function comprehensively considers obstacle distance, joint motion energy consumption, and execution time. The energy consumption weight is dynamically adjusted based on the motor temperature data fed back by S7. When the temperature exceeds 60℃, the weight is increased to 1.3 times the original set value. The local path is corrected in real time using a model predictive control algorithm with a control cycle of 10ms. During the motion parameter generation stage, the execution accuracy is adjusted according to the confidence level of the intention in S5. When the confidence level is higher than 0.8, a high-precision mode is activated, and the allowable range of joint angle control error is reduced to ±0.5°. When the confidence level is lower than 0.6, a robust mode is switched, and the allowable error range is expanded to ±1.2° by increasing the safety margin. The generated execution instructions are encapsulated in XML format, including the joint angle sequence, motion speed curve, and force feedback threshold, and are sent to the execution unit via the CAN bus.

8. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S7, the execution unit consists of a 6-DOF robotic arm and an end effector. Each joint is equipped with an absolute encoder and a torque sensor. The motion control adopts a position-force hybrid control strategy, with a position loop control frequency of 1kHz and a force loop sampling frequency of 2kHz. During execution, three key indicators are monitored in real time: the deviation between the actual joint position and the commanded position, the contact force of the end effector, and the smoothness of the motion trajectory. When the position deviation exceeds the threshold or the contact force suddenly increases by more than 15%, the system triggers an emergency deceleration mechanism and initiates path replanning. The path adjustment adopts a Bezier curve interpolation algorithm, and the new path generation time is controlled within 50ms.

9. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S8, the interface supports three input methods: voice commands, touch operation, and facial recognition. Voice feedback adopts an end-to-end recognition model based on Transformer, with a recognition accuracy set to over 95%. When the recognition confidence is below 80%, a secondary confirmation mechanism is triggered. Touch feedback is achieved through virtual buttons on the interface, with six standard feedback options such as "correct execution," "excessive force," and "trajectory deviation." After the user clicks, the system records the feedback type and trigger timestamp.

10. The robot human-computer interaction method based on visual gesture recognition as described in claim 1, characterized in that: In step S9, the optimization objective includes three dimensions: recognition accuracy, action execution efficiency, and user satisfaction, with weights set to 0.4, 0.3, and 0.3, respectively. The system parameter library is divided into two categories: static parameters and dynamic parameters. Static parameters, such as sensor sampling frequency, are evaluated and adjusted quarterly, while dynamic parameters, such as model learning rate, are updated daily. The optimization process is divided into two stages: offline and online. In the offline stage, historical data from the past 7 days is used for batch optimization, and a genetic algorithm is used to search for the optimal parameter combination. The population size is set to 50, and the number of iterations is 50 generations. During the online phase, the system makes fine adjustments based on the sentiment scores pushed by S8 in real time. When the satisfaction score is below 3.5 for 3 consecutive hours, a rapid optimization process is initiated, temporarily increasing the learning rate by 20% and adjusting the confusion matrix penalty coefficient. The optimized parameters are sent to S1 through an encrypted channel, using an incremental update mechanism to transmit only the changed parameter items to reduce the amount of data. The system continues to operate normally during the parameter update process, and a smooth transition strategy is adopted when switching to new parameters to avoid abrupt changes in action execution.

Images