A low-latency communication system and a method for controlling a human-machine interaction system thereof

Abstract

The application relates to a low-latency communication system and a human-computer interaction system control method thereof, and the low-latency communication system comprises the following: training data, containing the action data of one or more sending ends; corresponding data characteristics are determined according to the kinematic characteristics of the sending end; a deep learning model is used for processing time sequence training data; the deep learning model is trained and learned according to the training data and is deployed on a receiving end; the sending end sends the action data to the receiving end in real time; the receiving end receives the action data from the sending end in real time, a microcontroller analyzes and processes the action data, and action prediction is performed through the deep learning model; an advance response module outputs predicted data and a prediction trend according to the action prediction result, and controls the receiving end to complete corresponding actions in advance. The system can effectively reduce the transmission time between the sending end and the receiving end, reduce the influence of time delay in remote interaction, and thus improve the experience and safety of interaction.

Description

Technical Field

[0001] This invention relates to the field of remote control communication, specifically a low-latency communication system and its human-computer interaction system control method. Background Technology

[0002] Latency refers to the time required for a message or packet (signal) to travel from one end of a network (sender) to the other (receiver). In human-computer interaction, the sender and receiver need to perform a large amount of bidirectional signal transmission, and latency is one of the important factors in communication quality. The latency caused by the time required for a signal to travel from the sender to the receiver, and the processing time of the mechanical and computing systems at the sender and receiver ends, is unavoidable. If the signal latency is too high, it will lead to a large difference between the signal at the receiver and the signal at the sender, which will affect the interactive experience and pose security risks. In response, more and more integrated circuit companies are investing in the research and development of low-latency communication systems.

[0003] To address the challenge of latency, researchers have explored various approaches, including optimizing hardware and network connectivity, to reduce overall communication system latency. Hardware optimization involves improving processing speed through optimized circuit layout, the use of high-speed interfaces, and adjustments to clock frequencies. Network connectivity optimization focuses on increasing transmission speed through high-speed networks, developing more advanced communication algorithms, and enhancing network stability. While these methods directly reduce latency, the overall system latency remains a significant bottleneck due to signal transmission time and the processing time of the mechanical and computational systems. Even with further optimization, a limitation on further latency reduction persists. This unreduced latency remains an uncertain factor in remote human-computer interaction, impacting user experience and posing security risks. Severe latency significantly limits the application of communication systems, particularly in scenarios with extremely high latency requirements, such as remote robot control and autonomous driving.

[0004] It is evident that communication latency is a significant bottleneck in human-computer interaction and remote control. Current technologies are unable to effectively reduce its impact on communication systems, necessitating further improvements. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a low-latency communication system and a human-computer interaction system control method thereof. This system can effectively reduce the transmission time between the sending end and the receiving end, reduce the impact of latency in remote interaction, and thus improve the interactive experience and security.

[0006] The objective of this invention is achieved as follows:

[0007] A low-latency communication system, comprising,

[0008] Training data includes motion data from one or more transmitters; it includes motion data from one or more transmitters; the corresponding data features are determined based on the kinematic characteristics of the transmitter.

[0009] Deep learning models are used to process time series training data; deep learning models are trained and learned based on training data and deployed on the receiving end.

[0010] The sending end sends action data to the receiving end in real time;

[0011] The receiving end receives motion data from the sending end in real time. The microcontroller analyzes and processes the motion data and performs motion prediction through a deep learning model.

[0012] The advance response module outputs the predicted data and trends based on the action prediction results, and controls the receiving end to respond and complete the corresponding actions in advance.

[0013] As a specific solution, data features are expanded using limited training data. Expanded features can improve prediction accuracy. The training data is obtained through random free movement at the sending end. The data features of the training data are expanded as follows:

[0014] (1) The measurement transmitter outputs motion data from the corresponding sensor module under different conditions, and the microcontroller obtains the corresponding motion transformation curve based on the motion data;

[0015] (2) The transmitting end moves freely, which includes acceleration and deceleration, initial state and stationary state;

[0016] (3) Using the motion transition curve, and based on the sampling frequency, each data point is spaced 1-20ms apart to obtain the motion time curve;

[0017] (4) Difference is performed on the motion transition curve and the motion time curve to obtain the approximate motion speed time data; the approximate motion acceleration time data is obtained by difference of the approximate motion speed time data, thereby expanding the data characteristics of the training data.

[0018] As another specific solution, the deep learning model is based on the dataset of motion relationships corresponding to the sensor modules on the transmitting end and a model suitable for the characteristics of the dataset. It predicts the motion data, trends and probabilities for the next period of time based on the motion data and trends of the previous period.

[0019] As another specific embodiment, the sensor module is used to monitor the motion of the transmitting end; the sensor module includes one or more of the following: a strain sensor for monitoring the joint motion of the transmitting end, a force sensor for monitoring the motion of the transmitting end based on mechanical principles, an attitude sensor for monitoring the posture of the transmitting end, an optical sensor for monitoring the motion of the transmitting end based on optical principles, and a physiological signal sensor for monitoring the motion of the transmitting end based on biological signals; the microcontroller is communicatively connected to the strain sensor, and / or the force sensor, and / or the attitude sensor, and / or the optical sensor, and / or the physiological signal sensor.

[0020] As another specific solution, the deep learning model is an ARIMA model, LSTM model, CNN model, RNN model, SSM model, or their derivative models used to process time series data.

[0021] As another specific solution, the human-computer interaction system control method applying the above-mentioned low-latency communication system includes the following steps:

[0022] Step 1: Collect a dataset of motion data and movement relationships, and train a deep learning model to learn the data.

[0023] Step 2: Start the human-computer interaction system. The sending end can move freely. The action data transmission path is, in sequence, the sensor module, the microcontroller, the deep learning model, and the early response module.

[0024] Step 3: When the action data changes, the action data is transmitted to the deep learning model for action prediction and the prediction result is output.

[0025] Step four: The advance response module controls the receiver to respond in advance based on the prediction results.

[0026] As another specific solution, the training and learning of deep learning models:

[0027] a. Use time-series motion data to augment the motion data through differencing;

[0028] b) Use a smooth window method with a step size of 1 to group the data, and use each group for training the deep learning model.

[0029] As another specific solution, action prediction using a deep learning model: Input real-time data, the model outputs data after the target event, and the prediction result is evaluated using the loss function MSE; the algorithm formula is as follows:

[0030] The beneficial effects of this invention are as follows:

[0031] By using a deep learning model based on multi-action data and deep learning (or other prediction algorithms) for action prediction, combined with the early response mechanism of the early response module, the communication latency of the communication system can be reduced and responses can be made in advance, effectively reducing the transmission time of information between the sender and receiver, and significantly reducing latency. Specifically, firstly, a sensor module (which can fuse data from other sensors, including strain sensors and surface electromyography sensors, etc.) is used to acquire a portion of the sender's motion data as a training set, enabling the deep learning model to predict the sender's motion. Then, the deep learning model is connected to the data communication module. When the deep learning model predicts the action that may be performed next, it makes a prediction in advance (action prediction), and the early response module controls the receiver to respond in advance, thereby reducing the latency of the communication system. A low-latency communication system can reduce the impact of latency in remote interaction, thereby improving the interactive experience and security. Attached Figure Description

[0032] Figure 1 This is an architecture diagram of a low-latency communication system according to an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the transmitting end, the receiving end, and the signal after algorithm processing in one embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram illustrating the prediction result in one embodiment of the present invention.

[0035] Figure 4 and Figure 5 These are experimental data graphs from one embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] See Figures 1-5 This low-latency communication system includes,

[0038] The training data includes motion data monitored by one or more transmitters; the corresponding data features are determined based on the kinematic characteristics of the transmitter. However, the differentiation in this embodiment is only one of the many forms of feature establishment. Differential form: first, the relationship between bending angle and time is obtained, then the relationship between bending angular velocity and time is obtained by difference, and finally the relationship between bending angular acceleration and time is obtained by difference, thereby determining the three features of bending angle, bending angular velocity, and bending angular acceleration.

[0039] Deep learning models are used to process training data over long periods of time. Deep learning models are trained and learned based on the training data and then deployed on the receiving end. Through training and learning, deep learning models can simulate different actions and plan corresponding motion trajectories, acceleration and deceleration, and other information.

[0040] The transmitting end sends motion data collected by the sensor module to the receiving end in real time; the digital signal of the sensor module is repeatedly transmitted at a predetermined frequency (the transmission method can be wired communication or wireless communication).

[0041] The receiving end receives motion data from the sending end in real time. The microcontroller analyzes and processes the motion data and performs motion prediction through a deep learning model.

[0042] The advance response module sets different judgment conditions according to different needs. When the prediction result meets the judgment condition, it outputs the predicted data and prediction trend after prediction based on the action prediction result, and controls the receiver to respond and complete the corresponding action in advance. Here, "need" refers to the minimum prediction probability that the response can be received in milliseconds in advance; "judgment condition" refers to the number of milliseconds in advance and the prediction probability (i.e., MSE used in this example).

[0043] Furthermore, data features are expanded using limited training data. Expanded features improve prediction accuracy. The training data is obtained through random free movement at the sending end. The data features of the training data are expanded as follows:

[0044] (1) The corresponding sensor module measures and outputs motion data under different conditions (performing different actions) at the transmitter end. The microcontroller on the transmitter end obtains the corresponding motion transition curve based on the motion data. The motion transition curve is obtained by: first, obtaining the motion data graph that changes over time under different motions at the transmitter end; then, obtaining the motion transition curve by finding the correspondence between "sensor data - bending angle".

[0045] (2) The transmitting end makes free movement, which includes various situations such as acceleration and deceleration, initial state and stationary state. The free movement covers the left and right movement modes of the transmitting end as much as possible.

[0046] (3) Using the motion transition curve, and based on the sampling frequency with a 10ms interval between each data point, obtain the motion time curve;

[0047] (4) Difference is performed on the motion transition curve and the motion time curve to obtain the approximate motion speed time data; difference is performed on the approximate motion speed time data to obtain the approximate motion acceleration time data, thereby expanding the data characteristics of the training data; the methods for obtaining the approximate motion speed time data and the approximate motion acceleration time data are as follows: first, obtain the relationship between the bending angle and time, then obtain the relationship between the bending angular velocity and time through difference, and finally obtain the relationship between the bending angular acceleration and time through difference, thereby determining the three characteristics of bending angle, bending angular velocity, and bending angular acceleration, and calculating the approximate motion speed time data and the approximate motion acceleration time data through formula.

[0048] Furthermore, the deep learning model is based on the dataset of motion relationships corresponding to the sensor modules on the transmitting end and a model suitable for the characteristics of the dataset. It predicts the motion data, trends, and probabilities for a subsequent period based on the motion data and trends of the previous period.

[0049] Furthermore, the sensor module includes one or more of the following: a strain sensor for monitoring the joint movement of the transmitting end (operator); a force sensor for monitoring the movement of the transmitting end based on mechanical principles; an attitude sensor for monitoring the posture of the transmitting end; an optical sensor for monitoring the movement of the transmitting end based on optical principles; and a physiological signal sensor for monitoring the movement of the transmitting end based on biosignals. The microcontroller is communicatively connected to the strain sensor, and / or a surface electromyography (SEMG) sensor (e.g., an SEMG sensor for monitoring muscle movement at the transmitting end), and / or a force sensor, and / or an attitude sensor, and / or an optical sensor, and / or a physiological signal sensor. It should be noted that the sensor module can include all sensors capable of monitoring the movement of the transmitting end; their types and models can be diverse, and the richer the sensor data, the better the prediction effect.

[0050] Furthermore, deep learning models are ARIMA (Autoregressive Moving Average), LSTM (Long Short-Term Memory), CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), SSM (State Space Model), and any derivative models of the above models used to process time series data.

[0051] Furthermore, the human-computer interaction system control method applying the above-mentioned low-latency communication system includes the following steps:

[0052] Step 1: Collect motion data and motion relationship datasets through sensor modules, train and learn with a suitable deep learning model, and deploy the pre-trained deep learning model.

[0053] Step 2: Start the human-computer interaction system (connect all electrical components on the system). The transmitter moves freely, and the sensor module collects automatic activity information to generate corresponding action data. The action data transmission path is, in sequence, the sensor module, the microcontroller, the deep learning model, and the early response module.

[0054] Step 3: When the action data changes, the action data is transmitted to the deep learning model for action prediction and the prediction result is output.

[0055] Step four: The advance response module controls the receiver to respond in advance based on the prediction results, thereby effectively reducing the system latency.

[0056] See Figure 2 ,

[0057] Curve 1 in the figure is the real-time curve of the signal at the transmitting end changing over time;

[0058] Curve 2 in the figure is the time delay curve of the signal at the receiving end as a function of time;

[0059] Curve 3 in the figure is the predicted curve of the signal changing over time after algorithm prediction and early response;

[0060] This embodiment takes a single motion as an example. For the transmitting end, at time t1, the data value begins to rise significantly. For the receiving end, due to the impact of communication delay, the same upward trend occurs at time t2. That is, the interval s1 between time t2 and time t1 is the main factor of communication delay. This system outputs prediction results through a deep learning model. For the receiving end, it can predict the upward trend after time t2 in advance at time t3, thereby making an early response and generating the signal shown in Figure 3. In this way, the communication delay is reduced from the interval s1 between time t2 and time t1 to the interval s2 between time t3 and time t1. This example is only a simplified illustration of the present invention. The present invention is also applicable to arbitrarily complex signals, arbitrarily early warning times, and arbitrarily similar motion intention prediction algorithms.

[0061] Furthermore, the deep learning model can be LSTM and its derivatives for processing time series data, or other models suitable for motion intention prediction. LSTM is a popular recurrent neural network that can handle long-term series data while avoiding the vanishing and exploding gradient problems of traditional RNNs. In RNNs, due to the multiplicative effect of gradients during backpropagation, gradients gradually vanish as the sequence length increases, resulting in the inability to transmit information from long sequences. LSTM, by introducing a threshold mechanism, can selectively forget or update some information, thus better handling long-sequence data. Although the prediction performance of LSTM is far superior to that of RNN, the LSTM network encodes all input features into a fixed-length vector representation, ignoring the correlation between input features and predictions, and therefore cannot selectively utilize historical data. Therefore, this embodiment introduces an attention mechanism into LSTM, assigning more weights to key parts of the input data that affect the output result, in order to better learn the information in the input sequence. The following uses the LSTM-Attention model as an example to illustrate the training and learning of the deep learning model in this embodiment:

[0062] a. Use time-series motion data to augment the motion data through differencing;

[0063] b) The data is grouped using a smoothing window method with a step size of 1 (e.g., if there are 100 data points labeled 1-100, and the window size is 10, then the first group is 1-10, the second group is 2-11, the third group is 3-12, and so on). These groups are then used to train the deep learning model (LSTM-Attention model). The prediction result is the data value after the target time. By comparing the output value with the data value after the target time, the weights are continuously adjusted for iterative training to improve the training effect. The Attention mechanism is as follows: Figure 3 .

[0064] Furthermore, the deep learning model predicts action by taking real-time data as input and outputting data after the target event, then evaluating the prediction result using the loss function MSE; the algorithm formula is as follows:

[0065] ;

[0066] Mean Squared Error (MSE) is a method for measuring the difference between two arrays of numbers with the same shape. Specifically, it compares each element of each array and calculates the squared differences between them, ultimately averaging these differences. The loss function MSE is primarily used to measure the magnitude of the error between the predicted output and the actual output of a neural network, and is one of the common loss functions used to train models; in the formula above, the first yi is the true bending angle, the second yi is the predicted bending angle, and the average of their squared differences is calculated.

[0067] Furthermore, Figure 4 and Figure 5 These are experimental data obtained by testing with a trained deep learning model; the "true" line represents the real data in the test set, and the "predict" line represents the data predicted 100 milliseconds later.

[0068] The experimental data shows that the predicted line has the same data trend as the true line, which can be effectively used for early prediction. It also shows that this method can effectively predict data 100ms later, thereby responding in advance and greatly reducing latency.

[0069] The above describes the preferred embodiments of the present invention, illustrating and describing the basic principles, main features, and advantages of the invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A low latency communication system, characterized by: The application relates to a motion prediction method and device. The training data comprises action data of one or more sending ends; the corresponding data features are determined according to the kinematic characteristics of the sending ends; the features are established as follows: firstly, the bending angle-time relationship is obtained, then the bending angular velocity-time relationship is obtained by difference, and finally the bending angular acceleration-time relationship is obtained by difference, so as to determine the bending angle, the bending angular velocity and the bending angular acceleration; The deep learning model is used for processing the time sequence training data; the deep learning model is trained and learned according to the training data and is deployed on the receiving end; through the training and learning, the deep learning model simulates different actions and plans corresponding motion trajectories and motion acceleration and deceleration; The sending end sends the action data collected by the sensor module to the receiving end in real time; the digital signals of the sensor module are repeatedly sent at a predetermined frequency; The receiving end receives the action data from the sending end in real time, the microcontroller analyzes and processes the action data, and the deep learning model is used for action prediction; The early response module outputs the predicted data and the prediction trend according to the action prediction result, and controls the receiving end to complete the corresponding action in advance; The data features of the training data are expanded through limited training data; the training data are obtained through random free motion of the sending end; the data features of the training data are expanded as follows: (1) the action data output by the corresponding sensor module of the sending end under different conditions is measured, and the microcontroller obtains the corresponding motion condition conversion curve according to the action data; the motion condition conversion curve is obtained as follows: firstly, the action data-time curve is obtained under different motions of the sending end; then the motion condition conversion curve is obtained by looking up the corresponding relationship between the sensor data and the bending angle; (2) the sending end is subjected to free motion, and the free motion comprises acceleration and deceleration, an initial state and a static state; (3) the motion condition time curve is obtained according to the sampling frequency and the interval of each data point of 1-20 ms by using the motion condition conversion curve; (4) the motion speed approximate value time data are obtained by differentiating the motion condition conversion curve and the motion condition time curve; the motion acceleration approximate value time data are obtained by differentiating the motion speed approximate value time data, so as to expand the data features of the training data; the motion speed approximate value time data and the motion acceleration approximate value time data are obtained as follows: firstly, the bending angle-time relationship is obtained, then the bending angular velocity-time relationship is obtained by difference, and finally the bending angular acceleration-time relationship is obtained by difference, so as to determine the bending angle, the bending angular velocity and the bending angular acceleration; the motion speed approximate value time data and the motion acceleration approximate value time data are calculated through a formula; The deep learning model is based on the data set of the corresponding motion relationship of the sensor module of the sending end and a model suitable for the features of the data set; the action data, the trend and the probability of a subsequent period of time are predicted through the action data and the trend of a previous period of time.

2. The low-latency communication system of claim 1, wherein: The sensor module is used for monitoring the motion of the sending end; the sensor module includes one or more of a strain sensor for monitoring the joint motion of the sending end, a force sensor for monitoring the motion of the sending end based on the principle of mechanics, a posture sensor for monitoring the posture of the sending end, an optical sensor for monitoring the motion of the sending end based on the principle of optics, and a physiological signal sensor for monitoring the motion of the sending end based on biological signals; the microcontroller is communicatively connected to the strain sensor, and / or the force sensor, and / or the posture sensor, and / or the optical sensor, and / or the physiological signal sensor.

3. The low-latency communication system of claim 1, wherein: The deep learning model is an ARIMA model, an LSTM model, a CNN model, an RNN model, an SSM model, and a derivative model thereof for processing time series data.

4. A method for controlling a human-machine interface system for use with a low-latency communication system as claimed in any one of claims 1 to 3, characterized in that: The method comprises the following steps: Step one, collect the action data and the motion relationship data set, and train and learn by the deep learning model; Step two, start the human-computer interaction system, and the free activity of the sending end, the action data transmission path is in turn the sensor module, the microcontroller, the deep learning model and the advanced response module; Step three, when the action data changes, the action data is transmitted to the deep learning model for action prediction, and the prediction result is output; Step four, the advanced response module controls the receiving end to make a response in advance according to the prediction result.

5. The method of claim 4, wherein: Training and learning of the deep learning model: a, use the time series action data to expand the action data by difference method; b, use the step length of 1 smooth window method to group the data, which are respectively used for training and learning of the deep learning model.

6. The method of claim 4, wherein: Action prediction of the deep learning model: input real-time data, the model outputs data after the target event, and evaluates the prediction result by the loss function MSE; the algorithm formula is as follows: 。

Images