System and method for controlling external devices via human-machine interface

Abstract

The present technical solution relates to human-machine interfaces based on a system of sensors, including optomyographic sensors, and can be used for controlling external interfaces and devices for a wide range of tasks. The technical result achieved in solving the aforementioned technical problem is that of providing more accurate recognition of data and identification of movements obtained from optomyographic sensors and processed with the aid of AI algorithms. A system for controlling external devices via a human-machine interface comprises: a user-worn wearable device containing at least one optomyographic sensor capable of capturing a signal about the current state of a muscle of the user, said optomyographic sensor having a switching frequency of 10-50 Hz and consisting of a phototransistor and an infrared light-emitting diode; a microcontroller capable of controlling the on-time of the infrared light-emitting diode and the time for which data is read from the phototransistor and transmitting the obtained data to a data processing server; a data processing server capable of processing the obtained data with the aid of AI algorithms to produce control signals and transmit control signals to external devices.

Description

[0001] SYSTEM AND METHOD FOR CONTROLLING EXTERNAL DEVICES THROUGH A HUMAN-MACHINE INTERFACE

[0002] AREA OF TECHNOLOGY

[0003] This technical solution relates to human-machine interaction interfaces based on a sensor system including optomiography sensors and can be used to control external interfaces and devices for a wide range of tasks.

[0004] LEVEL OF TECHNOLOGY

[0005] Patent RU2775757C1, "Opto-Myographic Sensor for an Electromechanical Prosthesis," published July 7, 2022, is known in the prior art. This patent describes an opto-myographic sensor for an electromechanical prosthesis. The sensor comprises a housing. The housing has a surface for interacting with the user's limb. Optical sensors for measuring tissue light transmittance are mounted on the housing surface and connected to a circuit board. The sensor contains an element for sequentially activating the optical sensors, which is connected to the optical sensors. The housing contains an element for adjusting the intensity of light emitted by the optical sensor's receiver. The light intensity adjustment element is connected to the circuit board and is located on the opposite side of the housing from the surface for interacting with the user's limb.Optical sensors are designed to measure relative changes in the light transmittance of tissue located under the sensor due to changes in the light scattering of muscle fibers during compression / stretching, or blood filling of muscle tissue, or movement of tendons.

[0006] Patent RU2775756C1, "Opto-myographic sensor for an electromechanical prosthesis and a method for adjusting an electromechanical prosthesis," published July 7, 2022, is known from the prior art. This solution describes an opto-myographic sensor for an electromechanical prosthesis and a method for adjusting an electromechanical prosthesis.

[0007] The optomyographic sensor comprises a housing. The housing has a surface for interaction with the user's limb. Optical sensors are located on the housing surface to obtain data on processes occurring in the subcutaneous structure of the user's limb stump. The optomyographic sensor is equipped with a means for sequentially activating the optical sensors. The housing contains a control element for adjusting the intensity of light emitted by the optical sensor's receiver. The control element is located on the opposite side of the housing from the surface for interaction with the user's limb. The optical sensors are designed to obtain data on changes in muscle fiber curvature, muscle tissue perfusion, or tendon movement occurring beneath the skin of the user's limb stump.

[0008] The prior art also includes patent US10945863B2 "Method for controlling an artificial orthotic or prosthetic knee joint", published on March 16, 2021. This patent describes a method for controlling an artificial orthopedic or prosthetic knee joint, on which a tibia component is located and to which a resistance device is connected, wherein the resistance changes depending on sensor data, which are determined by at least one sensor during use of the orthopedic or prosthetic knee joint, wherein the linear acceleration of the tibia component is determined, which is compared with at least one threshold value, and if the threshold value of the linear acceleration is reached by the tibia component, then the bending resistance changes.

[0009] A disadvantage of the known solution in this field of technology is the lack of use of optomiography data (OMG) as control input data.

[0010] ESSENCE OF THE INVENTION

[0011] The technical problem that the claimed technical solution is aimed at solving is the creation of a method and system for controlling external devices via a human-machine interface using opto-myographic sensors.

[0012] The technical result achieved by solving the above-mentioned technical problem is increased data recognition and motion detection accuracy obtained from optometry data and processed using artificial intelligence algorithms. The signal obtained from optometry data is noise-resistant and has high spatial resolution, allowing the artificial intelligence algorithm to accurately determine the type of motion based on the input data. The control algorithm requires little memory and does not require large amounts of RAM, so it can be integrated even into miniature control systems.

[0013] The claimed technical result is achieved through the operation of a control system for external devices, via a human-machine interface, including: a wearable device placed on the user, containing at least one optomyographic sensor, configured to receive a signal about the current state of the user's muscle, with a frequency of 10-50 Hz, wherein the optomyographic sensor consists of a phototransistor and an infrared light-emitting diode; a microcontroller configured to control the time of switching on the infrared light-emitting diode and the time of reading data from the phototransistor and transmitting the received data to a data processing server; a data processing server configured to process the received data by means of artificial intelligence algorithms to receive control signals and transmit control signals to external devices.

[0014] In a particular embodiment of the proposed system, a bracelet is used as a wearable device.

[0015] In another particular embodiment of the proposed system, the external devices are at least prosthetic limbs or virtual reality devices, or a personal computer, or an industrial robot, or a medical robot.

[0016] The claimed technical result is also achieved through the operation of a method for controlling external devices via a human-machine interface, comprising the following steps: recording user movement signals via at least one optomiographic sensor consisting of a phototransistor and an infrared LED, at a frequency of 10-50 Hz and transmitting the obtained data to a microcontroller of a wearable device; the microcontroller of the wearable device controls the optomiographic sensors and transmits the obtained data from the sensors to an information processing server; the information processing server receives data from the optomiographic sensors and processes them using artificial intelligence algorithms to obtain control signals; the obtained control signals are transmitted to external devices.

[0017] DESCRIPTION OF DRAWINGS

[0018] The invention will be further described in accordance with the accompanying drawings, which are provided to illustrate the invention and in no way limit its scope. The following drawings are attached to the application:

[0019] Fig. 1 illustrates the change in the optomiography signal across 50 channels when performing various movements with the right hand.

[0020] Figure 2 illustrates the confusion matrix for the motion recognition paradigm.

[0021] Figure 3 illustrates an example of handwriting data collected as a result of the experiment.

[0022] Fig. 4 illustrates the results of trajectory reconstruction.

[0023] DETAILED DESCRIPTION OF THE INVENTION

[0024] The following detailed description of the invention includes numerous implementation details to provide a clear understanding of the present invention. However, one skilled in the art will readily understand how the present invention may be used with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the features of the present invention.

[0025] Furthermore, it will be clear from the foregoing description that the invention is not limited to the embodiment described. Numerous possible modifications, changes, variations, and substitutions, while preserving the spirit and form of the present invention, will be apparent to those skilled in the art.

[0026] The proposed solution is a hardware and software system with a human-machine interface based on a sensor system, including optometry sensors. It can be used to control external interfaces and devices for a wide range of tasks. Application areas include medical diagnostics and rehabilitation, assistive technologies for people with disabilities, and control of joysticks and computer accessories, including those for virtual reality.

[0027] The proposed solution makes it possible to create human-machine interfaces that can quickly and accurately control a wide range of devices through the movement of specific muscle groups. These devices could include prosthetic limbs, virtual reality devices, personal computers, industrial or medical robots, and more. The proposed solution consists of a wearable device and a data processing server.

[0028] The wearable device is designed as a bracelet and includes at least one optomyographic sensor and a microcontroller. The optomyographic sensor consists of a phototransistor and an infrared LED. The microcontroller is configured to control at least one optomyographic sensor. The microcontroller controls the on-time of the infrared LED and the time it takes to read data from the phototransistor. The optomyographic sensor operates at a frequency of 10-50 Hz. After collecting data, the microcontroller sends it to the server for processing.

[0029] On the data processing server, the received data is filtered, processed, and transmitted to the input of a mathematical model or artificial neural network (the architecture of which depends on the task at hand) or a classical machine learning algorithm (decision-making algorithm) in order to obtain the necessary control signals and then send such signals to external devices.

[0030] The input of a mathematical model or artificial neural network or classical machine learning algorithm is the input optomiography data, which is a matrix X of size MxN, where M is the number of channels (in the materials of this application, 50 channels are used), and N is the number of time samples, and the input data of decoded states, which is a matrix Y of size KxN, where K is the dimension of the state space, and N is the number of time samples.

[0031] Spatial states describe all possible states of a hand. For example, in the case of mouse movement recognition, state spaces are described using the cosine and sine of the angle through which the hand is rotated. And in the case of handwriting recognition, the state space is described using a Cartesian coordinate system (x1,y1), where the points describe the points in the plane where the pen is currently writing.

[0032] The algorithm for training the model to predict the required control signals includes the following steps.

[0033] The calibration data is divided into training and testing subsamples.

[0034] Outliers (those located at a large distance from the distribution of the majority of the data) can be discarded from the test data matrix. For each point in time, its state is decoded, that is, a prediction is made of which signal should be sent to the external control device. Accordingly, a feature characteristic is formed for each state.

[0035] Sometimes the feature characteristic is the vector of the channels' GMG at a given point in time.

[0036] Sometimes the feature characteristic is a concatenation of the vectors of the GMG channels at a given moment in time and in the previous L time samples.

[0037] Sometimes the feature characteristic is a vector of the GMF channels at a given time passed through the principal component method to reduce dimensionality.

[0038] The solution uses a standard architecture (e.g., kNN, SVM, MLP, RNN) in which weights are randomly initialized. The weights are then adjusted using the gradient descent algorithm, the random gradient algorithm, or the Adam gradient descent algorithm to achieve the best result. N gradient descent steps are performed, updating the model weights. The updated model is then fed with the test set data, and the prediction accuracy is assessed using various metrics. When the weight changes resulting from a gradient descent step do not improve the prediction quality, training is terminated.

[0039] Example of using algorithm 1.

[0040] Collection of calibration data

[0041] To implement the algorithm, calibration data is collected. This data collection produces optomiographic time series data (an X matrix of size M x N, where M = 50) and hand displacement data (a Y matrix of size K x N, where K = 1). Hand displacement data represents a coded value characterizing one of five states: up, down, right, left, and relaxed (see Fig. 1).

[0042] Preprocessing

[0043] During the preprocessing stage, the data from each optometry channel is standardized. The input value at each point in time is calculated as the difference in the mean for each channel, divided by the standard error for that channel.

[0044] X st = (x - u) / sx st - standardized data and - mean value s - standard deviation

[0045] Processing with SVM algorithm with RBF kernel The RBF kernel function calculates the similarity between two samples x and x'.

[0046] K(x, x') = exp(-gamma ||x — x'|| 2 )

[0047] An SVM with an RBF kernel seeks to find the optimal hyperplane separating data points from different classes in the feature space. The decision boundary is defined by a subset of training samples, called support vectors, which are the closest data points to the decision boundary.

[0048] Education

[0049] During training, an SVM with an RBF kernel optimizes the decision boundary position, maximizing the margin and minimizing classification error on the training data. This optimization was performed using the gradient descent method.

[0050] Evaluation of the quality of predictions

[0051] The quality of predictions is assessed using a confusion matrix, which shows the probability of each condition being recognized correctly and the probability of being recognized incorrectly. Figure 2 shows the confusion matrix for a group of six subjects.

[0052] An example of implementation of algorithm 2.

[0053] Collection of calibration data

[0054] To implement the algorithm, calibration data is collected. This collection produces optomiographic time series data (an X matrix of size M x N, where M = 50) and handwriting input data (a Y matrix of size K x N, where K = 2). The handwriting input data consists of two time series of X, Y coordinates that alternate during the writing process, as shown in Figure 3.

[0055] Preprocessing

[0056] During the preprocessing stage, the channels from which the signal will be extracted for input into the neural network are selected. Channels with a high-quality signal are selected based on the principle that the mean of the first half of the calibration record does not differ by more than a factor of two from the mean of the second half of the calibration record.

[0057] In-network processing

[0058] The time series is fed to the input of a 5-layer GRU network, with the hidden matrix dimension at each layer being 50. GRLI contains two gates:

[0059] Reset gate

[0060] Update gate z, = (W l: >x, + U^h,^)

[0061] Gates are mechanisms that allow the network to decide what information to retain or forget when processing new inputs. The update vector determines what portion of the previous hidden state should be updated with the new value. The reset vector determines what portion of the previous hidden state should be ignored when computing the new state. Based on the input, the previous hidden state, and the update vector, the network computes an output vector, which is then passed to the next time step or used to solve a specific task. t = tanh(

[0062] After 5 layers of GRU, the output goes through the LeakyRELU activation function

[0063] LeakyReLU(x)=max(0,x)+negative_slope*min(0,x)

[0064] Next comes a data normalization layer, after which it is passed to a linear layer, which compresses the input vector into two values—the predicted x- and y-coordinates of the imaged space. This processing occurs for each time sample of the optomiographic data.

[0065] Education

[0066] During training, the most optimal network parameter values ​​are calculated using the backpropagation method with the Adam optimizer over the MSELoss error function. MSELoss is calculated as the mean squared difference between the predicted coordinates and the actual coordinates.

[0067] The result of restoring the letter trajectory

[0068] The quality of the reconstructed trajectory is assessed using the determination coefficient and visually, where the original trajectory is compared with the obtained one. Figure 4 shows the trajectory reconstruction results. The images in the left column show the original symbols. The images in the right column show the reconstructed symbols. The algorithm predicts the control signal, and this control signal is transmitted to an external device, such as via a serial connection, a UDP socket, or a TCP socket.

[0069] Control signals. Example 1

[0070] For handwriting input, the control commands are the X,Y coordinates of points on the tablet display. In the proposed solution, signals are transmitted using the Android Debug Bridge (adb) software development kit (SDK). After processing the data on the host computer, control commands are transmitted to the tablet via the adb SDK, which colors the pixels on the tablet display according to the coordinates predicted by the model.

[0071] Control signals. Example 2

[0072] To control the prosthesis, the control commands are the codes 1 and 0, which encode the closing and opening of the prosthesis, respectively. These codes are then converted into corresponding byte messages, which are sent to the prosthetic device via Bluetooth using a Bleak packet. Thus, the observed state is calculated on the main computing device, and the control command is then transmitted to the prosthesis via a wireless connection.

[0073] A computing system that provides the data processing necessary for the implementation of the claimed solution generally contains the following components: one or more processors, at least one memory, a data storage means, input / output interfaces, an input means, and network interaction means.

[0074] When executing machine-readable commands contained in the RAM, the processor of the device is configured to perform the basic computing operations necessary for the operation of the device or the functionality of one or more of its components.

[0075] Memory is typically implemented as RAM, where the necessary software logic is loaded to provide the required functionality. When implementing the proposed solution, the memory capacity required for its implementation is allocated.

[0076] The data storage device can be a HDD, SSD, RAID array, network storage, flash memory, etc. It enables long-term storage of various types of information, such as the aforementioned user / passenger dataset files, databases containing records of time intervals measured for each user, user IDs, etc. The interfaces are standard for connecting and operating peripherals and other devices, such as USB, RS232, RJ45, COM, HDMI, PS / 2, Lightning, etc.

[0077] The choice of interfaces depends on the specific design of the device, which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

[0078] A keyboard may be used as a data input device in any embodiment of the system implementing the described method. The keyboard hardware may be any known device: it could be a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computing device. The connection may be either wired, in which the keyboard cable is connected to a PS / 2 or USB port located on the desktop computer's system unit, or wireless, in which the keyboard communicates wirelessly, such as via radio, with a base station, which is in turn directly connected to the system unit, such as via a USB port.In addition to the keyboard, data input devices may also include: a joystick, display (touch screen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

[0079] Network communication tools are selected from a device that provides network data reception and transmission, such as an Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDA, RFID module, GSM modem, etc. These tools facilitate data exchange via a wired or wireless data transmission channel, such as WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN, or GSM.

[0080] The device components are connected via a common data bus.

[0081] In these application materials, a preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other particular embodiments of its implementation, which do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

Formula 1. A system for controlling external devices via a human-machine interface, comprising: a wearable device placed on the user, containing at least one optomyographic sensor configured to receive a signal about the current state of the user's muscle, with a frequency of 10-50 Hz, wherein the optomyographic sensor consists of a phototransistor and an infrared LED; a microcontroller configured to control the time of switching on the infrared LED and the time of reading data from the phototransistor and transmitting the received data to a data processing server; a data processing server configured to process the received data using artificial intelligence algorithms to receive control signals and transmit control signals to external devices.

2. The system according to paragraph 1, characterized in that a bracelet is used as a wearable device.

3. The system according to claim 1, characterized in that the external devices are at least prosthetic limbs or virtual reality devices, or a personal computer, or an industrial robot, or a medical robot.

4. A method for controlling external devices by means of a human-machine interface, comprising the following steps: recording user movement signals by means of at least one optomiographic sensor consisting of a phototransistor and an infrared LED, with a frequency of 10-50 Hz and transmitting the obtained data to a microcontroller of a wearable device; the microcontroller of the wearable device controls the optomiographic sensors and transmits the obtained data from the sensors to an information processing server; the information processing server receives data from the optomiographic sensors and processes them by means of artificial intelligence algorithms to obtain control signals; the obtained control signals are transmitted to external devices.

Images