Tactile decoding normal form, model acquisition method, scene reconstruction method, device, equipment and medium

Abstract

The invention relates to the technical field of brain-computer interfaces, and discloses a tactile decoding normal form, a model acquisition method, a scene reconstruction method, a device, equipment and a medium, the acquisition method comprises the following steps: acquiring tactile related features in an electroencephalogram signal through a feature extractor; wherein the electroencephalogram signals are electroencephalogram signals which are acquired by the subject through a tactile decoding normal form when pressure is applied to a target part of the subject; inputting the touch related characteristics into a decoding model to obtain a pressure signal of each pressure sensor channel of the target part, and further generating a stress thermodynamic diagram; comparing the stress thermodynamic diagram with an actual thermodynamic diagram to obtain a decoding error; and iteratively updating the parameters of the decoding model according to the decoding error until the updated decoding model meets a preset condition, obtaining a tactile scene reconstruction model, and inputting the target tactile related features into the tactile scene reconstruction model during scene reconstruction to obtain a target stress thermodynamic diagram. The invention provides a natural and accurate tactile scene reconstruction mode.

Description

Technical Field

[0001] This application relates to the technical field of brain-computer interfaces, and in particular, to a tactile decoding paradigm, a model acquisition method, a scene reconstruction method, a device, a device, and a medium. Background Art

[0002] In the existing technical field of brain-computer interfaces, although there have been studies on realizing speech decoding and visual reconstruction through electroencephalogram (EEG) signals, there is currently no related technology for accurately reconstructing tactile perception using EEG signals. In the existing tactile brain-computer interface technology, the main method is to activate tactile signals in the nervous system through electrical stimulation or other stimulation methods, so that users can perceive pressure tactile information. However, the tactile feedback induced by these technologies is different from the natural perception mode of the human body to pressure, and long-term use may lead to feedback fatigue or increase cognitive burden, thus limiting the potential of its practical application. In addition, in the existing force feedback brain-computer interface technology, the main method is to transmit the reaction force information of the overall end force to help users perceive and control the mechanical properties of objects. The overall end force refers to the total force borne by the end of the limb, such as the resistance experienced by a person when pushing a handle. However, these technologies are only limited to the perception of the overall end force, and do not fully consider the magnitude and distribution of the forces borne by each region of the end parts such as the hand, so they lack the ability to perceive fine mechanical information such as the palm. This results in the inability of the force feedback technology to achieve accurate tactile perception reconstruction and effectively model more fine-grained tactile information, further limiting its application potential. Summary of the Invention

[0003] In view of this, the embodiments of this application provide a tactile decoding paradigm, a model acquisition method, a scene reconstruction method, a device, a device, and a medium, which can effectively realize tactile scene reconstruction.

[0004] In a first aspect, this application proposes a tactile decoding paradigm, including:

[0005] The paradigm is to set pressure sensors at multiple target point positions on the target part of the subject to collect the pressure distribution signal when pressure is applied to the target part of the subject, and synchronously collect the EEG signal when pressure is applied to the target part of the subject.

[0006] In a second aspect, this application proposes a method for obtaining a tactile scene reconstruction model, including:

[0007] Obtaining tactile-related features in the EEG signal through a feature extractor; wherein, the EEG signal is the EEG signal when pressure is applied to the target part of the subject collected by the subject through the above-mentioned tactile decoding paradigm;

[0008] Input the tactile-related features into a decoding model to obtain the pressure signals of each pressure sensor channel of the target part, and generate a force heat map based on the pressure signals;

[0009] Compare the force heat map with an actual heat map obtained based on the actual pressure values of the target part of the current subject to obtain a decoding error;

[0010] Iteratively update the parameters of the decoding model according to the decoding error until the updated decoding model meets a preset condition, then obtain a tactile scene reconstruction model.

[0011] In some embodiments, the tactile-related features include: the spatial features of the pressure distribution of the target part, and the electroencephalogram signal features when different pressures are applied at different spatial positions of the target part.

[0012] In some embodiments, the decoding model is a U-Net model, a regression model, or a fusion model of a U-Net model and a regression model.

[0013] In some embodiments, the U-Net model includes an encoder, a bottleneck layer, a decoder, a skip connection path, and an output layer;

[0014] The step of inputting the tactile-related features into a decoding model to obtain the pressure signals of each pressure sensor channel of the target part includes:

[0015] Input the transformed tactile-related features into the encoder, so that the encoder performs a downsampling operation on the transformed tactile-related features to obtain downsampled tactile-related features; wherein, the transformed tactile-related features are pictures of a preset size converted from the tactile-related features;

[0016] Perform feature enhancement on the downsampled tactile-related features through the bottleneck layer to obtain enhanced tactile-related features;

[0017] Process the enhanced tactile-related features through the decoder to increase the spatial resolution of the enhanced tactile-related features and obtain upsampled tactile-related features;

[0018] Stitch the upsampled tactile-related features and the downsampled tactile-related features through the skip connection path to obtain stitched tactile-related features;

[0019] Perform convolution processing on the stitched tactile-related features through the output layer to obtain the pressure signals of each pressure sensor channel of the target part.

[0020] In some embodiments, the regression model is a fully connected neural network;

[0021] Inputting the tactile-related features into a decoding model to obtain the pressure signals of each pressure sensor channel of the target part, including:

[0022] Inputting the tactile-related features into a fully-connected neural network to obtain the pressure signals of each pressure sensor channel of the target part; wherein, the pressure signal of each pressure sensor channel of the target part is the predicted value of the force magnitude at each pressure sensing electrode position corresponding to the target part.

[0023] In a third aspect, the present application proposes a tactile scene reconstruction method, including:

[0024] Obtaining target electroencephalogram (EEG) signals when a target person imagines applying pressure to a target part, and obtaining target tactile-related features in the target EEG signals based on a feature extractor;

[0025] Inputting the target tactile-related features into the tactile scene reconstruction model obtained by the above tactile scene reconstruction method to obtain a target force heat map.

[0026] In a fourth aspect, the present application proposes a device for obtaining a tactile scene reconstruction model, including:

[0027] A tactile-related feature acquisition module, configured to obtain tactile-related features in EEG signals through a feature extractor; wherein, the EEG signals are the EEG signals when a subject applies pressure to a target part of the subject collected through the above tactile decoding paradigm;

[0028] A force heat map acquisition module, configured to input the tactile-related features into a decoding model to obtain the pressure signals of each pressure sensor channel of the target part, and generate a force heat map according to the pressure signals;

[0029] A decoding error acquisition module, configured to compare the force heat map with an actual heat map obtained according to the actual pressure value of the target part of the current subject to obtain a decoding error;

[0030] A tactile scene reconstruction model acquisition module, configured to iteratively update the parameters of the decoding model according to the decoding error until the updated decoding model meets a preset condition, and then obtain a tactile scene reconstruction model.

[0031] In a fifth aspect, the present application proposes a terminal device, where the terminal device includes a processor and a memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the above method for obtaining a tactile scene reconstruction model, or implement the above tactile scene reconstruction method.

[0032] Sixth aspect, the present application provides a computer-readable storage medium storing a computer program, which when executed on a processor, implements the method for obtaining a tactile scene reconstruction model as described above, or implements the tactile scene reconstruction method as described above.

[0033] The embodiments of the present application have the following beneficial effects: When obtaining tactile pressure signals, the task paradigm in the present application adopts the natural process of human tactile pressure perception, and does not rely on external methods such as electrical stimulation to activate tactile signals in the nervous system, which can effectively improve the recognition accuracy of tactile information. This method of the present application provides a more efficient and safe way for the rehabilitation of tactile perception ability, especially suitable for patients who need tactile reconstruction; in addition, by extracting tactile-related features from electroencephalogram signals, the present application can accurately reconstruct tactile scenes, such as the process of applying pressure with the palm, thereby improving the feature extraction and recognition ability of human tactile information and providing support for the accurate reconstruction of tactile information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0035] Figure 1 Shows a first flowchart of the method for obtaining a tactile scene reconstruction model according to an embodiment of the present application;

[0036] Figure 2 Shows a schematic diagram of obtaining electroencephalogram signals according to an embodiment of the present application;

[0037] Figure 3 Shows a second flowchart of the method for obtaining a tactile scene reconstruction model according to an embodiment of the present application;

[0038] Figure 4 Shows a flowchart of a tactile scene reconstruction method according to an embodiment of the present application;

[0039] Figure 5 Shows a schematic diagram of the process of obtaining a target force heat map according to an embodiment of the present application;

[0040] Figure 6 Shows a schematic structure diagram of a device for obtaining a tactile scene reconstruction model according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0042] Generally, the components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0043] In the following, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are only intended to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be construed as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0044] Unless otherwise defined, all terms (including technical terms and scientific terms) used here have the same meaning as commonly understood by those of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in a commonly used dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or being overly formal, unless clearly defined in the various embodiments of the present application.

[0045] The following will describe in detail some embodiments of the present application with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0046] A brain-computer interface (BCI) is a technology used to connect the brain with external devices, which interprets an individual's intention or state by detecting and analyzing electroencephalogram (EEG) activities. This technology has been applied in many fields, such as assisting disabled people to control smart home systems and participating in entertainment activities.

[0047] Currently, BCI technologies are mainly divided into two categories:

[0048] 1. Invasive BCI: Electrodes need to be implanted into the cerebral cortex through surgery. Although it can obtain higher-quality neural signals, the surgical risk is relatively high and it may cause a series of biocompatibility problems.

[0049] 2. Non-invasive BCI: A head-mounted device is used to collect electroencephalogram (EEG) signals on the scalp surface. It has high safety and a better user experience, so it is more popular in practical applications.

[0050] At present, non-invasive BCI technology has made remarkable progress in speech decoding and visual reconstruction. However, the research in the field of tactile perception is relatively lagging behind. Especially in the precise perception of the pressure distribution of human body parts (such as palms or soles), the existing research is still in its infancy and lacks systematic exploration and breakthroughs.

[0051] In the future, in-depth research on how to use EEG signals to efficiently capture and analyze tactile information, especially the precise recognition of the pressure distribution of human body parts, will become one of the key directions to promote the comprehensive development of BCI technology.

[0052] The following will illustrate the method for obtaining the tactile scene reconstruction model in combination with some specific embodiments.

[0053] Figure 1 FIG. shows a schematic flow chart of a method for obtaining a tactile scene reconstruction model according to an embodiment of the present application. Exemplarily, the method for obtaining the tactile scene reconstruction model includes steps S100-S400:

[0054] S100, obtaining tactile-related features in the EEG signal through a feature extractor.

[0055] Before step S100, there is also step S500, collecting the pressure distribution signal when pressure is applied to the target part of the subject through pressure sensors arranged at multiple target point positions on the target part of the subject, and synchronously collecting the EEG signal when pressure is applied to the target part of the subject.

[0056] Exemplarily, the target part includes but is not limited to the limb end parts such as palms, soles, fingers, and toes. When training the decoding model, the training samples used are the heat maps drawn according to the actual pressure values of the target part of the subject, and the EEG signals of the subject collected when pressure is applied to the target part.

[0057] Such as Figure 2As shown, when the subject is performing an active control task (applying pressure to a horizontal plane), the pressure distribution on the target area can be obtained by setting pressure sensors at multiple target points on the target area. Among them, the target points on the target area need to be evenly distributed throughout the target area. The more target points there are, the better the collected pressure distribution will be. Usually, the pressure distribution on the palm is recorded through a 256-channel full-palm high-density pressure sensing electrode array. If the target area is the palm, the pressure distribution on the palm is recorded through a 256-channel full-palm high-density pressure sensing electrode array. While collecting the pressure distribution of the target area, the electroencephalogram (EEG) signals of the subject during the task execution are synchronously collected. This process is consistent with the natural perception mode of the human body to pressure. Compared with the existing tactile brain-computer interface technology and force feedback brain-computer interface technology, the tactile feedback provided in this embodiment is more natural and comfortable.

[0058] Then, subsequently, the collected EEG signals are input into a decoding model (a U-Net model, a regression model, or a fusion model of a U-Net model and a regression model) for task scenario reconstruction to obtain a corresponding heat map. Then, the obtained heat map is compared with the heat map obtained based on the actual pressure. Then, the decoding model is updated in combination with the comparison result, and finally, a tactile scenario reconstruction model is obtained.

[0059] In this embodiment, after obtaining the electroencephalogram (EEG) signals, preprocessing of the EEG signals is performed before obtaining tactile-related features from the EEG signals. The process of preprocessing the EEG signals includes denoising, filtering, and normalization to remove interference signals and improve data quality. The preprocessing process includes artifact removal (Artifact removal is a process of removing noise from EEG signals, aiming to improve the quality of EEG signals. Artifacts may come from various sources, including eye movements, muscle activities, power supply interference, etc. Artifact removal methods include, but are not limited to, independent component analysis (ICA), principal component analysis (PCA), and threshold-based signal rejection, etc. These methods can help identify and remove the noise sources to ensure that the finally obtained signals are as pure as possible) and band-pass filtering (In EEG signal processing, a band-pass filter with a frequency range between 0.5 Hz and 40 Hz is usually adopted. Such a setting can effectively filter out low-frequency slow-wave activities (such as fluctuations caused by breathing and heartbeat) and high-frequency electronic device interference, thus retaining the intermediate-frequency signals closely related to brain activities); meanwhile, the signals are normalized to eliminate individual differences and ensure the stability and consistency of the subsequent trained model. Among them, during signal normalization, since there may be differences in the physiological characteristics and electrode placement positions of different individuals, it is necessary to normalize the EEG signals to eliminate the influence brought by these factors. This process usually includes scaling and translation of the EEG signals to make the data distributions of all individuals tend to be consistent. Commonly used methods include Z-score normalization, Min-Max normalization, etc. Through these processes, the data can be made more suitable for subsequent machine learning or statistical analysis, thereby ensuring the consistency and stability of subsequent model training.)

[0060] In some embodiments, after preprocessing the EEG signals, it is necessary to obtain tactile-related features from the preprocessed EEG signals. In this embodiment, the feature extractor is formed by combining a Transformer model and a convolutional neural network. Among them, the core of the Transformer model lies in the self-attention mechanism. In this embodiment, the EEG signals are regarded as a series of "sequences" through this Transformer model, and the long-term dependencies in the signals are captured through the self-attention mechanism. In this embodiment, the convolutional neural network (CNN) is used to capture the local spatial features in the EEG signals.

[0061] When obtaining tactile-related features in this embodiment, a convolutional layer can be added after the Transformer model. Specifically, a Transformer model can be constructed first. The Transformer model includes an input layer for taking the preprocessed electroencephalogram (EEG) signal as input; an embedding layer for embedding the input signal into a high-dimensional space so that the subsequent self-attention mechanism can work better; a multi-head self-attention layer for using multiple self-attention heads, each head independently calculating the attention weights of the input signal to capture different features in the signal; and a feed-forward neural network for performing a linear transformation on the output of each position and then processing it through a non-linear activation function (such as ReLU). Then, in combination with the convolutional layer, a convolutional layer can be added after the Transformer module to capture the local spatial features in the EEG signal. By combining the features extracted by the Transformer module and the features extracted by the convolutional layer, the tactile-related features in this embodiment can be obtained.

[0062] Among them, the tactile-related features include the spatial distribution features of the pressure on the target part and the EEG signal features (such as EEG signal spatial patterns and temporal dynamic features) when different pressures are applied at different spatial positions of the target part. Here, the palm is taken as an example to illustrate the tactile-related features, and the other target parts have the same tactile-related features as the palm. If the target part is the palm, the tactile-related features include the spatial distribution features of the palm pressure and the EEG signal features when different pressures are applied at different spatial positions of the palm. Among them, the spatial distribution features of the palm pressure involve the pressure distribution of different parts of the palm (such as the palm center, fingertips, etc.), and the spatial patterns in the EEG signal will be affected by these pressure distributions. In this embodiment, the spatial patterns of the EEG signal include EEG spatial distribution features, and the spatial patterns of the EEG signal are mainly reflected in the intensity differences of different electrode signals, the correlations between electrodes, the spatial topological structure, and the distribution of event-related potentials (ERPs). Specifically, certain brain regions will show stronger activities in specific tasks, forming local "hot spots", and the correlations between different electrode signals reflect the functional connections between brain regions. In this embodiment, convolutional methods are used to extract the spatial features in the EEG signal that reflect different pressure distributions, helping to capture the local response patterns related to tactile (mainly the feeling of force) stimuli; for the EEG signal response features when different pressures are applied at different spatial positions of the palm, different positions of the palm (such as the palm center, fingers, etc.) and different magnitudes of the applied force (for example, light touch or heavy pressure) will produce different responses in the EEG signal.

[0063] In this embodiment, by analyzing the spatial characteristics and temporal dynamics of electroencephalogram (EEG) signals, the variation rules of EEG signals under different force conditions are identified; for the spatial patterns and temporal dynamic characteristics of EEG signals, tactile perception not only involves spatial distribution but also the variation characteristics of signals in the time dimension. In this embodiment, by extracting the variation rules of EEG signals at different time points, the temporal dynamic characteristics related to tactile stimuli (the force condition of the palm) are captured, so as to completely reflect the process of tactile perception (the perception of the force on the palm).

[0064] S200. Input the tactile-related features into the decoding model to obtain the pressure signals of each pressure sensor channel at the target part, and generate a force heat map according to the pressure signals.

[0065] In this embodiment, the decoding model can be a U-Net model and a regression model.

[0066] First, the case where the decoding model is a U-Net model is described below.

[0067] The U-Net model of this embodiment includes an encoder, a bottleneck layer, a decoder, a skip connection path, and an output layer.

[0068] In some embodiments, as Figure 3 shown, inputting the tactile-related features into the decoding model to obtain the pressure signals of each pressure sensor channel at the target part includes steps S210 - S250:

[0069] S210. Input the transformed tactile-related features into the encoder, so that the encoder performs a downsampling operation on the transformed tactile-related features to obtain downsampled tactile-related features.

[0070] Among them, the transformed tactile-related features are pictures of a preset size converted from the tactile-related features.

[0071] The encoder is mainly responsible for extracting the features of the preprocessed EEG signals. The main task of the encoder is to extract high-level abstract features from the EEG signals. For the data captured by the tactile sensors, the encoder can identify different types of tactile patterns or textures. The purpose of this process is to reduce the dimensionality of the data while retaining the most important information. The encoder includes a signal processing module, a self-attention module (ResAttention), and a downsampling module. Among them, the signal processing module contains convolutional layers, activation functions (such as ReLU or StarReLU), and layer normalization, which are used to extract the local features of the EEG signals. The self-attention module combines residual connections and attention mechanisms, which may enable the U-Net model to better capture key information and avoid information loss. The downsampling module gradually reduces the spatial resolution of the input features through downsampling operations (such as using a convolutional kernel of size 4x4 with a stride of 2) to extract higher-level features. In layer normalization, the convolutional layers use convolutional kernels of 3x3 or 5x5 with a stride of 2 for downsampling.

[0072] S220. Feature enhancement is performed on the downsampled tactile-related features through the bottleneck layer to obtain enhanced tactile-related features.

[0073] The bottleneck layer is used to enhance the features extracted from the encoder. Specifically, the bottleneck layer is located between the encoder and the decoder and is a hidden layer with a very low dimension. Its function is to further compress the input EEG signals into a more compact form. This compression not only helps improve the efficiency of the model but also enables the model to focus on the most critical information, thereby potentially improving the generalization ability of the model. The bottleneck layer includes 2 signal processing modules and 1 self-attention module. The signal processing module and the self-attention module in the bottleneck layer are the same as those in the encoder.

[0074] S230. The decoder processes the enhanced tactile-related features to increase the spatial resolution of the enhanced tactile-related features and obtain upsampled tactile-related features.

[0075] The decoder is responsible for receiving the compressed data from the bottleneck layer and attempting to reconstruct the original input. In tactile information processing, the decoder needs to recover detailed tactile-related features from highly compressed information. Through inverse operations (such as deconvolution), the decoder can generate an output close to the original tactile signal or generate a specific representation required for tactile feedback according to application requirements. That is, the decoder is responsible for recovering the original high-resolution image from the features extracted from the bottleneck layer. Through a series of upsampling operations, the spatial resolution of the feature map is gradually restored. Among them, each layer in the decoder will gradually restore the spatial dimension of the feature map. The decoder includes a signal processing module with a convolution kernel size of 4X4 and a stride of (2,2), and the rest is the same as the encoder; a self-attention module, which is the same as the encoder; an upsampling module that gradually restores the spatial resolution of the feature map through deconvolution operations.

[0076] S240, splice the upsampled tactile-related features and the downsampled tactile-related features through the skip connection path to obtain the spliced tactile-related features.

[0077] The skip connection path connects the corresponding layers between the encoder and the decoder, ensuring that the flow of information is not blocked and avoiding feature loss. These connections enable the network to retain low-level features while using the decoder to recover higher-level detailed information.

[0078] S250, perform convolution processing on the spliced tactile-related features through the output layer to obtain the pressure signals of each pressure sensor channel at the target site.

[0079] The output layer is used to convert the final output of the decoder into a final image for visualizing the pressure distribution of the palm. The output layer includes a StarReLU activation function and a signal processing module with a convolution kernel size of 3X3 or 5X5, and the rest is the same as the encoder.

[0080] If the decoding model is a regression model, the regression model adopted in this embodiment is a fully connected neural network. Input the tactile-related features into the decoding model to obtain the pressure signals of each pressure sensor channel at the target site, including: input the tactile-related features into the fully connected neural network to obtain the pressure signals of each pressure sensor channel at the target site; among them, the pressure signals of each pressure sensor channel at the target site are the predicted values of the force magnitudes at each pressure sensing electrode position corresponding to the target site; draw a force heat map according to the pressure prediction values.

[0081] S300, compare the force heat map with the actual heat map obtained according to the actual pressure values of the target site of the current subject to obtain the decoding error.

[0082] The S400 iteratively updates the parameters of the decoding model according to the decoding error until the updated decoding model meets the preset conditions, and then a tactile scene reconstruction model is obtained.

[0083] Exemplarily, after obtaining the corresponding force thermogram through the above decoding model, in order to verify the accuracy of the model, the force thermogram output by the obtained model is compared with the actual thermogram obtained from the actual pressure value of the target part of the current subject. If the similarity between the force thermogram output by the model and the actual force thermogram reaches the preset requirement, it indicates that the model is accurate. If the similarity does not reach the preset requirement, it indicates that the decoding model still needs to be improved and updated, and then the model is updated according to the obtained decoding error.

[0084] Figure 4 Fig. shows a schematic flowchart of a tactile scene reconstruction method according to an embodiment of the present application. Exemplarily, the tactile scene reconstruction method includes:

[0085] S10, obtaining target electroencephalogram signals when a target person imagines applying pressure to a target part, and obtaining target tactile-related features in the target electroencephalogram signals based on a feature extractor.

[0086] When reconstructing the tactile scene, the preprocessing of the target electroencephalogram signals and the obtaining of the target tactile-related features are the same as those in the above tactile scene reconstruction method for the preprocessing of electroencephalogram signals and the extraction of tactile-related features, and will not be elaborated here.

[0087] S20, inputting the target tactile-related features into the above tactile scene reconstruction model to obtain a target force thermogram.

[0088] As Figure 5 shown, in this embodiment, the obtained target tactile-related features can be input only into the U-Net model, or only into the regression model, or can be input into the U-Net model and at the same time input into the regression model, so as to obtain a target force thermogram.

[0089] If the target tactile-related feature is input into the U-Net model and at the same time input into the regression model, that is, two models are used to reconstruct the tactile scenario simultaneously, then each of the two models obtains a target force heat map. For example, the target force heat map output by the U-Net model is the first target force heat map, and the target force heat map output by the regression model is the second target force heat map. In this way, after obtaining the first target force heat map and the second target force heat map, the first target force heat map and the second target force heat map can be compared; if the difference between the first target force heat map and the second target force heat map is not within the preset difference range, the target EEG signal when the target person imagines applying pressure to the target part is re-obtained until the difference between the first target force heat map and the second target force heat map is within the preset difference range. In this way, the accuracy of tactile scenario reconstruction can be improved.

[0090] When this application acquires the tactile pressure signal, the task paradigm adopts the natural process of human tactile pressure perception and does not rely on external methods such as electrical stimulation to activate the tactile signal in the nervous system, which can effectively improve the recognition accuracy of tactile information. This method of this application provides a more efficient and safe way for the rehabilitation of tactile perception ability, especially suitable for patients who need tactile reconstruction; in addition, this application extracts tactile-related features from EEG signals through an end-to-end deep learning method, which can accurately reconstruct the tactile scenario, such as the process of applying pressure with the palm, so as to improve the feature extraction and recognition ability of human tactile information and provide support for the accurate reconstruction of tactile information.

[0091] This method of tactile scenario reconstruction of this application can not only be applied to the field of rehabilitation medicine. For example, for patients who have lost limbs, the method of this application realizes prosthetic control and tactile feedback by accurately decoding the pressure distribution of the palm. This tactile feedback brain-computer interface enables patients to perceive the pressure distribution of objects through the prosthesis, improves the fine control ability of the prosthesis, and improves the tactile perception experience (such as prosthesis users can perceive the pressure when holding an object, so as to achieve more natural and accurate control and realize the control of a dexterous hand).

[0092] The method of this application can also be applied to neurorehabilitation training. For example, for patients with nerve injuries such as stroke and spinal cord injury, the tactile feedback brain-computer interface can be used as an auxiliary tool to help patients carry out rehabilitation training. By providing tactile feedback, it can help patients restore the tactile perception and motor function of the hand and improve the rehabilitation effect.

[0093] In addition, during the rehabilitation process after limb reconstruction surgery, the method of this application can help patients reconstruct the sensation of the hand, provide more accurate movement control and pressure perception, thereby accelerating the rehabilitation process. In virtual reality and augmented reality environments, tactile feedback can also enhance the user experience, make the tactile perception of virtual objects more real, and improve the immersion.

[0094] Figure 6 FIG. 2 shows a schematic structural diagram of a tactile scene reconstruction model acquisition device according to an embodiment of the present application. Exemplarily, the tactile scene reconstruction model acquisition device includes:

[0095] A tactile-related feature acquisition module 100, configured to acquire tactile-related features in the electroencephalogram signal through a feature extractor; wherein, the electroencephalogram signal is the electroencephalogram signal when a subject applies pressure to a target part collected by the above-mentioned tactile decoding paradigm.

[0096] A force heat map acquisition module 200, configured to input the tactile-related features into a decoding model to obtain a pressure signal for each pressure sensor channel of the target part, and generate a force heat map according to the pressure signal.

[0097] A decoding error acquisition module 300, configured to compare the force heat map with an actual heat map obtained according to the actual pressure value of the target part of the current subject to obtain a decoding error.

[0098] A tactile scene reconstruction model acquisition module 400, configured to iteratively update the parameters of the decoding model according to the decoding error until the updated decoding model meets a preset condition, and then obtain a tactile scene reconstruction model.

[0099] It can be understood that the device in this embodiment corresponds to the tactile scene reconstruction model acquisition method in the above embodiment, and the optional items in the above embodiment also apply to this embodiment, so they will not be described again here.

[0100] The present application also provides a terminal device. Exemplarily, the terminal device includes a processor and a memory. Among them, the memory stores a computer program, and the processor runs the computer program, so that the terminal device executes the above-mentioned tactile scene reconstruction model acquisition method or executes the above-mentioned tactile scene reconstruction method.

[0101] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0102] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Among them, the memory is used to store a computer program, and after receiving an execution instruction, the processor can execute the computer program accordingly.

[0103] This application also provides a computer-readable storage medium for storing the computer program used in the above terminal device. For example, the computer-readable storage medium can include, but is not limited to: various media such as USB flash drives, mobile hard disks, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disks, or optical discs that can store program codes.

[0104] In several embodiments provided by this application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions, and operations of the device, method, and computer program product according to multiple embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the blocks can occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the structure diagram and / or flowchart, as well as the combination of blocks in the structure diagram and / or flowchart, can be implemented by a dedicated hardware-based system that executes the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0105] In addition, each functional module or unit in various embodiments of this application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0106] When the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application.

[0107] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application.

Claims

1. A tactile decoding paradigm, characterized in that: include: The paradigm is to set pressure sensors at multiple target point positions of the subject's target part to collect pressure distribution signals when pressure is applied to the subject's target part, and simultaneously collect EEG signals when pressure is applied to the subject's target part.

2. A method for obtaining a tactile scene reconstruction model, characterized in that: include: Acquire tactile-related features in the EEG signal through a feature extractor; wherein the EEG signal is an EEG signal when the subject applies pressure to the target part of the subject, which is collected by the subject through the tactile decoding paradigm described in claim 1; Inputting the tactile-related features into a decoding model to obtain a pressure signal of each pressure sensor channel of the target part, and generating a force heat map according to the pressure signal; Comparing the stress thermogram with an actual thermogram obtained according to the actual pressure value of the target part of the subject to obtain a decoding error; The parameters of the decoding model are iteratively updated according to the decoding error until the updated decoding model meets a preset condition, thereby obtaining a tactile scene reconstruction model.

3. The method for obtaining a tactile scene reconstruction model according to claim 2, characterized in that: The touch-related characteristics include: spatial characteristics of pressure distribution on the target part, and EEG signal characteristics when different pressures are applied to different spatial positions of the target part.

4. The method for acquiring a tactile scene reconstruction model according to claim 2, characterized in that: The decoding model is a U-Net model, a regression model, or a fusion model of the U-Net model and the regression model.

5. The method for acquiring a tactile scene reconstruction model according to claim 4, characterized in that: The U-Net model includes an encoder, a bottleneck layer, a decoder, a skip connection path and an output layer; The step of inputting the tactile-related features into a decoding model to obtain a pressure signal of each pressure sensor channel of the target part includes: Inputting the converted tactile-related features to the encoder, so that the encoder performs a downsampling operation on the converted tactile-related features to obtain downsampled tactile-related features; wherein the converted tactile-related features are images of a preset size converted from the tactile-related features; Performing feature enhancement on the downsampled tactile-related features through the bottleneck layer to obtain enhanced tactile-related features; Processing the enhanced tactile-related features through the decoder to increase the spatial resolution of the enhanced tactile-related features to obtain upsampled tactile-related features; splicing the upsampled tactile-related features and the downsampled tactile-related features through the skip connection path to obtain a spliced ​​tactile-related feature; The output layer performs convolution processing on the stitched touch-related features to obtain the pressure signal of each pressure sensor channel of the target part.

6. The method for obtaining a tactile scene reconstruction model according to claim 4, characterized in that: The regression model is a fully connected neural network; The step of inputting the tactile-related features into a decoding model to obtain a pressure signal of each pressure sensor channel of the target part includes: The tactile-related features are input into a fully connected neural network to obtain a pressure signal of each pressure sensor channel of the target part; wherein the pressure signal of each pressure sensor channel of the target part is a predicted value of the force magnitude of each pressure sensing electrode point corresponding to the target part.

7. A tactile scene reconstruction method, characterized in that: include: Acquire a target EEG signal when the target person imagines applying pressure to the target part, and acquire a target tactile-related feature in the target EEG signal based on a feature extractor; The target tactile-related features are input into the tactile scene reconstruction method according to any one of claims 2 to 6 to obtain a tactile scene reconstruction model, so as to obtain a target force thermal map.

8. A tactile scene reconstruction model acquisition device, characterized in that: include: A tactile-related feature acquisition module, used to acquire tactile-related features in an EEG signal through a feature extractor; wherein the EEG signal is an EEG signal when pressure is applied to the target part of the subject, which is collected by the subject through the tactile decoding paradigm described in claim 1; A force thermogram acquisition module, used for inputting the tactile-related features into a decoding model to obtain a pressure signal of each pressure sensor channel of the target part, and generating a force thermogram according to the pressure signal; A decoding error acquisition module, used for comparing the stress thermogram with an actual thermogram obtained according to the actual pressure value of the target part of the subject at present, so as to obtain a decoding error; The tactile scene reconstruction model acquisition module is used to iteratively update the parameters of the decoding model according to the decoding error until the updated decoding model meets a preset condition, thereby obtaining a tactile scene reconstruction model.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the tactile scene reconstruction model acquisition method described in any one of claims 2 to 6, or implement the tactile scene reconstruction method described in claim 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed on a processor, implements the method for acquiring a tactile scene reconstruction model according to any one of claims 2 to 6, or implements the method for reconstructing a tactile scene according to claim 7.

Images