Upper limb rehabilitation training method based on emotion feedback, storage medium and product

Abstract

The invention discloses an upper limb rehabilitation training method based on emotion feedback, a storage medium and a product, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a face image of a user, carrying out the feature extraction and resolution recovery of the face image, and generating a face segmentation region; color signals and physiological features are extracted in a motion artifact removal mode; inputting the personal information, the behavior features and the physiological features of the user into the initial personalized emotion recognition model for fusion, and performing regression prediction on a fusion result to obtain a prediction result; acquiring a user state, and performing excitation adjustment on the user state according to a closed-loop strategy of self-adaptive adjustment of a preset task to obtain a training result; obtaining a keyword and a data set according to the prediction result and the training result, and inputting the keyword and the data set into the initial large model for fine tuning to obtain an optimized large model; and assisting the user in upper limb rehabilitation training through the optimized large model. The psychological state of the user is considered, and the rehabilitation effect is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an upper limb rehabilitation training method, storage medium and product based on emotional feedback. Background Art

[0002] Rehabilitation training is an important means of restoring function. Its core lies in the ability of the nervous system to self-repair and reorganize according to changes in the internal and external environment to help the target users regain lost motor skills. With the rise of artificial intelligence technology and the rapid development of smart medicine, the widespread application of rehabilitation robots has provided new impetus and hope for the rehabilitation of target users. In the process of robot-assisted rehabilitation training, enhancing the user's sense of participation is a key factor in promoting neural plasticity and accelerating the rehabilitation process, which is of great significance for improving the rehabilitation effect.

[0003] Currently, current research is insufficient in terms of quantitative evaluation and real-time feedback of user engagement, especially the psychological state of the user during the rehabilitation process is not taken into account, resulting in poor rehabilitation results.

[0004] Therefore, how to improve the user's rehabilitation effect is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide an upper limb rehabilitation training method, storage medium and product based on emotional feedback, aiming to solve the technical problem of poor rehabilitation effect of users.

[0006] To achieve the above objectives, the present application proposes an upper limb rehabilitation training method based on emotional feedback, the method comprising:

[0007] Acquire a facial image of a user, perform feature extraction and resolution restoration on the facial image, and generate a facial segmentation region;

[0008] Based on the facial segmentation area, extracting color signals and physiological features by removing motion artifacts;

[0009] Inputting the user's personal information, behavioral characteristics, and the physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion result to obtain a prediction result;

[0010] Acquire the user state, and perform incentive adjustment on the user state according to a closed-loop strategy of adaptive adjustment of preset tasks to obtain a training result;

[0011] Acquire keywords and data sets according to the prediction results and the training results, and input the keywords and data sets into the initial large model for fine-tuning to obtain an optimized large model;

[0012] The optimized large model is used to assist users in upper limb rehabilitation training.

[0013] In one embodiment, the step of obtaining a facial image of a user, performing feature extraction and resolution restoration on the facial image, and generating a facial segmentation region includes:

[0014] Obtaining local details and resolution of feature maps through encoders and decoders in neural networks;

[0015] Downsampling the local details and the resolution based on a neural convolutional network module to obtain global information;

[0016] The global information is processed according to multiple spatial compression modules to generate facial segmentation regions.

[0017] In one embodiment, the step of extracting color signals and physiological features based on the facial segmentation region by removing motion artifacts further includes:

[0018] Extracting color signals from the facial segmentation area according to the segmentation network, and denoising the color signals based on a three-axis cascade adaptive denoising algorithm to obtain denoised signals;

[0019] Based on the denoised signal, physiological features are extracted through iterative forward and backward dynamic programming and adaptive compensation.

[0020] In one embodiment, the step of inputting the user personal information, the behavioral characteristics, and the physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion result to obtain the prediction result includes:

[0021] Inputting the user's personal information, behavioral characteristics and the physiological characteristics into an initial personalized emotion recognition model, and obtaining a fusion result through feature encoding and feature fusion;

[0022] The fusion result is subjected to regression prediction based on a feedforward neural network to obtain a prediction result.

[0023] In one embodiment, the step of inputting the user personal information, behavioral characteristics and physiological characteristics into the initial personalized emotion recognition model and obtaining a fusion result through feature encoding and feature fusion includes:

[0024] By fitting recurrence graph, Markov transition field and Gramian angle field, the 1D time series feature sequence is converted into a 2D image;

[0025] Based on the 2D image, respectively encode the physiological feature and the behavioral feature through a residual feature encoder to obtain encoded physiological features and behavioral features;

[0026] The user's personal information and the encoded physiological and behavioral characteristics are input into a corresponding feedforward neural network, and the output results of the feedforward neural network are subjected to feature fusion through a symmetrical cross-attention mechanism to obtain a fusion result.

[0027] In one embodiment, the step of performing feature fusion on the output results of the feedforward neural network through a symmetric cross attention mechanism to obtain a fusion result includes:

[0028] Perform linear transformation on the output results of the feedforward neural network to obtain image features and data features;

[0029] Inputting the image features and data features into different multi-head cross attention networks respectively to obtain different fusion matrices;

[0030] The initial features and the different fusion matrices are residually connected to obtain a fusion result.

[0031] In one embodiment, the step of obtaining the user state, and performing incentive adjustment on the user state according to a closed-loop strategy of adaptive adjustment of preset tasks to obtain a training result includes:

[0032] Acquiring the user status according to the user personal information, behavioral characteristics and physiological characteristics;

[0033] Evaluate the user's real-time task performance through the task difficulty index to obtain the evaluation result;

[0034] Calculate an incentive function according to the evaluation result and the corresponding user status;

[0035] Based on the incentive function, task parameters are dynamically adjusted through reinforcement learning to obtain training results, and the adjusted task parameters are used to adaptively optimize task difficulty.

[0036] In one embodiment, the step of obtaining keywords and data sets according to the prediction results and the training results, and inputting the keywords and data sets into the initial large model for fine-tuning to obtain an optimized large model includes:

[0037] Obtaining prompt words according to the user status and the user's real-time task performance;

[0038] Inputting the prompt words into the initial large model for model training to obtain an intermediate large model;

[0039] A dialogue dataset is obtained, and the intermediate large model is fine-tuned by quantized low-rank adaptation to obtain an optimized large model.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the upper limb rehabilitation training method based on emotional feedback as described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the upper limb rehabilitation training method based on emotional feedback as described above.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] The user's facial image is obtained, and the facial image is feature extracted and the resolution is restored to generate a facial segmentation area; based on the facial segmentation area, the color signal and physiological features are extracted by removing motion artifacts; the user's personal information, behavioral characteristics and physiological characteristics are input into the initial personalized emotion recognition model for fusion, and the fusion results are regressed and predicted to obtain the prediction results; the user's state is obtained, and the user's state is stimulated and adjusted according to the closed-loop strategy of adaptive adjustment of the preset task to obtain the training results; the keywords and data sets are obtained according to the prediction results and training results, and the keywords and data sets are input into the initial large model for fine-tuning to obtain the optimized large model; the optimized large model is used to assist the user in upper limb rehabilitation training. By extracting features and restoring resolution of facial images, the maximum effective area of ​​the face can be accurately extracted, ensuring edge details and global consistency. Physiological and behavioral signals are feature encoded and fused with personal information, while quantifying concentration, assisting in the prediction task of emotion, enhancing the model's ability to understand emotions, and ensuring the robustness of the prediction of individual differences. By inputting the user's personal information, behavioral characteristics, and physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion results, the user's state is stimulated and adjusted according to the closed-loop strategy of adaptive adjustment of preset tasks. It can learn and self-optimize based on the user's real-time feedback, and build a "human in the loop" dual feedback closed-loop system. According to the user's emotional and behavioral responses, the task difficulty is adjusted intelligently and dynamically, and the training results are combined with the large model, which can improve the user's rehabilitation effect according to the user's real-time psychological state and task performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 This is a schematic diagram of the upper limb rehabilitation hardware system for this application;

[0047] Figure 2 This is a flow chart of the first embodiment of the upper limb rehabilitation training method based on emotional feedback of the present application;

[0048] Figure 3 This is the overall module framework diagram of the system of the embodiment of the present application;

[0049] Figure 4 This is a framework diagram of the robust detection algorithm of contactless pulse wave (rPPG) in the rehabilitation scenario of this application;

[0050] Figure 5 Schematic diagram of synchronizing ACC signals for this application;

[0051] Figure 6 This is the framework diagram of the facial region segmentation algorithm for this application;

[0052] Figure 7 This is a framework diagram of the motion artifact removal algorithm based on motion compensation in this application;

[0053] Figure 8 This is a flow chart of the second embodiment of the upper limb rehabilitation training method based on emotional feedback of the present application;

[0054] Fig. 9 This is the residual-based feature encoder network structure diagram of this application;

[0055] Fig.10 This is a schematic diagram of the entire personalized emotion recognition model for this application;

[0056] Fig.11 This is a partial diagram of the entire personalized emotion recognition model schematic of this application;

[0057] Fig.12 This is a flow chart of the third embodiment of the upper limb rehabilitation training method based on emotional feedback of the present application;

[0058] Fig.13 This is a schematic diagram of the intrinsic motivation algorithm of this application;

[0059] Fig.14 This is a schematic diagram of the module for adaptively adjusting the difficulty of tasks in this application.

[0060] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0062] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0063] In the process of robot-assisted rehabilitation training, enhancing the user's sense of engagement is a key factor in promoting neural plasticity and accelerating the rehabilitation process, which is of great significance for improving the rehabilitation effect. Since current research has obvious deficiencies in quantitative evaluation and real-time feedback of patient engagement, there are problems such as signal acquisition being easily disturbed by limb movement, bulky wearable devices affecting user comfort, and expensive equipment. In addition, in the process of traditional rehabilitation training, personalized adjustment of the difficulty of training tasks is the key, and the timely introduction of intelligent voice is also of great significance to improving the treatment effect. The current research interaction module does not take into account the user's psychological state during the training process. Sometimes, the user may seem to perform well during the training process, but in fact may feel nervous or anxious, which may affect the continuity of the training and lead to poor rehabilitation results.

[0064] In view of the above problems, the present application provides a solution, which is to obtain the user's facial image, extract features and restore the resolution of the facial image, and generate a facial segmentation area; based on the facial segmentation area, extract color signals and physiological features by removing motion artifacts; input the user's personal information, behavioral characteristics and physiological characteristics into the initial personalized emotion recognition model for fusion, and perform regression prediction on the fusion results to obtain the prediction results; obtain the user's state, and stimulate and adjust the user's state according to the closed-loop strategy of adaptive adjustment of the preset task to obtain the training results; obtain keywords and data sets according to the prediction results and training results, and input the keywords and data sets into the initial large model for fine-tuning to obtain the optimized large model; assist the user in upper limb rehabilitation training through the optimized large model. By extracting features and restoring the resolution of the facial image, the maximum effective area of ​​the face can be accurately extracted, ensuring edge details and global consistency. Physiological and behavioral signals are feature encoded and fused with personal information, while quantifying the concentration, assisting in the prediction task of emotion, enhancing the model's ability to understand emotion, and ensuring the robustness of the prediction of individual differences. By inputting the user's personal information, behavioral characteristics, and physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion results, the user's state is stimulated and adjusted according to the closed-loop strategy of adaptive adjustment of preset tasks. It can learn and self-optimize based on the user's real-time feedback, and build a "human in the loop" dual feedback closed-loop system. According to the user's emotional and behavioral responses, the task difficulty is adjusted intelligently and dynamically, and the training results are combined with the large model. According to the user's real-time psychological state and task performance, the user's rehabilitation effect is greatly improved.

[0065] It should be noted that the method provided in this application can be applied to the upper limb rehabilitation intelligent adjustment method and the upper limb rehabilitation hardware system, which can specifically include: an end-type rehabilitation robot, augmented reality (AR) glasses and a camera. For example, Figure 1 As shown, the AR glasses may be Hololens2 AR glasses. Hololens2AR glasses may have built-in scene cameras for three-dimensional eye tracking. The cameras may be fixed in front of the user. The terminal rehabilitation robot includes force sensors to assist the user in rehabilitation training. The user may wear the AR glasses and perform line drawing tasks with the assistance of the terminal robot. The AR glasses present an immersive 3D painting scene that is superimposed and integrated with the desktop in the real scene. The built-in scene cameras of the glasses track the hand position, and the cameras capture the user's facial image.

[0066] Based on this, the embodiment of the present application provides an upper limb rehabilitation training method based on emotional feedback, referring to Figure 2 , Figure 2This is a flow chart of the first embodiment of the upper limb rehabilitation training method based on emotional feedback of the present application.

[0067] In this embodiment, the upper limb rehabilitation training method based on emotional feedback includes steps S10 to S60:

[0068] Step S10, obtaining a facial image of the user, performing feature extraction and resolution restoration on the facial image, and generating a facial segmentation region.

[0069] It should be noted that the facial segmentation area can be understood as dividing the facial image into different functional areas to achieve accurate analysis of different facial areas. For example, the user's face after being blocked (after wearing AR glasses) can be captured in real time by a camera fixed in front of the user to obtain the user's facial image. A lightweight facial segmentation network can be used to extract facial features and restore the resolution of the user's facial image while ensuring real-time performance to generate facial segmentation areas.

[0070] Step S20, based on the facial segmentation area, extracting color signals and physiological features by removing motion artifacts.

[0071] It should be noted that motion artifacts are the phenomenon that images are blurred or distorted due to user movement. It is understandable that during rehabilitation training, body movement can cause head shaking, so motion artifact removal is used to overcome the interference caused by movement. Exemplarily, the method of removing motion artifacts may include optical flow analysis or deblurring algorithms. Physiological characteristics may include heart rate, respiratory rate, blood circulation, etc., and color signals may include color signals in the user's facial image, such as a reddish skin signal. It is understandable that physiological characteristics can also be analyzed by analyzing facial color signals, such as judging the user's blood oxygen saturation, heart rate, etc. by changes in skin color.

[0072] Step S30, inputting the user's personal information, behavioral characteristics and physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion result to obtain a prediction result.

[0073] It should be noted that user personal information may include information such as user gender, age, and five-dimensional personality traits. Behavioral characteristics may be understood as the user's behavior (such as movements or gestures) during rehabilitation training. Physiological characteristics may include at least one of heart rate, heart rate variability, and respiratory rate. The initial personalized emotion recognition model may be understood as a machine learning or deep learning model that can predict the user's emotional state based on the user's physiological characteristics, behavioral characteristics, and personal information. It is understandable that the prediction result is the user's emotional state obtained after the prediction.

[0074] Step S40, obtaining the user state, and performing incentive adjustment on the user state according to the closed-loop strategy of adaptive adjustment of the preset task to obtain the training result.

[0075] It should be noted that the user status can include the current user's personal information and the user's current behavioral and physiological characteristics, etc. In addition, the user status can also include the user's historical status data. The preset task can be understood as the line drawing task in the upper limb rehabilitation hardware system, and the closed-loop strategy can be understood as constantly monitoring the user's status and dynamically adjusting the task difficulty according to the monitoring results. Incentive regulation can adopt the form of internal and external dual incentives: internal emotional states such as concentration as internal incentives, and external task performance as external incentives.

[0076] Step S50, obtaining keywords and data sets according to the prediction results and the training results, and inputting the keywords and data sets into the initial large model for fine-tuning to obtain an optimized large model.

[0077] Exemplarily, the initial big model can be the PengCheng-Mind 7B big model. The keywords obtained according to the prediction results and training results can be used as prompt words for the big model. The data set can include a mental health dialogue data set and a doctor-patient dialogue data set. The prompt words and the data set are input into the PengCheng-Mind 7B big model for fine-tuning to obtain an optimized big model.

[0078] Step S60, assisting the user in upper limb rehabilitation training by using the optimized large model.

[0079] For example, the optimized large model can be applied to an upper limb rehabilitation hardware system to assist users in upper limb rehabilitation training.

[0080] In this embodiment, by extracting features and restoring resolution of facial images, the maximum effective area of ​​the face can be accurately extracted, ensuring edge details and global consistency. Physiological and behavioral signals are feature encoded and fused with personal information. While quantifying concentration, it assists in the prediction task of emotion, enhances the model's ability to understand emotion, and ensures the robustness of the prediction of individual differences. By inputting user personal information, behavioral characteristics, and physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion results, the user state is stimulated and adjusted according to the closed-loop strategy of adaptive adjustment of preset tasks, and learning and self-optimization can be performed based on the user's real-time feedback, thus building a "man in the loop" dual feedback closed-loop system. According to the user's emotional and behavioral responses, the task difficulty is intelligently and dynamically adjusted, and the training results are combined with the large model, which can improve the user's rehabilitation effect according to the user's real-time psychological state and task performance.

[0081] In order to more clearly illustrate the above embodiment, the present application provides a system overall module framework diagram, such as Figure 3 As shown in the figure, the system mainly consists of two closed-loop circuits: adaptive adjustment of task difficulty and emotional speech interaction based on a large model. Both closed-loop feedback mechanisms are based on multimodal emotion perception and task performance evaluation.

[0082] Based on the above multimodal signal measurement and emotion perception module, it should be noted that contactless pulse wave (remote Photoplethysmography, rPPG) measurement in rehabilitation scenarios faces two major challenges: (1) overcoming facial occlusion (such as AR glasses) and selecting the largest effective skin area; (2) overcoming motion interference (such as head shaking caused by body movement). Based on the above problems, this application proposes an implementation method, exemplarily, as Figure 4 As shown in the figure, real-time face detection uses the segmentation network to extract the largest effective area of ​​the face, that is, ROI detection, and then extracts the three-channel color signal S from it, removes the baseline drift and denoises to obtain Then, based on the complementarity between different color channel signals, the rPPG signal is constructed using the classic CHROM algorithm, and a preliminary rPPG signal p is obtained from the three-channel color signal. f Finally, combined with the ACC signal synchronously read from the AR glasses IMU (such as Figure 5 ) Remove motion artifact noise and obtain denoised signal And extract features such as heart rate, heart beat variability, respiratory rate, etc. In view of the above implementation, the present application specifically proposes the following implementation.

[0083] In one implementation, based on the above-mentioned first embodiment, step S10 includes: obtaining local details and resolution of feature maps through an encoder and a decoder in a neural network; downsampling local details and resolution based on a neural convolutional network module to obtain global information; processing the global information according to multiple spatial compression modules to generate a facial segmentation area.

[0084] For example, Figure 6As shown in the figure, the core of the neural network can include an encoder-decoder structure: the decoder extracts semantic features from the image, and the decoder is responsible for restoring local details and the resolution of the feature map. The encoder uses a squeeze-excitation block (SE block), a depthwise separable convolution (DS conv) module, etc. to downsample the feature map, and then uses multiple spatial squeeze modules (S2 modules) to process global information. The spatial compression module uses a split-merge structure to cover multiple receptive fields, and uses point-by-point convolution, channel shuffling and other structures to accelerate the calculation, which reduces the amount of calculation while ensuring the consistency of global information. In order to better restore the detail information with the decoder, an information blocking module can be introduced to filter the high-confidence area, that is, the part that the encoder has successfully segmented, so that the decoder can focus on restoring the features of the low-confidence area. Finally, a general bilinear upsampling or transposed convolution upsampling module is used to restore the high-resolution feature map, achieving fast and accurate facial effective area segmentation.

[0085] In this embodiment, the encoder and decoder structure can help extract and retain important local details in the image, extract high-level semantic information of the image by downsampling, and the decoder restores this information to an appropriate resolution, which helps to balance details with global information and avoid image information loss. The spatial compression module can reduce the spatial dimensions (width and height) of the feature map, and enable the network to process global information more efficiently by removing redundant information and emphasizing key features. Through downsampling and spatial compression, the amount of pixels and calculations that need to be processed are reduced, reducing the complexity of the model.

[0086] In one embodiment, based on the above-mentioned first embodiment, step S20 includes: extracting color signals from the facial segmentation area according to the segmentation network, denoising the color signals based on a three-axis cascade adaptive denoising algorithm to obtain denoised signals; based on the denoised signals, extracting physiological features through iterative forward and backward dynamic programming and adaptive compensation.

[0087] It should be noted that if Figure 7 As shown, the original rPPG signal p f Can be represented as a signal containing heart rate information and the motion artifact (MA) signal u f First, we use the energy-based frame screening scheme to determine whether the current frame contains a large number of MAs. The noise energy E of the fth frame is f It can be expressed as formula (1).

[0088]

[0089] In formula (1), x f (n), y f (n), z f (n) represents the three-axis component of the accelerometer (ACC). If the detected noise energy exceeds the threshold E T , then perform MA elimination. The motion artifact removal task is performed on the three-axis components in turn by using three cascade modules, in which the classic adaptive minimum mean square filter (LMS Filter) is used. In the denoising process, the algorithm will be based on the input signal p f (n) and the reference signal u f (n) Dynamically adjust the weight until convergence and recovery The modal component of the ACC signal, which is only related to motion artifacts but not to the heart rate signal, is used as the reference signal for LMS.

[0090] It should be noted that the ACC data is not used directly as the reference function here, but the variational model decomposition (VMD) or empirical mode decomposition (EMD) method is used to represent the uniaxial ACC signal as a linear combination of multiple intrinsic mode functions, each intrinsic mode function component has a specific center frequency, which corresponds to a separate LMS filter as a reference function. Since different MA modal components can be used to adjust the step size, filter order and other parameters of the individual LMS, the module has more degrees of freedom and can effectively denoise even when there is a high degree of frequency domain overlap between the MA and the heart rate signal.

[0091] It should be noted that since the signal-to-noise ratio (SNR) of the reconstructed rPPG signal is not high, how to accurately extract physiological characteristics related to cardiopulmonary activity (such as HR, HRV, RR, etc.) from rPPG is also a major challenge. In order to improve the accuracy of HR extracted from low signal-to-noise ratio signals, this embodiment proposes to adopt an accurate and efficient adaptive multi-trace carving (AMTC) method, which can track multiple frequency components in the noise signal through iterative forward and backward dynamic programming and adaptive compensation based on the frequency spectrogram. Without a lot of training, a high-temporal-resolution HR signal can be obtained.

[0092] Exemplarily, after rPPG denoising, peak detection is performed, and a narrow-bandpass infinite impulse response IIR filter is used to perform zero-phase filtering on the signal, and the bandwidth is determined according to the range of heart rate. It should be noted that due to the low frame rate (30Hz) of the video, the accuracy of the peak position is limited. Therefore, by performing secondary interpolation around the peak to refine the peak position, a more accurate heartbeat interval IBI sequence can be obtained. Due to the uneven sampling of this sequence, the Lomb-Scargle periodogram spectrum analysis tool can be used to calculate the corresponding power spectrum density PSD, and the relevant features of HRV time domain and frequency domain can be obtained: time domain related features include heartbeat interval standard deviation (SDNN), average heartbeat interval standard deviation (SDANN), root mean square rMSSD of the difference between adjacent heartbeat intervals, heartbeat interval ratio greater than 50ms pNN50, etc.; frequency domain related features include total power (TP), ultra-low frequency power (ULF), very low frequency power (VLF), low frequency power (LF), high frequency power (HF), LF / HF, etc. Since one of the sources of HR fluctuation is the respiratory sinus rate related to the respiratory cycle, the HRV spectrum is used to estimate the respiratory frequency RR. The frequency with the maximum energy response within the respiratory experience range (5-30 BrPM) can be selected to obtain RR.

[0093] In this embodiment, the segmentation network can accurately extract the facial area from the image and focus on the color signal of the area, which helps to focus on the key parts related to the physiological signal. The three-axis cascade adaptive denoising algorithm effectively reduces these noises through multiple stages of denoising processing, improving the clarity and accuracy of the signal. The iterative forward and backward dynamic programming algorithms can continuously optimize the analysis of the color signal during the processing process, and extract more accurate physiological features through multiple iterations in time or space. The adaptive compensation technology can dynamically adjust to factors such as environmental changes and user movements to improve the accuracy of physiological feature extraction. By compensating for signal deviations in real time, the reliability of the extraction results is further enhanced.

[0094] It should be noted that in view of the problems that the existing rehabilitation system does not have the ability to recognize personalized continuous emotions, has a single modality and weak generalization, this application proposes a new personalized emotion recognition model that integrates physiological behavioral signals and personal information, and adopts a multi-task learning approach to perform three-dimensional emotion (arousal, valence, dominance) prediction tasks while estimating concentration to assist learning, improve the model's understanding of emotions, and make it robust to individual differences. Figure 8 , Figure 8 This is a flow chart of the second embodiment of the upper limb rehabilitation training method based on emotional feedback of this application. Figure 2 The first embodiment shown proposes a second embodiment of the upper limb rehabilitation training method based on emotional feedback of the present application.

[0095] In the second embodiment, the step S30 includes:

[0096] Step S301, inputting user personal information, behavioral characteristics and physiological characteristics into an initial personalized emotion recognition model, and obtaining a fusion result through feature encoding and feature fusion.

[0097] It should be noted that feature encoding can be understood as converting raw, different types of data (such as text, images, audio signals, etc.) into a numerical form that can be input into a machine learning model. Feature fusion can be used to integrate features from different sources or dimensions to generate a unified feature vector. Personal information, behavioral features, and physiological features can be combined so that the model can consider information from multiple dimensions at the same time, thereby improving the accuracy of sentiment prediction. Fusion methods can include simple splicing, weighted summation, or the use of more complex learning models for feature integration.

[0098] Step S302: Perform regression prediction on the fusion result based on a feedforward neural network to obtain a prediction result.

[0099] It should be noted that the feedforward neural network can be a basic neural network structure, in which information is passed from the input layer through a series of hidden layers and finally to the output layer. It is understandable that in a feedforward neural network, the flow of data from input to output is unidirectional. Exemplarily, the feedforward neural network can be a multi-layer perceptron (Multi-Layer Perceptron, MLP), which is composed of multiple neurons (neurons are also called "nodes"), which are organized into layers, usually including at least an input layer, a hidden layer and an output layer, which can process the results obtained by feature encoding and fusion, and predict the user's emotional state based on the input fusion features through regression prediction.

[0100] In this embodiment, by inputting user personal information, behavioral characteristics and physiological characteristics, the emotion recognition model can integrate information from multiple sources, so that emotion prediction does not only rely on universal rules, but is optimized according to individualized behavior and physiological patterns, thereby providing more personalized emotional feedback. Feature encoding converts different data sources (such as text, behavior, images, sensor data, etc.) into a unified numerical form, so that the model can effectively process these heterogeneous data. Through feature fusion, the model can comprehensively consider multiple factors such as user behavior, psychology, and physiology, reduce the deviation that may be caused by a single feature, and improve the robustness and accuracy of emotion recognition. Through feedforward neural networks, emotion recognition does not only rely on simple rules or linear models, but can self-adjust according to the complex features of the input to obtain more accurate emotion predictions.

[0101] In one embodiment, based on the above-mentioned second embodiment, the step S301 includes: converting the 1D time series feature sequence into a 2D image by fitting a recursive graph, a Markov transition field and a Gramian angular field; based on the 2D image, encoding the physiological characteristics and the behavioral characteristics respectively through a residual feature encoder to obtain encoded physiological characteristics and behavioral characteristics; inputting the user personal information and the encoded physiological characteristics and behavioral characteristics into the corresponding feedforward neural network, and performing feature fusion on the output results of the feedforward neural network through a symmetric cross attention mechanism to obtain a fusion result.

[0102] It should be noted that the 1D time series feature sequence is converted into a 2D image through the recurrence plot (RP), Markov transition field (MTF) and Gramian angular field (GAF). RP can visualize the 1D non-stationary time series signal into 2D to present its repetitive pattern and structure; MTF is created by dividing the time series data into different states and then calculating the probability of transitioning from one state to another. The constructed 2D image contains the dynamic information of the state transition in the time series data; GAF converts the 1D time series into a polar coordinate system and then uses trigonometric functions to produce a 2D matrix. It has the advantage of maintaining the correlation between time series. The three grayscale images obtained by these three methods are fused into a color image in the RGB color space, corresponding to the RGB three-channel color data respectively. Exemplarily, a residual based feature encoder (RBFE) is used to encode physiological features and behavioral features respectively. The encoder includes multiple residual blocks (such as Fig. 9 As shown in the figure, the correlation information between different signals can be fully extracted. The encoded physiological and behavioral characteristics are fused and input into the corresponding MLP network. Personal data includes information such as gender, age, and five-dimensional personality characteristics, which are encoded by the soft label encoder and input into the corresponding MLP network.

[0103] Among them, the step of performing feature fusion on the output results of the feedforward neural network through a symmetric cross-attention mechanism to obtain a fusion result includes: linearly transforming the output results of the feedforward neural network to obtain image features and data features; inputting the image features and data features into different multi-head cross-attention networks respectively to obtain different fusion matrices; and performing residual connection between the initial features and different fusion matrices to obtain a fusion result.

[0104] For example, the output of the previous MLP layer is first transformed into query (q), key (k) and value (v) through linear transformation, and the transformation process is expressed as formula (2).

[0105]

[0106] A symmetric structure can be established, where q from image features in formula (2) i and k from personal data characteristics m and v m Input multi-head cross attention module, and q from data features m and k from image features i and v i Input another multi-head cross attention module. k and q generate an attention score map through the scaled dot product, and then convert it into a probability distribution through softmax, expressed as formula (3).

[0107]

[0108] In formula (3), Softmax can be a function that converts a vector into a probability distribution, that is, the value of each element is between 0 and 1, and the sum of all elements is 1. Fusedmatrix h×1×d It is represented as the final fusion matrix with the size of h×1×d, where h represents the number of attention heads and d represents the feature dimension. h The query vector representing the h attention heads can come from image features; k n Represented as the key vector of the nth attention head, v n represents the value vector of the nth attention head, k n and v n Both can come from personal data features. The dimension of the key vector, d, is used to scale the dot product to prevent the dot product result from being too large, causing the gradient to disappear or explode. Formula (3) describes how to combine image features and personal data features through the attention mechanism. First, the scaled dot product of the query vector and the key vector is calculated to obtain the attention score, that is, the query vector q h and key vector k n The transpose of (k nT ), and then divided by We then scale these scores to obtain attention scores. These scores are then converted to probability distributions using the softmax function. Finally, these probabilities are combined with the value vector v n Multiply them together to get the fused feature representation, namely Fusedmatrix h×1×d The output features are fused in a cascade manner, expressed as formula (4).

[0109] Fusedmatrix 1×d =Concat(head1,...,head n)W 0 (4)

[0110] In formula (4), W 0 represents the weight matrix, head1,...,head n Describe the output features. Perform residual connections on the two fusion matrices obtained from different symmetric cross attention mechanisms and the initial features, and output the final fusion result, which is expressed as formula (5).

[0111]

[0112] In formula (5), I is the encoded physiological and behavioral characteristics, M is the encoded personal information characteristics, and formula (5) represents the variable after I is added to the fusion characteristics, and M is connected to the variable after the fusion characteristics are added and multiplied by the weight matrix W. 0 Finally, four specialized MLP layers are used to perform regression prediction on the four tasks (attention, valence, arousal, and dominance). These four tasks share the feature encoding layer and feature fusion layer. The entire personalized emotion recognition model is as follows Fig.10 To avoid unclear issues, Fig.11 , as Fig.10 A partial diagram of .

[0113] In this embodiment, the 1D time series feature sequence is converted into a 2D image, which provides a richer representation method for the model input, and the spatial relationship features can be extracted while maintaining the time series relationship. RP can visualize the 1D non-stationary time series signal as 2D, presenting its repetitive pattern and structure. The 2D image constructed by MTF contains the dynamic information of the state transition in the time series data. GAF has the advantage of maintaining the correlation between time series. By integrating the advantages of the three methods, the loss of time information is minimized. The use of a residual-based feature encoder can fully extract the correlation information between different signals. A symmetric cross-attention mechanism is used in the feature fusion stage to fuse image data and metadata features. When using two different multi-head cross-attention blocks to help minimize the information loss of the two types of features and enhance their interaction, this mechanism can integrate them in a fairer way than simple cascading or element-by-element summation.

[0114] It should be noted that in order to keep users focused during the training process, this application proposes a closed-loop strategy for adaptive difficulty adjustment based on emotional feedback. The rehabilitation scenario is set as the augmented reality-based line drawing rehabilitation task proposed in the previous work, requiring users to complete the task with the assistance of the planar robot end or based on the preset task difficulty. The task difficulty can be adjusted in two ways: (1) by applying Perlin mechanical noise disturbances of different amplitudes at the end of the machine to interfere with the movement (the noise amplitude index σnoise ); (2) Adjust the complexity of the line drawing graphics (image complexity η LDJ ). Based on this, refer to Fig.12 , Fig.12 This is a flow chart of the third embodiment of the upper limb rehabilitation training method based on emotional feedback of this application. Figure 8 The second embodiment shown proposes the third embodiment of the upper limb rehabilitation training method based on emotional feedback of the present application.

[0115] In the third embodiment, the step S40 includes:

[0116] Step S401, obtaining the user status according to the user's personal information, behavioral characteristics and physiological characteristics.

[0117] For example, the user state is a description of the current emotional, psychological or physiological state calculated based on the user's personal information, behavioral characteristics and physiological characteristics. The user state can reflect the user's current adaptation or load level, providing a basis for task difficulty adjustment and motivation.

[0118] Step S402, evaluating the user's real-time task performance through the task difficulty index to obtain an evaluation result.

[0119] It should be noted that the task difficulty index can be used to quantify the parameter of the challenge or complexity of the task for the user. For example, the task difficulty index can be defined as the noise amplitude index σ noise With image complexity η LDJ The linear combination of is expressed as formula (6).

[0120] difficulty=α n σ noise +α j η LDJ (6)

[0121] The user's real-time task performance is evaluated according to the difficulty index to obtain an evaluation result. For example, a query table can be constructed to map the difficulty value difficulty to a two-dimensional game difficulty parameter. The mapping does not have a one-to-one correspondence and increases the randomness of the task.

[0122] Step S403: Calculate the incentive function according to the evaluation result and the corresponding user status.

[0123] It should be noted that the incentive function adopts the form of internal and external dual incentives, with internal emotional states such as concentration as internal incentives and external task performance as external incentives. Exemplarily, the incentive function can be expressed as formula (7).

[0124]

[0125] In formula (7), the total incentive r can be expressed as the intrinsic incentive With extrinsic motivation The convex combination of , where the weighting coefficient λ provides the trade-off between emotional feedback (intrinsic motivation) and task performance (extrinsic motivation). Intrinsic motivation in round n Depends on the concentration e(n) and arousal a(n) output by the emotion model, It is expressed as formula (8):

[0126]

[0127] In formula (8), is the emotional reward value obtained by the model algorithm after the end of the nth round of training. If the concentration e(n) of the nth round is higher than the concentration e(n-1) of the n-1th round, and the difference is greater than the threshold c e , while ensuring that the awakening degree a(n) is within a certain range [t a -c a , t a +c a ], such as Fig.13 As shown, a reward value of 1 is given. If the concentration e(n) of the nth round is lower than the concentration e(n-1) of the n-1th round, and the difference is less than the threshold value - c e , while the arousal level is in the interval [t a -c a , t a +c a ], a reward value of -1 is given. If none of the above is true, a reward value of 0 is given. It can be understood that the goal expressed by formula (8) can maximize the degree of concentration and keep the arousal at a moderate level. A decrease in concentration and a low arousal level indicate boredom, which is usually caused by a low actual difficulty; a decrease in concentration and a high arousal level indicate anxiety, which is usually caused by a high actual difficulty. External incentives for the nth round It depends on the total score p(n) of the task performance in this round, which is determined by five aspects: speed, accuracy, and smoothness, and the value range is [0, 1]. It is expressed as formula (9):

[0128]

[0129] In formula (9), is the task performance reward value obtained by the model algorithm after the nth round of training. p(n) is the performance score of the nth round. If p(n) is within a certain interval [t p -c p , t p +c p], a reward value of 1 is given; if p(n) is higher than t p +c p , then the reward value (1-|p(n)-t p -c p |) 2 , if p(n) is lower than t p -c p , then the reward value (1-|p(n)-t p +c p |) 2 It can be understood that the goal expressed in formula (9) can keep the sports performance within the target level range. We use proximal policy optimization (PPO) as the learning strategy, which is applicable to continuous action-state space and can avoid excessive updates by limiting the update range of the strategy, thereby achieving efficient and stable learning.

[0130] Step S404: Based on the incentive function, dynamically adjust the task parameters through reinforcement learning to obtain training results.

[0131] It should be noted that the task parameters are adjusted to adaptively optimize the task difficulty. Fig.14 As shown in the figure, the value of the incentive function can be used to evaluate whether the difficulty of the user's current task is appropriate. The reinforcement learning agent selects the optimal action (adjusts the task parameters) through a strategy based on the current state and incentive value. For example, if the incentive value is low (the user is in a poor state and performs poorly), the agent reduces the difficulty of the task (reduces the complexity of the task or extends the completion time); if the incentive value is high (the user is in a good state and performs well), the agent increases the challenge of the task (shortens the time limit or increases the accuracy requirement). After adjusting the task parameters, the system observes the user's state and performance changes, and feeds them back to the reinforcement learning agent as new input to continuously optimize the strategy.

[0132] In this embodiment, based on the incentive function, the task parameters are dynamically adjusted through reinforcement learning. On the one hand, the system can respond to emotions more intelligently, and on the other hand, more continuous and intensive incentives can be obtained, thereby avoiding problems such as low learning efficiency caused by sparse incentives. The reinforcement learning agent gradually enhances its decision-making ability through training, enabling the system to quickly respond to changes in user status and make reasonable adjustments, achieving accurate evaluation of user status, dynamic evaluation of task performance, and adaptive optimization of task parameters, and ultimately continuously improving the intelligence of task design and user experience through reinforcement learning.

[0133] Based on the third embodiment above, a fourth embodiment of the upper limb rehabilitation training method based on emotional feedback of the present application is proposed. In the fourth embodiment, the step S50 includes: obtaining a prompt word according to the user state and the user's real-time task performance; inputting the prompt word into the initial large model for model training to obtain an intermediate large model; obtaining a dialogue data set, and fine-tuning the intermediate large model through quantized low-rank adaptation to obtain an optimized large model.

[0134] Exemplarily, the user's psychological state (concentration, valence, arousal, dominance) and ability level predicted in real time by the emotion detection module and the task performance evaluation module can be made into prompt words and input into the PengCheng-Mind 7B model. These prompt words can indicate that the big model adopts an empathetic personality in the conversation and follows the empathy chain reasoning in the conversation to understand the content and difficulties encountered by the user. The big model can also be instructed to follow the principles of various psychotherapy techniques, such as cognitive behavioral therapy (CBT) and active listening (Active Listening), to enhance the empathy of the chatbot. The prompt words can be input into the initial big model for model preliminary to obtain an intermediate big model. Exemplarily, the mental health dialogue data set and the doctor-patient dialogue data set can be used and the quantized low-rank adaptation (QLoRA) technology can be used to optimize the 7B brain sea big model. Among them, the QLoRA technology can normalize the weights of the model to a 4-bit quantized format, which can reduce the memory requirements and the delay time of the conversation. The fine-tuned large model is designed with prompts, taking the predicted quantified user emotions and task performance scores as input, so that the large model can communicate with users in accordance with the principles of CBT, just like a well-trained psychologist and therapist, and provide reasonable explanations when it cannot answer questions.

[0135] In this implementation, prompt words are generated through the user's psychological state (such as concentration, valence, arousal, dominance) and task performance, and the user's emotional and ability information is passed to the large model, which can enhance the personalization and context relevance of the conversation content and improve the user experience. Fine-tuning the intermediate large model by quantizing low-rank adaptation can significantly reduce memory requirements and conversation latency, while making the model easy to expand to new tasks and fields, reducing development and deployment costs.

[0136] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the upper limb rehabilitation training method based on emotional feedback of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0137] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the upper limb rehabilitation training method based on emotional feedback in the above-mentioned embodiment.

[0138] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0139] The computer-readable storage medium may be included in the upper limb rehabilitation training device based on emotional feedback; or may exist independently without being assembled into the upper limb rehabilitation training device based on emotional feedback.

[0140] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the upper limb rehabilitation training device based on emotional feedback, the upper limb rehabilitation training device based on emotional feedback: obtains the user's facial image, and performs feature extraction and resolution restoration on the facial image to generate a facial segmentation area; based on the facial segmentation area, extracts color signals and physiological characteristics by removing motion artifacts; inputs the user's personal information, behavioral characteristics and the physiological characteristics into the initial personalized emotion recognition model for fusion, and performs regression prediction on the fusion result to obtain a prediction result; obtains the user state, and stimulates and adjusts the user state according to a closed-loop strategy of adaptive adjustment of preset tasks to obtain a training result; obtains keywords and data sets based on the prediction result and the training result, and inputs the keywords and data sets into the initial large model for fine-tuning to obtain an optimized large model; assists the user in upper limb rehabilitation training through the optimized large model.

[0141] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0144] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned upper limb rehabilitation training method based on emotional feedback, and can solve the technical problem of poor rehabilitation effect of users. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the upper limb rehabilitation training method based on emotional feedback provided by the above-mentioned embodiment, and will not be repeated here.

[0145] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the upper limb rehabilitation training method based on emotional feedback as described above.

[0146] The computer program product provided by the present application can solve the technical problem of poor rehabilitation effect of users. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the upper limb rehabilitation training method based on emotional feedback provided by the above embodiment, which will not be repeated here.

[0147] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An upper limb rehabilitation training method based on emotional feedback, characterized in that: The method includes: Acquire a facial image of a user, perform feature extraction and resolution restoration on the facial image, and generate a facial segmentation region; Based on the facial segmentation area, extracting color signals and physiological features by removing motion artifacts; Inputting the user's personal information, behavioral characteristics, and the physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion result to obtain a prediction result; Acquire the user state, and perform incentive adjustment on the user state according to a closed-loop strategy of adaptive adjustment of preset tasks to obtain a training result; Acquire keywords and data sets according to the prediction results and the training results, and input the keywords and data sets into the initial large model for fine-tuning to obtain an optimized large model; The optimized large model is used to assist users in upper limb rehabilitation training.

2. The method according to claim 1, characterized in that The step of obtaining a facial image of a user, extracting features and restoring resolution of the facial image, and generating a facial segmentation region comprises: Obtaining local details and resolution of feature maps through encoders and decoders in neural networks; Downsampling the local details and the resolution based on a neural convolutional network module to obtain global information; The global information is processed according to multiple spatial compression modules to generate facial segmentation regions.

3. The method according to claim 1, characterized in that The step of extracting color signals and physiological features based on the facial segmentation area by removing motion artifacts also includes: Extracting color signals from the facial segmentation area according to the segmentation network, and denoising the color signals based on a three-axis cascade adaptive denoising algorithm to obtain denoised signals; Based on the denoised signal, physiological features are extracted through iterative forward and backward dynamic programming and adaptive compensation.

4. The method according to claim 1, characterized in that The step of inputting the user personal information, behavioral characteristics and physiological characteristics into the initial personalized emotion recognition model for fusion, and performing regression prediction on the fusion result to obtain the prediction result includes: Inputting the user's personal information, behavioral characteristics and the physiological characteristics into an initial personalized emotion recognition model, and obtaining a fusion result through feature encoding and feature fusion; The fusion result is subjected to regression prediction based on a feedforward neural network to obtain a prediction result.

5. The method according to claim 4, characterized in that The step of inputting the user's personal information, behavioral characteristics and physiological characteristics into the initial personalized emotion recognition model and obtaining a fusion result through feature encoding and feature fusion includes: By fitting recurrence graph, Markov transition field and Gramian angle field, the 1D time series feature sequence is converted into a 2D image; Based on the 2D image, respectively encode the physiological feature and the behavioral feature through a residual feature encoder to obtain encoded physiological features and behavioral features; The user's personal information and the encoded physiological and behavioral characteristics are input into a corresponding feedforward neural network, and the output results of the feedforward neural network are subjected to feature fusion through a symmetrical cross-attention mechanism to obtain a fusion result.

6. The method according to claim 5, characterized in that The step of performing feature fusion on the output results of the feedforward neural network through a symmetrical cross attention mechanism to obtain a fusion result comprises: Perform linear transformation on the output results of the feedforward neural network to obtain image features and data features; Inputting the image features and data features into different multi-head cross attention networks respectively to obtain different fusion matrices; The initial features and the different fusion matrices are residually connected to obtain a fusion result.

7. The method according to any one of claims 1 to 6, characterized in that The step of obtaining the user state, and performing incentive adjustment on the user state according to a closed-loop strategy of adaptive adjustment of preset tasks to obtain a training result includes: Acquiring the user status according to the user personal information, behavioral characteristics and physiological characteristics; Evaluate the user's real-time task performance through the task difficulty index to obtain the evaluation result; Calculate an incentive function according to the evaluation result and the corresponding user status; Based on the incentive function, task parameters are dynamically adjusted through reinforcement learning to obtain training results, and the adjusted task parameters are used to adaptively optimize task difficulty.

8. The method according to claim 7, characterized in that The step of obtaining keywords and data sets according to the prediction results and the training results, and inputting the keywords and data sets into the initial large model for fine-tuning to obtain an optimized large model includes: Obtaining prompt words according to the user status and the user's real-time task performance; Inputting the prompt words into the initial large model for model training to obtain an intermediate large model; A dialogue dataset is obtained, and the intermediate large model is fine-tuned by quantized low-rank adaptation to obtain an optimized large model.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the intelligent adjustment method for upper limb rehabilitation based on multimodal emotional feedback as described in any one of claims 1 to 8 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the steps of the intelligent adjustment method for upper limb rehabilitation based on multimodal emotional feedback as described in any one of claims 1 to 8.

Images