A health care training method and system based on virtual simulation technology

Abstract

The application provides a health and pension care training method and system based on virtual simulation technology, relates to the technical field of intelligent pension, and comprises the following steps: acquiring life data of the elderly, constructing a memory palace virtual environment by using virtual simulation technology according to the life data; capturing and recording first multi-modal data of the elderly in the constructed memory palace virtual environment in real time to obtain an emotional analysis result; collecting all adjusted health and pension care strategies and second multi-modal data generated by the elderly in the virtual training environment; extracting a feature vector of the third multi-modal data to obtain a comprehensive evaluation result of the emotional state of the elderly; and fusing the social behavior mode and the comprehensive evaluation result of the emotional state to customize a health and pension care training strategy. The application combines virtual simulation technology and augmented reality technology to construct a personalized memory palace virtual environment for the elderly, which is helpful to promote the improvement of the cognitive function and emotional state of the elderly.

Description

Technical Field

[0001] This invention relates to the field of smart elderly care technology, and more specifically, to a health and elderly care training method and system based on virtual simulation technology. Background Technology

[0002] In nursing homes, it is common to encounter elderly individuals experiencing cognitive decline or illness, leading to memory loss and emotional problems. These issues negatively impact their quality of life and health, and also pose challenges to the nursing home's care.

[0003] Traditional methods of elderly care are often limited to training in daily living skills and medical care, lacking personalized interventions for cognitive and emotional aspects, making it difficult to effectively improve the cognitive and emotional state of the elderly. For example, existing virtual simulation technologies often lack consideration for the individual needs of the elderly, failing to provide customized care training programs for different cognitive and emotional problems; the importance of emotional factors is often overlooked in care training, lacking effective assessment and intervention of the elderly's emotional state, resulting in poor training outcomes; furthermore, the application of virtual simulation technology in elderly care suffers from insufficient data integration, failing to fully utilize multimodal data (such as voice, facial expressions, physiological indicators, etc.) for comprehensive analysis and personalized care. Summary of the Invention

[0004] The purpose of this invention is to provide a health and elderly care training method and system based on virtual simulation technology to improve the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0005] Firstly, this application provides a health and elderly care training method based on virtual simulation technology, including:

[0006] The project aims to acquire the life data of the elderly and construct a virtual environment of a memory palace using virtual simulation technology. The life data includes personal information, living habits and health status. The virtual environment of the memory palace includes multiple rooms and scenes to simulate the daily life and memory training environment of the elderly.

[0007] The first multimodal data of the elderly in the constructed memory palace virtual environment is captured and recorded in real time. The first multimodal data is input into the multimodal attention generation network for sentiment analysis to obtain the sentiment analysis results. The first multimodal data includes the corresponding first behavior, first voice and first facial expression data of the elderly in the memory palace virtual environment.

[0008] Reinforcement learning algorithms are used to train virtual agents to perform tasks and interact with the elderly in a virtual environment. Feedback during the training and interaction process is monitored and adjusted in real time. Based on feedback and sentiment analysis results, health care strategies for the elderly are adjusted autonomously.

[0009] All adjusted health and elderly care strategies and second multimodal data generated by the elderly in a virtual training environment were collected. The second multimodal data included the corresponding second behavioral data, second speech data, and second facial expression data of the elderly in all health and elderly care strategies. Features were extracted from the second behavioral data, second speech data, and second facial expression data in the second multimodal data according to each data type to obtain second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors. The extracted second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors were then fused to obtain the second multimodal feature vector.

[0010] The third multimodal data of the elderly during music therapy is collected using sensors and devices. The feature vector of the third multimodal data is extracted, and the feature vectors of the second and third multimodal data are integrated. The integrated feature vector is then input into a preset assessment model for comprehensive assessment to obtain a comprehensive assessment result of the elderly’s emotional state.

[0011] Collect social data of the elderly, including interaction data such as friend lists, comments, and likes; identify the social behavior patterns of the elderly through machine learning algorithms; and integrate the comprehensive assessment results of social behavior patterns and emotional state to customize health care training strategies for the elderly.

[0012] Preferably, the construction of the memory palace virtual environment using virtual simulation technology based on life data includes:

[0013] The process involves obtaining personal information and health status data of the elderly, performing data analysis using association rule mining algorithms, obtaining mining results, and establishing a filtered feature set, which includes filtered features related to the health status and personal information of the elderly.

[0014] Data on the lifestyle habits of the elderly were obtained, and clustering analysis algorithms were used to divide the elderly into different lifestyle habit groups to obtain the clustering results of lifestyle habits;

[0015] The feature set and clustering results are input into virtual simulation technology, and augmented reality technology is used to construct a memory palace virtual environment, which includes a scene that generates a personalized memory palace for each elderly person.

[0016] Preferably, the real-time capture and recording of the elderly person's first multimodal data in the constructed memory palace virtual environment, and the input of the first multimodal data into a multimodal attention generation network for sentiment analysis to obtain sentiment analysis results, wherein the first multimodal data includes the corresponding first behavior, first speech, and first facial expression data of the elderly person in the memory palace virtual environment, including:

[0017] Based on real-time capture of elderly people's behavior, speech and facial expression data in the constructed memory palace virtual environment, behavioral features, speech data and facial features are extracted. Among them, the speech data feature extraction includes sound spectrum features and speech emotion features, the facial expression data performs facial action and emotion recognition, and the behavioral data performs behavior pattern recognition.

[0018] The extracted data on the first behavior, first speech, and first facial expression of the elderly in the memory palace virtual environment are fused to construct a multimodal feature vector. A multimodal attention generation network is designed and classified using the softmax function to obtain the probability distribution of the elderly’s emotional state in the virtual environment. The emotional state of the elderly in the virtual environment is then output, which is the result of the emotion analysis.

[0019] Preferably, the step of training a virtual agent using reinforcement learning algorithms to perform tasks and interact with the elderly in a virtual environment, monitoring and adjusting feedback during the training interaction process in real time, and autonomously adjusting health and elderly care strategies based on feedback and sentiment analysis results includes:

[0020] The task specifications for the virtual agent are obtained, including task definition, interaction method, environmental state space, and action space. Based on the task specifications of the virtual agent, a deep Q-network algorithm is used for training to learn the optimal action strategy. The interaction between the virtual agent and the elderly is monitored in real time to obtain feedback data during the interaction process.

[0021] Based on feedback data, trajectory analysis algorithms and behavior recognition algorithms are used to analyze and identify behavior, thereby obtaining behavioral data of the elderly in the virtual environment;

[0022] Based on behavioral data and sentiment analysis results, action strategies trained using deep Q-networks enable virtual agents to perform tasks and interact with the elderly in a virtual environment, autonomously adjusting health and elderly care strategies.

[0023] Preferably, the method of collecting third-modal data of elderly people during music therapy using sensors and devices, and extracting feature vectors from the third-modal data, includes:

[0024] Acquire third-modal data of older adults during music therapy, including audio, video, and other sensor data;

[0025] A deep learning-based sound signal processing method is used to extract the spectral features of audio data. At the same time, a deep learning model is used for facial expression recognition to extract facial expression features from video data. Sensors are used to extract physiological indicators, including heart rate and skin conductance.

[0026] The features extracted from spectral features, facial expression features, and physiological indicators are combined into a unified feature vector, which is the feature vector of the third multimodal data.

[0027] Secondly, this application also provides a health and elderly care training system based on virtual simulation technology, including:

[0028] Acquisition Module: Used to acquire the elderly’s life data, and to construct a memory palace virtual environment based on the life data using virtual simulation technology. The life data includes personal information, living habits and health status. The memory palace virtual environment includes multiple rooms and scenes to simulate the elderly’s daily life and memory training environment.

[0029] Analysis module: Used to capture and record the first multimodal data of the elderly in the constructed memory palace virtual environment in real time. The first multimodal data is input into the multimodal attention generation network for sentiment analysis to obtain the sentiment analysis results. The first multimodal data includes the corresponding first behavior, first voice and first facial expression data of the elderly in the memory palace virtual environment.

[0030] Adjustment module: Used to train virtual agents to perform tasks and interact with the elderly in a virtual environment using reinforcement learning algorithms, monitor and adjust feedback during the training interaction process in real time, and autonomously adjust health and elderly care strategies based on feedback and sentiment analysis results;

[0031] Extraction Module: This module collects all adjusted health and elderly care strategies and the second multimodal data generated by the elderly in the virtual training environment. The second multimodal data includes corresponding second behavioral data, second speech data, and second facial expression data reflected by the elderly in all health and elderly care strategies. Features are extracted from the second behavioral data, second speech data, and second facial expression data in the second multimodal data according to each data type, resulting in second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors. The extracted second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors are then fused to obtain the second multimodal feature vector.

[0032] Assessment module: Used to collect third-modal data of the elderly during music therapy using sensors and devices, extract feature vectors from the third-modal data, integrate the feature vectors of the second-modal data and the feature vectors of the third-modal data, and input the integrated feature vectors into a preset assessment model for comprehensive assessment to obtain a comprehensive assessment result of the elderly's emotional state.

[0033] Fusion Module: This module collects social data from the elderly, including interaction data such as friend lists, comments, and likes. It uses machine learning algorithms to identify the social behavior patterns of the elderly and integrates the comprehensive assessment results of social behavior patterns and emotional states to customize health care training strategies for the elderly.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention integrates multimodal data such as behavior, voice, and facial expressions of the elderly in a virtual environment, and conducts comprehensive assessment and personalized care for the elderly through a comprehensive evaluation model. This multimodal data fusion and comprehensive evaluation method can more fully understand the needs and status of the elderly, and provide them with more personalized and precise health and elderly care services.

[0036] This invention combines virtual simulation technology, artificial intelligence algorithms, and reinforcement learning algorithms to provide personalized health and elderly care training for the elderly through comprehensive detection and diagnosis. At the same time, it can make full use of the advantages of virtual simulation technology and combine multimodal data to carry out personalized cognitive and emotional intervention, thereby improving the quality of life and health of the elderly.

[0037] This invention utilizes association rule mining algorithms to analyze personal information and health status data, establishing a filtered feature set, thereby enabling personalized assessment of the health status of the elderly. This allows for the development of individualized care plans for elderly individuals with different health conditions, improving care outcomes.

[0038] This invention processes data on the lifestyle habits of the elderly through clustering analysis algorithms, dividing them into different lifestyle groups. This helps identify the lifestyles and habits of the elderly, providing objective data support for developing targeted care plans. Furthermore, it utilizes deep learning models to model the social behavior patterns and emotional states of the elderly, combining fuzzy set theory and association analysis algorithms to achieve a comprehensive assessment of these patterns. This facilitates a better understanding of the behavior and emotional states of the elderly and provides a scientific basis for developing personalized health and elderly care training strategies.

[0039] This invention combines virtual simulation technology and augmented reality technology to create a personalized memory palace virtual environment for the elderly. This virtual environment can be customized according to the needs and characteristics of each elderly person, providing a safe and comfortable training place that helps to improve the cognitive function and emotional state of the elderly.

[0040] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the health and elderly care training method based on virtual simulation technology described in this embodiment of the invention;

[0043] Figure 2 This is a schematic diagram of the health and elderly care training system based on virtual simulation technology described in an embodiment of the present invention.

[0044] In the diagram: 701, Acquisition Module; 7011, First Acquisition Unit; 7012, Clustering Unit; 7013, Construction Unit; 702, Analysis Module; 7021, Extraction and Recognition Unit; 7022, Classification Unit; 703, Adjustment Module; 7031, Training Unit; 7032, Analysis Unit; 7033, Adjustment Unit; 704, Extraction Module; 705, Evaluation Module; 7051, Second Acquisition Unit; 7052, Feature Extraction Unit; 7053, Combination Unit; 706, Fusion Module; 7061, Clustering and Recognition Unit; 7062, Fuzzy Unit; 7063, Association Unit; 7064, Design Unit. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0046] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] Example 1:

[0048] This embodiment provides a health and elderly care training method based on virtual simulation technology.

[0049] See Figure 1 The figure shows that the method includes steps S100, S200, S300, S400, S500 and S600.

[0050] S100. Obtain the elderly person's life data, and construct a memory palace virtual environment based on the life data using virtual simulation technology. The life data includes personal information, living habits and health status. The memory palace virtual environment includes multiple rooms and scenes to simulate the elderly person's daily life and memory training environment.

[0051] Understandably, in this step, the various rooms and scenes of the memory palace are designed based on the elderly’s life experiences and preferences; virtual simulation technology is used to transform the designed scenes into a three-dimensional virtual environment, and textures, lighting and sound effects are added; and an interface for the elderly to interact with the virtual environment is designed, including a controller, voice recognition and a helmet device.

[0052] It should be noted that step S100 includes S101, S102, and S103, wherein:

[0053] S101. Obtain the personal information and health status of the elderly, use the association rule mining algorithm to perform data analysis, obtain the mining results, and establish a filtered feature set, wherein the feature set includes the filtered features related to the health status and personal information of the elderly.

[0054] Understandably, the first step is to collect personal information and health data on the elderly, which may include age, gender, physical indicators, medical history, medication use, etc. This data is inherently multi-dimensional and multimodal, but directly analyzing this raw data may result in excessive dimensionality and feature redundancy; therefore, feature filtering and dimensionality reduction are necessary. The core idea of ​​association rule mining algorithms is to discover potential relationships between data by mining frequent itemsets and association rules. In this process, the algorithm automatically identifies frequently occurring combinations of features and assesses the degree of association between features by calculating their support and confidence. In this step, based on the degree of association, a series of feature combinations related to the elderly's health status and personal information are obtained. These features may cover different aspects of health information, such as lifestyle habits, dietary habits, and exercise. To further optimize the feature set, feature selection and dimensionality reduction are required. In this process, feature selection methods (such as filtering, wrapping, and embedding) and dimensionality reduction techniques (such as principal component analysis and linear discriminant analysis) can be used to eliminate redundant features and reduce the dimensionality of features, thereby extracting features that are relevant to health status and representative, and establishing a more concise and efficient feature set. This provides reliable technical support for the personalized development of subsequent health and elderly care training programs.

[0055] S102. Obtain data on the living habits of the elderly, and use cluster analysis algorithms to divide the elderly into different living habit groups to obtain the clustering results of living habits;

[0056] S103. Input the feature set and clustering results into the virtual simulation technology, and use augmented reality technology to construct a memory palace virtual environment, wherein the memory palace virtual environment includes a scene that generates a personalized memory palace for each elderly person.

[0057] Understandably, by acquiring data on the lifestyle habits of the elderly, including daily activities, sleep habits, and dietary habits, this data reflects their lifestyles and behaviors. Cluster analysis algorithms are then used to process this data, dividing the dataset into several categories. Data points within each category have high similarity, while data points between different categories have low similarity. This clustering algorithm then yields the results of the elderly's lifestyle habits, dividing them into different lifestyle groups. This step includes, but is not limited to, using k-means clustering and DBSCAN. Next, for each elderly person, based on their personalized feature set and their respective lifestyle group, a corresponding memory palace scene is designed. Augmented reality technology is used to integrate these memory palace scenes with the real world, achieving a virtual overlay in the real environment. Various elements in the memory palace scene are designed and implemented, including buildings, furniture, decorations, and special functional areas customized according to the elderly person's personal preferences and needs. Using augmented reality technology, through wearable devices, smart glasses, or mobile phones, the elderly can interact with the virtual memory palace in the real environment, experiencing memory training and daily activities.

[0058] S200: Real-time capture and recording of the first multimodal data of the elderly in the constructed memory palace virtual environment; inputting the first multimodal data into a multimodal attention generation network for sentiment analysis to obtain sentiment analysis results; wherein the first multimodal data includes the corresponding first behavior, first speech and first facial expression data of the elderly in the memory palace virtual environment.

[0059] It is understood that step S200 includes S201 and S202, wherein:

[0060] S201. Real-time capture of the elderly’s behavior, speech and facial expression data in the completed virtual environment of the memory palace is performed, and behavioral features, speech data and facial features are extracted. Among them, the feature extraction of speech data includes sound spectrum features and speech emotion features, facial expression data is used for facial action and emotion recognition, and behavioral data is used for behavior pattern recognition.

[0061] It should be noted that in step S201, firstly, feature extraction is performed on the speech data, including sound spectrum features and speech emotion features. Secondly, facial expression data undergoes facial action and emotion recognition to extract facial features. Thirdly, behavioral pattern recognition is performed on the behavioral data to extract behavioral features. Finally, the extracted speech features, facial features, and behavioral features are input into a multimodal attention generation network for sentiment analysis.

[0062] The process involves feature extraction from speech data, including spectral features and emotional features. First, the speech data undergoes preprocessing, including noise removal and audio signal standardization. Then, a Short-Time Fourier Transform (STFT) is used to convert the time-domain speech signal to the frequency-domain signal, followed by the extraction of spectral features such as Mel-frequency coefficients and Mel-frequency cepstral coefficients. Finally, machine learning or deep learning methods are employed to extract emotion-related features from the speech signal, such as emotion recognition models based on deep neural networks. The STFT converts the time-domain signal to the frequency-domain signal, and its calculation formula is as follows:

[0063]

[0064] In the formula, X(m,ω) is the spectral component with frequency ω at time m, x(n) is the speech signal in the time domain, w(nm) is the window function (usually a Hanning window or a Hamming window), and N is the length of the window.

[0065] The process involves facial expression data to perform facial action and emotion recognition and extract facial features: using a camera or depth camera to collect facial expression data of the elderly in a virtual environment; using face recognition and action detection algorithms to identify actions in facial expressions, such as smiling and frowning; and using deep learning models, such as convolutional neural networks or recurrent neural networks, to classify facial expression images for emotion and extract emotion-related facial features.

[0066] This involves behavioral pattern recognition and feature extraction from behavioral data: using sensors or cameras to collect behavioral data of the elderly in virtual environments, including walking paths and interactions with objects in the virtual environment; using machine learning or deep learning methods to perform pattern recognition and classification on the behavioral data, such as using hidden Markov models to model behavioral sequences and identify different behavioral patterns; and extracting behavioral features related to emotional states based on the behavioral pattern recognition results, such as activity frequency and regularity of walking paths.

[0067] S202. The extracted data of the first behavior, first speech and first facial expression of the elderly in the memory palace virtual environment are fused to construct a multimodal feature vector. A multimodal attention generation network is designed and classified by the softmax function to obtain the probability distribution of the emotional state of the elderly in the virtual environment. The emotional state of the elderly in the virtual environment is then output, which is the result of the emotion analysis.

[0068] It should be noted that in step S202, features such as speech, facial expressions, and behavior need to be extracted from the input multimodal data. These features may include spectral features of speech, key point coordinates of facial expressions, and temporal data of behavior. After feature extraction, an attention mechanism is used to learn the importance between different features in order to better capture emotion-related information. The attention mechanism can be calculated using the softmax function, as shown below:

[0069]

[0070] In the formula, hi is the hidden representation of the i-th feature, f(·) is a mapping function, and αi is the attention weight of the feature.

[0071] Then, the features are multiplied by the attention weights to obtain a weighted feature representation, as shown below:

[0072]

[0073] In the formula, This is the fused feature representation. The fused features are input into the sentiment classifier, and classification is performed using the softmax function to obtain the probability distribution of the emotional state of the elderly in the virtual environment. It can be understood that the generation process of the multimodal attention-based generative network includes steps such as feature extraction, attention mechanism, feature fusion, and sentiment classification. These steps can effectively perform sentiment analysis on multimodal data.

[0074] S300 uses reinforcement learning algorithms to train virtual agents to perform tasks and interact with the elderly in a virtual environment, monitors and adjusts feedback during the training interaction process in real time, and autonomously adjusts health and elderly care strategies based on feedback and sentiment analysis results.

[0075] It is understood that step S300 includes S301, S302, and S303, wherein:

[0076] S301. Obtain the task specifications of the virtual agent, which include task definition, interaction method, environmental state space and action space; according to the task specifications of the virtual agent, use the deep Q network algorithm to train it to learn the optimal action strategy, and monitor the interaction between the virtual agent and the elderly in real time to obtain feedback data during the interaction process.

[0077] S302. Based on feedback data, use trajectory analysis algorithms and behavior recognition algorithms to analyze and identify behavior, and obtain behavioral data of the elderly in the virtual environment;

[0078] S303. Based on behavioral data and sentiment analysis results, action strategies trained using deep Q-networks are used to enable virtual agents to perform tasks and interact with the elderly in a virtual environment, and to autonomously adjust health care strategies for the elderly.

[0079] It's important to note that the virtual agent's tasks in interacting with the elderly must first be defined. This might include performing daily tasks, such as assisting with memory training, providing recreational activities, and monitoring health. The task specification also needs to clarify the interaction methods between the virtual agent and the elderly, which may include voice dialogue, graphical interface interaction, and gesture recognition. Furthermore, the environmental state space and action space need to be defined. The environmental state space refers to the various states that may occur in the virtual environment, such as the elderly person's emotional state and behavioral state; the action space refers to the various actions the virtual agent can take, such as providing advice and performing specific tasks. Based on the virtual agent's task specification, a deep Q-network algorithm can be used for training to learn the optimal action strategy. DQN is a deep reinforcement learning algorithm that learns to select the optimal action in different states by training in an environment with both state and action spaces.

[0080] Understandably, the training process in this step first requires modeling the actions chosen by the virtual agent in a specific state and the feedback from the environment, constructing a state-action-reward tuple. Then, a neural network is used to approximate the Q-value function to estimate the expected reward of performing different actions in each state. Through continuous interaction with the environment, the virtual agent can update the parameters of the neural network using techniques such as experience replay, gradually learning the optimal action strategy. During real-time monitoring of the interaction between the virtual agent and the elderly, feedback data can be obtained by monitoring the actions chosen by the virtual agent in specific states and the elderly's reactions to these actions. This feedback data can include the elderly's emotional state and behavioral responses, helping to evaluate the effectiveness of the virtual agent's actions and adjust and optimize the virtual agent's action strategies based on the feedback data to provide better service and support.

[0081] S400: Collect all adjusted health and elderly care strategies and the second multimodal data generated by the elderly in the virtual training environment. The second multimodal data includes the corresponding second behavioral data, second speech data, and second facial expression data of the elderly in all health and elderly care strategies. Extract features from the second behavioral data, second speech data, and second facial expression data in the second multimodal data according to each data type to obtain the second behavioral feature vector, the second speech feature vector, and the second facial expression feature vector. Then, fuse the extracted second behavioral feature vector, second speech feature vector, and second facial expression feature vector to obtain the second multimodal feature vector.

[0082] It should be noted that the collected data includes adjusted health and elderly care strategies, which may include individualized care plans tailored to different seniors. Simultaneously, it is necessary to acquire second-modal data generated by seniors in the virtual training environment, including second-modal behavioral data, second-modal speech data, and second-modal facial expression data. This data reflects the seniors' responses to different care strategies. For second-modal behavioral data, behavioral recognition algorithms can be used, including but not limited to using Hidden Markov Models or Convolutional Neural Networks, to extract behavioral feature vectors. For second-modal speech data, techniques such as Short-Time Fourier Transform can be used to extract speech feature vectors, such as Mel-frequency cepstral coefficients. For second-modal facial expression data, deep learning models, such as Convolutional Neural Networks or Recurrent Neural Networks, can be used to extract facial expression feature vectors.

[0083] In this step, after obtaining the feature vectors of each data type, these feature vectors need to be fused to obtain the second multimodal feature vector. Various methods can be used for data fusion, such as weighted averaging and feature concatenation. By combining the second behavioral feature vector, the second speech feature vector, and the second facial expression feature vector, the resulting second multimodal feature vector can more comprehensively reflect the elderly's overall response to different care strategies, providing an important basis for further analysis and optimization of health and elderly care strategies.

[0084] S500: Collect third-modal data of elderly people during music therapy using sensors and devices, extract feature vectors from the third-modal data, integrate the feature vectors of the second-modal data and the feature vectors of the third-modal data, and input the integrated feature vectors into a preset assessment model for comprehensive assessment to obtain a comprehensive assessment result of the emotional state of the elderly people.

[0085] It is understood that step S500 includes S501, S502, and S503, wherein:

[0086] S501. Acquire third-modal data of elderly people during music therapy, wherein the third-modal data includes sound, video and other sensor data;

[0087] S502. A deep learning-based sound signal processing method is used to extract the spectral features of the audio data. At the same time, a deep learning model is used to perform facial expression recognition to extract facial expression features from the video data. Sensors are used to extract physiological indicators, including heart rate and skin conductance.

[0088] It should be noted that for audio data, deep learning-based sound signal processing methods can be used to extract spectral features. This includes using deep learning models such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to input audio data into the network for feature extraction. This process includes, but is not limited to, using techniques such as spectrograms or Mel-frequency coefficients to represent spectral features. Secondly, for extracting facial expression features from video data, this includes, but is not limited to, using deep learning models for facial expression recognition. In this step, a common method is to use convolutional neural networks (CNNs) to extract facial features. By using the face region in the video data as input, a CNN model can be trained to recognize different facial expressions and represent them as feature vectors. Finally, physiological indicators, including heart rate and electrodermal response (EDS), are extracted using sensors. Using devices such as heart rate sensors and EDS sensors, the physiological indicators of the elderly can be monitored in real time. In this step, it is understood that heart rate data can be directly obtained through a heart rate sensor, while EDS can be indirectly obtained by measuring the skin's conductivity using an EDS sensor. These physiological indicators can provide important information about the physical condition of the elderly.

[0089] S503. Combine the features extracted from the spectral features, facial expression features, and physiological indicators into a unified feature vector, that is, extract the feature vector of the third multimodal data.

[0090] It should be noted that after acquiring the spectral features of the audio data, facial expression features from the video data, and physiological indicators extracted by sensors, these are combined into a unified feature vector. This can be achieved by concatenating the individual feature vectors to form a higher-dimensional feature vector, or by using feature fusion methods such as weighted summation or principal component analysis. Such a feature vector can more comprehensively reflect the multimodal data information of the elderly, providing a more comprehensive information foundation for subsequent analysis and applications. This involves weighted summation of these feature vectors. Assume the given weights are w... audio w facial w physiological Then the weighted summed eigenvector X can be expressed as:

[0091] X = w audio ·X audio +w facial ·X facial +w physiological ·X physiological

[0092] Next, Principal Component Analysis (PCA) can be used to reduce the dimensionality of the features. This involves first centering the eigenvectors, then calculating the covariance matrix and performing eigenvalue decomposition, selecting the eigenvectors corresponding to the top k eigenvalues ​​as the projection matrix. Finally, the original eigenvectors are multiplied by the projection matrix to obtain the dimensionality-reduced eigenvectors. In summary, the spectral features of audio data, facial expression features from video data, and physiological indicators extracted by sensors are combined into a unified feature vector, while also reducing dimensionality for subsequent processing and analysis.

[0093] Step S600: Collect social data of the elderly, including interaction data such as friend lists, comments and likes; identify the social behavior patterns of the elderly through machine learning algorithms, and integrate the comprehensive assessment results of social behavior patterns and emotional state to customize health care training strategies for the elderly.

[0094] It is understood that step S600 includes S601, S602, S603, and S604, wherein:

[0095] S601. Convert social data into matrix form. Based on the set neighborhood radius and minimum number of neighbors, divide the data points into core points, boundary points and noise points. Form clusters by connecting the density reachability relationships between core points until all data points are accessed, and identify different social behavior patterns.

[0096] S602. Use a deep learning model for sequence modeling to capture different social behavior patterns, including positive interaction, neutral interaction and negative interaction; divide the comprehensive evaluation results of social behavior patterns and emotional states into fuzzy sets to obtain fuzzy social behavior pattern data and fuzzy emotional state data.

[0097] It should be noted that social data (which can be time-series records of social behavior, including various social interactions at different points in time, such as verbal communication and body posture) is organized in the form of sequences, with each sequence representing a social behavior record over a period of time. The prepared social behavior data is serialized to facilitate input into a deep learning model for processing. Typically, each social behavior can be mapped to a feature vector, and the entire sequence can be represented as a sequence with one time step. A suitable deep learning model is then selected for sequence modeling. Therefore, in this step, a Recurrent Neural Network (RNN) or a Long Short-Term Memory (LSTM) network can be chosen because they can effectively process sequence data and capture the temporal relationships between sequences. The sequenced social behavior data is input into the deep learning model for training. During training, the model learns how to extract features from the sequence data and capture the temporal relationships between different social behaviors. Backpropagation and gradient descent can be used to optimize the model to adapt it to the characteristics of elderly people's social behavior data. Finally, after training, the model needs to be evaluated to ensure its performance on social behavior data. This includes, but is not limited to, using evaluation metrics such as accuracy and recall to assess the model's performance, and using methods such as cross-validation to verify the model's generalization ability. After training and evaluation, the deep learning model can be used for sequence modeling of elderly people's social behavior data. The model can be used to predict future social behavior trends, analyze the relationships between different social behavior patterns, and identify abnormal behaviors. These models can capture the temporal relationships between social behaviors and effectively identify and classify different types of social behavior patterns.

[0098] Understandably, dividing the comprehensive evaluation results of social behavior patterns and emotional states into fuzzy sets is a way to address the uncertainty and ambiguity of the evaluation results. By fuzzifying the comprehensive evaluation results, the complexity of emotional states and social behavior patterns can be better handled, and a more reliable data foundation can be provided for subsequent correlation analysis.

[0099] S603. Using association analysis algorithms, perform association analysis on fuzzy social behavior pattern data and fuzzy emotional state data to obtain association analysis results, extract association rules, and integrate and fuse them to generate the final comprehensive evaluation results.

[0100] It's important to note that before conducting association analysis, the fuzzy social behavior pattern data and emotional state data need to be integrated, and relevant features extracted. Then, association analysis algorithms, such as the Apriori algorithm or FP-Growth algorithm, can be used to perform association analysis on this data to discover the relationships between different social behavior patterns and emotional states. By extracting association rules and integrating and fusing them, a final comprehensive evaluation result can be generated, leading to a more comprehensive understanding of the social behavior and emotional state of older adults. Association rules are discovered by mining frequent itemsets in the dataset. Frequent itemsets are sets of items that appear frequently in the dataset, and association rules are patterns mined based on these frequent itemsets. By calculating the support and confidence of frequent itemsets, meaningful association rules can be selected, revealing the correlations and dependencies in the dataset. In this step, association rules refer to a data mining technique used to describe the relationships between different items in the social behavior data of older adults.

[0101] S604. Based on the final comprehensive evaluation results after integration, develop personalized health and elderly care training strategies, and provide corresponding measures and services for different social behavior patterns and emotional states of the elderly.

[0102] It should be noted that, based on the final comprehensive assessment results after integration, personalized health and elderly care training strategies are developed. These strategies can provide corresponding measures and services according to the different social behavior patterns and emotional states of the elderly, in order to meet their individual needs, promote their health and well-being, better provide effective elderly care services, and improve their quality of life and happiness.

[0103] Example 2:

[0104] like Figure 2 As shown, this embodiment provides a health and elderly care training system based on virtual simulation technology. See [link to documentation]. Figure 2 The system includes:

[0105] Acquisition Module 701: Used to acquire the elderly’s life data and construct a memory palace virtual environment based on the life data using virtual simulation technology. The life data includes personal information, living habits and health status. The memory palace virtual environment includes multiple rooms and scenes to simulate the elderly’s daily life and memory training environment.

[0106] Analysis module 702: used to capture and record the first multimodal data of the elderly in the constructed memory palace virtual environment in real time, input the first multimodal data into the multimodal attention generation network for sentiment analysis, and obtain sentiment analysis results. The first multimodal data includes the corresponding first behavior, first voice and first facial expression data of the elderly in the memory palace virtual environment.

[0107] Adjustment module 703: Used to train a virtual agent to perform tasks and interact with the elderly in a virtual environment using reinforcement learning algorithms, monitor and adjust feedback during the training interaction process in real time, and autonomously adjust health care strategies based on feedback and sentiment analysis results.

[0108] Extraction module 704: Used to collect all adjusted health and elderly care strategies and the second multimodal data generated by the elderly in the virtual training environment. The second multimodal data includes the corresponding second behavioral data, second voice data and second facial expression data of the elderly in all health and elderly care strategies. Features are extracted from the second behavioral data, second voice data and second facial expression data in the second multimodal data according to each data type to obtain the second behavioral feature vector, the second voice feature vector and the second facial expression feature vector. The extracted second behavioral feature vector, second voice feature vector and second facial expression feature vector are then fused to obtain the second multimodal feature vector.

[0109] Assessment module 705: Used to collect third multimodal data of the elderly during music therapy using sensors and devices, extract feature vectors of the third multimodal data, integrate the feature vectors of the second and third multimodal data, input the integrated feature vectors into a preset assessment model for comprehensive assessment, and obtain a comprehensive assessment result of the elderly's emotional state.

[0110] Fusion Module 706: Used to collect social data of the elderly, including interaction data such as friend lists, comments, and likes; it identifies the social behavior patterns of the elderly through machine learning algorithms, and integrates the comprehensive assessment results of social behavior patterns and emotional states to customize health care training strategies for the elderly.

[0111] Specifically, the acquisition module 701 includes:

[0112] First acquisition unit 7011: used to acquire personal information and health status of the elderly, use association rule mining algorithm to perform data analysis, obtain mining results, and establish a filtered feature set, wherein the feature set includes filtered features related to the health status and personal information of the elderly.

[0113] Clustering Unit 7012: Used to obtain data on the living habits of the elderly, and to divide the elderly into different groups based on their living habits using a clustering analysis algorithm, thereby obtaining the clustering results of their living habits;

[0114] Construction unit 7013: used to input feature sets and clustering results into virtual simulation technology, and use augmented reality technology to construct a memory palace virtual environment, wherein the memory palace virtual environment includes a scene that generates a personalized memory palace for each elderly person.

[0115] Specifically, the analysis module 702 includes:

[0116] Extraction and recognition unit 7021: It is used to capture real-time behavior, speech and facial expression data of the elderly in the completed memory palace virtual environment, and extract behavioral features, speech data and facial features. Among them, the feature extraction of speech data includes sound spectrum features and speech emotion features, the facial expression data performs facial action and emotion recognition, and the behavior data performs behavior pattern recognition.

[0117] Classification unit 7022: This unit is used to fuse the extracted data of the first behavior, first speech and first facial expression of the elderly in the memory palace virtual environment, construct a multimodal feature vector, design a multimodal attention generation network, classify the data through the softmax function, obtain the probability distribution of the elderly’s emotional state in the virtual environment, and output the emotional state of the elderly in the virtual environment, thus obtaining the emotion analysis result.

[0118] Specifically, the adjustment module 703 includes:

[0119] Training unit 7031: Used to acquire the task specifications of the virtual agent, which include task definition, interaction method, environmental state space and action space; based on the task specifications of the virtual agent, a deep Q-network algorithm is used for training to learn the optimal action strategy, and the interaction between the virtual agent and the elderly is monitored in real time to obtain feedback data during the interaction process.

[0120] Analysis unit 7032: Used to analyze and identify behavior based on feedback data using trajectory analysis algorithms and behavior recognition algorithms to obtain behavioral data of the elderly in the virtual environment;

[0121] Adjustment unit 7033: Based on behavioral data and sentiment analysis results, it uses action strategies trained by a deep Q-network to enable the virtual agent to perform tasks and interact with the elderly in a virtual environment, and autonomously adjust health care strategies for the elderly.

[0122] Specifically, the evaluation module 705 includes:

[0123] The second acquisition unit 7051 is used to acquire third multimodal data of the elderly during music therapy, wherein the third multimodal data includes sound, video and other sensor data;

[0124] Feature extraction unit 7052: used to extract spectral features of audio data using a deep learning-based sound signal processing method, and to perform facial expression recognition using a deep learning model to extract facial expression features from video data, and to extract physiological indicators using sensors, including heart rate and skin conductance response.

[0125] Combination unit 7053: used to combine the features extracted from spectral features, facial expression features and physiological indicators into a unified feature vector, that is, to extract the feature vector of the third multimodal data.

[0126] Specifically, the fusion module 706 includes:

[0127] Clustering identification unit 7061: It is used to convert social data into matrix form, and divide the data points into core points, boundary points and noise points based on the set neighborhood radius and minimum number of neighbors. Clustering is formed by connecting the density reachability relationship between core points until all data points are visited, and different social behavior patterns are identified.

[0128] Fuzzy Unit 7062: Used for sequence modeling using deep learning models to capture different social behavior patterns, including positive interaction, neutral interaction and negative interaction; the comprehensive evaluation results of social behavior patterns and emotional states are divided into fuzzy sets to obtain fuzzy social behavior pattern data and fuzzy emotional state data.

[0129] Association Unit 7063: Used to perform association analysis on fuzzy social behavior pattern data and fuzzy emotional state data using association analysis algorithms, obtain association analysis relationship results, extract association rules, and integrate and fuse them to generate the final comprehensive evaluation result;

[0130] Unit 7064: Used to develop personalized health and elderly care training strategies based on the final comprehensive assessment results after integration, and to provide corresponding measures and services for different social behavior patterns and emotional states of the elderly.

[0131] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0133] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A health and elderly care training method based on virtual simulation technology, characterized in that, include: The project aims to acquire the life data of the elderly and construct a virtual environment of a memory palace using virtual simulation technology. The life data includes personal information, living habits and health status. The virtual environment of the memory palace includes multiple rooms and scenes to simulate the daily life and memory training environment of the elderly. The first multimodal data of the elderly in the constructed memory palace virtual environment is captured and recorded in real time. The first multimodal data is input into the multimodal attention generation network for sentiment analysis to obtain the sentiment analysis results. The first multimodal data includes the corresponding first behavior, first voice and first facial expression data of the elderly in the memory palace virtual environment. Reinforcement learning algorithms are used to train virtual agents to perform tasks and interact with the elderly in a virtual environment. Feedback during the training and interaction process is monitored and adjusted in real time. Based on feedback and sentiment analysis results, health care strategies for the elderly are adjusted autonomously. All adjusted health and elderly care strategies and second multimodal data generated by the elderly in a virtual training environment were collected. The second multimodal data included the corresponding second behavioral data, second speech data, and second facial expression data of the elderly in all health and elderly care strategies. Features were extracted from the second behavioral data, second speech data, and second facial expression data in the second multimodal data according to each data type to obtain second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors. The extracted second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors were then fused to obtain the second multimodal feature vector. The third multimodal data of the elderly during music therapy is collected using sensors and devices. The feature vector of the third multimodal data is extracted, and the feature vectors of the second and third multimodal data are integrated. The integrated feature vector is then input into a preset assessment model for comprehensive assessment to obtain a comprehensive assessment result of the elderly’s emotional state. Collect social data of the elderly, including interaction data such as friend lists, comments, and likes; By identifying the social behavior patterns of the elderly through machine learning algorithms, and integrating the comprehensive assessment results of social behavior patterns and emotional states, training strategies for healthy aging care can be customized.

2. The health and elderly care training method based on virtual simulation technology according to claim 1, characterized in that, The construction of a memory palace virtual environment based on real-life data using virtual simulation technology includes: The process involves obtaining personal information and health status data of the elderly, performing data analysis using association rule mining algorithms, obtaining mining results, and establishing a filtered feature set, which includes filtered features related to the health status and personal information of the elderly. Data on the lifestyle habits of the elderly were obtained, and clustering analysis algorithms were used to divide the elderly into different lifestyle habit groups to obtain the clustering results of lifestyle habits; The feature set and clustering results are input into virtual simulation technology, and augmented reality technology is used to construct a memory palace virtual environment, which includes a scene that generates a personalized memory palace for each elderly person.

3. The health and elderly care training method based on virtual simulation technology according to claim 1, characterized in that, The system captures and records the first multimodal data of the elderly in the constructed memory palace virtual environment in real time. This first multimodal data is then input into a multimodal attention generation network for sentiment analysis to obtain the sentiment analysis results. The first multimodal data includes the corresponding first behavior, first speech, and first facial expression data of the elderly in the memory palace virtual environment, including: Based on real-time capture of elderly people's behavior, speech and facial expression data in the constructed memory palace virtual environment, behavioral features, speech data and facial features are extracted. Among them, the speech data feature extraction includes sound spectrum features and speech emotion features, the facial expression data performs facial action and emotion recognition, and the behavioral data performs behavior pattern recognition. The extracted data on the first behavior, first speech, and first facial expression of the elderly in the memory palace virtual environment are fused to construct a multimodal feature vector. A multimodal attention generation network is designed and classified using the softmax function to obtain the probability distribution of the elderly’s emotional state in the virtual environment. The emotional state of the elderly in the virtual environment is then output, which is the result of the emotion analysis.

4. The health and elderly care training method based on virtual simulation technology according to claim 1, characterized in that, The method involves training a virtual agent using reinforcement learning algorithms to perform tasks and interact with the elderly in a virtual environment, monitoring and adjusting feedback during the training interaction process in real time, and autonomously adjusting health and elderly care strategies based on feedback and sentiment analysis results. This includes: The task specifications for the virtual agent are obtained, including task definition, interaction method, environmental state space, and action space. Based on the task specifications of the virtual agent, a deep Q-network algorithm is used for training to learn the optimal action strategy. The interaction between the virtual agent and the elderly is monitored in real time to obtain feedback data during the interaction process. Based on feedback data, trajectory analysis algorithms and behavior recognition algorithms are used to analyze and identify behavior, thereby obtaining behavioral data of the elderly in the virtual environment; Based on behavioral data and sentiment analysis results, action strategies trained using deep Q-networks enable virtual agents to perform tasks and interact with the elderly in a virtual environment, autonomously adjusting health and elderly care strategies.

5. The health and elderly care training method based on virtual simulation technology according to claim 1, characterized in that, The method utilizes sensors and devices to collect third-modal data of elderly individuals during music therapy, and extracts feature vectors from the third-modal data, including: Acquire third-modal data of older adults during music therapy, including audio, video, and other sensor data; A deep learning-based sound signal processing method is used to extract the spectral features of audio data. At the same time, a deep learning model is used for facial expression recognition to extract facial expression features from video data. Sensors are used to extract physiological indicators, including heart rate and skin conductance. The features extracted from spectral features, facial expression features, and physiological indicators are combined into a unified feature vector, which is the feature vector of the third multimodal data.

6. The health and elderly care training method based on virtual simulation technology according to claim 1, characterized in that, The method involves identifying the social behavior patterns of the elderly through machine learning algorithms, integrating the comprehensive assessment results of social behavior patterns and emotional states, and thereby customizing health care and elderly care training strategies, including: Social data is converted into a matrix form. Based on a set neighborhood radius and minimum number of neighbors, data points are divided into core points, boundary points, and noise points. Clustering is formed by connecting the density reachability relationships between core points until all data points are accessed, thereby identifying different social behavior patterns. A deep learning model is used for sequence modeling to capture different social behavior patterns, including positive interaction, neutral interaction, and negative interaction. The comprehensive evaluation results of social behavior patterns and emotional states are divided into fuzzy sets to obtain fuzzy social behavior pattern data and fuzzy emotional state data. Using association analysis algorithms, we perform association analysis on fuzzy social behavior pattern data and fuzzy emotional state data to obtain association analysis results, extract association rules, and integrate and fuse them to generate the final comprehensive evaluation results. The development unit is used to develop personalized health and elderly care training strategies based on the final comprehensive assessment results after integration, and to provide corresponding measures and services for different social behavior patterns and emotional states of the elderly.

7. A health and elderly care training system based on virtual simulation technology, based on the health and elderly care training method based on virtual simulation technology as described in claim 1, characterized in that, include: Acquisition Module: Used to acquire the elderly’s life data, and to construct a memory palace virtual environment based on the life data using virtual simulation technology. The life data includes personal information, living habits and health status. The memory palace virtual environment includes multiple rooms and scenes to simulate the elderly’s daily life and memory training environment. Analysis module: Used to capture and record the first multimodal data of the elderly in the constructed memory palace virtual environment in real time. The first multimodal data is input into the multimodal attention generation network for sentiment analysis to obtain the sentiment analysis results. The first multimodal data includes the corresponding first behavior, first voice and first facial expression data of the elderly in the memory palace virtual environment. Adjustment module: Used to train virtual agents to perform tasks and interact with the elderly in a virtual environment using reinforcement learning algorithms, monitor and adjust feedback during the training interaction process in real time, and autonomously adjust health and elderly care strategies based on feedback and sentiment analysis results; Extraction Module: This module collects all adjusted health and elderly care strategies and the second multimodal data generated by the elderly in the virtual training environment. The second multimodal data includes corresponding second behavioral data, second speech data, and second facial expression data reflected by the elderly in all health and elderly care strategies. Features are extracted from the second behavioral data, second speech data, and second facial expression data in the second multimodal data according to each data type, resulting in second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors. The extracted second behavioral feature vectors, second speech feature vectors, and second facial expression feature vectors are then fused to obtain the second multimodal feature vector. Assessment module: Used to collect third-modal data of the elderly during music therapy using sensors and devices, extract feature vectors from the third-modal data, integrate the feature vectors of the second-modal data and the feature vectors of the third-modal data, and input the integrated feature vectors into a preset assessment model for comprehensive assessment to obtain a comprehensive assessment result of the elderly's emotional state. Fusion module: Used to collect social data of the elderly, including interaction data such as friend lists, comments, and likes; By identifying the social behavior patterns of the elderly through machine learning algorithms, and integrating the comprehensive assessment results of social behavior patterns and emotional states, training strategies for healthy aging care can be customized.

8. The health and elderly care training system based on virtual simulation technology according to claim 7, characterized in that, The acquisition module includes: The first acquisition unit is used to acquire the personal information and health status of the elderly, conduct data analysis using association rule mining algorithms, obtain mining results, and establish a filtered feature set, which includes the filtered features related to the health status and personal information of the elderly. Clustering unit: Used to obtain data on the living habits of the elderly, and to divide the elderly into different groups based on their living habits using clustering analysis algorithms, thereby obtaining clustering results of their living habits; Construction Unit: Used to input feature sets and clustering results into virtual simulation technology, and to construct a memory palace virtual environment using augmented reality technology. The memory palace virtual environment includes a scene that generates a personalized memory palace for each elderly person.

9. The health and elderly care training system based on virtual simulation technology according to claim 7, characterized in that, The analysis module includes: Extraction and recognition unit: used to capture real-time behavior, speech and facial expression data of the elderly in the constructed memory palace virtual environment, extract behavioral features, speech data and facial features, including sound spectrum features and speech emotion features for speech data, facial movement and emotion recognition for facial expression data, and behavior pattern recognition for behavioral data. The classification unit is used to fuse the extracted data of the first behavior, first speech and first facial expression of the elderly in the memory palace virtual environment, construct a multimodal feature vector, design a multimodal attention generation network, classify through the softmax function, obtain the probability distribution of the elderly’s emotional state in the virtual environment, and output the emotional state of the elderly in the virtual environment, that is, obtain the emotion analysis result.

10. The health and elderly care training system based on virtual simulation technology according to claim 7, characterized in that, The adjustment module includes: Training unit: Used to acquire the task specifications of the virtual agent, which include task definition, interaction method, environmental state space and action space; based on the task specifications of the virtual agent, a deep Q-network algorithm is used for training to learn the optimal action strategy, and the interaction between the virtual agent and the elderly is monitored in real time to obtain feedback data during the interaction process. Analysis Unit: Used to analyze and identify behavior based on feedback data using trajectory analysis algorithms and behavior recognition algorithms, to obtain behavioral data of the elderly in the virtual environment; Adjustment Unit: Based on behavioral data and sentiment analysis results, this unit utilizes action strategies trained using a deep Q-network to enable the virtual agent to perform tasks and interact with the elderly in a virtual environment, autonomously adjusting health and elderly care strategies.

Images