Cooperative intervention system and method based on multi-mode emotion perception and five-tone therapy

Abstract

The invention relates to the technical field of wisdom medical treatment, in particular to a cooperative intervention system and method based on multi-modal emotion perception and five-tone therapy, and realizes precision of emotion recognition and individuation of intervention strategies through cooperative intervention of multi-modal emotion perception and traditional Chinese medicine five-tone therapy. Physiological and behavior signals of a user in different states are comprehensively captured, emotion recognition is carried out in combination with a deep learning model, the accuracy and real-time performance of emotion recognition are remarkably improved, the system can intelligently match music tracks most suitable for the current emotion state based on the five-internal-organ-five-sound theory of traditional Chinese medicine and modern music psychology, and the user experience is improved. And an intervention strategy is continuously optimized through reinforcement learning, so that more targeted and effective emotion regulation is realized, and the method is particularly suitable for long-term management and intervention of negative emotions such as anxiety and depression.

Description

Technical Field

[0001] This invention relates to the field of smart healthcare technology, and in particular to a collaborative intervention system and method based on multimodal emotion perception and five-tone therapy. Background Technology

[0002] With the increasing aging population, the number of elderly cancer patients is rising year by year. Cancer treatment is lengthy and has significant side effects, often accompanied by various psychological and physical symptoms such as anxiety, depression, pain, and insomnia, severely impacting their quality of life and treatment adherence. Simultaneously, the primary caregivers (often spouses or children) also experience immense psychological stress during long-term care, making them prone to emotional breakdowns, sleep disorders, and depression. Research indicates a significant emotional contagion effect between patients and their families; the emotional state of family members directly influences the patient's mental state and treatment outcomes, creating a negative cycle. Therefore, psychological interventions targeting only the patient are insufficient to achieve holistic improvement within the family system; a collaborative intervention system that simultaneously addresses the psychological states of both patients and their families, achieving two-way regulation, is urgently needed.

[0003] However, existing technical solutions have the following significant limitations:

[0004] First, the intervention targets are singular. Most patents or products focus only on the patient's emotional regulation, neglecting the psychological state of family members, resulting in a one-sided intervention effect and difficulty in achieving synergistic improvement within the family system. Second, the means of emotion perception are limited. Some systems rely on single physiological signals (such as heart rate or skin conductance), which cannot comprehensively and accurately reflect an individual's emotional state, resulting in limited accuracy. Third, the intervention methods are static. Traditional Chinese medicine's five-tone therapy lacks a dynamic adjustment mechanism during the digitization process, making it impossible to provide personalized music recommendations based on real-time emotional states, resulting in low compatibility and poor intervention effects.

[0005] In summary, existing technologies have significant shortcomings in terms of intervention targets, emotion perception, intervention methods, and collaborative mechanisms. There is an urgent need for a new system that can integrate multimodal emotion perception, artificial intelligence decision-making, traditional Chinese medicine five-tone therapy, and family collaborative intervention to achieve synchronous monitoring, personalized intervention, and two-way regulation of patients' and their families' emotions, thereby improving overall mental health and quality of life. Summary of the Invention

[0006] To achieve the above objectives, this invention proposes a collaborative intervention system and method based on multimodal emotion perception and five-tone therapy, including a multimodal perception module, an emotion recognition and state assessment module, an intelligent decision-making module, a music intervention execution module, a feedback and optimization module, and a user interaction and management module;

[0007] The multimodal perception module is responsible for collecting multimodal physiological and behavioral signals from patients and their families in real time, serving as the basic data source for emotion recognition;

[0008] The emotion recognition and state assessment module uses deep learning algorithms to recognize emotional states based on the physiological and behavioral data provided by the multimodal perception module.

[0009] The intelligent decision-making module formulates personalized music intervention strategies based on emotion recognition results and the five-tone theory of traditional Chinese medicine.

[0010] The music intervention execution module is responsible for pushing the music content selected by the decision-making module to the user in an appropriate manner to execute personalized intervention;

[0011] The feedback and optimization module is used to monitor the intervention effect in real time and dynamically adjust the intervention strategy based on user feedback and physiological response;

[0012] The user interaction and management module provides an interface for interaction between the system and users, which is divided into the patient / family side and the doctor / administrator side.

[0013] In one example, the multimodal perception module mainly includes millimeter-wave radar, a smart bracelet, and a voice acquisition device. The millimeter-wave radar is deployed at a fixed location at the bedside or in the room for non-intrusive monitoring at night or in a resting state. The smart bracelet is used for continuous monitoring during daily activities. The voice acquisition device collects the user's voice signals and extracts emotional tendencies and semantic sentiments using natural language processing technology. All collected data is transmitted to an edge computing terminal via Bluetooth or Wi-Fi for preliminary data cleaning, synchronization, and feature extraction, providing data support for subsequent emotion recognition and intervention decisions.

[0014] In one example, the emotion recognition and state assessment module adopts a convolutional neural network model structure, with multi-channel physiological feature sequences as input and emotion state labels and their intensity scores as outputs.

[0015] In one example, the intelligent decision-making module includes a Traditional Chinese Medicine (TCM) Five-Tone Mapping Unit, a Music Feature Matching Unit, and a Weight Calculation Unit. The TCM Five-Tone Mapping Unit constructs a "Five Organs-Five Tones" mapping rule library, associating emotional states with organ functions. The Weight Calculation Unit sets weights for five dimensions: emotion recognition results, TCM diagnosis, music matching, user preferences, and historical feedback. It calculates the intervention suitability score for each track using a weighted summation method and selects the track with the highest score as the current intervention music. The Music Feature Matching Unit combines music psychology parameters and user preferences to screen candidate tracks from the music database.

[0016] In one example, the music intervention execution module supports synchronous and asynchronous modes. In synchronous mode, patients and their families listen to the same music simultaneously, enhancing emotional resonance and family collaboration. In asynchronous mode, different music is pushed according to individual emotional state differences, achieving personalized adjustment.

[0017] In one example, the feedback and optimization module introduces a music segment recording and segment analysis mechanism. The system sets an emotion improvement threshold, identifies effective music segments, numbers and stores these segments, associates them with user emotion and physiological data, and forms an intervention record that is uploaded to the cloud.

[0018] In one example, the patient / family interface in the user interaction and management module displays the current emotional state, music playback information, intervention effect feedback entry, and music preference settings, while the doctor / administrator interface provides a multi-user emotion monitoring panel, intervention record statistics, weight adjustment tools, and system operation status monitoring functions.

[0019] In one example, the approach of a synergistic intervention system based on multimodal emotion perception and pentatonic therapy includes the following steps:

[0020] Step 1: Multimodal data acquisition. The system uses millimeter-wave radar, smart bracelets, and voice acquisition devices to collect multimodal physiological and behavioral data from patients and their families in real time.

[0021] Step 2: Data preprocessing and feature extraction. The edge computing terminal performs operations such as denoising, filtering, normalization, and time alignment on the raw data to ensure data quality and consistency. The system extracts key features from the multimodal data, including physiological features, behavioral features, and motion features. These features constitute a multidimensional feature vector, which serves as the input to the emotion recognition model.

[0022] Step 3: Emotion state recognition. The system inputs the preprocessed feature vector into the emotion recognition model. The model is built based on deep learning algorithms. The input is a multi-channel physiological feature sequence, and the output is an emotion state label and its intensity score. The emotion recognition results are uploaded to the cloud server for subsequent decision-making modules to use.

[0023] Step 4: Matching Traditional Chinese Medicine (TCM) diagnosis with music. Based on the results of emotion recognition and physiological indicators, combined with the TCM theory of "five organs and five tones", the current state of the internal organs is determined, and candidate tracks with corresponding tones are selected from the music database.

[0024] Step 5: Weight Calculation and Intervention Decision. Based on the set multi-dimensional weights, calculate the suitability score of each candidate track and select the one with the highest score as the intervention music. During the process, based on user feedback and intervention effects, use reinforcement learning algorithms such as Q-learning or deep reinforcement learning (DRL) to optimize the weight allocation.

[0025] Step Six: Music Intervention Execution. Play the selected music to the user through noise-canceling headphones or a smart speaker to execute personalized intervention;

[0026] Step 7: Monitoring and segment analysis of intervention effects. During music playback, users' physiological indicators are continuously collected, and emotional changes are assessed in real time. Effective segments are numbered and stored, and correlated with users' emotional state, physiological indicators, and playback time information to form a complete music intervention record. The differences in responses of different users to different segments are analyzed to construct a personalized music segment preference model.

[0027] Step 8: System optimization and strategy adjustment. Based on feedback data and physiological responses, use reinforcement learning algorithms to optimize weights and recommendation strategies to achieve continuous evolution of the personalized model.

[0028] The synergistic intervention system and method based on multimodal emotion perception and five-tone therapy proposed in this invention can bring the following beneficial effects:

[0029] 1. This invention achieves precise emotion recognition and personalized intervention strategies through the synergistic intervention of multimodal emotion perception and traditional Chinese medicine five-tone therapy. It comprehensively captures users' physiological and behavioral signals in different states, and combines deep learning models for emotion recognition, significantly improving the accuracy and real-time performance of emotion recognition. Based on the theory of "five organs and five tones" in traditional Chinese medicine and modern music psychology, the system can intelligently match the most suitable music for the current emotional state, and continuously optimize the intervention strategy through reinforcement learning, thereby achieving more targeted and effective emotion regulation. It is particularly suitable for the long-term management and intervention of negative emotions such as anxiety and depression.

[0030] 2. This invention, through the introduction of a music segment recording and segment analysis mechanism, breaks through the traditional extensive mode of whole-song intervention and achieves refined tracking and optimization of the music intervention effect. By real-time monitoring and evaluation of the user's physiological response during the playback of each music segment, the system can identify "effective music segments" and build a personalized music segment preference model, thereby prioritizing the playback of high-effect segments in subsequent interventions, significantly improving intervention efficiency and user experience. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0032] Figure 1 System architecture diagram of the present invention;

[0033] Figure 2 A schematic diagram of the method flow of the present invention. Detailed Implementation

[0034] To more clearly illustrate the overall concept of the present invention, a detailed description will be provided below with reference to the accompanying drawings and examples.

[0035] In the description of this invention, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0036] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0037] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0038] In this invention, unless otherwise expressly specified and limited, the first feature "on" or "below" the second feature may be in direct contact with the first and second features, or indirect contact through an intermediate medium. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0039] like Figures 1 to 2 As shown, this invention proposes a collaborative intervention system and method based on multimodal emotion perception and five-tone therapy, including a multimodal perception module, an emotion recognition and state assessment module, an intelligent decision-making module, a music intervention execution module, a feedback and optimization module, and a user interaction and management module.

[0040] The multimodal perception module is responsible for collecting multimodal physiological and behavioral signals from patients and their families in real time, serving as the basic data source for emotion recognition. It mainly includes millimeter-wave radar, smart bracelets, and voice acquisition devices.

[0041] Millimeter-wave radar is deployed at a fixed location at the head of the bed or in the room to collect signals such as respiratory rate, chest cavity micro-movement, and changes in body position in a non-contact manner, making it suitable for non-intrusive monitoring at night or in a resting state.

[0042] The smart bracelet is worn on the user's wrist and collects indicators such as heart rate variability, skin conductance, body temperature, and exercise status in real time, making it suitable for continuous monitoring during daily activities.

[0043] Voice acquisition devices, such as smart speakers or microphones, collect user voice signals and extract emotional tendencies and semantic sentiments using natural language processing technology. All collected data is transmitted to edge computing terminals via Bluetooth or Wi-Fi for preliminary data cleaning, synchronization, and feature extraction, providing data support for subsequent emotion recognition and intervention decisions.

[0044] The emotion recognition and state assessment module uses deep learning algorithms to identify emotion states based on the physiological and behavioral data provided by the multimodal perception module.

[0045] The system employs model structures such as LSTM, Transformer, or convolutional neural networks. The input is a multi-channel physiological feature sequence, and the output is an emotion state label and its intensity score. Emotion states include anxiety, depression, calmness, and pleasure. The model is trained on a large-scale emotion dataset and has high recognition accuracy and generalization ability. The emotion recognition results are uploaded to a cloud server for subsequent decision-making modules to use.

[0046] The intelligent decision-making module is the core decision-making unit of the system, responsible for formulating personalized music intervention strategies based on emotion recognition results and the five-tone theory of traditional Chinese medicine.

[0047] This module includes a Traditional Chinese Medicine (TCM) Five-Tone Mapping Unit, a Music Feature Matching Unit, and a Weight Calculation Unit. The TCM Five-Tone Mapping Unit constructs a "Five Organs - Five Tones" mapping rule library, associating emotional states with organ functions. For example, the liver governs anger and corresponds to the Jiao tone, the heart governs joy and corresponds to the Zhi tone, the spleen governs thought and corresponds to the Gong tone, the lungs govern worry and correspond to the Shang tone, and the kidneys govern fear and correspond to the Yu tone. The Music Feature Matching Unit combines music psychology parameters and user preferences to select candidate tracks from the music database.

[0048] The weighting calculation unit assigns weights to five dimensions: emotion recognition result, traditional Chinese medicine diagnosis, music matching, user preference, and historical feedback. It calculates the intervention suitability score for each track using a weighted summation method, selecting the track with the highest score as the current intervention music. Initial weight values ​​can be set as follows: emotion recognition 0.3, traditional Chinese medicine diagnosis 0.25, music matching 0.2, user preference 0.15, and historical feedback 0.1. The system supports dynamic optimization of weight allocation through reinforcement learning algorithms.

[0049] The music intervention execution module is responsible for pushing the music content selected by the decision module to the user in an appropriate manner to perform personalized intervention. This module plays music through noise-canceling headphones or smart speakers and supports setting the playback duration, such as 10, 20, or 30 minutes.

[0050] The system supports synchronous and asynchronous modes. In synchronous mode, patients and their families listen to the same music at the same time, enhancing emotional resonance and family collaboration. In asynchronous mode, different music is pushed according to individual emotional state differences, achieving personalized adjustment. The music database includes traditional pentatonic music, modern healing music and AI-generated music. All tracks are labeled with metadata such as pitch type, rhythm, mode, timbre and applicable emotions.

[0051] The feedback and optimization module monitors the intervention effect in real time and dynamically adjusts the intervention strategy based on user feedback and physiological responses. This module continuously collects indicators such as GSR and HRV during music playback to assess emotional changes. After playback, users can rate the intervention effect via their mobile devices. The system records feedback and uses reinforcement learning algorithms to optimize weights and recommendation strategies, enabling system self-evolution. Furthermore, the system constructs a family-patient emotional linkage model to analyze emotional contagion pathways and dynamically adjust intervention priorities, achieving coordinated regulation within the family system.

[0052] The system incorporates a music segment recording and analysis mechanism. Each piece of music is divided into several 30-second segments based on its duration. During playback, the system continuously collects the user's physiological indicators (such as GSR, HRV, and respiratory rate) and records the physiological response data corresponding to each music segment in real time. After playback, the system evaluates the effect of each segment to determine whether the music had a positive impact on the user's emotions.

[0053] The system sets a threshold for mood improvement; for example, a GSR decrease of more than 10% or an HRV increase of more than 5% is considered a "valid music segment." Once a valid music segment is identified, the system automatically marks the start and end times of the music segment and extends it forward and backward by 30 seconds each, forming a complete "valid segment." The segment's range is 30 seconds before and after the valid music segment. This is done to capture the complete process of emotional change in the music and avoid missing key emotional guidance.

[0054] The system will number and store these effective segments, and link them with information such as the user's emotional state, physiological indicators, and playback time to form a complete "music intervention record." These records will be uploaded to a cloud database as an important basis for subsequent personalized interventions. In subsequent interventions, the system will prioritize playing music segments that were previously marked as "effective segments," improving the targeting and effectiveness of the intervention. Simultaneously, the system will analyze the differences in different users' responses to different segments, constructing a personalized "music segment preference model" to achieve more refined music recommendations.

[0055] Furthermore, the system allows doctors or users to manually mark certain music segments as "effective" or "ineffective," adding notes such as "the patient's mood was significantly relaxed during playback" or "the tempo of this music segment was too fast, causing discomfort to the patient." This subjective feedback, combined with objective physiological data, further enhances the system's intelligence. Through music segment recording and segment analysis mechanisms, the system can not only identify the effect of the entire piece of music but also precisely locate specific music fragments, truly achieving personalized music intervention "according to the person, the time, and the situation." This refined intervention approach not only improves the controllability and interpretability of the intervention effect but also provides strong technical support for the modernization and digitalization of Traditional Chinese Medicine's Five-Tone Therapy.

[0056] The user interaction and management module provides an interactive interface between the system and users, divided into a patient / family interface and a doctor / administrator interface. The patient / family interface displays the current emotional state, music playback information, an entry point for intervention effect feedback, and music preference settings. The doctor / administrator interface provides a multi-user emotion monitoring panel, intervention record statistics, weight adjustment tools, and system operation status monitoring functions. All interfaces support graphical displays, are easy to operate, and suitable for elderly users.

[0057] The present invention also provides a collaborative intervention method based on the above system, which specifically includes the following steps: multimodal data acquisition, data preprocessing and feature extraction, emotional state recognition, TCM syndrome differentiation and music matching, weight calculation and intervention decision-making, music intervention execution, intervention effect monitoring and stage analysis, system optimization and strategy adjustment.

[0058] In the multimodal data acquisition step, the system uses millimeter-wave radar, smart bracelets, and voice acquisition devices to collect real-time multimodal physiological and behavioral data from patients and their families, including respiration, heart rate, skin conductance, and speech. In the data preprocessing and feature extraction step, the edge computing terminal performs noise reduction, filtering, normalization, and time alignment on the raw data to ensure data quality and consistency. Subsequently, the system extracts key features from the multimodal data, including physiological features such as RMSSD of HRV, LF / HF ratio, peak frequency of GSR, and respiratory rate changes; behavioral features such as emotional tone, speech rate, and volume changes in speech; and motion features such as cadence, activity intensity, and sleep state. These features constitute a multidimensional feature vector, which serves as the input to the emotion recognition model.

[0059] In the emotion state recognition step, the system inputs the preprocessed feature vector into the emotion recognition model. The model is built based on deep learning algorithms, such as LSTM, Transformer, or convolutional neural networks. The input is a multi-channel physiological feature sequence, and the output is an emotion state label and its intensity score. Emotion states include anxiety, depression, calmness, and pleasure. The model is trained on a large-scale emotion dataset and possesses high recognition accuracy and generalization ability. The emotion recognition results are uploaded to a cloud server for subsequent decision-making modules to use.

[0060] In the TCM diagnosis and music matching process, the system determines the current state of the internal organs based on emotion recognition results and physiological indicators, combined with the TCM theory of "five organs - five tones." The specific mapping relationships are as follows: the liver governs anger and corresponds to the Jiao tone; the heart governs joy and corresponds to the Zhi tone; the spleen governs thought and corresponds to the Gong tone; the lungs govern sorrow and correspond to the Shang tone; and the kidneys govern fear and correspond to the Yu tone. The system then selects candidate tracks with corresponding tones from a music database. The music database includes traditional five-tone pieces, modern healing music, and AI-generated music, with all tracks labeled with metadata such as rhythm, mode, timbre, frequency, and applicable emotions.

[0061] In the weighting calculation and intervention decision-making process, the system calculates the suitability score of each candidate track based on the set multi-dimensional weights, and selects the track with the highest score as the intervention music. The initial weight values ​​can be set as follows: emotion recognition result 0.3, traditional Chinese medicine diagnosis result 0.25, music psychology parameter 0.2, user preference 0.15, and historical feedback 0.1. The system uses a weighted summation formula to calculate the intervention suitability score of each candidate track: Score = W1×E + W2×T + W3×M + W4×P + W5×H, where E is the emotion recognition matching degree, T is the traditional Chinese medicine diagnosis matching degree, M is the music psychology matching degree, P is the user preference matching degree, and H is the historical feedback score. The system selects the track with the highest score as the current intervention music. The weight settings support dynamic adjustment. During system operation, based on user feedback and intervention effects, reinforcement learning algorithms such as Q-learning or deep reinforcement learning (DRL) are used to optimize weight allocation. Users or doctors can also manually adjust the weights to meet personalized needs.

[0062] During the music intervention process, the system plays selected music to the user through noise-canceling headphones or a smart speaker, providing a personalized intervention. Playback duration can be set to 10, 20, or 30 minutes depending on the user's emotional state. The system supports synchronous and asynchronous modes: in synchronous mode, the patient and family listen to the same music simultaneously, enhancing emotional resonance and family collaboration; in asynchronous mode, different music is pushed based on individual emotional states, achieving personalized adjustment. During music playback, the system can combine aromatherapy, lighting, and guided breathing with multi-sensory stimulation to further enhance the intervention effect.

[0063] In the intervention effect monitoring and segment analysis steps, the system continuously collects users' physiological indicators such as GSR and HRV during music playback to assess emotional changes in real time. The system divides each piece of music into several segments, each 30 seconds long. During playback, the system continuously collects users' physiological indicators (such as GSR, HRV, respiratory rate, etc.) and records the physiological response data corresponding to each segment of music in real time.

[0064] After playback, the system evaluates the effect of each segment to determine whether it positively impacted the user's emotions. The system sets an emotion improvement threshold; for example, a GSR decrease of more than 10% or an HRV increase of more than 5% is considered a "valid music segment." Once a valid music segment is identified, the system automatically marks its start and end times and extends it forward and backward by 30 seconds to form a complete "valid segment," meaning the segment's range is 30 seconds before and after the valid music segment. This is done to capture the complete emotional shift in the music, avoiding missing crucial emotional guidance. The system numbers and stores these valid segments and links them to the user's emotional state, physiological indicators, playback time, and other information to create a complete "music intervention record." These records are uploaded to a cloud database as an important basis for subsequent personalized interventions.

[0065] In subsequent interventions, the system will prioritize playing music segments previously marked as "effective segments," enhancing the targeting and effectiveness of the intervention. Simultaneously, the system will analyze the differences in responses to different segments among users, constructing a personalized "music segment preference model" for more refined music recommendations. Furthermore, the system allows doctors or users to manually mark certain music segments as "effective" or "ineffective," adding notes such as "The patient's mood was significantly relaxed during playback" or "The tempo of this music segment was too fast, causing discomfort to the patient." This subjective feedback will be combined with objective physiological data to further improve the system's intelligence.

[0066] In the system optimization and strategy adjustment steps, the system uses reinforcement learning algorithms to optimize weights and recommendation strategies based on feedback data and physiological responses, enabling continuous evolution of the personalized model. Specifically, this includes: weight optimization, dynamically adjusting the weights of each dimension based on intervention effects; model updates, periodically retraining the emotion recognition model with new data; music library expansion, continuously updating the music database based on user preferences and feedback; and the construction of an emotion linkage model, analyzing the emotional contagion pathways between patients and their families, dynamically adjusting intervention priorities, and achieving coordinated regulation within the family system. The system also constructs a family-patient emotion linkage model to identify emotional contagion pathways, dynamically adjust intervention priorities and strategies, achieve coordinated optimization of the family system, and break the cycle of negative emotions.

[0067] This system is applicable to various settings including hospital oncology wards, home care environments, rehabilitation centers, and elderly care facilities. In hospital oncology wards, inpatients and their families can use the system simultaneously, improving treatment adherence and satisfaction. In home care environments, portable devices enable home-based psychological adjustment, reducing the burden of caregiving. In rehabilitation centers and elderly care facilities, the system is used for long-term psychological support and emotion management, improving quality of life. The system can also be integrated with an app and cloud services to enable remote intervention and data management, expanding service coverage.

[0068] Regarding system security and privacy protection, all data transmissions are encrypted using AES-256 to ensure information security. User identity information and physiological data are stored separately to prevent data leakage. Users can view, export, or delete their personal data at any time.

[0069] After the system is operational, its effectiveness can be evaluated using indicators such as emotion recognition accuracy, intervention effectiveness rate, user satisfaction, reduction in family burden, and system response time. Emotion recognition accuracy refers to the consistency between the system's recognized emotions and human assessment. Intervention effectiveness rate refers to the proportion of individuals whose GSR (Growth Rate of Severity) decreases or whose HRV (Heart Rate of Resilience) increases after intervention. User satisfaction refers to users' subjective ratings of the intervention's effectiveness. Reduction in family burden refers to changes in family members' depression / anxiety scale scores before and after use. System response time refers to the average delay from emotion recognition to music playback.

[0070] This invention constructs a collaborative psychological intervention platform integrating multimodal perception, intelligent decision-making, personalized intervention, and dynamic optimization through the synergistic design of systems and methods. This platform not only modernizes and digitizes the Five-Tone Therapy of Traditional Chinese Medicine, but also improves the accuracy of emotion recognition and the level of personalized intervention through artificial intelligence and multimodal perception technology. It is particularly suitable for the mental health management of elderly cancer patients and their families, and has broad clinical application prospects and social value.

[0071] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0072] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A synergistic intervention system based on multimodal emotion perception and five-tone therapy, characterized in that: It includes a multimodal perception module, an emotion recognition and state assessment module, an intelligent decision-making module, a music intervention execution module, a feedback and optimization module, and a user interaction and management module; The multimodal perception module is responsible for collecting multimodal physiological and behavioral signals from patients and their families in real time, serving as the basic data source for emotion recognition; The emotion recognition and state assessment module uses deep learning algorithms to recognize emotional states based on the physiological and behavioral data provided by the multimodal perception module. The intelligent decision-making module formulates personalized music intervention strategies based on emotion recognition results and the five-tone theory of traditional Chinese medicine. The music intervention execution module is responsible for pushing the music content selected by the decision-making module to the user in an appropriate manner to execute personalized intervention; The feedback and optimization module is used to monitor the intervention effect in real time and dynamically adjust the intervention strategy based on user feedback and physiological response; The user interaction and management module provides an interface for interaction between the system and users, which is divided into the patient / family side and the doctor / administrator side.

2. The synergistic intervention system based on multimodal emotion perception and five-tone therapy according to claim 1, characterized in that: The multimodal perception module mainly includes millimeter-wave radar, a smart bracelet, and a voice acquisition device. The millimeter-wave radar is deployed at a fixed location at the bedside or in the room for non-intrusive monitoring at night or in a resting state. The smart bracelet is used for continuous monitoring during daily activities. The voice acquisition device collects the user's voice signals and extracts emotional tendencies and semantic sentiments using natural language processing technology. All collected data is transmitted to an edge computing terminal via Bluetooth or Wi-Fi for preliminary data cleaning, synchronization, and feature extraction, providing data support for subsequent emotion recognition and intervention decisions.

3. The synergistic intervention system based on multimodal emotion perception and five-tone therapy according to claim 1, characterized in that: The emotion recognition and state assessment module adopts a convolutional neural network model structure. The input is a multi-channel physiological feature sequence, and the output is an emotion state label and its intensity score.

4. The synergistic intervention system based on multimodal emotion perception and five-tone therapy according to claim 1, characterized in that: The intelligent decision-making module includes a Traditional Chinese Medicine (TCM) Five-Tone Mapping Unit, a Music Feature Matching Unit, and a Weight Calculation Unit. The TCM Five-Tone Mapping Unit constructs a "Five Organs - Five Tones" mapping rule library, linking emotional states with organ functions. The Weight Calculation Unit sets weights for five dimensions: emotion recognition results, TCM diagnosis, music matching, user preferences, and historical feedback. It calculates the intervention suitability score for each track using a weighted summation method and selects the track with the highest score as the current intervention music. The Music Feature Matching Unit combines music psychology parameters and user preferences to select candidate tracks from the music database.

5. The synergistic intervention system based on multimodal emotion perception and five-tone therapy according to claim 1, characterized in that: The music intervention execution module supports synchronous and asynchronous modes. In synchronous mode, patients and their families listen to the same music at the same time, enhancing emotional resonance and family collaboration. In asynchronous mode, different music is pushed according to individual emotional state differences, achieving personalized adjustment.

6. The synergistic intervention system based on multimodal emotion perception and five-tone therapy according to claim 1, characterized in that: The feedback and optimization module introduces a music segment recording and segment analysis mechanism. The system sets an emotion improvement threshold, identifies effective music segments, numbers and stores these segments, associates them with user emotion and physiological data, and forms an intervention record that is uploaded to the cloud.

7. The synergistic intervention system based on multimodal emotion perception and five-tone therapy according to claim 1, characterized in that: The user interaction and management module includes a patient / family interface that displays the current emotional state, music playback information, an intervention effect feedback entry point, and music preference settings, and a doctor / administrator interface that provides a multi-user emotion monitoring panel, intervention record statistics, weight adjustment tools, and system operation status monitoring functions.

8. A method for implementing the synergistic intervention system based on multimodal emotion perception and five-tone therapy as described in claim 1, characterized in that, The following steps are involved: Step 1: Multimodal data acquisition. The system uses millimeter-wave radar, smart bracelets, and voice acquisition devices to collect multimodal physiological and behavioral data from patients and their families in real time. Step 2: Data preprocessing and feature extraction. The edge computing terminal performs operations such as denoising, filtering, normalization, and time alignment on the raw data to ensure data quality and consistency. The system extracts key features from the multimodal data, including physiological features, behavioral features, and motion features. These features constitute a multidimensional feature vector, which serves as the input to the emotion recognition model. Step 3: Emotion state recognition. The system inputs the preprocessed feature vector into the emotion recognition model. The model is built based on deep learning algorithms. The input is a multi-channel physiological feature sequence, and the output is an emotion state label and its intensity score. The emotion recognition results are uploaded to the cloud server for subsequent decision-making modules to use. Step 4: Matching Traditional Chinese Medicine (TCM) diagnosis with music. Based on the results of emotion recognition and physiological indicators, combined with the TCM theory of "five organs and five tones", the current state of the internal organs is determined, and candidate tracks with corresponding tones are selected from the music database. Step 5: Weight Calculation and Intervention Decision. Based on the set multi-dimensional weights, calculate the suitability score of each candidate track and select the one with the highest score as the intervention music. During the process, based on user feedback and intervention effects, use reinforcement learning algorithms such as Q-learning or deep reinforcement learning (DRL) to optimize the weight allocation. Step Six: Music Intervention Execution. Play the selected music to the user through noise-canceling headphones or a smart speaker to execute personalized intervention; Step 7: Monitoring and segment analysis of intervention effects. During music playback, users' physiological indicators are continuously collected, and emotional changes are assessed in real time. Effective segments are numbered and stored, and correlated with users' emotional state, physiological indicators, and playback time information to form a complete music intervention record. The differences in responses of different users to different segments are analyzed to construct a personalized music segment preference model. Step 8: System optimization and strategy adjustment. Based on feedback data and physiological responses, use reinforcement learning algorithms to optimize weights and recommendation strategies to achieve continuous evolution of the personalized model.

Images