Man-machine conversation psychological stress identification and intervention method, system, medium and product

Abstract

The invention discloses a man-machine dialogue psychological stress identification and intervention method and system, a medium and a product, and relates to the field of natural language data processing, and the method comprises the steps: carrying out man-machine dialogue based on a man-machine dialogue strategy through employing a preset man-machine dialogue model; acquiring a multi-modal signal of the detected object in a man-machine conversation process in real time, and performing feature extraction and mapping on the multi-modal signal to obtain a multi-modal fusion answer space; performing psychological stress identification on the multi-modal fusion answer space to obtain a real-time psychological stress level of the tested object; a man-machine conversation strategy of the man-machine conversation model is adaptively adjusted according to the real-time psychological stress level; and executing corresponding operation based on the adjusted man-machine dialogue strategy by utilizing the man-machine dialogue model. The accuracy and robustness of psychological pressure recognition are improved, the man-machine conversation strategy can be automatically adjusted according to the recognition result, and the emotional stability and psychological assistance value of man-machine conversation are improved.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, physiological signal analysis, and human-computer interaction, and in particular to an adaptive human-computer dialogue method, system, medium, and product for identifying and intervening in psychological stress based on multimodal information. Background Technology

[0002] With the accelerating pace of modern life, people face increasing psychological stress. Accurate identification and assessment of psychological stress are crucial for mental health monitoring and intervention. Traditional methods for emotion and stress recognition primarily rely on single-modal data, such as facial expressions, voice tone, or physiological signals (e.g., EEG, ECG). However, single-modal data has limitations in psychological stress recognition, is easily affected by environmental interference leading to low accuracy, and cannot comprehensively reflect an individual's psychological state, potentially missing important information and impacting the recognition results. Furthermore, current technologies lack real-time intervention capabilities after stress recognition, failing to automatically adjust dialogue strategies based on changes in user stress levels. This can lead to further accumulation of psychological stress, affecting the interactive experience and even causing psychological burden. Therefore, there is an urgent need for an integrated technological solution capable of stress recognition and adaptive intervention, supporting real-time stress relief during conversations. Summary of the Invention

[0003] To address the shortcomings of existing technologies that can only identify stress but lack real-time intervention, this invention proposes an adaptive human-computer dialogue method, system, medium, and product for psychological stress identification and intervention based on multimodal information. By integrating multimodal information such as EEG, audio, and vision, the psychological stress level is determined, and personalized human-computer dialogue strategies are triggered according to the psychological stress level, thereby improving the safety, comfort, and psychological friendliness of human-computer dialogue.

[0004] In a first aspect, the present invention proposes a method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information, comprising: Human-computer dialogue is conducted using a pre-defined human-computer dialogue model and a human-computer dialogue strategy. The system acquires EEG, audio, and video signals of the tested subject during human-computer interaction in real time; the EEG, audio, and video signals are acquired synchronously with unified timestamps. Feature extraction is performed on the electroencephalogram (EEG) signals to obtain EEG feature representations; Feature extraction is performed on the audio signal to obtain an audio feature representation; Feature extraction is performed on the video signal to obtain a visual feature representation; EEG feature representations, audio feature representations, and visual feature representations are cascaded by channel to form a high-dimensional fusion vector; EEG feature representations, audio feature representations, and visual feature representations are spatially mapped through a linear mapping layer to form a multimodal fusion answer space; Psychological stress is identified by multimodal fusion answer space to obtain the real-time psychological stress level of the test subjects; the psychological stress level is divided into three levels: low stress, medium stress, and high stress. The human-computer dialogue strategy of the human-computer dialogue model is adaptively adjusted according to the real-time psychological stress level. The corresponding operation is performed based on the adjusted human-computer dialogue strategy using the human-computer dialogue model.

[0005] Preferably, the strategy for adaptively adjusting the current human-computer dialogue based on the real-time psychological stress level specifically includes: When the real-time psychological stress level is low, use normal task-oriented dialogue strategies. When the real-time psychological stress level is moderate, a relief-type response is selected and output from the preset low-load dialogue template library; among them, relief-type responses include breathing guidance, attention-shifting topics, emotional empathy statements or simplified task prompts; When the real-time psychological stress level is high, the current task-oriented dialogue is automatically paused and switched to pure companionship mode, psychological comfort mode, or the subject is proactively advised to take a short rest. At the same time, the real-time psychological stress level of the test subject is dynamically injected into the system prompts of the human-computer dialogue model.

[0006] Preferably, the wording templates for the slow-release type of response are marked by psychological experts and optimized through controlled experiments and A / B testing. Before output, the large model rewrites them in real time according to the personalized characteristics of the test subjects.

[0007] Preferably, the feature extraction process for EEG feature representation includes: The EEG signal is segmented using a sliding window of preset duration to obtain multiple EEG signal sample segments. After bandpass filtering and ICA artifact removal, the multiple EEG signal sample segments are input into a multilayer perceptron (MLP) to obtain deep EEG feature vectors. The deep EEG feature vectors are then compressed and refined using a Q-Former model to obtain EEG feature representations.

[0008] Preferably, in the process of extracting audio feature representation, the wav2vec2 model is used to extract features to obtain a deep speech feature vector, and the deep audio feature vector is compressed and refined by the Q-Former model to obtain the audio feature representation.

[0009] Preferably, the process of extracting visual feature representations includes: The VIT model is used for feature extraction to obtain deep visual feature vectors, and the Q-Former model is used to compress and refine the deep visual feature representation to obtain the visual feature representation.

[0010] Secondly, this invention also proposes a human-computer dialogue psychological stress recognition and intervention system based on multimodal information, comprising: The human-computer dialogue module is used to conduct human-computer dialogue based on a preset human-computer dialogue model and a human-computer dialogue strategy. The acquisition module is used to acquire the EEG signals, audio signals, and video signals of the tested object during the human-computer dialogue process; The feature extraction module is used to extract features from EEG signals, audio signals, and video signals respectively, and obtain EEG feature representations, audio feature representations, and video feature representations. The fusion module is used to cascade EEG feature representations, audio feature representations, and visual feature representations by channel to form a high-dimensional fusion vector; and to spatially map EEG feature representations, audio feature representations, and visual feature representations through a linear mapping layer to form a multimodal fusion answer space. The recognition module is used to identify psychological stress in the multimodal fusion answer space based on preset prompts using a predefined large model, so as to obtain the real-time psychological stress level of the test subject.

[0011] The psychological intervention module is used to adaptively adjust the human-computer dialogue strategy of the human-computer dialogue model based on the real-time psychological stress level. The human-computer dialogue module is also used to perform corresponding operations based on the adjusted human-computer dialogue strategy using a preset human-computer dialogue model.

[0012] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal information human-computer dialogue psychological stress identification and intervention method described in any one of the first aspects.

[0013] Fourthly, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal information human-computer dialogue psychological stress identification and intervention method described in any one of the first aspects.

[0014] This invention proposes a method, system, medium, and product for identifying and intervening in psychological stress during human-computer dialogue. It acquires three multimodal information signals from the subject during the dialogue: electroencephalogram (EEG), audio, and video signals. Feature extraction is performed on these multimodal signals to obtain EEG, audio, and visual feature representations. These multimodal EEG, audio, and visual feature representations are mapped to obtain a multimodal fusion answer space. A predefined large model is used to identify psychological stress within this multimodal fusion answer space based on preset prompts, yielding the subject's real-time psychological stress level. This invention not only significantly improves the accuracy of psychological stress identification through multimodal fusion but also proposes an integrated "identification-intervention" technical approach. By determining the psychological stress level in real time, it enables dynamic adjustment of dialogue content, early warning of emotional risks, and automatic adaptation of the interaction mode to the subject's psychological state. This enhances the agent's psychological friendliness in long-term dialogues and effectively reduces the subject's real-time psychological stress level during interaction. Therefore, this invention has significant application and social value in real-world human-computer dialogue scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a human-computer dialogue psychological stress identification and intervention method based on multimodal information, as proposed in one embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] Firstly, referring to Figure 1 This invention proposes a method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information, comprising: Acquire multimodal information of the test subject during human-computer dialogue; the multimodal information includes EEG signals, audio signals, and video frames. Feature extraction is performed on multimodal information to obtain multimodal feature data; the multimodal feature data includes: EEG feature representation, audio feature representation, and visual feature representation; By mapping the multimodal feature data, a multimodal fusion answer space is obtained; Psychological stress is identified by multimodal fusion answer space to obtain the real-time psychological stress level of the test subjects; the psychological stress level is divided into three levels: low stress, medium stress, and high stress. Human-computer dialogue strategy based on real-time psychological stress level adaptive adjustment of human-computer dialogue model; The human-computer dialogue model performs corresponding operations based on the adjusted human-computer dialogue strategy. This involves mapping multimodal feature data to obtain a multimodal fusion answer space, specifically including: EEG feature representations, audio feature representations, and visual feature representations are cascaded by channel to form a high-dimensional fusion vector; EEG feature representations, audio feature representations, and visual feature representations are spatially mapped through a linear mapping layer to form a multimodal fusion answer space.

[0018] This invention acquires three multimodal information signals from the test subject during human-computer interaction: EEG signals, audio signals, and video signals. It then extracts features from these multimodal information through three branches, obtaining EEG feature representations, audio feature representations, and visual feature representations. These multimodal feature data are mapped to obtain a multimodal fusion answer space. A predefined large model is used to identify psychological stress within this multimodal fusion answer space based on preset prompts, thus obtaining the test subject's real-time psychological stress level. This overcomes the limitations of single-modal data and improves the accuracy and robustness of psychological stress identification. Furthermore, after stress identification, the invention triggers corresponding human-computer interaction strategies based on the psychological stress level (low, medium, high). This allows the human-computer interaction model to execute corresponding operations based on the adjusted strategies. By binding psychological stress identification with human-computer interaction strategies, continuous monitoring and intelligent adjustment of the test subject's psychological state during human-computer interaction are achieved. This realizes an integrated identification-intervention process, giving the invention significant advantages in real-time performance, user-friendliness, and practical application value.

[0019] Moreover, this invention forms a high-dimensional fusion vector by cascading EEG feature representations, audio feature representations, and visual feature representations by channel; and obtains a multimodal fusion answer space by spatially mapping the high-dimensional fusion vector through a shared Linear mapping layer. This solves the problems of handling heterogeneity and spatiotemporal asynchrony between different modalities and the difficulty of data acquisition and processing in multimodal data fusion. The multimodal fusion answer space is not only used for stress identification, but also serves as an important basis for subsequent intervention decisions, enabling the system to automatically select appropriate interaction strategies based on the identification results, thereby achieving personalized psychological adjustment.

[0020] In this embodiment, the strategy for adaptively adjusting the current human-computer dialogue based on real-time psychological stress levels specifically includes: When the real-time psychological stress level is low, use normal task-oriented dialogue strategies. When the real-time psychological stress level is moderate, a relief-type response is selected and output from the preset low-load dialogue template library; among them, relief-type responses include breathing guidance, attention-shifting topics, emotional empathy statements or simplified task prompts; When the real-time psychological stress level is high, the current task-oriented dialogue is automatically paused and switched to pure companionship mode, psychological comfort mode, or the subject is proactively advised to take a short rest. Furthermore, the real-time psychological stress level of the test subjects is dynamically injected into the system prompts of the human-computer dialogue model, making the subsequent responses generated by the human-computer dialogue model tend to be shorter, more soothing, and have lower cognitive load.

[0021] With this configuration, this embodiment can dynamically adjust the dialogue content according to changes in stress, adapt the interaction mode to the psychological state of the test subject in real time, provide early warning of emotional risks under high stress, improve the psychological friendliness of the system in long-term human-computer dialogue, and effectively reduce the real-time psychological stress level of the test subject.

[0022] It's important to understand that when psychological stress levels are low, normal task-oriented dialogue strategies should be used, with mild reassuring statements only when necessary (there is some psychological stress, but it's at a low level). For example, short, friendly prompts can express understanding without over-interfering.

[0023] When the stress level is at a central level, initiate stress-relieving responses, such as empathic language, and simultaneously write the stress level into the system prompt words to make subsequent responses tend to be gentle, slow, and low-confrontational, so as to mitigate the intervention. When the psychological stress level is high, the system automatically pauses the task-oriented dialogue, switches to "emotional companionship mode", and automatically outputs soothing instructions (deep breathing, mindfulness pause, etc.). If necessary, it suggests that the subject take a short rest, or notify the guardian in an authorized scenario to carry out safety-first intervention.

[0024] Among them, the wording templates for the slow-release type of response were marked by psychological experts and optimized through controlled experiments and A / B testing. Before output, the large model rewrote them in real time according to the personalized characteristics of the test subjects.

[0025] For example, the system prompts dynamically injected into the human-computer dialogue model are specifically: "You are having a conversation with a test subject whose stress level is 'high'. Please reply with calm, concise, and empathetic language, and avoid complex reasoning and high-load questions."

[0026] Of course, this embodiment can also use hints that have an equivalent structured meaning to this example.

[0027] It should be noted that the multimodal information in this embodiment is obtained by synchronously acquiring data using an EEG acquisition module, a camera module, and a microphone module. During the acquisition process, millisecond-level clocks are used for alignment and to unify timestamps, thereby improving the accuracy of psychological stress recognition.

[0028] In one specific embodiment, the EEG acquisition module is a portable EEG acquisition module, the camera module is a high-definition camera, and the microphone module is a directional microphone array. The portable EEG acquisition module is used to capture the frontal / whole-brain EEG signals of the subject in real time; the high-definition camera is used to acquire facial video streams at 30 fps or higher; and the directional microphone array is used to simultaneously acquire speech / ambient audio.

[0029] In this embodiment, feature extraction is performed on the multimodal information to obtain multimodal feature data, specifically including: Feature extraction is performed on the electroencephalogram (EEG) signals to obtain EEG feature representations; Feature extraction is performed on the audio signal to obtain an audio feature representation; Feature extraction is performed on the video signal to obtain a visual feature representation.

[0030] In a further embodiment, feature extraction is performed on the EEG signal to obtain an EEG feature representation, specifically including: The EEG signal is segmented using a sliding window of preset duration to obtain multiple EEG signal sample segments. After bandpass filtering and ICA artifact removal, the multiple EEG signal sample segments are input into a multilayer perceptron (MLP) to obtain deep EEG feature vectors. The deep EEG feature vectors are then compressed and refined using a Q-Former model to obtain EEG feature representations.

[0031] In a further embodiment, during the extraction of audio feature representation, the wav2vec2 model is used to extract features to obtain a deep speech feature vector, and the deep audio feature vector is compressed and refined using the Q-Former model to obtain an audio feature representation.

[0032] In a further embodiment, feature extraction is performed on the video signal to obtain a visual feature representation, specifically including: The VIT model is used to extract features from the video signal to obtain a depth visual feature vector. The Q-Former model is then used to compress and refine the depth visual feature vector to obtain a visual feature representation.

[0033] In one specific embodiment, psychological stress identification is performed on the multimodal fusion answer space to obtain the real-time psychological stress level of the tested subject, specifically including: By using a predefined large model to identify psychological stress in the multimodal fusion answer space based on preset prompts, the real-time psychological stress level of the test subjects can be obtained.

[0034] Of course, in this embodiment, an existing neural network model for identifying psychological stress can also be used to identify psychological stress.

[0035] Secondly, this invention also proposes a human-computer dialogue psychological stress recognition and intervention system based on multimodal information, comprising: The human-computer dialogue module is used to conduct human-computer dialogue based on a preset human-computer dialogue model and a human-computer dialogue strategy. The acquisition module is used to acquire the EEG signals, audio signals, and video signals of the tested object during the human-computer dialogue process; The feature extraction module is used to extract features from the EEG signal, audio feature representation and visual feature representation respectively, to obtain EEG feature representation, audio feature representation and visual feature representation; The fusion module is used to cascade EEG feature representations, audio feature representations, and visual feature representations by channel to form a high-dimensional fusion vector; and to spatially map EEG feature representations, audio feature representations, and visual feature representations through a linear mapping layer to form a multimodal fusion answer space. The recognition module is used to identify psychological stress in the multimodal fusion answer space based on preset prompts using a predefined large model, so as to obtain the real-time psychological stress level of the test subject. The psychological intervention module is used to adaptively adjust the human-computer dialogue strategy of the human-computer dialogue model based on the real-time psychological stress level. The human-computer dialogue module is also used to perform impact operations based on the adjusted human-computer dialogue strategy using a preset human-computer dialogue model.

[0036] This invention overcomes the limitations of single-modal data, improves the accuracy and robustness of psychological stress recognition, and solves the problems of heterogeneity and spatiotemporal asynchrony between different modalities and the difficulty of data acquisition and processing in multimodal data fusion. Furthermore, this invention triggers corresponding human-computer dialogue strategies based on psychological stress levels (low, medium, high), enabling the human-computer dialogue model to perform corresponding operations based on the adjusted strategies. This achieves an integrated recognition-intervention processing flow, giving this invention significant advantages in real-time performance, user-friendliness, and practical application value.

[0037] Among these, the strategy for adaptively adjusting the current human-computer dialogue based on real-time psychological stress levels specifically includes: When the real-time psychological stress level is low, use normal task-oriented dialogue strategies. When the real-time psychological stress level is moderate, a relief-type response is selected and output from the preset low-load dialogue template library; among them, relief-type responses include breathing guidance, attention-shifting topics, emotional empathy statements or simplified task prompts; When the real-time psychological stress level is high, the current task-oriented dialogue is automatically paused and switched to pure companionship mode, psychological comfort mode, or the subject is proactively advised to take a short rest. The real-time psychological stress level of the test subject is dynamically injected into the system prompts of the human-computer dialogue model.

[0038] In this embodiment, the wording templates for the slow-release type of response were annotated by psychological experts and optimized through controlled experiments and A / B testing. Before output, the large model rewrote them in real time according to the personalized characteristics of the test subjects.

[0039] In this embodiment, it also includes: The EEG acquisition module is used to capture EEG signals from the frontal / whole brain of the subject in real time; The camera module is used to acquire real-time video streams of the faces of the subjects being tested, i.e., video signals; The microphone module is used to acquire the voice / environmental audio of the test subject in real time, i.e., the audio signal.

[0040] During the data acquisition process, the EEG acquisition module, camera module, and microphone module are aligned using millisecond-level clocks to ensure consistent timestamps.

[0041] In one specific embodiment, the EEG acquisition module is a portable EEG acquisition module, the camera module is a high-definition camera, and the microphone module is a directional microphone array. The portable EEG acquisition module is used to capture the forehead / whole brain EEG signals of the subject in real time; the high-definition camera is used to acquire facial video streams at 30fps or higher; and the directional microphone array is used to simultaneously acquire speech / ambient audio. In practice, the EEG acquisition module, camera module, and microphone module all upload data to the acquisition module via Wi-Fi / Bluetooth / USB wired connection.

[0042] This embodiment, through a low-cost, scalable multimodal acquisition scheme and an end-to-end deep learning fusion and classification framework, significantly improves the accuracy and robustness of psychological stress identification while maintaining portable deployment. It can be widely applied in scenarios such as online mental health screening, intelligent customer service emotion monitoring, and remote medical follow-up. Combined with automatic intervention functionality, the system can issue risk warnings when high-stress states are detected, implement soothing interaction strategies when medium-stress states are detected, and maintain normal, friendly dialogue when low-stress states are detected, thereby improving psychological safety and user experience in real human-computer interactions.

[0043] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal information human-computer dialogue psychological stress identification and intervention method described in any one of the first aspects.

[0044] Fourthly, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal information human-computer dialogue psychological stress identification and intervention method described in any one of the first aspects.

[0045] The present invention will now be described in conjunction with specific embodiments.

[0046] Example 1

[0047] This invention proposes a method for identifying and intervening in human-computer dialogue psychological stress using a multimodal information-based human-computer dialogue psychological stress identification and intervention system, comprising: Human-computer dialogue is conducted using a pre-defined human-computer dialogue model and a human-computer dialogue strategy. The system acquires multimodal information of the test subject during human-computer interaction in real time; the multimodal information includes EEG signals, audio signals, and video signals. Feature extraction is performed on multimodal information to obtain multimodal feature data; the multimodal feature data includes: EEG feature representation, audio feature representation, and visual feature representation; The feature extraction process of the EEG feature representation of the EEG-language branch includes: using a 30 s sliding window, performing 1–50 Hz bandpass filtering and ICA artifact removal on the original EEG and then inputting it into a three-layer MLP network to obtain a deep EEG feature vector. The deep EEG feature vector is then compressed and refined using the Q-Former model to obtain the EEG feature representation. The feature extraction process for audio feature representation in the audio-language branch includes: using the wav2vec2 model to extract features from the audio signal to obtain a deep speech feature vector, and then using the Q-Former model to compress and refine the deep audio feature vector to obtain the audio feature representation; The feature extraction process for visual feature representation in the video-language branch includes: The video signal is extracted frame by frame, and the VIT model is used for feature extraction to obtain a depth visual feature vector. The depth visual feature vector is then compressed and refined using the Q-Former model to obtain a visual feature representation. Multimodal feature data, namely EEG feature representation, audio feature representation, and visual feature representation, are mapped to obtain a multimodal fusion answer space. Specifically, the EEG feature representation, audio feature representation, and visual feature representation are cascaded and then spatially mapped through a shared Linear mapping layer to obtain the multimodal fusion answer space. Using a predefined large model, psychological stress is identified in the multimodal fusion answer space based on preset prompt words to obtain the real-time psychological stress level of the test subjects; Specifically, the multidimensional fusion representation is input into the large language model, and the output real-time psychological stress level is divided into three categories: low, medium and high stress. Among them, the psychological stress classification threshold can be adaptively adjusted according to the norms of different populations. The human-computer dialogue strategy of the human-computer dialogue model is adaptively adjusted according to the real-time psychological stress level. The adjustments to the human-computer dialogue strategy include: dynamic adjustment of dialogue content and personalized psychological intervention output. In the dynamic adjustment of dialogue content, for low-stress situations, informational and task-oriented interactions are maintained to improve efficiency; for medium-stress situations, emotionally supportive expressions are added, and the pace of speech is appropriately slowed; for high-stress situations, soothing dialogue templates are used to reduce cognitive load and negative stimuli. In the personalized psychological intervention output, 3-minute breathing relaxation guidance, brief emotional stabilization prompts, reminders for the test subject to take a short break, or reminders for severe stress levels to seek professional help can be used. The corresponding operation is performed based on the adjusted human-computer dialogue strategy using the human-computer dialogue model.

[0048] In this embodiment, the multimodal information acquisition process involves capturing the frontal / whole-brain EEG signals of the subject in real time using a portable EEG acquisition module; acquiring a facial video stream at over 30 fps using a high-definition camera; and simultaneously acquiring speech / environmental audio using a directional microphone array. It is important to note that the multimodal information is acquired synchronously, using millisecond-level clocks for alignment and a unified timestamp.

[0049] To verify the effectiveness of the present invention, this embodiment conducted a total of 100 experiments on 50 students in two schools. In each experiment, the students conducted a complete interview dialogue with the human-computer dialogue psychological stress identification and intervention system based on multimodal information proposed in this invention. The data were averaged to ensure the scientificity and reliability of the results.

[0050] Table 1

[0051] This embodiment of the experiment used the PSS-10 scale for comparison, and the PSS-10 scores were used to preliminarily classify psychological stress levels: low stress (0~13 points), medium stress (14~26 points), and high stress (27~40 points). The PSS-10 scale is shown in Table 1. The results of the human-computer dialogue psychological stress recognition are shown in Table 2.

[0052] Table 2

[0053] As can be calculated from Table 2, the accuracy rate of the psychological stress recognition result of human-computer dialogue in this embodiment is 84%.

[0054] The psychological intervention effect of this system was further evaluated in 100 experiments involving the aforementioned 50 students. The intervention module was automatically triggered after identifying moderate / high stress, and the subjective stress level of the subjects was measured using the PSS-10 scale before and after the intervention. The experimental results are shown in Table 3.

[0055] Table 3

[0056] As can be seen from Table 3, the psychological intervention module in this invention can effectively alleviate the psychological stress of the tested subjects, verifying the feasibility of the integrated identification-intervention technical route.

[0057] This invention, through a low-cost, scalable multimodal acquisition scheme and an end-to-end deep learning fusion and classification framework, not only significantly improves the accuracy and robustness of psychological stress identification while maintaining portable deployment, but also enables real-time personalized intervention. It can be widely applied to online mental health screening, intelligent customer service emotion monitoring, remote medical follow-up, and other scenarios requiring psychological stress regulation.

[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information, characterized in that, include: Human-computer dialogue is conducted using a pre-defined human-computer dialogue model and a human-computer dialogue strategy. The system acquires EEG, audio, and video signals of the tested subject during human-computer interaction in real time; the EEG, audio, and video signals are acquired synchronously with unified timestamps. Feature extraction is performed on EEG signals, audio signals, and video signals to obtain EEG feature representations, audio feature representations, and visual feature representations. The EEG feature representations, audio feature representations, and visual feature representations are cascaded by channel to form a high-dimensional fusion vector. The EEG feature representations, audio feature representations, and visual feature representations are spatially mapped through a linear mapping layer to form a multimodal fusion answer space. Psychological stress is identified by multimodal fusion answer space to obtain the real-time psychological stress level of the test subjects; the psychological stress level is divided into three levels: low stress, medium stress, and high stress. The human-computer dialogue strategy of the human-computer dialogue model is adaptively adjusted according to the real-time psychological stress level. The corresponding operation is performed based on the adjusted human-computer dialogue strategy using the human-computer dialogue model.

2. The method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information according to claim 1, characterized in that, The strategy for adaptively adjusting the current human-computer dialogue based on real-time psychological stress levels includes: When the real-time psychological stress level is low, use normal task-oriented dialogue strategies. When the real-time psychological stress level is moderate, a relief-type response is selected and output from the preset low-load dialogue template library; among them, relief-type responses include breathing guidance, attention-shifting topics, emotional empathy statements or simplified task prompts; When the real-time psychological stress level is high, the current task-oriented dialogue is automatically paused and switched to pure companionship mode, psychological comfort mode, or the subject is proactively advised to take a short rest. At the same time, the real-time psychological stress level of the test subject is dynamically injected into the system prompts of the human-computer dialogue model.

3. The method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information according to claim 2, characterized in that, The templates for responses designed to mitigate stress are annotated by psychological experts and optimized through controlled experiments and A / B testing. Before output, the large model rewrites the responses in real time based on the individual characteristics of the test subjects.

4. The method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information according to claim 1, characterized in that, The feature extraction process of EEG feature representation includes: The EEG signal is segmented using a sliding window of preset duration to obtain multiple EEG signal sample segments. After bandpass filtering and ICA artifact removal, the multiple EEG signal sample segments are input into a multilayer perceptron (MLP) to obtain deep EEG feature vectors. The deep EEG feature vectors are then compressed and refined using a Q-Former model to obtain EEG feature representations.

5. The method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information according to claim 1, characterized in that, In the process of extracting audio feature representation, the wav2vec2 model is used to extract features from the audio signal to obtain a deep speech feature vector, and the Q-Former model is used to compress and refine the deep audio feature vector to obtain the audio feature representation.

6. The method for identifying and intervening in psychological stress through human-computer dialogue based on multimodal information according to claim 1, characterized in that, In the process of visual feature extraction, the VIT model is used to extract features from the video signal to obtain a depth visual feature vector; and the Q-Former model is used to compress and refine the depth visual feature vector to obtain a visual feature representation.

7. A human-computer dialogue system for identifying and intervening in psychological stress based on multimodal information, characterized in that, include: The human-computer dialogue module is used to conduct human-computer dialogue based on a preset human-computer dialogue model and a human-computer dialogue strategy. The acquisition module is used to acquire the EEG signals, audio signals, and video signals of the tested object during the human-computer dialogue process; The feature extraction module is used to extract features from EEG signals, audio signals, and video signals respectively, and obtain EEG feature representations, audio feature representations, and visual feature representations. The fusion module is used to cascade EEG feature representations, audio feature representations, and visual feature representations by channel to form a high-dimensional fusion vector; and to spatially map EEG feature representations, audio feature representations, and visual feature representations through a linear mapping layer to form a multimodal fusion answer space. The recognition module is used to identify psychological stress in the multimodal fusion answer space to obtain the real-time psychological stress level of the test subject. The psychological intervention module is used to adaptively adjust the human-computer dialogue strategy of the human-computer dialogue model based on the real-time psychological stress level. The human-computer dialogue module is used to perform corresponding operations based on the adjusted human-computer dialogue strategy using a preset human-computer dialogue model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the human-computer dialogue psychological stress identification and intervention method based on multimodal information as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the human-computer dialogue psychological stress identification and intervention method based on multimodal information as described in any one of claims 1-6.

Images