Cognitive training emotion interaction method and system based on large model

Abstract

The invention discloses a cognitive training emotion interaction method and system based on a large model. The method comprises the following steps: acquiring a cognitive evaluation result of a patient, and pushing a corresponding cognitive training scheme to the patient; before cognitive training, acquiring the current emotional psychological state of the user to judge whether the cognitive training requirement is met or not, and if not, performing emotion regulation until the cognitive training requirement is met; during cognitive training, acquiring multi-modal data of the patient and inputting the multi-modal data into a preset large model to perform emotion calculation, so as to identify the current emotional psychological state of the patient again; and based on the current emotional psychological state of the patient, carrying out interaction feedback of a corresponding mode on the patient so as to carry out personalized emotional interaction with the patient. According to the method, the emotional state of the user is recognized in advance before cognitive training, emotion feedback of different modes can be performed according to the current emotional psychological state of the patient in the cognitive training, so that the training compliance of the patient is improved, and the cognitive training effect is improved.

Description

Technical Field

[0001] The present invention relates to a cognitive training emotional interaction method based on a large model, and also relates to a corresponding cognitive training emotional interaction system, belonging to the technical field of healthcare informatics. Background Art

[0002] There are various ways to express emotions, and their carriers are extremely rich. Different from other psychological processes, specific behavioral manifestations are usually accompanied during the emotional activity process. Among many behavioral manifestations, speech intonation, facial expressions, and body movements are key modal information for emotion recognition, and good results have been achieved in the field of emotion recognition.

[0003] The cognitive rehabilitation of patients with cognitive impairment is a long and complex process. It is difficult to meet the needs of patients relying only on limited medical resources. Therefore, more community and family resources need to be utilized for assisted rehabilitation. However, the existing computerized cognitive training programs have obvious limitations. On the one hand, the training programs cannot be adaptively updated according to the actual situation of patients; on the other hand, the training process only relies on one-way feedback from the device to the patient and lacks emotional interaction support. This results in a poor experience and low compliance for patients during cognitive function rehabilitation training, ultimately affecting the effect of rehabilitation training.

[0004] Therefore, how to provide a cognitive rehabilitation training method for patients that is applicable to both hospital and home environments, can improve the patient experience, enhance compliance, and effectively promote rehabilitation has become a key clinical and technical problem to be solved urgently. Summary of the Invention

[0005] The primary technical problem to be solved by the present invention is to provide a cognitive training emotional interaction method based on a large model.

[0006] Another technical problem to be solved by the present invention is to provide a cognitive training emotional interaction system based on a large model.

[0007] To achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] According to the first aspect of the embodiments of the present invention, a cognitive training emotional interaction method based on a large model is provided, including the following steps:

[0009] Obtain the cognitive assessment results of the patient and push the corresponding cognitive training program to the patient;

[0010] Before conducting cognitive training, identify the emotional and psychological state of the patient in advance to obtain the current emotional and psychological state of the patient;

[0011] Determine whether the patient's current emotional and psychological state meets the training requirements; if it meets, directly conduct cognitive training; if it does not meet, adjust the patient's emotions until the patient's emotional and psychological state meets the training requirements;

[0012] Obtain the multi-modal data of the patient during cognitive training based on the cognitive training plan; wherein, the multi-modal data at least includes: text information, visual information, auditory information, and tactile information;

[0013] Input the multi-modal data into a preset large model to perform emotional calculation on the multi-modal data through the large model, so as to re-identify the patient's current emotional and psychological state;

[0014] Based on the patient's current emotional and psychological state, conduct corresponding modal interaction feedback on the patient to conduct personalized emotional interaction with the patient;

[0015] Among them, different emotional and psychological states correspond to different modal interaction feedbacks, and different modal interaction feedbacks correspond to different feedback combination forms; the feedback combination form is freely composed of multiple emotional feedback methods.

[0016] Among them, preferably, the patient's current emotional and psychological state includes: positive state, neutral state, and negative state; and before cognitive training, when the patient's current emotional and psychological state is in a positive state, the training requirements are met;

[0017] During cognitive training, when the patient's current emotional and psychological state is in a positive state, conduct at most dual-modal interaction feedback on the patient in a way that combines auditory feedback and text feedback; and, in order of priority: the auditory feedback > the text feedback;

[0018] During cognitive training, when the patient's current emotional and psychological state is in a neutral state, conduct at most triple-modal interaction feedback on the patient in a way that combines auditory feedback, visual feedback, and text feedback; and, in order of priority: the auditory feedback > the visual feedback > the text feedback;

[0019] During cognitive training, when the patient's current emotional and psychological state is in a negative state, conduct at most quadruple-modal interaction feedback on the patient in a way that combines auditory feedback, visual feedback, text feedback, and tactile feedback; and, in order of priority: the auditory feedback > the visual feedback > the text feedback > the tactile feedback.

[0020] Among them, preferably, the patient's current emotional and psychological state is identified by the following method:

[0021] Obtain cognitive training data related to the cognitive training task based on the text information, so as to identify the patient's cognitive training level through the large model;

[0022] Obtain the facial expression and gesture data of the patient during cognitive training collected in real time by the camera device based on the visual information, so as to identify the patient's negative emotions and the cumulative duration of non-touch gestures through the large model;

[0023] Obtain the audio information of the patient during the entire training process based on the auditory information, so as to identify the patient's emotional state through the large model;

[0024] Based on the tactile information, identify the touch pressure value and the number of clicks of the patient during the cognitive training process through the large model;

[0025] Comprehensively output the current emotional and psychological state of the patient according to the patient's cognitive training level, the patient's negative emotions and the cumulative duration of non-touch gestures, the patient's emotional state, and the patient's touch pressure value and the number of clicks.

[0026] Preferably, the text feedback includes: when the current emotional and psychological state of the patient is a negative state, ask the patient about their feelings in words and give positive encouragement and guidance to the patient; when the current emotional and psychological state of the patient is a positive state or a neutral state, give neutral feedback to the patient in words and give affirmation;

[0027] The visual feedback includes: when the current emotional and psychological state of the patient is a negative state, adjust the patient's emotions through video animations or images; when the current emotional and psychological state of the patient is a positive state or a neutral state, give neutral feedback to the patient through video animations or images and give affirmation;

[0028] The auditory feedback includes: when the current emotional and psychological state of the patient is a negative state, adjust the patient's emotions in the form of voice playback; when the current emotional and psychological state of the patient is a positive state or a neutral state, give neutral feedback to the patient in the form of voice playback and give affirmation;

[0029] The tactile feedback includes: when the current emotional and psychological state of the patient is a negative state, give tactile feedback to the patient by vibrating the touch screen device according to the rhythm of the voice; when the current emotional and psychological state of the patient is a positive state or a neutral state, no tactile feedback is given.

[0030] Preferably, when the current emotional and psychological state of the patient is a negative state, obtain the current emotional level n of the patient; where n is a positive integer and the smaller the value, the more negative the emotion.

[0031] If the n value is within the first preset interval, continuously perform emotional regulation on the patient and abort the cognitive training; and, the feedback combination form of the psychological regulation is a four-modal interactive feedback combining auditory feedback, visual feedback, text feedback, and tactile feedback.

[0032] If the n value is within the second preset interval, perform cognitive training and emotional regulation simultaneously; and, the feedback combination form of the psychological regulation is a three-modal interactive feedback combining auditory feedback, visual feedback, and tactile feedback.

[0033] If the n value is within the third preset interval, perform cognitive training and emotional regulation simultaneously; and, the feedback combination form of the psychological regulation is a two-modal interactive feedback combining auditory feedback and visual feedback.

[0034] If the n value is within the fourth preset interval, perform cognitive training and emotional regulation simultaneously; and, the feedback combination form of the psychological regulation is a two-modal interactive feedback combining auditory feedback and text feedback.

[0035] If the n value is within the fifth preset interval, perform cognitive training and emotional regulation simultaneously; and, the feedback combination form of the psychological regulation is a two-modal interactive feedback combining auditory feedback and tactile feedback.

[0036] Among them, sorted by numerical size: the first preset interval < the second preset interval < the third preset interval < the fourth preset interval < the fifth preset interval.

[0037] Preferably, when the cognitive training scenario of the patient is a home scenario, the auditory information is the audio information of the patient himself and his family members during the entire training process collected by a voice collection device.

[0038] When the emotional states of the patient and his family members are negative, play voice content that empathizes with the patient and his family members to perform positive regulation on the patient and his family members; when the emotional states of the patient and his family members are positive or neutral, give neutral feedback to the patient and his family members and give affirmation.

[0039] Among them, the priority of emotional feedback to the patient is greater than the priority of emotional feedback to the family member.

[0040] Preferably, when the cognitive training scenario of the patient is a hospital scenario, the auditory information is the audio information of the patient himself and medical staff during the entire training process collected by a voice collection device.

[0041] When the emotional state of the patient is negative, positive adjustment is performed on the patient by playing voice content that empathizes with the patient; when the emotional state of the patient is positive or neutral, neutral feedback is given to the patient and affirmation is provided.

[0042] When the emotional state of the medical staff is negative, positive adjustment is performed on the medical staff by playing voice content that empathizes with the medical staff.

[0043] Among them, the priority of providing emotional feedback to the patient is higher than that of providing emotional feedback to the medical staff.

[0044] Preferably, when the patient is undergoing cognitive training in a home or hospital scenario, the emotional state of the family member or medical staff during the current training task is obtained.

[0045] If the emotional state of the family member or medical staff during the current training task is negative, after the current training task ends, the emotional adjustment of the family member or medical staff is preferentially performed until the emotional state of the family member or medical staff improves to neutral or positive; and, after the current training task ends, the patient is asked through human-computer interaction whether they have been affected.

[0046] If the patient is not affected, the next training task is directly started; if the patient is affected, the emotional adjustment of the patient is performed until the emotional state of the patient improves to neutral or positive, and then the next training task is started.

[0047] Preferably, the large model is constructed in the following manner:

[0048] The multi-modal data of the historical patient group is obtained; among them, the multi-modal data at least includes: text information, visual information, auditory information, and tactile information.

[0049] The multi-modal data of the historical patient group is input into a general large model to train and generate a vertical large model for emotional interaction with cognitive disorder patients, so as to use the vertical large model for real-time feature extraction, decoding, and analysis to determine the emotional type of the historical patient in the previous task state.

[0050] According to the different emotional types of different historical patients, the vertical large model is trained again to output the different emotional psychological states of different historical patients, and then the large model is constructed.

[0051] According to the second aspect of the embodiments of the present invention, a cognitive training emotional interaction system based on a large model is provided, including:

[0052] A cognitive assessment unit for cognitively assessing a patient to obtain the patient's cognitive assessment result;

[0053] An emotion regulation unit for, before cognitive training, pre-identifying the emotional and psychological state of the patient to obtain the patient's current emotional and psychological state; and judging whether the patient's current emotional and psychological state meets the training requirements; if it meets, directly conduct cognitive training, if it does not meet, regulate the patient's emotions until the patient's emotional and psychological state meets the training requirements;

[0054] A cognitive training unit connected to the cognitive assessment unit to push a corresponding cognitive training plan to the patient according to the patient's cognitive assessment result, thereby being used for cognitive interaction;

[0055] A multi-modal data acquisition unit connected to the cognitive training unit to obtain multi-modal data of the patient during cognitive training based on the cognitive training plan; wherein, the multi-modal data at least includes: text information, visual information, auditory information, and tactile information;

[0056] An affective computing unit pre-set with a large model and connected to the multi-modal data acquisition unit to perform affective computing on the multi-modal data, thereby identifying the patient's current emotional and psychological state;

[0057] An affective interaction unit connected to the affective computing unit to perform different-modal interaction feedback on the patient based on the patient's current emotional and psychological state to perform personalized affective interaction with the patient;

[0058] Among them, the different-modal interaction feedback corresponds to different feedback combination forms, and the feedback combination forms are freely combined by four emotion feedback methods: text feedback, visual feedback, auditory feedback, and tactile feedback.

[0059] Compared with the prior art, the present invention has the following technical effects:

[0060] (1) Before the patient's cognitive training, it is possible to pre-identify the patient's emotions. If the emotional state is negative, the patient's emotions are preferentially regulated to ensure that the patient enters cognitive training with a positive or neutral emotional state, avoiding affecting the cognitive training effect due to the emotional state before training.

[0061] (2) During the patient's cognitive training, multi-modal data fusion is performed through text, vision, audition, and touch to perform emotional recognition on the patient from multiple aspects, and thus different emotional feedbacks are performed according to the patient's current emotional and psychological state to help the patient regulate emotions during cognitive training, thereby greatly improving the patient's training compliance and being beneficial to enhancing the patient's cognitive training effect.

[0062] (3) For the two scenarios of home and hospital, by identifying and interacting with family members or medical staff in different ways, different forms of emotional interaction are carried out in different scenarios to help patients improve the effect of cognitive training. Moreover, it is able to identify the emotions of family members or medical staff, so as to adjust their emotions when their emotional state is negative, to avoid the emotional state of family members or medical staff affecting the patients. In addition, when the patients are affected by the emotions of family members or medical staff, their emotions will be adjusted again.

[0063] (4) The preset large model has stronger generalization ability, which is conducive to reducing the model iteration frequency and shortening the response time, so as to perform real-time interactive calculations based on the massive information of multi-modal data and enhance the emotional interaction experience of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1A It is the overall flowchart of a cognitive training emotional interaction method based on a large model provided by the first embodiment of the present invention;

[0065] Figure 1B It is the flowchart of pre-identifying the emotions of patients in the first embodiment of the present invention;

[0066] Figure 2 It is the schematic diagram of auditory feedback in the home scenario in the first embodiment of the present invention;

[0067] Figure 3 It is the schematic diagram of auditory feedback in the hospital scenario in the first embodiment of the present invention;

[0068] Figure 4 It is the schematic diagram of emotional calculation of the large model in the first embodiment of the present invention;

[0069] Figure 5 It is the flowchart of emotional regulation for negative emotions in the first embodiment of the present invention;

[0070] Figure 6 It is the structural diagram of a cognitive training emotional interaction system based on a large model provided by the second embodiment of the present invention;

[0071] Figure 7 It is the structural diagram of a cognitive training emotional interaction system based on a large model provided by the third embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] The technical content of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] An embodiment of the present invention aims to provide a cognitive training emotional interaction method based on a large model. By identifying and collecting multimodal data information generated by patients during training (including training task scores, facial expression information, voice information, and limb movement information such as touch screen pressure, click times, and frequencies during the answering process), and transmitting the multimodal data to the large model in the cloud through a network for emotional calculation. Thus, the current emotional and psychological state of the patient can be accurately identified, and through an interactive tablet device, multimodal information feedback matching different emotional states of the patient, such as positive and negative emotions, can be given. Among them, the feedback content forms include but are not limited to text, voice comfort, video guidance, device vibration, etc., so as to support the emotional interaction of the patient during the acceptance of this rehabilitation training mode, and improve the training compliance, effect, and human-computer interaction experience.

[0074] The first embodiment

[0075] As Figure 1A shown, a cognitive training emotional interaction method based on a large model provided by the first embodiment of the present invention specifically includes the following steps:

[0076] S1: Obtain the cognitive assessment results of the patient and push the corresponding cognitive training plan to the patient.

[0077] It can be understood that corresponding cognitive training plans need to be obtained according to different patients first, so as to be able to carry out cognitive training emotional interaction with the patient during cognitive training.

[0078] Therefore, in this embodiment, it is necessary to first conduct a cognitive assessment and a personality assessment on the patient to obtain the patient's cognitive level and personality type. Preferably, the patient's demographic information can also be used, such as personal information such as age, education level, gender, and native place; and, in combination with the patient's personal medical history and family history, comprehensively determine the possibility of improving the patient's cognitive ability. Finally, a personalized cognitive training plan is comprehensively output and pushed to the patient for cognitive training.

[0079] In addition, it should be understood that this step S1 is not a specific step of cognitive training emotional interaction, but a pre-step before cognitive training emotional interaction.

[0080] S2: Before cognitive training, pre-identify the patient's emotions and feelings.

[0081] Specifically, after the cognitive assessment is completed, according to the patient's choice, cognitive rehabilitation training is carried out in the home or hospital setting. The cognitive rehabilitation training needs to be carried out in a quiet environmental space and can be guided by a medical staff or family member. The patient's cognitive rehabilitation training mainly uses a mobile tablet. According to medical orders and expert consensus, the training is carried out at least 3 times a week, with each training session lasting at least 30 minutes. For people over 60 years old, each training session lasts at least 50 minutes, and for people under 60 years old, each training session lasts at least 25 minutes.

[0082] As Figure 1B shown, after the patient logs in and activates the account each time they turn on the device and before entering cognitive training, an identification assessment of the emotional state is first carried out. The assessment form is a voice interaction consultation and answering questions through screen options. That is, the system asks the patient through voice whether their current emotion is positive, neutral, or negative, and different emotional state expressions or animations appear on the screen. When the patient gives a voice feedback and selects the expression or animation representing their emotional state, at the same time, the system captures the patient's voice, facial expression, and touch screen tactile information in real time based on the camera, and comprehensively analyzes the above several identified information to obtain the patient's current emotional state.

[0083] If it is identified that the patient's current emotional and psychological state is negative, they will not enter cognitive training. Instead, emotional regulation and counseling will be carried out first until the system identifies that the patient's emotion has turned neutral or positive, and then the cognitive training process will be entered.

[0084] S3: Obtain the multimodal data of the patient during cognitive training based on the cognitive training plan.

[0085] In this embodiment, the multimodal data at least includes: text information, visual information, auditory information, and tactile information.

[0086] Next, the acquisition methods of each piece of information will be described separately:

[0087] (1) Text information

[0088] In this embodiment, the text information is cognitive training data related to the cognitive training task. Among them, based on this cognitive training data, the cognitive training level of the patient can be identified.

[0089] For example: If the score of the patient's previous training task is lower than the norm average level, it means that the task is relatively difficult for the patient, and emotional regulation feedback needs to be initiated. On the contrary, if the score of the patient's previous training task is higher than the norm average level, it indicates that the patient can easily handle the training task.

[0090] In addition, it can be understood that the cognitive training data can include not only the cognitive training score but also data information such as reaction duration and error rate, so as to comprehensively judge the patient's cognitive training level.

[0091] (2) Visual information

[0092] In this embodiment, the visual information is the facial expression and gesture data of the patient collected in real time by the imaging device. Among them, the negative emotions of the patient can be recognized according to the facial expression, and the cumulative duration of non-touch gestures of the patient can be recognized according to the gesture data.

[0093] Specifically, an open-source expression and gesture recognition software is used to capture and analyze the facial expression and gesture data of the patient during the entire training process in real time, so as to identify whether the facial expression of the patient is a negative expression and identify the cumulative duration of non-touch gestures.

[0094] (3) Auditory information

[0095] In this embodiment, the auditory information is the audio information of the patient himself, medical staff and family members during the entire training process collected by the voice acquisition device. The voice subject can be judged by using the audio information, and moreover, the emotional state of the voice subject can be recognized according to the speech semantics of the audio information, and the emotional state includes: positive, neutral and negative.

[0096] Specifically, the cognitive training emotional interaction method is mainly applied to home scenes or hospital scenes, and the audio information of different personnel needs to be collected in different scenes. For example Figure 2 As shown, when the cognitive training scene of the patient is a home scene, the auditory information is the audio information of the patient himself and family members during the entire training process collected by the voice acquisition device. As Figure 3 shown, when the cognitive training scene of the patient is a hospital scene, the auditory information is the audio information of the patient himself and medical staff during the entire training process collected by the voice acquisition device.

[0097] Moreover, in an embodiment of the present invention, an open-source speech recognition software is used to capture and analyze the audio information of the patient himself, family members and medical staff during the entire training process. First, by extracting and analyzing the speech information in the audio file, the voice subject is distinguished and analyzed to determine whether the voice subject is the patient himself, the patient's family member or the medical staff. Then, according to the audio information, the speech and semantics are extracted and analyzed to determine the emotional state of the voice subject.

[0098] It should be noted that for auditory information, not only the emotional state of the patient himself is concerned, but also the emotional state of the patient's family members (home scene), and the emotional state of the medical staff (hospital scene). Thus, emotional monitoring is carried out from multiple aspects to achieve the best emotional interaction effect. And when performing emotional regulation, the priority of emotional feedback to the patient is the highest, followed by the emotional regulation of family members or medical staff.

[0099] (4) Tactile information

[0100] In this embodiment, the tactile information is the touch pressure value and the number of clicks of the patient during the cognitive training process collected in real time by the touch screen controller.

[0101] It can be understood that the greater the touch pressure value and the more the number of clicks on the same target, the more unstable the patient's mood is, and the patient needs emotional regulation. Among them, the touch pressure value and the number of clicks can be collected through the touch screen controller of the human-computer interaction device (such as: tablet computer).

[0102] It can be understood that in this embodiment, only the above four data types form multi-modal data for subsequent emotion analysis. In other embodiments, other types of data can be added or reduced, and specific adaptive adjustments can be made according to needs.

[0103] S4: Perform emotion calculation on the multi-modal data through a large model to identify the current emotional and psychological state of the patient.

[0104] In this embodiment, a large model needs to be constructed in advance. After obtaining the multi-modal data based on step S2, the multi-modal data is input into the preset large model to perform emotion calculation on the multi-modal data through the large model, so as to identify the current emotional and psychological state of the patient.

[0105] Specifically, in this embodiment, the large model is constructed in the following way:

[0106] S41: Obtain the multi-modal data of the historical patient group.

[0107] In this embodiment, the multi-modal data of the historical patient group (including multiple historical patients) is obtained according to historical data. Specifically, the multi-modal data is multi-modal emotion and sentiment data such as the original scores of the cognitive behaviors of historical patients, task response times, gesture behaviors, facial expressions, speech and semantics, tactile pressure values, click frequencies, etc.

[0108] S42: Train and generate a vertical large model for emotional interaction with cognitive disorder patients.

[0109] In this embodiment, with the general large model as the base, after obtaining the multi-modal data of the historical patient group based on step S31, the data is input into the general large model for model training, so as to train and form a vertical large model for emotional interaction with cognitive disorder patients.

[0110] It is understandable that the vertical large model in this embodiment is a large model formed for the field of cognitive impairment. Through this vertical large model, real-time feature extraction, decoding, and analysis can be performed on historical patients to determine the emotional type of historical patients in the previous task state.

[0111] S43: Retrain the model to construct and form a large model.

[0112] Specifically, in this embodiment, after the vertical large model is formed, the vertical large model is used to output the emotional types of each historical patient in the previous task state. Then, according to the different emotional types of different historical patients, the vertical large model is retrained again to output different emotional and psychological states for different patients, and then the large model is constructed and formed. It should be understood that different emotional and psychological states correspond to different modal interaction feedbacks.

[0113] Refer to Figure 4 As shown, the large model in this embodiment performs comprehensive emotion calculation according to the data types of multi-modal data. After obtaining text information, visual information, auditory information, and tactile information based on the above step S3, the current emotional and psychological state of the patient is output through model calculation.

[0114] S5: Based on the current emotional and psychological state of the patient, perform different modal interaction feedbacks to conduct personalized emotional interaction with the patient.

[0115] In this embodiment, different modal interaction feedbacks correspond to different feedback combination forms. The feedback combination form is freely composed of four emotional feedback methods: text feedback, visual feedback, auditory feedback, and tactile feedback.

[0116] Next, the four emotional feedback methods will be described in detail:

[0117] (1) Text feedback

[0118] After obtaining the patient's cognitive training level based on step S3, the patient's cognitive training level is compared with the preset cognitive training level (for example: norm training level). If the patient's cognitive training level is lower than the preset cognitive training level, it means that the cognitive training task is difficult for this patient. At this time, the first interaction instruction of text feedback needs to be activated. Specifically, give feedback by asking about feelings and positive encouragement through text, "Do you think the task is very difficult? It doesn't matter. Many peers are also stuck on this task! Similar feedback expressing empathy, normalization, etc."; otherwise, give neutral feedback to the patient through text and give affirmation.

[0119] (2) Visual feedback

[0120] After obtaining the patient's negative emotions and the cumulative duration of non-touch gestures based on step S3, if the proportion of the patient's negative emotions exceeds the first threshold (30% in this embodiment) and reaches the preset monitoring duration (30s in this embodiment), then it is necessary to initiate the second interaction instruction of visual feedback to regulate the patient's emotions in the form of video playback; otherwise, provide neutral feedback to the patient through the video and give affirmation.

[0121] Similarly, when the cumulative duration of the patient's non-touch gestures exceeds the second threshold (30s in this embodiment), then initiate the second interaction instruction of visual feedback to regulate the patient's emotions in the form of video playback; otherwise, provide neutral feedback to the patient through the video and give affirmation.

[0122] (3) Auditory feedback

[0123] After determining the sound subject based on the audio information in step S3 and recognizing the emotional state of the sound subject according to the speech semantics of the audio information, when the emotional state of the sound subject is negative, then initiate the third interaction instruction of auditory feedback to regulate the patient's emotions in the form of voice playback; otherwise, provide neutral feedback to the patient through the voice and give affirmation.

[0124] For example: Recognize the patient himself / herself, and judge whether his / her current emotional state is positive, neutral, or negative. If negative emotional characteristics such as sighing, fluctuating tone beyond the normal voice threshold are recognized in the patient, then it is prompted that emotional feedback needs to be initiated to provide the patient with encouragement, comfort, empathy (understanding of the current emotion), and normalization (normal understanding of the event affecting the current emotion) feedback support;

[0125] Another example: If the voice information of medical staff is recognized, then based on its voice and semantics, feature extraction and analysis are carried out to judge whether the current feedback on the patient's training needs support is positive (related to encouragement and praise), neutral (routine guidance), or negative (related to blame and criticism). If the medical staff information is recognized as negative, then initiate the feedback mechanism for regulating the negative emotions of medical staff to appropriately soothe the emotional psychology of medical staff.

[0126] Still another example: If the voice information of the caregiver is recognized, then based on its voice and semantics, feature extraction and analysis are carried out to judge whether the current feedback on the patient's training needs support is positive (related to encouragement and praise), neutral (routine guidance), or negative (related to blame and criticism). If the caregiver's voice information is recognized as negative, then initiate the emotional regulation feedback mechanism for the caregiver to soothe the current emotional psychology of the caregiver, expressing empathy, understanding, support, and empowerment; if the voice information is positive or neutral, then provide encouraging words and empowerment.

[0127] (4) Tactile feedback

[0128] After the touch pressure value and the number of clicks of the patient during the cognitive training are collected based on the tactile information in step S3, if the touch pressure value of the patient exceeds the preset pressure threshold, or the number of clicks on the same target exceeds 3 times, the fourth interaction instruction for tactile feedback is activated, so as to perform tactile feedback on the patient by vibrating according to the rhythm of the voice through the touch screen device.

[0129] In this embodiment, different emotional and psychological states correspond to different modalities of interaction feedback, and different modalities of interaction feedback correspond to different feedback combination forms. The feedback combination form is freely composed of a variety of emotional feedback methods (in this embodiment, the above four emotional feedback methods of auditory feedback, visual feedback, text feedback, and tactile feedback). Moreover, the degrees of significance of the four emotional feedback methods for evoking the patient's emotions are different, that is: sound has the highest arousal degree, followed by images, text ranks third, and touch ranks fourth. Therefore, they are sorted according to priority, auditory feedback > visual feedback > text feedback > tactile feedback.

[0130] The logic of the specific feedback is as follows:

[0131] (1) If the system recognizes that the emotional state of the patient after the previous task is a positive emotion (emotional level is n (n≥1)), then providing a dual-modal emotional interaction of sound + text is sufficient. For example, artistic text "You are wonderful" appears on the screen, or there is only a voice prompt, or the three combination forms of corresponding voice output are configured at the same time, and the sound is preferred in the three feedback forms.

[0132] (2) If the system recognizes that the emotional state of the patient in the previous task is a neutral emotion (emotional level is n (n = 0)), then providing a triple-modal emotional interaction of sound + image + text. For example, a cartoon animation similar to "You are wonderful. If you need to rest or adjust yourself, please tell me" (video duration 5s) appears on the screen, and corresponding text and voice outputs are configured at the same time.

[0133] (3) If the system recognizes that the emotional state of the patient after the previous task is a negative emotion (emotional level is n, n≤ -1), the smaller the n value, the more negative the emotion. Specifically, as Figure 5 shown, the emotional regulation method for negative emotions in this embodiment is as follows:

[0134] ① If the n value is in the first preset interval (n≤ -10), then continuously perform emotional regulation on the patient and abort the cognitive training. Moreover, the feedback combination form of this psychological regulation is a quadruple-modal interaction feedback combining auditory feedback, visual feedback, text feedback, and tactile feedback.

[0135] ② If the value of n is within the second preset interval (-10 < n ≤ -7), cognitive training and emotional regulation are carried out simultaneously. Moreover, the feedback combination form of psychological regulation is a three-modal interactive feedback combining auditory feedback, visual feedback, and tactile feedback;

[0136] ③ If the value of n is within the third preset interval (-7 < n ≤ -5), cognitive training and emotional regulation are carried out simultaneously. Moreover, the feedback combination form of psychological regulation is a two-modal interactive feedback combining auditory feedback and visual feedback;

[0137] ④ If the value of n is within the fourth preset interval (-5 < n ≤ -3), cognitive training and emotional regulation are carried out simultaneously. Moreover, the feedback combination form of psychological regulation is a two-modal interactive feedback combining auditory feedback and text feedback;

[0138] ⑤ If the value of n is within the fifth preset interval (-3 < n ≤ -1), cognitive training and emotional regulation are carried out simultaneously. Moreover, the feedback combination form of psychological regulation is a two-modal interactive feedback combining auditory feedback and tactile feedback.

[0139] For example: when the patient's emotion n = -15, provide a four-modal emotional interaction of sound + image + text + touch. The specific feedback content is: a video animation of the image of a psychological counselor appears on the screen, and corresponding voices, texts, and the touch screen device vibrates following the rhythm of the voice, such as "If you are not in a good state today, you can talk to me...". This process is guided by the actual interaction needs and has no time limit until the negative emotion value n of the patient obtained through the interaction > -10. Then ask if to continue the training. If yes, continue; otherwise, terminate today's training.

[0140] In addition, when the patient conducts cognitive training in a home or hospital scenario, obtain the emotional state of the family member or medical staff during the current training task. If the emotional state of the family member or medical staff is negative during the current training task, then after the current training task ends, give priority to regulating the emotion of the family member or medical staff until the emotional state of the family member or medical staff improves to neutral or positive; and, after the current training task ends, ask the patient through human-computer interaction whether they are affected. If the patient is not affected, directly start the next training task; if the patient is affected, then regulate the emotion of the patient until the emotional state of the patient improves to neutral or positive and then start the next training task.

[0141] Second Embodiment

[0142] Such as Figure 6As shown, based on the above first embodiment, a cognitive training emotional interaction system based on a large model according to a second embodiment of the present invention includes a cognitive evaluation unit 1, a cognitive training unit 2, a multimodal data acquisition unit 3, an emotion calculation unit 4, and an emotional interaction unit 5.

[0143] Among them,

[0144] The cognitive evaluation unit 1 is used to conduct a cognitive evaluation on the patient to obtain the patient's cognitive evaluation result. Specifically, the cognitive evaluation unit 1 can use a built-in scale for cognitive assessment or can conduct a task-based assessment through a preset assessment task. In addition, preferably, the cognitive evaluation unit 1 can also include a personality assessment module to conduct a personality assessment on the patient while conducting a cognitive evaluation.

[0145] The emotion regulation unit 6 is used to pre-identify the emotional and psychological state of the patient before cognitive training to obtain the patient's current emotional and psychological state. Among them, if the patient's current emotional and psychological state is positive or neutral, the training requirements are met, and thus cognitive training can be directly carried out. On the contrary, if the patient's current emotional and psychological state is negative, the training requirements are not met, and thus the patient needs to be emotionally regulated until the patient's emotional and psychological state improves to positive or neutral.

[0146] The cognitive training unit 2 is connected to the cognitive evaluation unit 1 to push a corresponding cognitive training plan to the patient according to the patient's cognitive evaluation result, so as to be used for cognitive interaction. It can be understood that the cognitive training unit 2 can form different cognitive training plans for different patients to meet the personalized needs of the patients, which is beneficial to improving the compliance of the patients' cognitive training.

[0147] The multimodal data acquisition unit 3 is connected to the cognitive training unit 2 to obtain multimodal data of the patient during cognitive training based on the cognitive training plan. Specifically, the multimodal data in this embodiment at least includes: text information, visual information, auditory information, and tactile information. The acquisition method of the multimodal data refers to the description of the above first embodiment and will not be elaborated here.

[0148] The emotion calculation unit 4 is preset with a large model 41 and is connected to the multimodal data acquisition unit 3 to be used for emotion calculation of the multimodal data, so as to identify the patient's current emotional and psychological state. Among them, the large model 41 is constructed in the manner of the above first embodiment and will not be elaborated here.

[0149] The emotional interaction unit 5 is connected to the emotional computing unit 4 to control the interaction device to execute different interaction instructions based on the patient's current emotional and psychological state, so as to perform personalized emotional interaction with the patient. Specifically, after calculating the patient's current emotional and psychological state through the four sub-computing modules respectively, different interaction instructions are executed according to different emotional calculation results, so as to match the patient's current emotional and psychological state to achieve the adjustment of the patient's emotions. Among them, the specific interaction instructions refer to the description in the above first embodiment and will not be elaborated here.

[0150] It can be understood that the above-mentioned each unit is a module structure for implementing each step of the above first embodiment, but is not limited to this structural form. In other embodiments, other types of module structures can also be adopted to implement each step in the above first embodiment.

[0151] Third Embodiment

[0152] As Figure 7 shown, on the basis of the above first embodiment, the third embodiment of the present invention further provides a cognitive training emotional interaction system based on a large model. The cognitive training emotional interaction system includes one or more processors 21 and a memory 22. Among them, the memory 22 is coupled to the processor 21 and is used to store one or more programs. When the program is executed by the processor 21, the processor 21 is enabled to implement the cognitive training emotional interaction method based on the large model as in the above embodiment.

[0153] Among them, the processor 21 is used to control the overall operation of the cognitive training emotional interaction system to complete all or part of the steps of the above cognitive training emotional interaction method based on the large model. The processor 21 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory 22 is used to store various types of data to support the operation of the cognitive training emotional interaction system. These data can include, for example, instructions for any application program or method operating on the cognitive training emotional interaction system, as well as data related to the application program. The memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, etc.

[0154] In an exemplary embodiment, the cognitive training emotional interaction system may be specifically implemented by a computer chip or entity, or by a product with certain functions, for executing the above-mentioned cognitive training emotional interaction method based on a large model and achieving the same technical effects as the above method. A typical embodiment is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0155] In another exemplary embodiment, the present invention also provides a computer-readable storage medium including program instructions, which, when executed by a processor, implement the steps of the cognitive training emotional interaction method based on a large model in any one of the above embodiments. For example, the computer-readable storage medium may be the above-mentioned memory including program instructions, and the above program instructions may be executed by the processor of the cognitive training emotional interaction system to complete the above-mentioned cognitive training emotional interaction method based on a large model and achieve the same technical effects as the above method.

[0156] In summary, the cognitive training emotional interaction method and system provided by the embodiments of the present invention have the following beneficial effects:

[0157] (1) During the cognitive training of patients, multi-modal data fusion is performed through text, vision, audition, and touch to perform emotional recognition on patients from multiple aspects, so as to perform different emotional feedbacks according to the current emotional and psychological state of the patients, thereby helping the patients adjust their emotions during cognitive training and further improving the cognitive training effect of the patients.

[0158] (2) For the two scenarios of home and hospital, different forms of emotional interaction are performed in different scenarios by performing different identifications and interaction feedbacks on family members or medical staff, so as to help patients improve the cognitive training effect.

[0159] (3) The preset large model has stronger generalization, which is beneficial to reducing the model iteration frequency and shortening the response time, so as to perform real-time interaction calculations based on the massive information of multi-modal data and improve the patient emotional interaction experience.

[0160] It should be noted that the above-mentioned multiple embodiments are only examples. The technical solutions of each embodiment can be combined, and all are within the protection scope of the present invention.

[0161] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality of" means two or more unless otherwise specifically defined.

[0162] The above has provided a detailed description of the cognitive training emotion interaction method and system based on a large model provided by the present invention. For those of ordinary skill in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims

1. A cognitive training emotional interaction method based on a large model, characterized by The steps include: Obtaining the patient's cognitive assessment results and pushing a corresponding cognitive training program to the patient; Before conducting cognitive training, the patient's emotional and psychological state is identified in advance to obtain the patient's current emotional and psychological state; Determine whether the patient's current emotional and psychological state meets the training requirements; if so, directly conduct cognitive training; if not, perform emotional regulation on the patient until the patient's emotional and psychological state meets the training requirements; Acquiring multimodal data of the patient when performing cognitive training based on the cognitive training program; wherein the multimodal data includes at least: text information, visual information, auditory information and tactile information; Inputting the multimodal data into a preset macro model, so as to perform sentiment calculation on the multimodal data through the macro model, thereby identifying the patient's current emotional and psychological state again; Based on the patient's current emotional and psychological state, providing the patient with interactive feedback of a corresponding modality to conduct personalized emotional interaction with the patient; Among them, different emotional psychological states correspond to interactive feedback of different modes, and interactive feedback of different modes corresponds to different feedback combination forms; the feedback combination form is formed by freely combining multiple emotional feedback modes.

2. The cognitive training emotional interaction method according to claim 1, characterized in that: The patient's current emotional and psychological state includes: a positive state, a neutral state, and a negative state; and before cognitive training, when the patient's current emotional and psychological state is a positive state, the training needs are met; In cognitive training, when the patient's current emotional and psychological state is positive, the patient is provided with at most bimodal interactive feedback by combining auditory feedback with textual feedback; and, in order of priority: the auditory feedback > the textual feedback; In cognitive training, when the patient's current emotional and psychological state is a neutral state, the patient is provided with up to three-modal interactive feedback in a combination of auditory feedback, visual feedback and textual feedback; and the priorities are as follows: the auditory feedback> the visual feedback> the textual feedback; In cognitive training, when the patient's current emotional and psychological state is negative, the patient is provided with up to four-modal interactive feedback through a combination of auditory feedback, visual feedback, textual feedback and tactile feedback; and the feedback is arranged in priority: the auditory feedback > the visual feedback > the textual feedback > the tactile feedback.

3. The cognitive training emotional interaction method as claimed in claim 2, characterized in that The patient's current emotional and psychological state is identified by: Acquiring cognitive training data related to cognitive training tasks based on the text information, so as to identify the cognitive training level of the patient through the macro model; Acquire the facial expression and gesture data of the patient during cognitive training collected in real time by the camera device based on the visual information, so as to identify the negative emotion of the patient and the cumulative duration of non-touch screen gestures through the large model; Acquiring audio information of the patient during the entire training process based on the auditory information, so as to identify the patient's emotional state through the large model; Based on the tactile information, identifying the touch pressure value and the number of clicks of the patient during the cognitive training process through the large model; The current emotional and psychological state of the patient is comprehensively outputted according to the patient's cognitive training level, the patient's negative emotions and the cumulative duration of non-touch screen gestures, the patient's emotional state, and the patient's touch pressure value and click times.

4. The cognitive training emotional interaction method according to claim 2, characterized in that: The text feedback includes: when the patient's current emotional and psychological state is a negative state, asking the patient about his / her feelings through text and providing positive encouragement and guidance to the patient; when the patient's current emotional and psychological state is a positive or neutral state, providing neutral feedback to the patient through text and giving affirmation; The visual feedback includes: when the patient's current emotional and psychological state is a negative state, the patient's emotions are regulated through video animations or images; when the patient's current emotional and psychological state is a positive state or a neutral state, the patient is given neutral feedback and affirmation through video animations or images; The auditory feedback includes: when the patient's current emotional and psychological state is a negative state, the patient's emotions are regulated in the form of voice playback; when the patient's current emotional and psychological state is a positive state or a neutral state, the patient is given neutral feedback and affirmation in the form of voice playback; The tactile feedback includes: when the patient's current emotional and psychological state is a negative state, tactile feedback is provided to the patient by vibrating the touch screen device according to the rhythm of the voice; when the patient's current emotional and psychological state is a positive state or a neutral state, no tactile feedback is provided.

5. The cognitive training emotional interaction method according to claim 2, characterized in that: When the patient's current emotional and psychological state is a negative state, obtaining the patient's current emotional level n; wherein n is a positive integer and the smaller the value, the more negative the emotion; If the value of n is in the first preset interval, the patient is continuously emotionally regulated and cognitive training is terminated; and the feedback combination form of the psychological regulation is a four-modal interactive feedback combining auditory feedback, visual feedback, text feedback and tactile feedback; If the value of n is in the second preset interval, cognitive training and emotional regulation are performed simultaneously; and the feedback combination form of the psychological regulation is a three-modal interactive feedback combining auditory feedback, visual feedback and tactile feedback; If the value of n is in the third preset interval, cognitive training and emotional regulation are performed simultaneously; and the feedback combination form of the psychological regulation is a two-modal interactive feedback combining auditory feedback and visual feedback; If the value of n is in the fourth preset interval, cognitive training and emotional regulation are performed simultaneously; and the feedback combination form of the psychological regulation is a two-modal interactive feedback combining auditory feedback and text feedback; If the value of n is in the fifth preset interval, cognitive training and emotional regulation are performed simultaneously; and the feedback combination form of the psychological regulation is a two-modal interactive feedback combining auditory feedback and tactile feedback; Among them, in order of numerical values, the first preset interval is less than the second preset interval < the third preset interval < the fourth preset interval < the fifth preset interval.

6. The cognitive training emotional interaction method according to claim 2, characterized in that: When the patient's cognitive training scenario is a home scenario, the auditory information is the audio information of the patient and his / her family members during the entire training process collected by a voice collection device; When the emotional state of the patient and family members is negative, the patient and family members are positively regulated by playing voice content that empathizes with the patient and family members; when the emotional state of the patient and family members is positive or neutral, the patient and family members are given neutral feedback and affirmation; Among them, the priority of providing emotional feedback to the patient is higher than the priority of providing emotional feedback to the family members.

7. The cognitive training emotional interaction method according to claim 2, characterized in that: When the patient's cognitive training scenario is a hospital scenario, the auditory information is the audio information of the patient and medical staff during the entire training process collected by a voice collection device; When the patient's emotional state is negative, the patient is positively regulated by playing voice content that empathizes with the patient; when the patient's emotional state is positive or neutral, the patient is given neutral feedback and affirmation; When the emotional state of the medical staff is negative, the medical staff is positively regulated by playing voice content that empathizes with the medical staff; Among them, the priority of providing emotional feedback to the patient is higher than the priority of providing emotional feedback to the medical staff.

8. The cognitive training emotional interaction method according to claim 2, characterized in that: When the patient is undergoing cognitive training at home or in a hospital setting, obtaining the emotional state of the family members or medical staff during the current training task; If the emotional state of the family member or medical staff is negative during the current training task, after the current training task is completed, the family member or medical staff will be given priority to emotional regulation until the emotional state of the family member or medical staff is improved to neutral or positive; and after the current training task is completed, the patient will be asked whether he is affected through human-computer interaction; If the patient is not affected, the next training task is started directly; if the patient is affected, the patient is emotionally regulated until the patient's emotional state improves to neutral or positive, and then the next training task is started.

9. The cognitive training emotional interaction method according to claim 1, characterized in that The large model is constructed in the following way: Acquire multimodal data of a historical patient group; wherein the multimodal data includes at least: text information, visual information, auditory information, and tactile information; Inputting the multimodal data of the historical patient group into the general big model to train and generate a vertical big model for emotional interaction for patients with cognitive impairment, thereby using the vertical big model to perform real-time feature extraction, decoding and analysis to determine the emotion type of the historical patients in the previous task state; According to the different emotional types of different historical patients, the vertical large model is trained again to output different emotional psychological states for different historical patients, thereby constructing the large model.

10. A cognitive training emotional interaction system based on a large model, characterized by include: A cognitive assessment unit, used to conduct cognitive assessment on patients to obtain cognitive assessment results of patients; An emotion regulation unit, used to identify the patient's emotional and psychological state in advance before cognitive training, so as to obtain the patient's current emotional and psychological state; Furthermore, it is determined whether the patient's current emotional and psychological state meets the training requirements; if so, cognitive training is directly performed; if not, the patient is emotionally regulated until the patient's emotional and psychological state meets the training requirements; A cognitive training unit connected to the cognitive assessment unit to push a corresponding cognitive training program to the patient according to the patient's cognitive assessment result, thereby being used for cognitive interaction; A multimodal data acquisition unit connected to the cognitive training unit to obtain multimodal data of the patient when performing cognitive training based on the cognitive training program; wherein the multimodal data at least includes: text information, visual information, auditory information and tactile information; An emotional computing unit, which is preset with a large model and connected to the multimodal data acquisition unit, is used to perform emotional computing on the multimodal data, thereby identifying the current emotional and psychological state of the patient; an emotion interaction unit connected to the emotion calculation unit to provide interactive feedback of different modalities to the patient based on the patient's current emotional and psychological state, so as to perform personalized emotion interaction with the patient; Among them, interactive feedback of different modes corresponds to different feedback combination forms, and the feedback combination form is freely combined by four emotional feedback modes: text feedback, visual feedback, auditory feedback and tactile feedback.

Images