Intelligent entertainment recommendation method for disabled people based on emotion recognition
By collecting and analyzing multimodal biological signals of people with disabilities, determining their emotional state and recommending content based on the disability type, the problem that the existing system cannot meet the needs of people with disabilities is solved, and precise personalized entertainment recommendations and non-traditional interaction methods are achieved.
Patent Information
- Application Number
- CN202510095196.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-21
- Publication Date
- 2025-05-13
AI Technical Summary
The existing entertainment recommendation system fails to effectively consider the special needs of people with disabilities, cannot achieve accurate recommendations based on real-time emotional fluctuations and physiological signals, and cannot help people with disabilities participate in entertainment activities through non-traditional interactive methods.
An intelligent entertainment recommendation method based on emotion recognition is adopted, and multimodal biological signals (such as electroencephalogram signals, electroskin reaction signals and electrocardiogram signals) are collected, pre-processing and feature extraction are performed to determine the user's emotional state, and the recommended content is determined based on the user's emotional state and disability type.
It realizes accurate analysis of the emotional status of people with disabilities and personalized matching of recommended content, and helps people with disabilities to independently control entertainment content through non-traditional interaction methods (such as brain-computer interfaces), improving their entertainment experience.
Smart Images
Figure CN119989092A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent recommendation, and in particular relates to an intelligent entertainment recommendation method for disabled people based on emotion recognition. Background Art
[0002] Current entertainment recommendation systems mainly rely on users' historical behavior data (such as viewing records, likes, comments, etc.) for personalized recommendations, but these systems fail to fully consider the special needs of people with disabilities. For people with visual impairments, they need to rely on auditory content (such as music or voice); people with hearing impairments need visual content (such as subtitles or sign language); and people with inconvenient hands and feet (such as limb disabilities, motor dysfunction, etc.) often cannot easily operate traditional devices (such as remote controls, touch screens, etc.), resulting in their inability to independently control the selection of entertainment content. Existing systems fail to effectively provide personalized recommendations for these specific groups, cannot achieve accurate recommendations based on real-time emotional fluctuations and physiological signals, and cannot help these groups effectively participate in entertainment activities through non-traditional interactive methods. Summary of the invention
[0003] The present invention provides an intelligent entertainment recommendation method for people with disabilities based on emotion recognition to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:
[0004] An intelligent entertainment recommendation method for people with disabilities based on emotion recognition, comprising:
[0005] Collecting multimodal bio-signals of the user;
[0006] Preprocessing the collected multimodal biological signals;
[0007] Acquire multimodal features based on the preprocessed multimodal biological signals;
[0008] Determining the user's emotional state based on the acquired multimodal features;
[0009] Recommendations are determined based on the user's emotional state and the user's disability type.
[0010] Furthermore, the specific method of collecting the user's multimodal biosignal is:
[0011] Collect the patient's EEG signals, galvanic skin response signals and ECG signals.
[0012] Furthermore, the specific method for preprocessing the collected multimodal biological signals is:
[0013] Noise reduction is performed on EEG signals, galvanic skin response signals and ECG signals.
[0014] Furthermore, the specific method for obtaining multimodal features based on the preprocessed multimodal biological signal is:
[0015] Based on the EEG signal, the first EEG feature is calculated by the following formula:
[0016]
[0017] Among them, X(f) is the EEG signal, f is the frequency, and T is the length of the signal;
[0018] Based on the skin electrical response signal, the skin electrical response characteristics are calculated by the following formula:
[0019] △GSR=GSR max —GSR min
[0020] Among them, GSR max is the peak value of the skin electrical response signal in the reaction stage, GSR max It is the lowest value of the skin electrical response signal in the reaction stage;
[0021] Based on the ECG signal, the ECG characteristics are calculated using the following formula:
[0022]
[0023] Average RR Internal represents the average time interval between each heartbeat.
[0024] Furthermore, the specific method for determining the user's emotional state based on the acquired multimodal features is:
[0025] Through the trained classification model, the emotion classification model is output based on the first EEG feature, galvanic skin response feature and electrocardiogram feature.
[0026] Further, determining the recommended content based on the user's emotional state and the user's disability type includes:
[0027] For visually impaired users, voice content is played based on the user's emotional state.
[0028] Further, determining the recommended content based on the user's emotional state and the user's disability type includes:
[0029] For hearing-impaired users, play visual content based on the user's emotional state.
[0030] Further, determining the recommended content based on the user's emotional state and the user's disability type includes:
[0031] For users with physical disabilities, content that can be selected through brain-computer interface control is played according to the user's emotional state.
[0032] Furthermore, based on the EEG signal, the second EEG feature is calculated by the following formula:
[0033]
[0034] Among them, P300 is an event-related potential in the EEG signal, EEG(t) represents the EEG signal at time point t, h(t) is a windowing function used to filter out irrelevant noise, and t0 and t1 are the time windows for calculating the P300 wave;
[0035]
[0036] Where EEG(t) represents the EEG signal at time point t, g(t) is a windowing function used to filter out irrelevant noise, and t0 and t1 are the time windows for calculating the N200 wave;
[0037]
[0038] Among them, C(f) represents coherence, S xy (f) is the cross power spectrum of the two signals, S xx (f) and S yy (f) are the auto-power spectra of the two signals;
[0039]
[0040] Among them, PDP represents the potential distribution pattern, EEG i is the potential of the electrode of the ith channel, EEG ref is the potential of the reference electrode, N is the sum of the channel electrodes;
[0041] The user's intention is identified based on the second EEG feature, and the currently playing content is regulated based on the user's intention.
[0042] Furthermore, the determination of recommended content based on the user's emotional state and the user's disability type is specifically:
[0043] Recommendations are determined based on the user's emotional state, the user's disability type, and the user's historical data.
[0044] The benefit of the present invention lies in the intelligent entertainment recommendation method for people with disabilities based on emotion recognition, which combines innovative multimodal brain-computer interface technology with emotion recognition technology to accurately analyze the user's emotional state and determine the recommended content based on the user's emotional state and disability type. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0046] Figure 1 This is a schematic diagram of the intelligent entertainment recommendation method for people with disabilities based on emotion recognition in this application. DETAILED DESCRIPTION
[0047] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0048] like Figure 1 The present invention shows an intelligent entertainment recommendation method for people with disabilities based on emotion recognition, comprising:
[0049] S1: Collect the user's multimodal bio-signals.
[0050] Specifically, a high-precision sensor is used to monitor and record the user's physiological state in real time. In the present application, the specific method of collecting the user's multimodal biosignals is: collecting the user's electroencephalogram signal, skin galvanic response and electrocardiogram signal.
[0051] S2: Preprocess the collected multimodal biological signals.
[0052] The specific method for preprocessing the collected multimodal biological signals is: performing noise reduction processing on the EEG signals, the galvanic skin response signals and the ECG signals. It is understandable that other relevant data preprocessing can also be performed on the EEG signals, the galvanic skin response signals and the ECG signals to improve the accuracy of the data.
[0053] S3: Acquire multimodal features based on the preprocessed multimodal biological signals.
[0054] The collected EEG signals are processed by denoising and feature extraction to extract signal features related to emotional state. Specifically, the EEG signals are decomposed in the frequency domain, and the power spectral density (PSD) features of the Delta (0.5-4Hz), Theta (4-8Hz), Alpha (8-13Hz), and Beta (13-30Hz) frequency bands are extracted for emotion analysis and recognition. The calculation formula is as follows.
[0055]
[0056] Among them, X(f) is the EEG signal, f is the frequency, and T is the length of the signal.
[0057] For the skin electrical response signal, the difference between its maximum and minimum values is extracted to calculate its variation range. Specifically, the electromyographic characteristics are calculated using the following formula:
[0058] △GSR=GSR max -GSR min
[0059] Among them, GSR max is the peak value of the skin electrical response signal in the reaction stage, GSR max It is the lowest value of the skin electrical response signal in the response stage.
[0060] For ECG signals, the average heart rate is calculated by the time interval between signal peaks. Specifically, the ECG feature is calculated by the following formula:
[0061]
[0062] Average RR Internal represents the average time interval between each heartbeat.
[0063] S4: Determine the user's emotional state based on the acquired multimodal features.
[0064] The specific method for determining the user's emotional state based on the acquired multimodal features is:
[0065] Through the trained classification model, the emotion classification model is output based on the first EEG feature, EMG feature and ECG feature. Specifically, the emotion calculation model based on deep learning is used to analyze the first EEG feature, EMG feature and ECG feature, identify the user's current emotional state, and output the emotion category (such as anxiety, pleasure, etc.). Assuming that the user's emotional state is "anxiety", in the EEG data, the PSD of the Beta and Theta frequency bands is higher, the PSD of the Alpha frequency band is lower, the skin electrical response changes greatly, and the heart rate is high.
[0066] S5: Determine recommended content based on the user's emotional state and the user's disability type.
[0067] In an embodiment of the present application, determining recommended content based on the user's emotional state and the user's disability type includes:
[0068] For visually impaired users, voice content is played based on the user's emotional state.
[0069] Specifically, for the "anxiety" emotion category, gentle and soothing music is recommended. For the "pleasure" emotion category, cheerful and fast-paced music is recommended. For the "calm" emotion category, relaxing background music is recommended. The user can select one from the recommended content to play. It is understandable that the user's disability type can be preset.
[0070] Specifically, customers can use rule engines or deep learning models (such as collaborative filtering or neural networks) to match emotional states with content types. For example, when the emotional state is anxiety, the model will recommend through emotion-content mapping rules (for example, anxiety->cheerful music).
[0071] For hearing-impaired users, play visual content based on the user's emotional state.
[0072] Correspondingly, for the "anxiety" emotion category, meditation or relaxation videos are recommended. For the "pleasure" emotion category, cheerful videos are recommended. For the "calm" emotion category, relaxing self-improvement videos are recommended.
[0073] For users with physical disabilities, content that can be selected through brain-computer interface control is played according to the user's emotional state.
[0074] Specifically, for the "anxiety" emotion category, gentle, soothing music or videos that can be controlled by the brain-computer interface are recommended. For the "pleasure" emotion category, cheerful, fast-paced music or videos that can be controlled by the brain-computer interface are recommended. For the "calm" emotion category, relaxing background music or videos that can be controlled by the brain-computer interface are recommended.
[0075] In terms of recommended content, compared with hearing-impaired and visually impaired users, physically impaired users can not only choose audio and video content, but also achieve interactive experience through brain-computer interface. Hearing-impaired users prefer visual content, such as subtitled videos or meditation videos; visually impaired users prefer audio content, such as music or audiobooks, and rely on hearing to perceive information. In terms of operation method, compared with hearing-impaired and visually impaired users, physically impaired users can choose to achieve non-contact operation through brain-computer interface, while hearing-impaired and visually impaired users may still rely on traditional touch, gesture or voice control.
[0076] For users with physical disabilities, they cannot select recommended content through remote controls, touch screens, etc. In this application, EEG signal decoding technology is further used to identify user intentions, thereby achieving contactless interactive control. An EEG signal decoding model (such as a support vector machine, SVM) is used to identify the user's intentions (such as selection, play, pause, etc.). The calculation formula for SVM classification for intent recognition is as follows:
[0077] f(x)=sign(w*x+b)
[0078] Among them, x is the EEG signal feature vector, w is the weight vector of the support vector machine, b is the bias term, and f(x) is the classification result. The EEG feature vector of x includes event-related potentials (such as P300 and N200 waves) and spatial features.
[0079] Based on the EEG signal, the second EEG feature is calculated by the following formula:
[0080] P300 is an event-related potential (ERP) in EEG signals, which usually appears in cognitive responses to sudden stimuli, especially when target stimuli appear. Its calculation formula usually relies on time domain analysis and frequency domain analysis of EEG signals. The specific calculation formula is as follows:
[0081]
[0082] Among them, EEG(t) represents the EEG signal at time point t, h(t) is a windowing function, which is usually used to filter out irrelevant noise, and t0 and t1 are the time windows for calculating the P300 wave, usually from 200 mm to 600 mm after stimulation.
[0083] N200 is another event-related potential in EEG signals related to cognitive load and attention allocation, which usually appears about 200 milliseconds after the stimulus when the negative wave (ie, N wave) appears. The calculation formula is as follows:
[0084]
[0085] Where EEG(t) represents the EEG signal at time point t, g(t) is a windowing function, which is usually used to filter out irrelevant noise, and t0 and t1 are the time windows for calculating the N200 wave, usually from 150 mm to 350 mm after stimulation.
[0086] The spatial characteristics mainly include coherence and potential distribution pattern (PDP).
[0087] Coherence indicates the degree of synchronization between two signals (such as EEG signals from different brain regions), and its calculation formula is as follows:
[0088]
[0089] Among them, S xy (f) is the cross power spectrum of two signals (such as different brain regions), S xx (f) and S yy (f) are the autopower spectra of the two signals respectively.
[0090] The Potential Distribution Pattern (PDP) evaluates the spatial distribution by measuring the potential difference between different electrodes. The calculation formula is as follows:
[0091]
[0092] Among them, EEG i is the potential of the electrode of the ith channel, EEG ref is the potential of the reference electrode and N is the sum of the channel electrodes.
[0093] The user's intention is identified based on the second EEG feature, and the currently playing content is regulated based on the user's intention.
[0094] Step S5 may further include: determining recommended content based on the user's emotional state, the user's disability type, and the user's historical data.
[0095] Specifically, the user's past entertainment preferences (such as favorite music genres, meditation videos watched, etc.) are collected. The emotional state is combined with the user's historical data through a personalized model (such as a recommendation network based on user preferences) to recommend the content that best suits the user's preferences. The user's historical data is further combined to improve the accuracy of the recommendation.
[0096] Recommended content evaluation and optimization:
[0097] By real-time monitoring of user emotional changes, the system continuously evaluates the suitability of recommended content. If the recommended content cannot meet the user's emotional needs, the system will automatically adjust and recommend more appropriate content.
[0098] Specifically, reinforcement learning (RL) is used for self-optimization, and the recommendation strategy is adjusted according to user feedback (such as emotional changes and content selection). The Q-learning update formula is as follows:
[0099]
[0100] Among them, s t is the state, a t For action, r t is the immediate reward, α is the learning rate, and γ is the discount factor. Assume that the user selects a relaxing video in an anxious state and his / her emotional state is relieved. The system updates the strategy through Q-learning and recommends more similar relaxing content in the future.
[0101] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.
Claims
1. An intelligent entertainment recommendation method for people with disabilities based on emotion recognition, characterized in that: Include: Collecting multimodal bio-signals of the user; Preprocessing the collected multimodal biological signals; Acquire multimodal features based on the preprocessed multimodal biological signals; Determining the user's emotional state based on the acquired multimodal features; Recommendations are determined based on the user's emotional state and the user's disability type.
2. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 1 is characterized in that: The specific method of collecting the user's multimodal bio-signal is: Collect the patient's EEG signals, galvanic skin response signals and ECG signals.
3. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 2 is characterized in that: The specific method for preprocessing the collected multimodal biological signals is: Noise reduction is performed on EEG signals, galvanic skin response signals and ECG signals.
4. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 2 is characterized in that: The specific method for obtaining multimodal features based on the preprocessed multimodal biological signal is: Based on the EEG signal, the first EEG feature is calculated by the following formula: Among them, X(f) is the EEG signal, f is the frequency, and T is the length of the signal; Based on the skin electrical response signal, the skin electrical response characteristics are calculated by the following formula: △GSR=GSR max -GSR min Among them, GSR max is the peak value of the skin electrical response signal in the reaction stage, GSR max It is the lowest value of the skin electrical response signal in the reaction stage; Based on the ECG signal, the ECG characteristics are calculated using the following formula: Average RR Internal represents the average time interval between each heartbeat.
5. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 4 is characterized in that: The specific method for determining the user's emotional state based on the acquired multimodal features is: Through the trained classification model, the emotion classification model is output based on the first EEG feature, galvanic skin response feature and electrocardiogram feature.
6. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 1, characterized in that: Determining the recommended content based on the user's emotional state and the user's disability type includes: For visually impaired users, voice content is played based on the user's emotional state.
7. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 1 is characterized in that: Determining the recommended content based on the user's emotional state and the user's disability type includes: For hearing-impaired users, play visual content based on the user's emotional state.
8. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 1, characterized in that: Determining the recommended content based on the user's emotional state and the user's disability type includes: For users with physical disabilities, content that can be selected through brain-computer interface control is played according to the user's emotional state.
9. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 8, characterized in that: Based on the EEG signal, the second EEG feature is calculated by the following formula: Among them, P300 is an event-related potential in the EEG signal, EEG(t) represents the EEG signal at time point t, h(t) is a windowing function used to filter out irrelevant noise, and t0 and t1 are the time windows for calculating the P300 wave; Where EEG(t) represents the EEG signal at time point t, g(t) is a windowing function used to filter out irrelevant noise, and t0 and t1 are the time windows for calculating the N200 wave; Among them, C(f) represents coherence, S xy (f) is the cross power spectrum of the two signals, S xx (f) and S yy (f) are the auto-power spectra of the two signals; Among them, PDP represents the potential distribution pattern, EEG i is the potential of the electrode of the ith channel, EEG ref is the potential of the reference electrode, N is the sum of the channel electrodes; The user's intention is identified based on the second EEG feature, and the currently playing content is regulated based on the user's intention.
10. The intelligent entertainment recommendation method for disabled people based on emotion recognition according to claim 1, characterized in that: The determination of recommended content based on the user's emotional state and the user's disability type is specifically: Recommendations are determined based on the user's emotional state, the user's disability type, and the user's historical data.