Blind person emotion analysis method and system and medium

Abstract

The invention discloses a blind person emotion analysis method and system and a storage medium, which are applied to a blind person communication auxiliary terminal device and aim to assist a blind person to effectively communicate with an interaction object. The method comprises the following steps: acquiring a facial image, a gesture and a voice signal of an interaction object, and analyzing to generate a first emotion vector; synchronously collecting a physiological signal, a behavior signal and a subjective question and answer result of the blind, and analyzing to generate a second emotion vector; calculating an emotion matching degree between the two vectors; determining a target social coping strategy and a risk level according to the matching degree; and finally, the control terminal outputs corresponding prompt information to guide the blind to adjust communication behaviors. The system comprises an acquisition module, an emotion analysis module, a matching and evaluation module, a prompt control module and the like, which cooperatively realize the above functions. According to the invention, through multi-mode emotion perception and intelligent strategy matching, the perception and coping ability of blind people to other people and own emotional states in social interaction is effectively improved, and the social risk is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction and emotion computing, and particularly relates to a blind person emotion analysis method and system and a medium. BACKGROUND

[0002] With the development of computer vision, speech recognition, wearable devices and other technologies, assistive technologies for visually impaired groups have emerged. For example, guide canes, speech screen reading software, object recognition and text reading devices based on cameras, etc. can help the blind to a certain extent to complete tasks such as travel navigation, environmental perception and information acquisition. Some intelligent glasses products can also recognize scenes, identities or simple expression categories, and prompt the blind user through voice broadcast, thereby assisting information perception in social scenes within a limited range.

[0003] On the other hand, emotion computing technology has also made great progress. Existing emotion recognition methods are usually based on single or multi-modal data such as facial images, speech signals, gestures, etc. and identify emotion categories such as "happy", "angry", "sad" through deep learning models, and are widely used in intelligent customer service, online education, human-computer interaction and other fields. However, the above-mentioned emotion recognition technology mainly serves the able-bodied population, and most of them only analyze the emotional state of a single party, and are mostly output in the form of label recognition or emotion score, and are not designed for the specific needs of the blind.

[0004] In the existing technology related to communication with the blind, some solutions can recognize the facial expressions or speech tones of the interaction object when the blind person communicates face-to-face, and prompt the blind person about the current general emotion of the other party through voice, such as "the other party is smiling" or "the other party is somewhat angry". However, this type of solution generally has the following limitations: first, it only focuses on the emotional state of the interaction object, ignoring the emotional fluctuations and physiological and behavioral reactions of the blind person, and cannot fully depict the two-way emotional relationship between the two parties in the social process; second, the emotion analysis result stays at the level of simple description, and lacks the ability to give specific social coping strategies based on the emotional relationship between the two parties, such as continuing to empathize, easing the tone, changing the topic or ending the communication.

[0005] In addition, the existing technology usually uses voice broadcast to feedback emotion-related information to the blind, which can easily interrupt the communication rhythm in real conversations, and even expose the assistive system used by the blind, affecting privacy and user experience; at the same time, the existing system generally lacks personalized modeling for the physiological baseline and behavioral habits of individual blind people, and the fusion method of multi-modal signals is relatively simple, making it difficult to accurately reflect the real emotional state of different blind users in different scenarios, and the ability to identify emotional changes and potential social risks in complex social situations is limited.

[0006] In summary, the prior art at least has the following technical problems: lacking of joint modeling mechanism considering the emotional state of the interactive object and the emotional state of the blind person, unable to evaluate the current social situation from the perspective of two-way emotion; lacking of decision mechanism for automatically generating social coping strategies and risk levels based on emotional matching relationship, the existing system is difficult to provide operational and specific situation-oriented communication suggestions for the blind; the existing prompt method is mainly based on voice broadcast, which is easy to interfere with the conversation and has poor privacy, lacking of low-interference and multi-channel feedback design for real social scenes of the blind; lacking of personalized emotional analysis means combining physiological signals, behavior signals and subjective feedback, difficult to accurately reflect the emotional characteristics and change rules of different blind users in the long-term use process. SUMMARY

[0007] The purpose of the present application is to solve the above-mentioned problems of insufficient social emotional perception and strategy assistance for the blind.

[0008] According to an aspect of the present application, a blind emotional analysis method is provided, which is applied to a blind communication assistance terminal device to assist the blind in communicating with interactive objects. The method comprises: S10, collecting facial images, gesture data and voice signals of an interactive object communicating with the blind, analyzing the facial images, gesture data and voice signals to obtain a first emotional vector for representing the emotional state of the interactive object; S20, collecting physiological signals, behavior signals and subjective questionnaire results of the blind, and analyzing the physiological signals, behavior signals and subjective questionnaire results to obtain a second emotional vector for representing the emotional state of the blind; S30, calculating the emotional matching degree between the first emotional vector and the second emotional vector based on the first emotional vector and the second emotional vector; S40, determining a target social coping strategy and a corresponding risk level according to the emotional matching degree; S50, based on the target social coping strategy and the risk level, controlling the blind communication assistance terminal device to output prompt information to remind the blind to adjust the communication behavior.

[0009] Preferably, the analysis of the facial images, gesture data and voice signals in S10 to obtain the first emotional vector for representing the emotional state of the interactive object comprises: S11, performing face detection and face alignment processing on the facial image sequence, inputting the aligned face image into a convolutional neural network model to extract a first facial feature vector for representing the intensity of facial muscle action units; S12. Input the gesture data into the human posture estimation model to obtain the skeletal key point coordinate sequence of the interactive object in each frame, and calculate the joint angle change, movement speed and acceleration features based on the skeletal key point coordinate sequence. Input the kinematic features into the temporal neural network model to obtain the first gesture feature vector. S13. Perform pre-emphasis, framing, and windowing processing on the speech signal, calculate Mel frequency cepstral coefficients and / or spectrogram, and input the time-frequency features into the speech emotion recognition model to obtain the first speech feature vector; S14. Normalize the first facial feature vector, the first gesture feature vector, and the first speech feature vector, and input them into the multimodal feature fusion network. Obtain the fused feature vector through attention calculation with the first facial feature vector as the query vector, the first gesture feature vector as the key vector, and the first speech feature vector as the value vector. S15. Input the fused feature vector into the fully connected layer and the normalized Softmax layer, and output the first emotion vector. The first emotion vector is used to characterize the probability distribution of the interactive object in the preset emotion category and / or emotion dimension.

[0010] Preferably, in step S20, the analysis of the second emotion vector of the blind person's emotional state includes: S21. Filter and remove artifacts from the photoplethysmography (PPG) signal of the blind person. Divide the time window into preset durations. Detect the heartbeat interval in each window and calculate the standard deviation of the heartbeat interval, the root mean square of the difference between adjacent heartbeat intervals, the proportion of heartbeat interval differences that meet preset thresholds, and the ratio of low-frequency energy to high-frequency energy to obtain heart rate variability characteristics. S22. Smooth and baseline-correct the skin conduction signal of the blind person, extract the slow-varying component and the fast-varying component, calculate the average skin conduction level, the number of skin conduction peaks per unit time and their peak amplitude, and obtain skin conduction characteristics. S23. The heart rate variability feature and the skin conductance feature are concatenated to form a physiological feature vector, and then input into the physiological emotion assessment network to obtain the physiological emotion embedding vector. S24. Perform gait cycle detection on the gait signal of the blind person, calculate the gait frequency, stride length, gait cycle fluctuation and / or gait symmetry index, input the index into the gait emotion assessment sub-network, and obtain the gait emotion feature vector. S25. Extract acoustic features from the speech signal of the blind person, input the obtained acoustic features into the speech emotion recognition sub-network of the blind person, and obtain the speech emotion feature vector of the blind person. S26. The gait emotion feature vector is concatenated with the blind person's speech emotion feature vector to form a behavioral emotion feature vector; S27. Based on the subjective question and answer results, convert the question and answer options into subjective emotion feature vectors according to a preset mapping relationship; S28. Input the physiological emotion embedding vector, the behavioral emotion feature vector, and the subjective emotion feature vector into the self-emotion fusion network, and output the second emotion vector through attention weighting and nonlinear transformation.

[0011] Preferably, in step S30, calculating the emotion matching degree between the first emotion vector and the second emotion vector includes: S31. Perform linear transformations on the first emotion vector and the second emotion vector respectively to align the dimensions, and obtain the first normalized emotion vector and the second normalized emotion vector. S32. Calculate the dot product of the first normalized emotion vector and the second normalized emotion vector in the same emotion feature space, and divide the dot product by the product of the magnitudes of the two vectors to obtain the cosine similarity. S33. The cosine similarity is converted into an emotion matching degree in the range of [0, 1] via interval mapping, wherein the closer the emotion matching degree is to 1, the more consistent the emotional state of the interactive object is with the emotional state of the blind person, and the closer the emotion matching degree is to 0, the greater the difference in their emotions.

[0012] Preferably, in step S40, determining the target social coping strategy and the corresponding risk level based on the emotion matching degree includes: S41. Construct a training sample set containing multiple social scenarios. Each training sample includes a first emotion vector, a second emotion vector, the corresponding emotion matching degree, a scenario type label, and social coping strategy labels and risk level labels labeled by professionals. S42. Using the combination of the first emotion vector, the second emotion vector, the emotion matching degree, and the scene type label as input features, and the social coping strategy label and the risk level label as supervision signals, the strategy generation model is trained under supervision, so that after training, similar combinations of input features will produce similar strategy outputs and risk level outputs through the strategy generation model. S43. During the method execution phase, the first emotion vector, the second emotion vector, the emotion matching degree, and the current scene type, calculated in real time, are input into the strategy generation model. The strategy generation model then outputs the target social coping strategy and the corresponding risk level, wherein: The target social coping strategies include at least one of the following: empathy strategies, de-escalation strategies, topic guidance strategies, and communication termination strategies. The risk levels include a first risk level, a second risk level, and a third risk level. Wherein, the emotion matching degree is a real value in the range of [0, 1]. The larger the value, the closer the emotional state of the interactive object is to the emotional state of the blind person. The smaller the value, the greater the difference between the two emotions. The component in the second emotion vector that represents the emotional arousal degree of the blind person is a normalized value in the range of [0, 1]. The larger the value, the higher the emotional arousal degree of the blind person. The first risk level is used to indicate the situation where the emotion matching degree is greater than the first matching threshold and the component in the second emotion vector that represents the emotional arousal degree of the blind person is lower than the first arousal threshold, corresponding to the state where both the emotion matching degree and the emotional arousal degree are in a preset low-risk range; The second risk level is used to indicate the situation where the emotional matching degree is between the first matching threshold and the second matching threshold and / or the component in the second emotional vector that represents the emotional arousal degree of the blind person is between the first arousal threshold and the second arousal threshold, corresponding to the state where the emotional matching degree and / or the emotional arousal degree are in a preset intermediate risk range; The third risk level is used to indicate the situation where the emotional matching degree is lower than the second matching threshold and / or the component in the second emotional vector that represents the emotional arousal degree of the blind person is higher than the second arousal threshold, corresponding to the state where the emotional matching degree and / or the emotional arousal degree are in a preset high-risk range; Wherein, the first matching threshold, the second matching threshold, the first wake-up threshold and the second wake-up threshold satisfy: 0 < the second matching threshold < the first matching threshold ≤ 1, 0 < the first wake-up threshold < the second wake-up threshold ≤ 1.

[0013] Preferably, in step S50, controlling the blind person's communication assistance terminal device to output a prompt message specifically includes: When the risk level is the first risk level, only a vibration prompt is output to indicate the current social situation, and no voice prompt is output to avoid interfering with normal communication; When the risk level is the second risk level, a vibration prompt is output, and a short voice prompt is output when a pause is detected in the current conversation, prompting the blind person to adjust their tone or topic appropriately. When the risk level is the third risk level, an enhanced vibration alert is output, and a voice prompt corresponding to the target social coping strategy is output in real time to remind the blind person to take strategies such as downgrading the conflict or ending the communication.

[0014] Preferably, the method further includes: S60. Based on the blind person's historical physiological signals, behavioral signals, and historical subjective question-and-answer results at different time periods, construct an individualized training sample set corresponding to the blind person. S61. Based on the individualized training sample set, some parameters of the strategy generation model are fine-tuned to correct the physiological baseline differences and behavioral habits of the blind person, thereby obtaining a personalized strategy generation model corresponding to the blind person. S62. In the process of performing sentiment analysis on the current social situation and outputting the target social coping strategy and / or risk level, reasoning is performed based on the personalized strategy generation model to generate the target social coping strategy and / or the risk level.

[0015] The present invention also provides a blind person emotion analysis system, applied to the blind person emotion analysis method as described in any of the above claims, the system comprising: The acquisition module is used to acquire interaction signals that characterize the social behavior of the interacting objects and state signals that characterize the blind person's own state. The emotion analysis module receives the interaction signal and generates first emotion information representing the emotional state of the interaction object, and receives the state signal and generates second emotion information representing the emotional state of the blind person. The matching and evaluation module calculates the emotion matching degree based on the first emotion information and the second emotion information, and generates decision information including target social coping strategies and risk levels based on the emotion matching degree. The prompt control module receives the decision information and, based on the target social response strategy and the risk level, controls the blind person's communication assistance terminal device to output a prompt signal, which is used to prompt the blind person to adjust their communication behavior.

[0016] Preferably, the system further includes: The storage module is used to store the interaction signals and status signals collected by the acquisition module, as well as the first emotion information, the second emotion information, the emotion matching degree and / or the decision information generated by the emotion analysis module and / or the matching and evaluation module, and to use them as historical reference data for subsequent emotion analysis and generation of target social coping strategies.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the blind emotion analysis method as described in any of the above.

[0018] This application offers the following advantages: it can simultaneously perceive the emotional state of both the interacting party and the blind person's own emotional state in real-world social scenarios, quantifying both parties' emotions using a unified emotion vector. By calculating the emotional matching degree between the two parties' emotion vectors, it assesses the risk level of the current social situation from the perspective of bidirectional emotional relationships, automatically generating social coping strategies adapted to different risk levels. Furthermore, it outputs prompts to the blind person in ways that are easily perceived by the blind, such as vibration and voice, with minimal interference, enabling the blind person to understand the current communication status and adjust their communication behavior accordingly. Therefore, this embodiment not only overcomes the limitations of existing technologies that only focus on the emotions of the interacting party and lack consideration for the blind person's own emotions, but also avoids the problem that simple emotional descriptions cannot be translated into specific behavioral guidance. Simultaneously, through tiered prompts and multi-channel output, it reduces the impact on the rhythm of the conversation and privacy, improving the blind person's emotional understanding and coping abilities in complex social scenarios, enabling the blind person to shift from passively receiving information to actively adjusting and optimizing their social behavior. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a logic block diagram of the blind emotion analysis method described in one embodiment of this application. Detailed Implementation

[0021] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] Please refer to Figure 1One embodiment of this application provides a method for analyzing the emotions of blind people, applied in a communication assistive terminal device for the blind, to assist blind people in communicating with interactive objects. The method includes: S10. Collect facial images, gesture data, and speech signals of the interaction subject communicating with the blind person. Analyze the facial images, gesture data, and speech signals to obtain a first emotion vector representing the emotional state of the interaction subject. In this step, it should be noted that the blind communication assistive terminal device can continuously collect information about the external performance of the interaction subject during the current communication process through sensing units such as cameras, posture sensors, and microphones configured on the terminal. Facial images reflect changes in facial expressions, gesture data reflects body posture and movement amplitude, and speech signals reflect speech rate, pitch, volume, and other emotional characteristics. After collecting the above multi-source signals, the terminal can preprocess and extract features from various signals, converting the emotion-related information contained in the face, gestures, and speech into a unified form of feature representation. Then, a multimodal fusion algorithm is used to map these features onto the same emotion feature space, thereby obtaining a first emotion vector that comprehensively represents the current emotional state of the interaction subject. This emotion vector can reflect information such as the emotion category and intensity of the interaction subject in the numerical space.

[0024] S20. Collect the blind person's physiological signals, behavioral signals, and subjective question-and-answer results, and analyze them to obtain a second emotion vector representing the blind person's emotional state. In this step, it should be noted that the blind person's communication assistive terminal device can acquire physiological and behavioral signals during the current communication process through physiological sensors (such as heart rate and skin conductance sensors) and motion sensors (such as inertial measurement units) worn by the blind person. These signals include physiological data reflecting changes in heart rate and skin conductance, and behavioral data reflecting gait, micro-movements, and speech patterns. Simultaneously, the terminal can initiate brief subjective question-and-answer sessions with the blind person via voice interaction at appropriate times, such as asking if the blind person is currently tense or feeling happy, and converting the blind person's voice responses into processable subjective feedback information. Subsequently, the terminal can extract features from the physiological and behavioral signals and perform multi-source information fusion combined with the subjective question-and-answer results, mapping the blind person's current physiological arousal level, behavioral performance, and self-perception into a unified emotional feature representation, obtaining a second emotion vector representing the blind person's emotional state, thereby quantitatively describing the blind person's emotions within the same emotional feature space.

[0025] S30. Calculate the emotion matching degree between the two based on the first and second emotion vectors. In this step, it's important to note that the emotion matching degree characterizes the similarity between the emotional state of the interacting object and the blind person's own emotional state in the emotion feature space. Specifically, the first and second emotion vectors can be aligned in the same dimensional space, and their proximity can be calculated using a similarity measurement algorithm. For example, cosine similarity or Euclidean distance mapping can be used to convert the relationship between the two emotion vectors into a scalar indicator. This scalar can be normalized to a preset interval to represent the degree of "synchronicity" of the emotions of both parties: a higher value indicates a greater consistency in their emotions, while a lower value indicates a greater difference in their emotions, thus providing a quantifiable basis for subsequent social strategy decisions.

[0026] S40. Determine the target social coping strategy and corresponding risk level based on the emotion matching degree. In this step, it should be noted that the blind communication assistive terminal device can comprehensively assess the risk level of the current social situation based on the magnitude of the emotion matching degree and the relevant components in the second emotion vector reflecting the blind person's emotional arousal level, and generate a target social coping strategy adapted to that risk level. Specifically, different matching degree intervals and arousal degree intervals can be pre-divided based on a large amount of historical communication data and expert experience. Situations with high emotional matching and the blind person in a low arousal state are classified as low risk; situations with some emotional difference or the blind person in a moderate arousal state are classified as medium risk; and situations with significant emotional mismatch or the blind person in a high arousal state are classified as high risk. Based on this, corresponding social coping strategies can be preset or learned through a model for different risk levels. For example, in low-risk situations, it is recommended to maintain the current communication style and provide appropriate empathy; in medium-risk situations, it is recommended to appropriately soften the tone or change the topic; and in high-risk situations, it is recommended that the blind person reduce the intensity of their emotions or end the communication as soon as possible. Through the above methods, the target social coping strategy and corresponding risk level can be automatically determined based on the emotion matching degree and related emotional characteristics.

[0027] S50. Based on the target social coping strategy and risk level, control the output of prompting information from the assistive communication terminal device for the blind to remind them to adjust their communication behavior. In this step, it should be noted that the assistive communication terminal device for the blind can select different output methods and content according to the target social coping strategy and risk level, in order to convey necessary emotional and strategic information to the blind person without interfering with the current conversation as much as possible. For example, in a low-risk situation, the terminal can simply use slight vibrations to indicate to the blind person that their current emotional state is good and no significant adjustment is needed; in a medium-risk situation, a specific vibration pattern combined with short voice prompts during pauses in the conversation can guide the blind person to adjust their tone or topic appropriately; in a high-risk situation, a more obvious vibration pattern and real-time voice prompts can be used to remind the blind person that there is a high social risk, guiding them to take strategies such as mitigation or ending the communication. The prompting information may include brief feedback on the current emotional state and suggestions for specific coping methods, enabling the blind person to adjust their behavior in a timely manner based on the prompts, thereby improving the communication effect.

[0028] The technical solution implemented in this embodiment can simultaneously perceive the emotional state of the interacting object and the blind person's own emotional state in actual social scenarios. It quantifies the emotions of both parties using a unified emotion vector format, calculates the emotional matching degree between their emotional vectors, assesses the risk level of the current social situation from the perspective of bidirectional emotional relationships, and automatically generates social coping strategies adapted to different risk levels. It also outputs prompts to the blind person in ways that are easily perceived by the blind and have minimal interference, such as vibration and voice, enabling the blind person to understand the current communication status and adjust their communication behavior accordingly. Therefore, this embodiment not only overcomes the limitations of existing technologies that only focus on the emotions of the interacting object and lack consideration for the blind person's own emotions, but also avoids the problem that simple emotional descriptions cannot be translated into specific behavioral guidance. Furthermore, through tiered prompts and multi-channel output, it reduces the impact on the rhythm of the conversation and privacy, improving the blind person's emotional understanding and coping abilities in complex social scenarios, enabling the blind person to shift from passively receiving information to actively adjusting and optimizing their social behavior.

[0029] Furthermore, in S10, facial images, gesture data, and voice signals are analyzed to obtain a first emotion vector representing the emotional state of the interactive object, including: S11. Perform face detection and face alignment processing on the facial image sequence. Input the aligned face image into a convolutional neural network model to extract the first facial feature vector representing the intensity of facial muscle action units (AUs). In this step, it should be noted that face detection can be achieved using a traditional feature classifier or a deep learning-based face detection network to locate the face region of the interactive object from the original image. Face alignment can be performed on the face using affine or similar transformations based on key points such as the corners of the eyes, the tip of the nose, and the corners of the mouth, unifying the face in different frames and poses to a standard reference position and reducing the influence of factors such as head rotation and camera angle. The aligned face image is scaled to a preset resolution and then input into the convolutional neural network model. Local texture and global structural features are extracted through multi-layer convolution, non-linear activation, and pooling operations, and the network outputs feature components corresponding to the intensity of facial muscle action units (AUs) at the end. The first facial feature vector obtained in this way can represent the micro-expression changes in areas such as eyebrows, eyes, and corners of the mouth with fine granularity, providing a reliable foundation of facial expression information for subsequent emotion estimation.

[0030] S12. Input the gesture data into the human pose estimation model to obtain the skeletal keypoint coordinate sequence of the interactive object in each frame. Calculate joint angle changes, motion velocity, and acceleration features based on the skeletal keypoint coordinate sequence. Input the kinematic features into the temporal neural network model to obtain the first gesture feature vector. In this step, it should be noted that the gesture data can come from a depth camera, an RGB camera combined with a pose estimation algorithm, or acceleration and angular velocity data output from a wearable inertial measurement unit. The human pose estimation model is used to parse the temporal coordinate sequence of skeletal keypoints of the head, torso, upper limbs, and lower limbs from video frames or sensor data. Based on these keypoint coordinates, the angle change curves of joints such as the shoulder, elbow, wrist, hip, knee, and ankle can be calculated, along with corresponding kinematic quantities such as motion velocity and acceleration, thereby characterizing the amplitude, rhythm, and direction changes of the interactive object's limb movements during speech. By inputting the aforementioned kinematic features into a temporal neural network model (such as a one-dimensional convolutional network or a recurrent neural network), it is possible to automatically learn emotion-related movement patterns from time series. For example, frequent and large-amplitude hand movements are often associated with high arousal emotions such as excitement and anger, while small-amplitude and slow-paced movements are more likely to correspond to calm or low-arousal emotions, thus forming a first gesture feature vector with emotion discrimination capabilities.

[0031] S13. Perform pre-emphasis, framing, and windowing processing on the speech signal, calculate Mel-frequency cepstral coefficients and / or spectrograms, and input the time-frequency features into the speech emotion recognition model to obtain the first speech feature vector. In this step, it should be noted that pre-emphasis processing enhances high-frequency components, while framing and windowing divide continuous speech into short, stationary segments, facilitating the extraction of stable acoustic features. Mel-frequency cepstral coefficients can describe the speech spectral envelope at a Mel scale close to human auditory perception, while spectrograms display the spectral energy distribution over time as a two-dimensional image. Both types of time-frequency features can effectively reflect emotion-related information such as intonation, speech rate, and stress in speech. Inputting these features into a speech emotion recognition model (e.g., a convolutional neural network, a model combining convolutional and recurrent networks, or a model based on self-attention mechanisms) allows the model to learn differences in speech rhythm, pitch variations, and intonation under different emotions, enabling the first speech feature vector output by the model to distinguish between different emotional states such as calm, joy, anger, and sadness. Since emotions are often directly reflected in the way people speak, this speech feature plays an important supplementary role in judging overall emotions.

[0032] S14. The first facial feature vector, first gesture feature vector, and first speech feature vector are normalized and input into a multimodal feature fusion network. An attention mechanism is used, with the first facial feature vector as the query vector, the first gesture feature vector as the key vector, and the first speech feature vector as the value vector, to obtain the fused feature vector. In this step, it should be noted that the numerical ranges and statistical distributions of different modal features often differ. Directly concatenating them without processing can easily lead to one modality having an excessively large or small influence on the result. Therefore, the feature vectors of each modality are first normalized to ensure a relatively uniform numerical scale. The multimodal feature fusion network introduces an attention mechanism, using the first facial feature vector as the query vector, calculating the action features most relevant to the current facial expression from the first gesture feature vector, and using the first speech feature vector as the value vector to weight and aggregate these relevances. This highlights gesture and speech patterns consistent with or corroborating the current expression in the fused feature vector, while weakening inconsistent or noisy components. This design allows the system to prioritize body language and voice changes that match the expression when the interaction object's facial expression is obvious; while when the expression is not obvious or partially obscured, the system can rely more on voice and gesture information, improving the robustness and accuracy of overall emotion judgment.

[0033] S15. Input the fused feature vector into the fully connected layer and the normalized Softmax layer, and output the first emotion vector. The first emotion vector is used to represent the probability distribution of the interactive object in the preset emotion category and / or emotion dimension.

[0034] In this step, it's important to note that the fully connected layer maps the high-dimensional features obtained from multimodal fusion to a predefined emotion space. This could be mapped to score vectors representing several discrete emotion categories, or to continuous representations of emotional dimensions such as pleasure, arousal, and tension. The normalized Softmax layer converts these scores into probability distributions or normalized weights, giving each component of the first emotion vector a clear probability or intensity meaning, facilitating subsequent unified processing and comparison with the emotion vectors of blind individuals. In this way, the complex multimodal behavior of the interacting object is compressed into a single emotion feature vector with clear semantics, preserving the comprehensive discriminative ability of multimodal information while providing a concise vector representation for subsequent mathematical operations and model inference.

[0035] The technical solution implemented in this embodiment can map the facial expressions, body movement patterns, and speaking styles of interactive objects during communication into a unified first emotion vector through the collaborative analysis of facial images, gesture data, and speech signals. This fully utilizes the complementarity and redundancy among the three modalities to improve the accuracy and robustness of emotion state recognition for interactive objects. Specifically, face detection and alignment ensure stable expression of facial features under different postures and perspectives; posture estimation and temporal modeling capture dynamic action information related to emotions; speech sentiment analysis characterizes acoustic emotion cues such as tone and speed; and multimodal attention fusion adaptively allocates the importance of each modality according to different contexts. Through these mechanisms, this embodiment can still obtain reliable emotion estimation results for interactive objects even in noisy environments, with partial occlusion, or with insufficient information from a single modality. This provides high-quality input for subsequent calculation of emotion matching degree and generation of social response strategies, thereby improving the overall emotion perception capability of the social assistance system for the blind.

[0036] Furthermore, in S20, the analysis of the second emotion vector of the blind person's emotional state includes: S21. Filter and remove artifacts from the photoplethysmography (PPG) signal of the blind person. Divide the time window into preset durations, detect the heartbeat interval within each window, and calculate the standard deviation of the heartbeat interval, the root mean square of the difference between adjacent heartbeat intervals, the proportion of heartbeat interval differences that meet preset thresholds, and the ratio of low-frequency energy to high-frequency energy to obtain heart rate variability characteristics. It should be noted in this step that PPG signals are usually acquired through photoelectric sensors worn on the wrist, finger, or earlobe. The original signal may contain motion artifacts, electrical noise, and other interference. Therefore, bandpass filtering and wavelet denoising techniques are first used to filter and remove artifacts from the signal to highlight the periodic components related to the heartbeat. The processed pulse wave signal is then divided into several time windows of preset duration. Within each window, a peak detection algorithm is used to identify the time interval between adjacent heartbeat peaks, thus obtaining a heartbeat interval sequence. Based on this sequence, several heart rate variability indicators can be calculated, including the standard deviation of the heartbeat interval, the root mean square of the difference between adjacent heartbeat intervals, the proportion of heartbeat interval differences exceeding preset thresholds, and the ratio of low-frequency energy to high-frequency energy. These indicators reflect the current physiological arousal level and stress load of the blind person from the perspectives of rhythm stability and autonomic nervous activity, serving as important inputs for subsequent physiological and emotional assessments.

[0037] S22. Smooth and baseline-correct the ESC signal of the blind person, extract the slow-varying and fast-varying components, and calculate the average ESC level, the number of ESC peaks per unit time, and their peak amplitudes to obtain ESC characteristics. In this step, it should be noted that the ESC signal can be obtained through electrodes placed on the palm, fingers, or forearm. The original signal contains both a slowly changing baseline level over time and rapid fluctuations caused by instantaneous emotional stimuli. For ease of analysis, the ESC signal is first smoothed to reduce high-frequency noise. Simultaneously, baseline correction is performed by subtracting the average value over a period of time or using a high-pass / low-pass decomposition method, decomposing the signal into a slow-varying component reflecting the overall activation level and a fast-varying component reflecting the instantaneous response. Based on this, the average ESC level within a certain time window, the number of ESC peaks exceeding a set threshold per unit time, and the amplitude of each peak can be calculated, thus characterizing the physiological activation level and emotional sensitivity of the blind person in the current situation from both intensity and frequency perspectives. The above ESC characteristics complement the heart rate variability characteristics, jointly describing the autonomic nervous system activity state of the blind person.

[0038] S23. The heart rate variability (HRV) and electrodermal transfer (EDT) features are concatenated to form a physiological feature vector, which is then input into the physiological emotion assessment network to obtain a physiological emotion embedding vector. In this step, it should be noted that multiple time-domain, frequency-domain, and nonlinear indicators from HRV analysis, as well as indicators such as average level, number of peaks, and peak amplitude from EDT analysis, can be concatenated into a single physiological feature vector in a fixed order. This physiological feature vector is input into a pre-trained physiological emotion assessment network, such as a deep neural network composed of several fully connected layers and nonlinear activation functions. The network learns the mapping relationship between different combinations of physiological features and different emotional states through training. The physiological emotion embedding vector output by the physiological emotion assessment network comprehensively represents information such as the blind person's physiological arousal level and tension level in a low-dimensional embedding space. Compared to the original feature dimensions, it is more compact and has higher discriminative power, facilitating fusion with other modalities of emotional features.

[0039] S24. Gait cycle detection is performed on the gait signals of blind individuals. Step frequency, stride length, gait cycle variability, and / or gait symmetry indices are calculated. These indices are then input into the gait emotion assessment subnetwork to obtain a gait emotion feature vector. In this step, it should be noted that gait signals can be acquired using inertial measurement units worn on the ankle, waist, or in shoes, containing information such as acceleration and angular velocity. By periodically detecting the gait signals, continuous gait cycles can be identified. From each cycle, indices such as step frequency, stride length, variability in gait cycle length, and left-right gait symmetry are calculated. These indices reflect the movement state of blind individuals during walking or standing micro-movements. Typically, increased step frequency and gait cycle variability may be associated with high arousal emotions such as tension and anxiety, while decreased stride length and gait symmetry may be associated with emotions such as unease and hesitation. Inputting these gait-related indices into the gait emotion assessment subnetwork maps gait patterns to a gait emotion feature vector with emotion discrimination capabilities, used to characterize the emotional expressions at the behavioral level of blind individuals.

[0040] S25. Acoustic features are extracted from the speech signal of the blind person. The obtained acoustic features are then input into the blind person's speech emotion recognition sub-network to obtain a blind person's speech emotion feature vector. In this step, it should be noted that the blind person's speech signal can originate from their natural speech or brief responses during communication. Similar to the processing of the speech of the interacting object, preprocessing operations such as pre-emphasis, framing, and windowing can be performed on the blind person's speech to extract acoustic features including Mel-frequency cepstral coefficients, fundamental frequency, energy envelope, and formants. If necessary, a spectrogram can also be constructed as the input representation. These acoustic features can reflect emotional cues such as speech rate, pitch, volume, and timbre changes during the blind person's speech. Inputting these acoustic features into the blind person's speech emotion recognition sub-network (e.g., a convolutional network or a network combining convolution and recurrent structures) allows the network to learn the correspondence between different acoustic patterns and different emotional states, thereby outputting a blind person's speech emotion feature vector to describe the emotional tendency and level of excitement expressed by the blind person through speech.

[0041] S26. Concatenate the gait emotion feature vector with the blind person's speech emotion feature vector to form a behavioral emotion feature vector. In this step, it should be noted that the gait emotion feature vector focuses on reflecting the blind person's emotional expression in overall movement and posture, while the blind person's speech emotion feature vector focuses on reflecting their emotional characteristics during language expression. Both depict the blind person's overt behavior from different perspectives. By concatenating these two feature vectors, a behavioral emotion feature vector that comprehensively reflects the blind person's emotional information at the behavioral level can be obtained. This allows the system to simultaneously consider emotional cues from both "action" and "speech style" aspects in a unified representation, thereby improving the completeness and robustness of behavioral emotion modeling.

[0042] S27. Based on the subjective question-and-answer results, convert the question-and-answer options into subjective emotion feature vectors according to a preset mapping relationship. In this step, it should be noted that subjective questions can be posed to the blind person in the form of short voice questions, such as asking if their current feeling is "relaxed," "neutral," or "tense," or asking for their subjective rating of the current communication atmosphere. The blind person's voice responses are converted into discrete options or numerical ratings after speech recognition, and then converted into subjective emotion feature vectors according to a preset mapping relationship, such as using one-hot encoding or linearly mapping the ratings to a preset emotion dimension range. The subjective emotion feature vector directly reflects the blind person's perception and evaluation of their own emotional state, and can, to some extent, compensate for the bias that may arise from relying solely on physiological and behavioral signals to infer emotions, providing an important reference for the final emotion estimation.

[0043] S28. Input the physiological emotion embedding vector, behavioral emotion feature vector, and subjective emotion feature vector into the self-emotion fusion network, and output the second emotion vector through attention weighting and nonlinear transformation.

[0044] In this step, it's important to note that the self-emotion fusion network is used to integrate emotional information from three sources: physiological, behavioral, and subjective. Since the reliability of each modality may vary in different situations—for example, physiological indicators may be affected by movement during strenuous activity, while subjective responses may be more reliable in quiet settings—the fusion network can incorporate an attention mechanism. This mechanism adaptively assigns different weights to the physiological emotion embedding vector, behavioral emotion feature vector, and subjective emotion feature vector based on the distribution of input features. Through multi-layer nonlinear transformations, the fusion network integrates the information from the three modalities into a unified emotion feature space, outputting a second emotion vector. This vector comprehensively represents the overall emotional state of the blind person at the current moment, ensuring that it reflects both objective physiological and behavioral responses as well as subjective feelings.

[0045] The technical solution implemented in this embodiment can simultaneously incorporate three types of information—physiological signals, behavioral signals, and subjective feedback—in the modeling of blind individuals' emotions. Physiological emotion embedding vectors reflecting autonomic nervous activity and physiological arousal levels are obtained through feature extraction from photoplethysmography (PPG) and electrodermal signals. Behavioral emotion feature vectors reflecting overt movement and speech patterns are obtained through gait and speech analysis. Subjective emotion feature vectors are constructed by combining the blind individual's self-perception. Finally, a self-emotional fusion network is used to weightedly integrate the above multi-source information within a unified emotional space. Therefore, this embodiment can more accurately and stably depict the true emotional state of blind individuals than single-signal or single-modality schemes, reducing the adverse effects of noise or environmental interference on a particular modality. This provides a highly reliable second emotion vector foundation for subsequent emotion matching calculations and social risk assessments, thereby improving the overall reliability and individual adaptability of blind emotion analysis and social assistance decision-making.

[0046] Furthermore, in S30, calculating the emotion matching degree between the first emotion vector and the second emotion vector includes: S31. Perform linear transformations on the first and second emotion vectors respectively to align their dimensions, resulting in a first normalized emotion vector and a second normalized emotion vector. It should be noted in this step that the first and second emotion vectors originate from the emotion analysis processes of the interacting object and the blind person themselves, respectively, and may differ in the number of dimensions, component meanings, or numerical ranges. To facilitate similarity calculation within a unified emotion feature space, these two emotion vectors need to be linearly transformed to map them to the same dimensional space, and each dimension needs to be scaled. Specifically, a linear mapping layer with learnable weights (e.g., matrix multiplication plus bias) can be used to transform the original emotion vector into a feature representation of a preset dimension, combined with mean-variance standardization or interval scaling to ensure that each component falls within a relatively consistent numerical range. Through the above linear transformation and normalization, the first and second normalized emotion vectors can be obtained, ensuring a one-to-one correspondence in dimensions while mitigating the impact of scale differences in different emotion dimensions on subsequent similarity calculations, thus creating conditions for a fair and stable comparison of the emotional states of both parties.

[0047] S32. Calculate the dot product of the first and second normalized emotion vectors in the same emotion feature space, and divide the dot product by the product of the magnitudes of the two vectors to obtain the cosine similarity. In this step, it should be noted that after dimensional alignment and normalization, the first and second normalized emotion vectors can be considered as two vector points located in the same emotion feature space, and the angle between them reflects the degree of consistency between the two emotional states in the overall trend. By calculating the dot product of the two vectors, their weighted overlap in each emotion dimension can be obtained; simultaneously, by calculating the magnitudes of the two vectors respectively and dividing the dot product by the magnitude product, their cosine similarity can be obtained. Cosine similarity reflects the similarity in the direction of the two vectors, rather than being directly affected by the overall magnitude, thus it is more suitable for measuring whether the "emotional structure" is similar, such as whether the interacting object and the blind person are both in a high-arousal or low-arousal state, or whether they are both biased towards positive or negative emotions. By using cosine similarity as the basic metric, the relative relationship between the two emotion vectors can be numerically and stably characterized.

[0048] S33. Convert the cosine similarity to an emotion matching score within the range of [0, 1] via interval mapping. A higher emotion matching score (closer to 1) indicates a closer match between the emotional state of the interacting object and the blind person's emotional state, while a lower score (closer to 0) indicates a greater difference in their emotions. It should be noted that the theoretical range of cosine similarity is typically [-1, 1]. To facilitate subsequent risk grading using thresholds and joint processing with other normalized indicators, this similarity value needs to be mapped to an emotion matching score within the range of [0, 1]. Specifically, a linear mapping method can be used to linearly compress the [-1, 1] interval to the [0, 1] interval. Alternatively, in actual training, the mapping function can be appropriately adjusted according to the sample distribution, so that neutral emotion combinations correspond to the median value, highly consistent emotion combinations correspond to a high matching value close to 1, and severely mismatched emotion combinations correspond to a low matching value close to 0. The emotion matching degree obtained through this interval mapping process has a unified numerical range and a clear physical meaning. It can be directly used as an input indicator for subsequent strategy generation and risk level determination, enabling the system to measure the degree of emotional resonance in different scenarios with a unified scale.

[0049] The technical solution implemented in this embodiment can perform scale alignment and standardization on the first emotion vector of the interactive object and the second emotion vector of the blind person within a unified emotion feature space. It then uses cosine similarity to characterize the degree of similarity between the two in the emotional direction, and converts this similarity into an emotion matching index with a clear numerical range and semantic interpretation through interval mapping. This emotion matching index can, on the one hand, quantitatively and intuitively determine whether the emotions of both parties are in sync in the current social scenario; on the other hand, it can serve as a core basis for subsequent social response strategy selection and risk level classification, thereby realizing the transformation from "one-way emotion recognition" to "decision-making based on two-way emotional relationships." This is beneficial for improving the interpretability and decision-making accuracy of the blind person's emotion analysis and social assistance system in complex scenarios. Furthermore, in S40, determining the target social coping strategy and corresponding risk level based on the emotional matching degree includes: S41. Construct a training sample set containing various social scenarios. Each training sample includes a first emotion vector, a second emotion vector, the corresponding emotion matching degree, a scenario type label, and social coping strategy labels and risk level labels labeled by professionals. In this step, it should be noted that the training sample set can be derived from real communication data of multiple blind users at different times and in different scenarios, such as daily greetings, work communication, service scenarios, and conflict scenarios. For each communication, the system can record the first emotion vector of the interacting object, the second emotion vector of the blind user, and the emotion matching degree obtained from the previous steps during system runtime, and assign corresponding scenario type labels based on the current application scenario, such as "casual chat," "formal report," "help / consultation," and "conflict / dispute." Simultaneously, professionals with experience in psychology, social work, or blind counseling are invited to annotate these historical samples offline, providing appropriate social coping strategies for the situation (e.g., continuing empathy, moderate de-escalation, changing the topic, or ending the communication) and corresponding risk levels (e.g., low, medium, high risk). The training sample set constructed in the above manner can cover the correspondence between "emotions of both parties + scenario type → strategy and risk" in various real social situations, providing a data foundation for subsequent model learning.

[0050] S42. Using the combination of the first emotion vector, the second emotion vector, the emotion matching degree, and the scene type label as input features, and the social coping strategy label and risk level label as supervision signals, the strategy generation model is trained under supervision. This ensures that after training, similar combinations of input features will produce similar strategy outputs and risk level outputs through the strategy generation model. In this step, it should be noted that the strategy generation model can be a deep neural network, gradient boosting tree, or other machine learning models with non-linear fitting capabilities. Its input is the concatenated combination of emotion and scene features, i.e., a high-dimensional feature vector composed of the first emotion vector, the second emotion vector, the emotion matching degree, and the scene type label encoding; the output is the corresponding social coping strategy category and risk level category. By constructing a loss function that includes strategy labels and risk level labels (e.g., a weighted sum of cross-entropy loss), the model parameters are continuously adjusted during training to ensure that the model can predict the strategies and risk levels labeled by professionals as accurately as possible on the training samples. After sufficient training, the model will learn a mapping relationship from "emotional state of both parties and scenario type" to "appropriate strategy and risk assessment". This enables it to output strategies and risk levels similar to historical annotations when faced with new but pattern-similar inputs, thus realizing a data-driven decision-making mechanism instead of simply relying on manually preset rule tables.

[0051] S43. During the method execution phase, the first emotion vector, the second emotion vector, the emotion matching degree, and the current scene type, calculated in real time, are input into the strategy generation model. The strategy generation model outputs the target social coping strategy and the corresponding risk level, wherein: Targeted social coping strategies include at least one of the following: empathy strategies, de-escalation strategies, topic guidance strategies, and communication termination strategies.

[0052] The risk levels include Level 1, Level 2, and Level 3. The emotion matching degree is a real value in the range [0, 1]. The larger the value, the closer the emotional state of the interactive object is to the emotional state of the blind person; the smaller the value, the greater the difference between the two emotions. The component representing the emotional arousal of the blind person in the second emotion vector is a normalized value in the range [0, 1]. The larger the value, the higher the emotional arousal of the blind person.

[0053] The first risk level is used to indicate a situation where the emotional matching degree is greater than the first matching threshold and the component in the second emotional vector representing the emotional arousal degree of the blind person is lower than the first arousal threshold, corresponding to a state where both the emotional matching degree and the emotional arousal degree are in the preset low-risk range.

[0054] The second risk level is used to indicate the situation where the emotional matching degree is between the first matching threshold and the second matching threshold and / or the component in the second emotion vector that represents the emotional arousal degree of the blind person is between the first arousal threshold and the second arousal threshold, corresponding to the state where the emotional matching degree and / or emotional arousal degree are in the preset intermediate risk range.

[0055] The third risk level is used to indicate situations where the emotional matching degree is lower than the second matching threshold and / or the component in the second emotional vector that represents the emotional arousal degree of the blind person is higher than the second arousal threshold, corresponding to a state where the emotional matching degree and / or emotional arousal degree are in a preset high-risk range.

[0056] The first matching threshold, second matching threshold, first arousal threshold, and second arousal threshold satisfy the following conditions: 0 < second matching threshold < first matching threshold ≤ 1, 0 < first arousal threshold < second arousal threshold ≤ 1. In this step, it should be noted that during actual system operation, after calculating the first emotion vector, second emotion vector, and emotion matching degree for the current moment, and knowing the current scenario type, this information can be used as input features and fed into the previously trained strategy generation model. Based on the mapping relationship learned during training, the model outputs one or more target social coping strategies. For example, when both parties' emotions are highly consistent and the blind person's emotions are stable, it tends to output an empathy strategy; when there is a certain difference in emotions, it tends to output a calming or topic-guiding strategy; and when both parties' emotions are significantly mismatched or the blind person is highly stressed, it tends to output an ending-of-communication or strong calming strategy. Simultaneously, the model also outputs the corresponding risk level, classifying the current situation as low risk (first risk level), medium risk (second risk level), or high risk (third risk level).

[0057] To facilitate a consistent interpretation of risk levels across different users and application environments, this embodiment uses preset first matching thresholds, second matching thresholds, first arousal thresholds, and second arousal thresholds to divide emotional matching degree and the emotional arousal degree of the blind person into three risk ranges: low, medium, and high. Based on these, risk level definitions are provided: when both parties' emotions are highly consistent and the blind person is in a low arousal state, the social situation is considered safe and stable, corresponding to the first risk level; when the matching degree or arousal degree is in the middle range, a certain degree of deviation or tension is considered, corresponding to the second risk level; when there is a severe emotional mismatch and / or the blind person is in a high arousal state, a significant social risk is considered, corresponding to the third risk level. These thresholds can be configured according to different groups or application scenarios or optimized based on historical data, thus balancing uniformity and flexibility.

[0058] The technical solution implemented in this embodiment enables supervised training of the strategy generation model by constructing multi-scenario, multi-sample labeled datasets. This allows the model to automatically learn the complex mapping relationship between "interaction object's emotion—blind person's emotion—emotion matching degree—scenario type" and "social coping strategies and risk levels." During actual system operation, real-time manual judgment is unnecessary. Only the currently calculated first emotion vector, second emotion vector, emotion matching degree, and scenario type need to be input. The model can then quickly output target social coping strategies and clear risk level classifications for specific situations. By leveraging the joint threshold design of emotion matching degree and blind person's emotional arousal, this embodiment can characterize social risk levels in a unified and quantitative manner. It provides differentiated strategy suggestions such as empathy, mitigation, guiding the topic, or ending the interaction in low-risk, medium-risk, and high-risk states, respectively. This avoids the shortcomings of traditional rule-based decision-making, which relies on experience and is difficult to adapt to complex scenarios. It significantly improves the accuracy, adaptability, and interpretability of the blind person's emotion analysis system in various social environments, providing a solid decision-making foundation for subsequent prompt output and blind person behavior adjustment.

[0059] Furthermore, in S50, the control of the blind communication assistive terminal device to output prompt messages specifically includes: When the risk level is the highest, only vibration alerts are output to indicate the current social context, and no voice alerts are output to avoid interfering with normal communication.

[0060] When the risk level is the second risk level, a vibration prompt is output, and a short voice prompt is output when a pause is detected in the current conversation, prompting the blind person to adjust their tone or topic appropriately.

[0061] When the risk level is level 3, an enhanced vibration alert is output, and voice prompts corresponding to the target social coping strategy are output in real time, reminding the blind person to take strategies such as de-escalating the conflict or ending the communication.

[0062] In this embodiment, it should be noted that the communication assistance terminal device for the blind may include one or more vibration actuators and a speaker or bone conduction sound unit placed close to the blind person's ear, for outputting vibration prompts and voice prompts respectively. When the risk level is the first risk level, the terminal, based on the current decision, only controls the vibration actuator to output slight vibrations at a preset vibration frequency and duration, such as periodic shaking with a low frequency and short pulse, to convey to the blind person that "the current social situation is generally safe and no significant adjustments are needed," while completely shutting down the voice broadcast channel to avoid introducing additional sound interference during normal communication. When the risk level is at the second risk level, the terminal maintains vibration prompts on the one hand, by appropriately increasing the vibration frequency or extending the vibration duration, so that the blind person can perceive that "appropriate adjustments are needed at present"; on the other hand, the terminal collects the sound signal of the current conversation through the microphone, uses a speech activity detection algorithm to identify the speaking and pause intervals, and only outputs a short voice prompt on the speaker when the continuous silence time exceeds a preset threshold, such as briefly reminding the blind person to "slow down their speech", "soften their tone", or "change the topic", thereby guiding the blind person to adjust their communication method without interrupting the other person's speech. When the risk level is at the third risk level, the terminal, based on the high-risk assessment, controls the vibration execution unit to output enhanced vibration prompts, such as using a higher vibration frequency, greater vibration intensity, or repeated vibration patterns, to ensure sufficient attention from the blind user. Simultaneously, the terminal no longer only broadcasts during pauses but prioritizes the real-time nature of voice prompts. Based on the target social response strategy output by the strategy generation model, it dynamically generates or invokes corresponding voice prompts, such as explicitly reminding the blind user that "the other party is emotionally agitated; please lower your volume and respond briefly" or "it is recommended to end this communication as soon as possible," to help the blind user take more proactive actions such as de-escalating the conflict or ending the communication. The vibration parameters, voice content length, and voice volume at the different risk levels can be configured or adaptively adjusted according to the blind user's usage habits and environmental noise levels to balance perceptibility and concealment.

[0063] The technical solution implemented in this embodiment can perform graded control of vibration and voice prompts according to different risk levels. In low-risk scenarios, only low-intensity vibrations are used to provide contextual feedback to the blind user discreetly, without affecting the natural fluency of the conversation or exposing the blind user's privacy in using the assistive system. In medium-risk scenarios, vibrations combined with brief voice prompts inserted during pauses in the conversation provide lightweight guidance for the blind user's communication behavior, reducing interference with both parties' communication while helping the blind user make timely minor adjustments. In high-risk scenarios, a combination of enhanced vibrations and real-time voice prompts sends clear and strong risk signals to the blind user and provides specific coping suggestions, enabling the blind user to quickly take strategies such as mitigation or ending the conversation, thereby effectively reducing the risk of potential social conflict. Overall, this embodiment implements a multi-channel, graded feedback mechanism based on risk levels on the output side, which improves the timeliness and understandability of the prompt information while taking into account low interference with the actual conversation and protection of the blind user's privacy. This is beneficial for significantly improving the blind user's emotional perception and communication control abilities in complex social situations.

[0064] Furthermore, the method also includes: S60. Based on the blind person's historical physiological signals, behavioral signals, and historical subjective question-and-answer results at different time periods, construct an individualized training sample set corresponding to the blind person. In this step, it should be noted that historical physiological signals can include photoplethysmography (PPG) signals, electrodermal signals, etc., accumulated over a long period in multiple social scenarios; historical behavioral signals can include gait signals and speech signals collected within the corresponding time period; and historical subjective question-and-answer results can include the blind person's subjective evaluation records of their emotional state at different times and in different situations. The system can organize the above historical data according to a timeline, dividing each social situation into several time windows. Within each time window, using the same feature extraction process as online processing, the corresponding physiological emotion embedding vector, behavioral emotion feature vector, and subjective emotion feature vector are calculated offline to obtain the blind person's second emotion vector at multiple moments. Simultaneously, this historical data can be organized into one or more sets of training samples by combining information such as the type of scene at the time, the target social coping strategy output by the system, and the risk level. By summarizing the historical data from the same blind person across multiple moments and scenarios, an individualized training sample set corresponding to that blind person can be constructed. This sample set can reflect the blind person's physiological baseline level, behavioral habits, and actual response to strategy feedback in different situations.

[0065] S61. Based on the individualized training sample set, fine-tuning is performed on some parameters of the policy generation model to correct for the differences in the blind person's physiological baseline and behavioral habits, thereby obtaining a personalized policy generation model corresponding to the blind person. In this step, it should be noted that the policy generation model is usually trained on group data from multiple users in the initial stage to obtain a basic model with general adaptability. However, different blind people have significant differences in heart rate, skin conductance levels, gait patterns, and speech habits at rest. If a general model is used directly, it may result in "oversensitivity" or "slow response" to some blind people. Therefore, a portion of the policy generation model that is highly correlated with the input feature mapping and decision output but has a relatively small parameter scale (e.g., several layers near the output end, or adaptation layers specifically set for personalization) can be selected. While maintaining the main structure of the model and most parameters unchanged, parameter fine-tuning is performed only on the individualized training sample set of the blind person. During fine-tuning, policy labels and risk level labels can continue to be used as supervision signals, or slight corrections can be made based on existing decisions. With a small learning rate and a limited number of iterations, the model gradually adapts to the blind person's physiological baseline and behavioral patterns, thereby outputting strategies and risk assessment results that better meet the blind person's actual needs. After this fine-tuning process, a personalized policy generation model corresponding to the blind person can be obtained. This model maintains general decision-making capabilities while being further corrected for the individual characteristics of the blind person.

[0066] S62. In the process of performing sentiment analysis on the current social situation and outputting the target social coping strategy and / or risk level, reasoning is performed based on the personalized strategy generation model to generate the target social coping strategy and / or risk level. In this step, it should be noted that after the system completes the calculation of the first emotion vector, the second emotion vector, and the emotion matching degree for the current social situation, and obtains the current scene type, it will select the inference model based on whether the current user has completed individualized training: For blind users who have accumulated some historical data and completed fine-tuning, the system will prioritize calling their corresponding personalized strategy generation model, using the current first emotion vector, second emotion vector, emotion matching degree, and scene type as input, and performing reasoning on this personalized model to directly output the target social coping strategy and / or risk level; for new users who have not yet completed individualized training, a general strategy generation model can be used temporarily for reasoning, and individualized training will be triggered after gradually accumulating historical data. In this way, the system can continuously use a strategy generation model that matches the individual characteristics of the same blind user during the operation phase. This ensures that each decision-making process is corrected based on the blind person's past physiological, behavioral, and subjective feedback, so that the output target social coping strategies and risk levels are more in line with the blind person's actual feelings and coping habits.

[0067] The technical solution implemented in this embodiment can utilize the historical physiological signals, behavioral signals, and subjective question-and-answer results accumulated by blind individuals in different time periods and scenarios to construct a training sample set reflecting the individual characteristics of the blind person. Based on this, the parameters of the general strategy generation model are fine-tuned in a targeted manner, resulting in a personalized strategy generation model that considers the differences in the blind person's physiological baseline and behavioral habits. In actual operation, by prioritizing the use of this personalized model for decision-making, the bias in emotional judgment and misjudgment of risk assessment caused by individual differences can be significantly reduced. This reduces situations where "some people are always judged as high-risk" or "some people are not warned in time," making the target social coping strategies and risk levels output by the system more closely match the blind person's true state and subjective feelings. Overall, this embodiment introduces an individualized adaptive mechanism on top of the general emotion analysis framework, improving the stability and reliability of the blind person's emotion analysis method in long-term use. This helps to enhance the blind person's trust and compliance with the system prompts, thereby further improving the actual effect of social assistance for the blind.

[0068] Furthermore, the present invention also provides a blind person emotion analysis system, applied to any of the above-mentioned blind person emotion analysis methods, the system comprising: The acquisition module is used to acquire interaction signals that characterize the social behavior of the interacting objects and state signals that characterize the blind person's own state.

[0069] The emotion analysis module receives interaction signals and generates first emotion information representing the emotional state of the interaction object, and receives state signals and generates second emotion information representing the emotional state of the blind person.

[0070] The matching and evaluation module calculates the emotional matching degree based on the first and second emotional information, and generates decision information including the target social coping strategy and risk level based on the emotional matching degree.

[0071] The prompt control module receives decision information and controls the output of prompt signals from the blind communication assistive terminal device based on the target social response strategy and risk level. The prompt signals are used to prompt the blind person to adjust their communication behavior.

[0072] In this embodiment, it should be noted that the blind person's emotion analysis system can be integrated into a single blind person's communication assistance terminal device, or it can be implemented collaboratively by a wearable device and an external terminal. Each module can be implemented using physically independent hardware units, or it can be logically divided by the same processor through software functional units. The acquisition module can be connected to sensing hardware such as cameras, microphones, inertial measurement units, photoplethysmography (PPG) sensors, and electrodermal sensors. It continuously acquires interaction signals (e.g., signals including facial expressions, gestures, and voice features) characterizing the social behavior of the interaction object, as well as state signals characterizing the blind person's own state (e.g., signals including physiological, behavioral, and subjective feedback signals), during the interaction between the blind person and the interaction object. The acquired signals are then output to the emotion analysis module in a preset data format. The emotion analysis module can be implemented by the processor inside the terminal executing corresponding programs. It extracts and fuses multimodal features from the interaction signals to generate first emotion information representing the emotional state of the interaction object; simultaneously, it analyzes and fuses physiological, behavioral, and subjective features of the state signals to generate second emotion information representing the blind person's emotional state. The first and second emotion information are then provided to the matching and evaluation module. The matching and evaluation module can calculate the emotion matching degree within a unified emotion feature space based on the aforementioned emotion information. It then combines this with a pre-trained or personalized strategy generation model to generate decision information containing the target social coping strategy and risk level, and forwards this decision information to the prompting control module. The prompting control module can connect to output devices such as vibration actuators, speakers, or bone conduction sound units. Based on the target social coping strategy and risk level in the decision information, it selects vibration, voice, or a combination thereof as a prompting method, controlling the blind person's communication assistance terminal to output corresponding prompting signals. This allows the blind person to perceive the current social situation and system suggestions without relying on vision, thereby adjusting their communication behavior. The modules can interact via a system bus or communication interface, or communicate via wired or wireless networks in a distributed architecture to adapt to different hardware deployment requirements.

[0073] The technical solution implemented in this embodiment can clearly decompose the functions involved in the emotion analysis method for the blind, such as signal acquisition, emotion calculation, matching evaluation, and prompt output, into several collaborative system modules, making the overall system structure clear and the signal flow definite. On the one hand, it is beneficial to decouple the front-end hardware such as cameras and sensors from the emotion analysis algorithm and strategy generation algorithm, making it easier to flexibly configure the location and computing power allocation of the acquisition end and the computing end according to different application scenarios. On the other hand, by uniformly managing the flow of first emotion information, second emotion information, emotion matching degree, and decision information at the system level, it is convenient to upgrade or replace each module independently in the future. For example, the emotion analysis model and strategy generation model can be replaced or optimized without changing the acquisition and prompt hardware. At the same time, the system structure is easy to implement on various carriers such as smart glasses, wristbands, mobile terminals, or edge computing boxes, which helps to improve the engineering feasibility and scalability of the method of the present invention, thereby providing blind people with an integrable, scalable, and easy-to-maintain emotion analysis and social assistance system.

[0074] Furthermore, the system also includes: The storage module is used to store the interaction signals and status signals collected by the acquisition module, as well as the first emotion information, second emotion information, emotion matching degree and / or decision information generated by the emotion analysis module and / or matching and evaluation module, and to serve as historical reference data for subsequent emotion analysis and generation of target social coping strategies.

[0075] In this embodiment, it should be noted that the storage module can be implemented using one or more combinations of terminal local non-volatile memory, external storage media, or remote storage space connected to a cloud server. It is used to structurally store the interaction signals and status signals collected by the acquisition module according to a preset timeline or session identifier, and to correlate and store the first emotion information and second emotion information output by the emotion analysis module, as well as the emotion matching degree and / or decision information output by the matching and evaluation module. Specifically, the storage module can assign a unique identifier to each social session and record the original or preprocessed multimodal signals, emotion vectors, and risk assessment results for the corresponding time period under that identifier. It can also attach metadata such as scene type tags and timestamps for subsequent retrieval and statistics by scene, time, or user. The storage module can also set a data retention strategy to compress, aggregate, or clean up data exceeding a preset retention period to balance the relationship between storage space usage and historical data utilization. In subsequent emotion analysis and target social coping strategy generation, the system can call relevant historical data through the storage module to statistically analyze the blind person's typical emotional patterns in different scenarios, verify the effectiveness of the current strategy, or provide training samples for fine-tuning the individualized model, thereby achieving long-term accumulation and continuous optimization of the emotion analysis and strategy generation process.

[0076] The technical solution implemented in this embodiment enables the orderly and correlated persistent storage of collected multi-source signals and emotional and decision-making information generated at each stage by introducing a dedicated storage module into the system. This allows the system to move beyond short-term analysis of single sessions, enabling it to trace and mine the long-term emotional trajectories and social behavior patterns of blind individuals. Utilizing this historical reference data, the stability and robustness of subsequent emotion analysis and risk assessment can be improved, for example, by comparing the current emotion vector with historical typical states to identify abnormal situations. Furthermore, it provides a data foundation for the continuous optimization and individualized adjustment of the strategy generation model, improving the model's adaptability and predictive accuracy for specific blind users. Simultaneously, the system can evaluate and iteratively update the actual effects of different strategies based on the stored data, thereby evolving from "static strategies" to "data-driven adaptive strategies," significantly enhancing the scalability, maintainability, and long-term value of the blind emotion analysis system.

[0077] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned method for analyzing the emotions of blind people.

[0078] The embodiments described above are merely illustrative of several implementations of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. A method for blind emotion analysis, applied to a blind communication auxiliary terminal device to assist a blind person in communicating with an interactive object, characterized in that, The method comprises: S10, collecting facial images, gesture data and voice signals of an interactive object communicating with the blind person, analyzing the facial images, gesture data and voice signals to obtain a first emotion vector for representing the emotion state of the interactive object; S20, collecting physiological signals, behavior signals and subjective questionnaire results of the blind person, and analyzing the physiological signals, behavior signals and subjective questionnaire results to obtain a second emotion vector for representing the emotion state of the blind person; S30, calculating the emotion matching degree between the first emotion vector and the second emotion vector based on the first emotion vector and the second emotion vector; S40, determining a target social coping strategy and a corresponding risk level according to the emotion matching degree; S50, controlling the blind person communication auxiliary terminal device to output prompt information based on the target social coping strategy and the risk level, and reminding the blind person to adjust the communication behavior.

2. The blind emotion analysis method of claim 1, wherein, The analysis of the facial images, gesture data and voice signals in S10 to obtain the first emotion vector for representing the emotion state of the interactive object comprises: S11, performing face detection and face alignment processing on the facial image sequence, inputting the aligned facial image into a convolutional neural network model to extract a first facial feature vector for representing the intensity of facial muscle action units; S12, inputting the gesture data into a human pose estimation model to obtain a sequence of skeletal key point coordinates of the interactive object in each frame, and calculating joint angle changes, motion speed and acceleration features based on the sequence of skeletal key point coordinates, and inputting the kinematic features into a time series neural network model to obtain a first gesture feature vector; S13, performing pre-emphasis, framing and windowing processing on the voice signal, calculating mel-frequency cepstrum coefficients and / or a spectrogram, and inputting the time-frequency features into a speech emotion recognition model to obtain a first voice feature vector; S14, performing normalization processing on the first facial feature vector, the first gesture feature vector and the first voice feature vector, and inputting them into a multi-modal feature fusion network to obtain a fusion feature vector through attention calculation with the first facial feature vector as a query vector, the first gesture feature vector as a key vector and the first voice feature vector as a value vector; S15, inputting the fusion feature vector into a fully connected layer and a normalized Softmax layer to output the first emotion vector, which is used to represent the probability distribution of the interactive object in a preset emotion category and / or emotion dimension. 3.The blind emotion analysis method of claim 1, wherein, The analysis of the second emotion vector for representing the emotion state of the blind person in S20 comprises: S21, filtering and artifact removal are performed on the photoplethysmogram signal of the blind person, time windows are divided according to a preset time length, interbeat intervals are detected in each window, and interbeat interval standard deviation, adjacent interbeat interval difference root mean square, interbeat interval difference proportion satisfying a preset threshold, and low frequency energy to high frequency energy ratio are calculated to obtain heart rate variability features; S22, the blind person's skin electrical signal is smoothed and baseline corrected, the slow varying component and the fast varying component are extracted, the average skin electrical level, the number of skin electrical peak values per unit time and the peak amplitude are calculated, and the skin electrical characteristics are obtained; S23, the heart rate variability characteristics and the skin electrical characteristics are spliced to form a physiological characteristic vector, and input to a physiological emotion evaluation network to obtain a physiological emotion embedding vector; S24, the gait signal of the blind person is detected to obtain the gait cycle, the step frequency, the step length, the gait cycle fluctuation and / or the gait symmetry index are calculated, and the index is input into a gait emotion evaluation subnetwork to obtain a gait emotion characteristic vector; S25, the acoustic characteristics of the speech signal of the blind person are extracted, and the obtained acoustic characteristics are input into a blind person speech emotion recognition subnetwork to obtain a blind person speech emotion characteristic vector; S26, the gait emotion characteristic vector and the blind person speech emotion characteristic vector are spliced to form a behavioral emotion characteristic vector; S27, according to the subjective question and answer result, the question and answer options are converted into a subjective emotion characteristic vector according to a preset mapping relationship; S28, the physiological emotion embedding vector, the behavioral emotion characteristic vector and the subjective emotion characteristic vector are input into a self-emotion fusion network, and through attention weighting and nonlinear transformation, the second emotion vector is output.

4. The blind emotion analysis method of claim 1, wherein, In the S30, the emotion matching degree between the first emotion vector and the second emotion vector is calculated based on the first emotion vector and the second emotion vector, which includes: S31, the first emotion vector and the second emotion vector are respectively linearly transformed to align the dimensions, to obtain a first normalized emotion vector and a second normalized emotion vector; S32, the dot product of the first normalized emotion vector and the second normalized emotion vector is calculated in the same emotion feature space, and the dot product is divided by the product of the two vector lengths to obtain the cosine similarity; S33, the cosine similarity is converted into an emotion matching degree in the range of [0, 1] through interval mapping, wherein the closer the emotion matching degree to 1, the more consistent the emotional state of the interaction object with that of the blind person, and the closer the emotion matching degree to 0, the greater the emotional difference between the two. 5.The blind emotion analysis method of claim 1, wherein, In the S40, the target social coping strategy and the corresponding risk level are determined according to the emotion matching degree, which includes: S41, a training sample set containing multiple social scenes is constructed, each training sample includes a first emotion vector, a second emotion vector, a corresponding emotion matching degree, a scene type label and a social coping strategy label and a risk level label annotated by a professional; S42, the combination of the first emotion vector, the second emotion vector, the emotion matching degree and the scene type label is used as the input feature, and the social coping strategy label and the risk level label are used as the supervision signal, and the strategy generation model is supervised trained, so that after the training is completed, similar input feature combinations obtain similar strategy output and risk level output through the strategy generation model; S43, in the method running stage, input the first emotion vector, the second emotion vector, the emotion matching degree and the current scene type calculated in real time into the strategy generation model, output the target social coping strategy and the corresponding risk level from the strategy generation model, wherein: The target social coping strategy includes at least one of empathy strategy, mitigation strategy, topic guiding strategy and end of communication strategy; The risk level includes first risk level, second risk level and third risk level, Wherein, the emotion matching degree is a real value in the range of [0, 1], the larger the value, the closer the emotional state of the interaction object to that of the blind person, the smaller the value, the greater the difference between the two emotions; The component representing the emotional arousal of the blind person in the second emotion vector is a normalized value in the range of [0, 1], the larger the value, the higher the emotional arousal of the blind person; The first risk level is used to represent the case that the emotion matching degree is greater than the first matching threshold and the component representing the emotional arousal of the blind person in the second emotion vector is lower than the first arousal threshold, corresponding to the state that the emotion matching degree and the emotional arousal are both in the preset low risk interval; The second risk level is used to represent the case that the emotion matching degree is between the first matching threshold and the second matching threshold and / or the component representing the emotional arousal of the blind person in the second emotion vector is between the first arousal threshold and the second arousal threshold, corresponding to the state that the emotion matching degree and / or the emotional arousal is in the preset intermediate risk interval; The third risk level is used to represent the case that the emotion matching degree is lower than the second matching threshold and / or the component representing the emotional arousal of the blind person in the second emotion vector is higher than the second arousal threshold, corresponding to the state that the emotion matching degree and / or the emotional arousal is in the preset high risk interval; Wherein, the first matching threshold, the second matching threshold, the first arousal threshold and the second arousal threshold satisfy: 0 6. The blind emotion analysis method of claim 5, wherein, In S50, the control of the blind communication auxiliary terminal device outputs the prompt signal, specifically including: When the risk level is the first risk level, only the vibration prompt for prompting the current social situation is output, and no voice prompt is output to avoid interfering with normal communication; When the risk level is the second risk level, output the vibration prompt, and output the short voice prompt when detecting that the current conversation has a pause, prompting the blind person to adjust the tone or topic appropriately; When the risk level is the third risk level, output the enhanced vibration prompt, and output the voice prompt information corresponding to the target social coping strategy in real time, prompting the blind person to take the strategy of downgrading conflict or ending communication.

7. The blind emotion analysis method of claim 1, wherein, The method further comprises: S60, based on the historical physiological signals, behavior signals and historical subjective question and answer results of the blind person in different time periods, construct an individualized training sample set corresponding to the blind person; S61, based on the individualized training sample set, fine-tuning part of parameters of the strategy generation model to correct physiological baseline differences and behavioral habits of the blind person, so as to obtain an individualized strategy generation model corresponding to the blind person; S62, in the process of emotion analysis on the current social situation and output of the target social coping strategy and / or risk level, reasoning based on the individualized strategy generation model to generate the target social coping strategy and / or the risk level.

8. A blind emotion analysis system applied to the blind emotion analysis method according to any one of claims 1 to 7, characterized in that, The system comprises: a collection module configured to collect interaction signals representing social behaviors of an interaction object and state signals representing a state of the blind person; an emotion analysis module configured to receive the interaction signals and generate first emotion information representing an emotional state of the interaction object, and receive the state signals and generate second emotion information representing an emotional state of the blind person; a matching and evaluation module configured to calculate an emotional matching degree based on the first emotion information and the second emotion information, and generate decision information including a target social coping strategy and a risk level according to the emotional matching degree; a prompt control module configured to receive the decision information, and control a blind person communication auxiliary terminal device to output a prompt signal based on the target social coping strategy and the risk level, the prompt signal being used to prompt the blind person to adjust communication behaviors.

9. The blind emotion analysis system of claim 8, wherein, The system further comprises: a storage module configured to store the interaction signals and the state signals collected by the collection module, and the first emotion information, the second emotion information, the emotional matching degree and / or the decision information generated by the emotion analysis module and / or the matching and evaluation module, and use the data as historical reference data for subsequent emotion analysis and target social coping strategy generation.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the blind emotion analysis method of any one of claims 1 to 7.

Images