Non-contact physiological and psychological detection system based on multiple modes

Abstract

The invention discloses a non-contact physiological and psychological detection system based on multiple modes, relates to the cross technical field of artificial intelligence and biomedical engineering, and can collect multi-mode data such as facial images and sound signals of a user at the same time by integrating multiple sensors such as a camera and a microphone. The fusion of the multi-source data enables the system to more comprehensively capture the physiological and psychological state changes of the user, thereby improving the accuracy and reliability of evaluation. The system collects data in a non-contact mode, and discomfort and limitation possibly brought by traditional physiological monitoring equipment are avoided. According to the system, various non-contact detection technologies such as computer vision, voice analysis and natural language processing are comprehensively applied, and the physiological and psychological states of the user can be evaluated from multiple dimensions. The non-contact detection technology reduces the interference to the user, so that the detection process is more natural and efficient.

Description

Technical Field

[0001] The present invention relates to the field of intersectional technology of artificial intelligence and biomedical engineering, and specifically to a multimodal contactless physiological and psychological detection system. Background Art

[0002] With the development of artificial intelligence technology, traditional physiological and mental health detection methods (such as blood pressure monitors, psychological questionnaires, etc.) have certain limitations, usually require special equipment or professional operation, and the detection process is not flexible and real-time. Existing research has gradually turned to non-contact, multimodal detection methods, using computer vision, speech analysis, deep learning and other technologies to accurately detect and predict an individual's physiological and psychological state in real time without contact.

[0003] The purpose of this invention is to design a contactless physiological and psychological detection system based on multimodal information fusion, which combines facial dynamic expression analysis, physiological index detection, sound change detection and natural language processing technologies to comprehensively and real-time evaluate the user's physiological and mental health status. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a multimodal contactless physiological and psychological detection system, which solves the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multimodal contactless physiological and psychological detection system, including a data acquisition module, a feature extraction module, a multimodal information fusion module and a psychological and physiological state assessment module;

[0006] The data acquisition module collects the user's facial image and sound signal through a camera and a microphone;

[0007] The feature extraction module includes a facial expression analysis unit, a physiological index detection unit, a sound change analysis unit and a natural language processing analysis unit;

[0008] Among them, the facial expression analysis unit: uses computer vision technology to extract facial expression features, such as eye expressions, micro-expressions, and facial muscle changes;

[0009] Physiological indicator detection unit: extracts heart rate, respiratory rate and blood oxygen saturation from facial images through computer vision and deep learning models;

[0010] Voice change analysis unit: uses voice signal processing technology to analyze the user's voice characteristics, including speech speed, pitch and volume changes;

[0011] Natural language processing analysis unit: Analyze the conversation between the user and the AI ​​psychologist and extract language features related to the psychological state;

[0012] The multimodal information fusion module is based on the Bayesian network method, which integrates multimodal information from facial expressions, physiological indicators, sound characteristics and language content for comprehensive analysis;

[0013] The psychological and physiological status assessment module combines the fused multimodal information, uses a machine learning model to predict the user's psychological and physiological status, and gives a health assessment report.

[0014] Preferably, the data acquisition module includes an image acquisition unit and an audio acquisition unit;

[0015] The image acquisition unit captures the user's facial image through a camera to obtain the user's facial expression and physiological feature information; the image acquisition unit obtains a continuous facial image sequence I from the camera face ;

[0016]

[0017] In the formula, Represents the preprocessed facial image. Preprocess refers to a series of image processing techniques, including noise removal, brightness adjustment, and geometric correction.

[0018] The audio acquisition unit collects the user's voice signal through a microphone to analyze the user's speech speed, pitch, volume and emotional state; the audio acquisition unit obtains the user's voice signal A and processes the signal through preprocessing steps, including noise suppression and volume normalization; the preprocessing process can be expressed as:

[0019] A pre =AudioPreprocess(A);

[0020] In the formula, A pre Represents the preprocessed audio signal. AudioPreprocess includes filtering, denoising, and volume adjustment.

[0021] Preferably, the feature extraction module includes a facial expression analysis unit, a physiological index detection unit, a sound change analysis unit and a natural language processing analysis unit;

[0022] The facial expression analysis unit extracts the user's facial expression features from the collected facial images through computer vision technology and convolutional neural network (CNN); it can recognize the user's eye expression, micro-expression, muscle movement, etc., and provide data support for judging the user's emotional state;

[0023] Use the preprocessed facial image As input, facial expression features are extracted through a convolutional neural network (CNN):

[0024]

[0025] In the formula, F face Represents the extracted facial feature vector, including eye movement, eyebrow position, and mouth corner change features.

[0026] Preferably, the physiological index detection unit extracts heart rate, respiratory rate and blood oxygen saturation from the facial image by photoplethysmography (PPG) technology;

[0027] From the preprocessed facial image Extract the color change signal of the skin area and calculate the heart rate and blood oxygen saturation through PPG technology:

[0028]

[0029] Where, T represents the heartbeat cycle time extracted from the PPG signal, and HR represents the heart rate per minute;

[0030] The respiratory rate RR is calculated by analyzing the periodic changes of image pixel values ​​through Fourier transform:

[0031] The respiratory rate RR is obtained by the following formula:

[0032]

[0033] Where, T resp Represents the breathing cycle.

[0034] Preferably, the sound change analysis unit uses speech signal processing technology to extract features from the user's speech signal, including speech speed,

[0035] pitch and volume;

[0036] The preprocessed audio signal A pre Perform MFCC (Mel-frequency cepstral coefficient) extraction to obtain the audio feature vector F audio :

[0037] The audio feature vector F audio Obtained by the following formula:

[0038] F audio =MFCC(A pre );

[0039] In the formula, F audio Represents the audio feature vector, which specifically includes the frequency characteristics and time domain characteristics of the speech signal.

[0040] The natural language processing analysis unit extracts language features related to the psychological state, including emotional vocabulary and semantic patterns, by analyzing the conversation content between the user and the AI ​​psychologist;

[0041] Use the preprocessed text conversation data T pre , perform semantic analysis through the pre-trained language model BERT and extract the language feature vector F nlp ;

[0042] The language feature vector F nlp Obtained by the following formula:

[0043] F nlp =BERT(T pre );

[0044] In the formula, F nlp Represents a language feature vector, specifically including the user's psychological emotional characteristics, which is used to identify changes in the user's emotional state, including anxiety and depression.

[0045] Preferably, the multimodal information fusion module includes a feature alignment and normalization unit and a feature fusion and weighted summation unit;

[0046] The feature alignment and normalization unit performs alignment and normalization processing on the features to ensure that all modal features are fused at the same scale;

[0047] F aligned (t)=Align(F face (t),F physio (t),F audio (t),F nlp (t));

[0048] In the formula, F aligned (t) represents the aligned feature vector, F face (t),F physio (t),F audio (t) and F nlp (t) represents the facial feature vector, physiological index feature vector, voice feature vector and natural language feature vector at time t respectively; Align(.) represents the alignment function;

[0049] The aligned features are normalized to eliminate the scale differences between different modalities and are obtained using the following formula:

[0050] F norm (t) = Normalize(F aligned (t));

[0051] In the formula, Normalize(.) represents the feature normalization operation.

[0052] Preferably, after completing feature alignment and normalization, the feature fusion and weighted summation unit is responsible for fusing features of different modalities; including a Bayesian network or a deep neural network, which integrates the features of each modality through learning a weighted summation model, thereby generating a unified evaluation feature vector;

[0053]

[0054] In the formula, F fusion (t) represents the normalized eigenvector of the ith mode, ω i Represents the weight parameter.

[0055] Preferably, the psychological and physiological state evaluation module uses the fused feature vector to evaluate the user's psychological and physiological state in real time through a pre-trained machine learning model. It is responsible for processing the current input data and outputting the current state evaluation result;

[0056] S(t)=ML_Model(F fusion (t));

[0057] Where S(t) represents the psychological and physiological state evaluation results of the user at time t, ML_Model represents the machine learning model, and F fusion (t) The fused feature vector, which represents the features extracted and fused from different modalities at time t;

[0058] The output state evaluation result S(t) is a multidimensional vector, which includes the user's psychological state of anxiety, depression and physiological state of abnormal heart rate and fatigue at the current moment.

[0059] The present invention provides a multi-modal contactless physiological and psychological detection system, which has the following beneficial effects:

[0060] (1) By integrating multiple sensors such as cameras and microphones, the system can simultaneously collect multimodal data such as the user's facial image and sound signals. This fusion of multi-source data enables the system to more comprehensively capture changes in the user's physiological and psychological state, thereby improving the accuracy and reliability of the assessment. The system uses a non-contact method to collect data, avoiding the discomfort and limitations that may be caused by traditional physiological monitoring equipment. Users only need to face the camera and microphone to detect their physiological and psychological state, which greatly improves the convenience and comfort of use.

[0061] The system can process and analyze the collected data in real time, and quickly provide feedback on the user's physiological and psychological state. This real-time performance enables the system to detect changes in the user's status in a timely manner, making it possible to intervene in time and provide personalized suggestions. At the same time, the system also supports dynamic monitoring, which can continuously track changes in the user's physiological and psychological state and provide users with long-term health management and guidance. The system uses machine learning models to conduct a comprehensive analysis of the fused multimodal information, and can provide personalized physiological and psychological state assessment reports based on the characteristics and needs of different users. This intelligent processing method makes the system closer to the actual needs of users and improves the pertinence and effectiveness of the assessment.

[0062] (2) By combining the data from the image acquisition unit and the audio acquisition unit, this embodiment can provide a more comprehensive user emotion analysis method. By analyzing the user's facial expression and physiological feature information, as well as the speech speed, pitch, volume and emotional state in the voice signal, the system can more accurately judge the user's emotional state.

[0063] The system uses a variety of non-contact detection technologies, such as computer vision, speech analysis and natural language processing, to assess the user's physiological and psychological state from multiple dimensions. Non-contact detection technology reduces interference to the user, making the detection process more natural and efficient.

[0064] The system uses time series models, such as long short-term memory networks (LSTM), to dynamically track and analyze the user's status in real time. This real-time nature enables the system to respond quickly when the user's status changes, thereby improving the accuracy and timeliness of the evaluation.

[0065] The system uses big data and machine learning technology to build a health status prediction model, which can detect users' psychological and physiological abnormalities in advance, provide early warnings and recommend corresponding intervention measures. This function helps users identify potential health problems at an early stage.

[0066] (3) By integrating the multimodal data extracted by the facial expression analysis unit, the physiological index detection unit, the sound change analysis unit, and the natural language processing analysis unit, a comprehensive assessment of the user's psychological state can be achieved. This comprehensive analysis method can provide more accurate and comprehensive psychological health assessment results than a single modality.

[0067] The facial expression analysis unit can capture subtle changes in the user's expression during communication with the AI ​​psychologist, and these changes are often closely related to the user's true emotional state. The physiological indicator detection unit extracts physiological indicators such as heart rate, respiratory rate and blood oxygen saturation from facial images in a non-invasive way. These indicators can reflect the user's physical reactions and autonomic nervous system activities, thereby indirectly reflecting the psychological state. The sound change analysis unit analyzes the voice signal to extract features such as speech speed, pitch and volume. These features can reveal the user's emotional fluctuations and psychological pressure. The natural language processing analysis unit analyzes the content of the conversation, extracts emotional vocabulary and semantic patterns, and thus identifies the user's psychological and emotional characteristics.

[0068] (4) Through multimodal data fusion, the system can comprehensively consider information from different sensors to provide a more comprehensive and accurate assessment of psychological and physiological status. Combining multiple physiological and behavioral signals such as heart rate, skin electrodermal activity, facial expressions, and voice characteristics can more accurately judge the user's emotional state and health status.

[0069] The system uses a real-time data processing and evaluation mechanism to provide instant feedback on the user's current status. This is especially important for scenarios that require real-time monitoring of physiological and psychological states. Because the system uses a machine learning model, it can be personalized based on the user's personal characteristics and historical data, thereby providing evaluation results that are more in line with individual characteristics. As the system is used for a longer period of time, the system's evaluation accuracy will gradually improve, better adapting to the user's specific needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the flow chart of the non-contact physiological and psychological detection system based on multi-modality of the present invention; DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] Example 1

[0073] The present invention provides a non-contact physiological and psychological detection system based on multi-modality, please refer to Figure 1 , including data acquisition module, feature extraction module, multimodal information fusion module and psychological and physiological state assessment module;

[0074] The data acquisition module collects the user's facial image and sound signal through a camera and a microphone;

[0075] The feature extraction module includes a facial expression analysis unit, a physiological index detection unit, a sound change analysis unit and a natural language processing analysis unit;

[0076] Among them, the facial expression analysis unit: uses computer vision technology to extract facial expression features, such as eye expressions, micro-expressions, and facial muscle changes;

[0077] Physiological indicator detection unit: extracts heart rate, respiratory rate and blood oxygen saturation from facial images through computer vision and deep learning models;

[0078] Voice change analysis unit: uses voice signal processing technology to analyze the user's voice characteristics, including speech speed, pitch and volume changes;

[0079] Natural language processing analysis unit: Analyze the conversation between the user and the AI ​​psychologist and extract language features related to the psychological state;

[0080] The multimodal information fusion module is based on the Bayesian network method, which integrates multimodal information from facial expressions, physiological indicators, sound characteristics and language content for comprehensive analysis;

[0081] The psychological and physiological status assessment module combines the fused multimodal information, uses a machine learning model to predict the user's psychological and physiological status, and gives a health assessment report.

[0082] In this embodiment, by integrating multiple sensors such as cameras and microphones, the system can simultaneously collect multimodal data such as the user's facial image and sound signals. This fusion of multi-source data enables the system to more comprehensively capture changes in the user's physiological and psychological state, thereby improving the accuracy and reliability of the assessment. The system uses a contactless method to collect data, avoiding the discomfort and limitations that may be caused by traditional physiological monitoring equipment. Users only need to face the camera and microphone to detect their physiological and psychological states, which greatly improves the convenience and comfort of use.

[0083] The system can process and analyze the collected data in real time, and quickly provide feedback on the user's physiological and psychological state. This real-time performance enables the system to detect changes in the user's status in a timely manner, making it possible to intervene in time and provide personalized suggestions. At the same time, the system also supports dynamic monitoring, which can continuously track changes in the user's physiological and psychological state and provide users with long-term health management and guidance. The system uses machine learning models to conduct a comprehensive analysis of the fused multimodal information, and can provide personalized physiological and psychological state assessment reports based on the characteristics and needs of different users. This intelligent processing method makes the system closer to the actual needs of users and improves the pertinence and effectiveness of the assessment.

[0084] Example 2

[0085] This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically: the data acquisition module includes an image acquisition unit and an audio acquisition unit;

[0086] The image acquisition unit captures the user's facial image through a camera to obtain the user's facial expression and physiological feature information; the image acquisition unit obtains a continuous facial image sequence I from the camera face ;

[0087]

[0088] In the formula, Represents the preprocessed facial image. Preprocess refers to a series of image processing techniques, including noise removal, brightness adjustment, and geometric correction.

[0089] The audio acquisition unit collects the user's voice signal through a microphone to analyze the user's speech speed, pitch, volume and emotional state; the audio acquisition unit obtains the user's voice signal A and processes the signal through preprocessing steps, including noise suppression and volume normalization; the preprocessing process can be expressed as:

[0090] A pre =AudioPreprocess(A);

[0091] In the formula, A pre Represents the preprocessed audio signal. AudioPreprocess includes filtering, denoising, and volume adjustment.

[0092] In this embodiment, by combining the data of the image acquisition unit and the audio acquisition unit, this embodiment can provide a more comprehensive user emotion analysis method. By analyzing the user's facial expression and physiological feature information, as well as the speech speed, pitch, volume and emotional state in the voice signal, the system can more accurately judge the user's emotional state.

[0093] The system uses a variety of non-contact detection technologies, such as computer vision, speech analysis and natural language processing, to assess the user's physiological and psychological state from multiple dimensions. Non-contact detection technology reduces interference to the user, making the detection process more natural and efficient.

[0094] The system uses time series models, such as long short-term memory networks (LSTM), to dynamically track and analyze the user's status in real time. This real-time nature enables the system to respond quickly when the user's status changes, thereby improving the accuracy and timeliness of the evaluation.

[0095] The system uses big data and machine learning technology to build a health status prediction model, which can detect users' psychological and physiological abnormalities in advance, provide early warnings and recommend corresponding intervention measures. This function helps users identify potential health problems at an early stage.

[0096] Example 3

[0097] This embodiment is explained in Example 2. Please refer to Figure 1 ,Specifically: the feature extraction module includes a facial expression analysis unit, a physiological index detection unit, a sound change analysis unit and a natural language processing analysis unit;

[0098] The facial expression analysis unit extracts the user's facial expression features from the collected facial images through computer vision technology and convolutional neural network (CNN); it can recognize the user's eye expression, micro-expression, muscle movement, etc., and provide data support for judging the user's emotional state;

[0099] Use the preprocessed facial image As input, facial expression features are extracted through a convolutional neural network (CNN):

[0100]

[0101] In the formula, F face Represents the extracted facial feature vector, including eye movement, eyebrow position, and mouth corner change features.

[0102] The physiological index detection unit extracts heart rate, respiratory rate and blood oxygen saturation from the facial image through photoplethysmography (PPG) technology;

[0103] From the preprocessed facial image Extract the color change signal of the skin area and calculate the heart rate and blood oxygen saturation through PPG technology:

[0104]

[0105] Where, T represents the heartbeat cycle time extracted from the PPG signal, and HR represents the heart rate per minute;

[0106] The respiratory rate RR is calculated by analyzing the periodic changes of image pixel values ​​through Fourier transform:

[0107] The respiratory rate RR is obtained by the following formula:

[0108]

[0109] Where, T resp Represents the breathing cycle.

[0110] The sound change analysis unit uses speech signal processing technology to extract features from the user's speech signal, including speech speed,

[0111] pitch and volume;

[0112] The preprocessed audio signal A pre Perform MFCC (Mel-frequency cepstral coefficient) extraction to obtain the audio feature vector F audio :

[0113] The audio feature vector F audio Obtained by the following formula:

[0114] F audio =MFCC(A pre );

[0115] In the formula, F audio Represents the audio feature vector, which specifically includes the frequency characteristics and time domain characteristics of the speech signal.

[0116] The natural language processing analysis unit extracts language features related to the psychological state, including emotional vocabulary and semantic patterns, by analyzing the conversation content between the user and the AI ​​psychologist;

[0117] Use the preprocessed text conversation data T pre , perform semantic analysis through the pre-trained language model BERT and extract the language feature vector F nlp ;

[0118] The language feature vector F nlp Obtained by the following formula:

[0119] F nlp =BERT(T pre );

[0120] In the formula, F nlp Represents a language feature vector, specifically including the user's psychological emotional characteristics, which is used to identify changes in the user's emotional state, including anxiety and depression.

[0121] In this embodiment, by integrating the multimodal data extracted by the facial expression analysis unit, the physiological index detection unit, the sound change analysis unit and the natural language processing analysis unit, a comprehensive assessment of the user's psychological state can be achieved. This comprehensive analysis method can provide a more accurate and comprehensive psychological health assessment result than a single modality.

[0122] The facial expression analysis unit can capture subtle changes in the user's expression during communication with the AI ​​psychologist, and these changes are often closely related to the user's true emotional state. The physiological indicator detection unit extracts physiological indicators such as heart rate, respiratory rate and blood oxygen saturation from facial images in a non-invasive way. These indicators can reflect the user's physical reactions and autonomic nervous system activities, thereby indirectly reflecting the psychological state. The sound change analysis unit analyzes the voice signal to extract features such as speech speed, pitch and volume. These features can reveal the user's emotional fluctuations and psychological pressure. The natural language processing analysis unit analyzes the content of the conversation, extracts emotional vocabulary and semantic patterns, and thus identifies the user's psychological and emotional characteristics.

[0123] By integrating and analyzing these multimodal data, we can more accurately identify the user's psychological state, such as anxiety, depression and other emotional problems. In addition, through real-time monitoring and analysis, the system can promptly detect abnormal changes in the user's emotional state and provide corresponding intervention measures, such as reminding users to do deep breathing exercises, guiding users to do relaxation training or suggesting users to seek professional psychological support.

[0124] Example 4

[0125] This embodiment is explained in Example 3, please refer to Figure 1 ,Specifically: the multimodal information fusion module includes a feature alignment and normalization unit and a feature fusion and weighted summation unit;

[0126] The feature alignment and normalization unit performs alignment and normalization processing on the features to ensure that all modal features are fused at the same scale;

[0127] F aligned (t)=Align(F face (t),F physio (t),F audio (t),F nlp (t));

[0128] In the formula, F aligned (t) represents the aligned feature vector, F face (t),F physio (t),F audio (t) and F nlp (t) represents the facial feature vector, physiological index feature vector, voice feature vector and natural language feature vector at time t respectively; Align(.) represents the alignment function;

[0129] The aligned features are normalized to eliminate the scale differences between different modalities and are obtained using the following formula:

[0130] F norm(t) = Normalize(F aligned (t));

[0131] In the formula, Normalize(.) represents the feature normalization operation.

[0132] After the feature alignment and normalization are completed, the feature fusion and weighted summation unit is responsible for fusing the features of different modalities; including a Bayesian network or a deep neural network, which integrates the features of each modality through the learning of a weighted summation model, thereby generating a unified evaluation feature vector;

[0133]

[0134] In the formula, F fusion (t) represents the normalized eigenvector of the ith mode, ω i Represents the weight parameter.

[0135] In this embodiment, through the design of the above-mentioned multimodal information fusion module, it is possible to achieve effective integration of different modal features, thereby improving the overall performance and accuracy of the system. Through the feature alignment and normalization unit, it is ensured that the features of different modalities are fused at the same scale, so that the system can more accurately capture the correlation between the modalities, thereby improving the feature expression ability. Since the features of different modalities are normalized during the fusion process, the system has stronger robustness to the noise and changes of the input data, and can perform feature fusion and evaluation more stably.

[0136] The feature fusion and weighted summation unit learns the weight parameters of different modal features, allowing the system to perform weighted summation according to the importance of each modal feature, thereby optimizing the decision-making process and improving the accuracy of the evaluation results.

[0137] The multimodal information fusion module design in this embodiment has good flexibility and scalability, and can flexibly add or delete modal features according to the needs of actual application scenarios to adapt to different tasks and environments.

[0138] Example 5

[0139] For a multimodal contactless physiological and psychological detection system, please refer to Figure 1 Specifically: The psychological and physiological state evaluation module uses the fused feature vector to evaluate the user's psychological and physiological state in real time through a pre-trained machine learning model. It is responsible for processing the current input data and outputting the current state evaluation result;

[0140] S(t)=ML_Model(F fusion (t));

[0141] Where S(t) represents the psychological and physiological status evaluation results of the user at time t, ML_Model represents the machine learning model, and F fusion (t) The fused feature vector, which represents the features extracted and fused from different modalities at time t;

[0142] The output state evaluation result S(t) is a multidimensional vector, which includes the user's psychological state of anxiety, depression and physiological state of abnormal heart rate and fatigue at the current moment.

[0143] In this embodiment, through multimodal data fusion, the system can comprehensively consider the information from different sensors to provide a more comprehensive and accurate psychological and physiological state assessment. Combining multiple physiological and behavioral signals such as heart rate, skin electrodermal activity, facial expressions and voice characteristics can more accurately judge the user's emotional state and health status.

[0144] The system uses real-time data processing and evaluation mechanisms to provide instant feedback on the user's current status. Because the system uses a machine learning model, it can be personalized based on the user's personal characteristics and historical data, thereby providing evaluation results that are more in line with individual characteristics. As the system is used for a longer period of time, the system's evaluation accuracy will gradually improve, better adapting to the user's specific needs.

[0145] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal contactless physiological and psychological detection system, characterized by: It includes data acquisition module, feature extraction module, multimodal information fusion module and psychological and physiological state assessment module; The data acquisition module collects the user's facial image and sound signal through a camera and a microphone; The feature extraction module includes a facial expression analysis unit, a physiological index detection unit, a sound change analysis unit and a natural language processing analysis unit; Among them, the facial expression analysis unit: uses computer vision technology to extract facial expression features, such as eye expressions, micro-expressions, and facial muscle changes; Physiological indicator detection unit: extracts heart rate, respiratory rate and blood oxygen saturation from facial images through computer vision and deep learning models; Voice change analysis unit: uses voice signal processing technology to analyze the user's voice characteristics, including speech speed, pitch and volume changes; Natural language processing analysis unit: Analyze the conversation between the user and the AI ​​psychologist and extract language features related to the psychological state; The multimodal information fusion module is based on the Bayesian network method, which integrates multimodal information from facial expressions, physiological indicators, sound characteristics and language content for comprehensive analysis; The psychological and physiological status assessment module combines the fused multimodal information, uses a machine learning model to predict the user's psychological and physiological status, and gives a health assessment report.

2. The multimodal non-contact physiological and psychological detection system according to claim 1, characterized in that: The data acquisition module includes an image acquisition unit and an audio acquisition unit; The image acquisition unit captures the user's facial image through a camera to obtain the user's facial expression and physiological feature information; The image acquisition unit obtains a continuous facial image sequence I from the camera face ; In the formula, Represents the preprocessed facial image. Preprocess refers to a series of image processing techniques, including noise removal, brightness adjustment, and geometric correction. The audio acquisition unit collects the user's voice signal through a microphone to analyze the user's speech speed, pitch, volume and emotional state; the audio acquisition unit obtains the user's voice signal A and processes the signal through preprocessing steps, including noise suppression and volume normalization; the preprocessing process can be expressed as: A pre =AudioPreprocess(A); In the formula, A pre Represents the preprocessed audio signal. AudioPreprocess includes filtering, denoising, and volume adjustment.

3. The multimodal non-contact physiological and psychological detection system according to claim 1, characterized in that: The feature extraction module includes a facial expression analysis unit, a physiological index detection unit, a sound change analysis unit and a natural language processing analysis unit; The facial expression analysis unit extracts the user's facial expression features from the collected facial images through computer vision technology and convolutional neural network (CNN); it can recognize the user's eye expression, micro-expression, muscle movement, etc., and provide data support for judging the user's emotional state; Use the preprocessed facial image As input, facial expression features are extracted through a convolutional neural network (CNN): In the formula, F face Represents the extracted facial feature vector, including eye movement, eyebrow position, and mouth corner change features.

4. The multimodal non-contact physiological and psychological detection system according to claim 3, characterized in that: The physiological index detection unit extracts heart rate, respiratory rate and blood oxygen saturation from the facial image through photoplethysmography (PPG) technology; From the preprocessed facial image Extract the color change signal of the skin area and calculate the heart rate and blood oxygen saturation through PPG technology: Where, T represents the heartbeat cycle time extracted from the PPG signal, and HR represents the heart rate per minute; The respiratory rate RR is calculated by analyzing the periodic changes of image pixel values ​​through Fourier transform: The respiratory rate RR is obtained by the following formula: Where, T resp Represents the breathing cycle.

5. The multimodal non-contact physiological and psychological detection system according to claim 3, characterized in that: The sound change analysis unit uses speech signal processing technology to extract features from the user's speech signal, including speech speed, pitch and volume; For the preprocessed audio signal A pre Perform MFCC (Mel-frequency cepstral coefficient) extraction to obtain the audio feature vector F audio : The audio feature vector F audio Obtained by the following formula: F audio MFCC(A pre )4 In the formula, F audio Represents the audio feature vector, which specifically includes the frequency characteristics and time domain characteristics of the speech signal.

6. The multimodal non-contact physiological and psychological detection system according to claim 3, characterized in that: The natural language processing analysis unit extracts language features related to the psychological state, including emotional vocabulary and semantic patterns, by analyzing the conversation content between the user and the AI ​​psychologist; Use the preprocessed text conversation data T pre , perform semantic analysis through the pre-trained language model BERT and extract the language feature vector F nlp ; The language feature vector F nlp Obtained by the following formula: F nlp =BERT(T pre ); In the formula, F nlp Represents a language feature vector, specifically including the user's psychological emotional characteristics, which is used to identify changes in the user's emotional state, including anxiety and depression.

7. The multimodal non-contact physiological and psychological detection system according to claim 1, characterized in that: The multimodal information fusion module includes a feature alignment and normalization unit and a feature fusion and weighted summation unit; The feature alignment and normalization unit performs alignment and normalization processing on the features to ensure that all modal features are fused at the same scale; F aligned (t)=Align(F face (t),F physio (t),F audio (t),F nlp (t)); In the formula, F aligned (t) represents the aligned feature vector, F face (t),F physio (t),F audio (t) and F nlp (t) represents the facial feature vector, physiological index feature vector, voice feature vector and natural language feature vector at time t respectively; Align(.) represents the alignment function; The aligned features are normalized to eliminate the scale differences between different modalities and are obtained using the following formula: F norm (t)=Normalize(F aligned (t)); In the formula, Normalize(.) represents the feature normalization operation.

8. The multimodal non-contact physiological and psychological detection system according to claim 7, characterized in that: After the feature alignment and normalization are completed, the feature fusion and weighted summation unit is responsible for fusing the features of different modalities; including a Bayesian network or a deep neural network, which integrates the features of each modality through the learning of a weighted summation model, thereby generating a unified evaluation feature vector; In the formula, F fusion (t) represents the normalized eigenvector of the ith mode, ω i Represents the weight parameter.

9. The multimodal non-contact physiological and psychological detection system according to claim 1, characterized in that: The psychological and physiological state evaluation module uses the fused feature vector to evaluate the user's psychological and physiological state in real time through a pre-trained machine learning model. It is responsible for processing the current input data and outputting the current state evaluation result; S(t)=ML_Model(F fusion (t)); Where S(t) represents the psychological and physiological status evaluation results of the user at time t, ML_Model represents the machine learning model, and F fusion (t) The fused feature vector, which represents the features extracted and fused from different modalities at time t; The output state evaluation result S(t) is a multidimensional vector, which includes the user's psychological state of anxiety, depression and physiological state of abnormal heart rate and fatigue at the current moment.

Images