Vital Sign Acquisition and Analysis System and Method Based on Multimodal Physiological Data

Abstract

The present application provides a vital sign acquisition and analysis system and method based on multimodal physiological data. The method classifies tags for each emotional feature corresponding to the multimodal physiological data to obtain multiple emotional tags and corresponding tag times; after extracting the vital sign information, windowing is performed on the vital sign information to obtain multiple information windows and health scores are given to the vital sign information in each information window; when the emotional relevance is higher than a preset threshold, according to the emotional relevance, each emotional tag and the corresponding tag time, determine the emotional correction coefficient corresponding to each information window, and generate the vital sign analysis result based on the health scores and emotional correction coefficients corresponding to each information window of the vital sign information. In this way, it is possible to correct the vital sign analysis result based on the emotional features of the multimodal physiological data, and reduce the interference of emotional fluctuations of the user during the acquisition of vital sign information on the vital sign analysis result.

Description

Technical Field

[0001] This application relates to the technical field of vital sign analysis, and more specifically, to a vital sign acquisition and analysis system and method based on multi-modal physiological data. Background Art

[0002] Vital signs refer to physiological indicators used to judge the severity and criticality of a patient's condition, including body temperature, pulse, respiration, blood pressure, heart rate, etc. With the rapid development of wearable devices, the public has generally improved their ability to independently manage their own health. By monitoring various physiological parameters through wearable devices and understanding vital sign data, changes in these vital sign data may contain early clues to chronic diseases.

[0003] In the prior art, the process of collecting users' vital sign information is relatively complex. Especially when users are collecting vital sign information, they often face a series of emotions such as tension and anxiety, resulting in large fluctuations in the obtained vital sign information, making the analysis results of users' vital signs inaccurate, which greatly restricts the development of vital sign acquisition and analysis systems. Summary of the Invention

[0004] This application provides a vital sign acquisition and analysis system and method based on multi-modal physiological data, which can correct the vital sign analysis results based on the emotional characteristics of multi-modal physiological data, and reduce the interference of emotional fluctuations during the collection of vital sign information on the vital sign analysis results.

[0005] In a first aspect, this application provides a vital sign acquisition and analysis method based on multi-modal physiological data. This method can be executed by a network device, or can also be executed by a chip configured in the network device. This application does not make any limitations in this regard.

[0006] Specifically, the method includes:

[0007] Collect multi-modal physiological data, extract emotional characteristics from the multi-modal physiological data based on a preset emotion correction time, and obtain multiple emotional characteristics corresponding to the multi-modal physiological data;

[0008] Classify the respective emotional characteristics corresponding to the multi-modal physiological data to obtain multiple emotional labels and corresponding label times;

[0009] Extract the vital sign information from the multi-modal physiological data, window the vital sign information according to the emotion correction time to obtain multiple information windows, and perform a health score on the vital sign information in each information window;

[0010] Analyze the correlation degree of the health scores of each information window based on each emotion label and the corresponding label time, and determine the emotion relevance corresponding to the health score;

[0011] When the emotion relevance is higher than a preset threshold, determine the emotion correction coefficient corresponding to each information window according to the emotion relevance, each emotion label and the corresponding label time, and generate a vital sign analysis result based on the health score and the emotion correction coefficient corresponding to each information window of the vital sign information.

[0012] Combined with the first aspect, in some implementation manners of the first aspect, extracting emotion features from the multimodal physiological data based on a preset emotion correction time, and obtaining multiple emotion features corresponding to the multimodal physiological data specifically includes:

[0013] Obtain the multimodal physiological data, segment the multimodal physiological data based on a preset emotion correction time to obtain multiple multimodal physiological data segments, and for any one multimodal physiological data segment, perform data fitting on the multimodal physiological data segment to obtain the fitting data of each modality;

[0014] Determine the basic level features corresponding to the fitting data of each modality, and remove the basic level differences from the fitting data of each modality according to the basic level features corresponding to the fitting data of each modality to obtain the physiological characteristic waves corresponding to each modality;

[0015] Perform Fourier transform on the physiological characteristic waves corresponding to each modality to obtain the energy spectrum corresponding to each modality, and extract feature vectors based on the physiological characteristic waves and the energy spectrum corresponding to each modality to obtain the emotion features corresponding to the multimodal physiological data segment;

[0016] Extract emotion features from other multimodal physiological data segments in the same manner until the emotion features corresponding to each multimodal data segment are obtained and used as the multiple emotion features corresponding to the multimodal physiological data.

[0017] Combined with the first aspect, in some implementation manners of the first aspect, obtaining multiple information windows and performing a health score on the vital sign information in each information window specifically includes:

[0018] Extract features from the vital sign information in each information window to obtain the key health score features corresponding to each information window;

[0019] Obtain a health score sample, and cluster each key health score feature based on the health score sample to obtain the health score corresponding to each information window.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the correlation degree analysis of the health scores of each information window is performed based on each emotion label and the corresponding label time to determine the emotion relevance corresponding to the health score, specifically including:

[0021] Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model;

[0022] Extract the emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence;

[0023] Based on the emotion interference quantization sequence and the health scores of each information window, perform a correlation degree analysis to determine the emotion relevance corresponding to the vital sign information.

[0024] In combination with the first aspect, in certain implementations of the first aspect, a single-hidden layer neural network is used to perform label classification on the emotion features corresponding to each multi-modal physiological data segment to obtain multiple emotion labels and the corresponding label times.

[0025] In combination with the first aspect, in certain implementations of the first aspect, when the emotion relevance is lower than a preset threshold, a vital sign analysis result is generated based on the health scores corresponding to each information window.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the multi-modal physiological data is collected by a wearable device worn by the user.

[0027] In a second aspect, the present application provides a vital sign acquisition and analysis system based on multi-modal physiological data. The vital sign acquisition and analysis system includes:

[0028] A physiological data acquisition module, configured to collect multi-modal physiological data, extract emotion features from the multi-modal physiological data based on a preset emotion correction time, and obtain multiple emotion features corresponding to the multi-modal physiological data;

[0029] A physiological data processing module, configured to perform label classification on each emotion feature corresponding to the multi-modal physiological data to obtain multiple emotion labels and the corresponding label times;

[0030] The physiological data processing module is further configured to extract vital sign information from the multi-modal physiological data, window the vital sign information according to the emotion correction time to obtain multiple information windows, and perform a health score on the vital sign information in each information window;

[0031] The physiological data processing module is further configured to analyze the correlation degree of the health scores of each information window based on each emotion label and the corresponding label time, and determine the emotion relevance corresponding to the health score;

[0032] The vital sign result analysis module is configured to, when the emotion relevance is higher than a preset threshold, determine the emotion correction coefficient corresponding to each information window according to the emotion relevance, each emotion label and the corresponding label time, and generate a vital sign analysis result based on the health score and the emotion correction coefficient corresponding to each information window of the vital sign information.

[0033] In a third aspect, the present application provides a computer terminal device, which includes a memory and a processor. The memory stores code, and the processor is configured to obtain the code and execute the above-mentioned vital sign collection and analysis method based on multimodal physiological data.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above-mentioned vital sign collection and analysis method based on multimodal physiological data.

[0035] The technical solution provided by the disclosed embodiments of the present application has the following beneficial effects:

[0036] In a vital sign collection and analysis system and method based on multimodal physiological data provided by the present application, first, multimodal physiological data is collected, emotion features are extracted from the multimodal physiological data based on a preset emotion correction time to obtain multiple emotion features corresponding to the multimodal physiological data; the respective emotion features corresponding to the multimodal physiological data are classified by labels to obtain multiple emotion labels and the corresponding label times; the vital sign information in the multimodal physiological data is extracted, windowing is performed on the vital sign information according to the emotion correction time to obtain multiple information windows, and a health score is given to the vital sign information in each information window; when the emotion relevance is higher than a preset threshold, the emotion correction coefficient corresponding to each information window is determined according to the emotion relevance, each emotion label and the corresponding label time, and a vital sign analysis result is generated based on the health score and the emotion correction coefficient corresponding to each information window of the vital sign information.

[0037] It can be seen that this application segments and analyzes multimodal physiological data based on a preset emotion correction time, thereby extracting features reflecting emotion changes, classifying the extracted emotion features, and generating quantitative emotion labels and their corresponding times. That is, classifying emotion fluctuations through labels clarifies the time points and intensities of emotion fluctuations, laying a foundation for introducing an emotion correction coefficient in subsequent analysis. Furthermore, windowing processing is performed on vital sign information, enabling the health score to correspond to emotion fluctuations within a short period of time, thereby separately correcting the influence of vital sign information in a local time period. Based on the preset emotion correction time, the specific time range in which emotion fluctuations occur can be accurately captured, enabling the system to focus on key adjustment of the vital sign data during this time period. When the emotion relevance of emotion fluctuations is higher than a preset threshold, that is, when emotion fluctuations significantly interfere with the stability of vital sign signals, an emotion correction coefficient is determined according to the intensity, type, and occurrence time of emotion fluctuations, and the health score is dynamically adjusted during the analysis process according to the emotion correction coefficient to eliminate or weaken the interference of emotion fluctuations, thereby improving the accuracy of vital sign analysis results.

[0038] In summary, this application determines an emotion correction coefficient based on emotion features, and generates a vital sign analysis result according to the health scores and emotion correction coefficients respectively corresponding to each information window of vital sign information, thereby enabling correction of the vital sign analysis result based on the emotion features of multimodal physiological data and reducing the interference of emotion fluctuations on the vital sign analysis result when users collect vital sign information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is an exemplary flowchart of a method for collecting and analyzing vital signs based on multimodal physiological data according to some embodiments of the present application;

[0040] Figure 2 is a schematic structural diagram of a vital sign collection and analysis system according to some embodiments of the present application;

[0041] Figure 3 is a schematic structural diagram of a computer terminal device for implementing a method for collecting and analyzing vital signs based on multimodal physiological data according to some embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] In this application, by collecting multimodal physiological data, extracting emotional features from the multimodal physiological data based on a preset emotional correction time, multiple emotional features corresponding to the multimodal physiological data are obtained; each emotional feature corresponding to the multimodal physiological data is classified by label to obtain multiple emotional labels and corresponding label times; the vital sign information in the multimodal physiological data is extracted, windowed according to the emotional correction time, multiple information windows are obtained, and the vital sign information in each information window is health scored; when the emotional relevance is higher than a preset threshold, according to the emotional relevance, each emotional label and the corresponding label time, the emotional correction coefficient corresponding to each information window is determined, and the way of generating the vital sign analysis result based on the health score and the emotional correction coefficient corresponding to each information window of the vital sign information can correct the vital sign analysis result based on the emotional features of the multimodal physiological data, reducing the interference of emotional fluctuations of the user during the collection of vital sign information on the vital sign analysis result.

[0043] To better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments. Refer to Figure 1 , which is an exemplary flowchart of a vital sign acquisition and analysis method based on multimodal physiological data according to some embodiments of the present application. The vital sign acquisition and analysis method 100 based on multimodal physiological data mainly includes the following steps:

[0044] In step S101, multimodal physiological data is collected, and emotional features are extracted from the multimodal physiological data based on a preset emotional correction time, obtaining multiple emotional features corresponding to the multimodal physiological data.

[0045] It should be noted that the multimodal physiological data in this application specifically includes: facial expression data, electrocardiogram data, skin conductance data, heart rate data, respiratory data, etc. The multimodal physiological data can be collected through wearable devices worn by users and corresponding other types of sensors.

[0046] It should be noted that the emotional correction time in this application refers to a time range or window length used to analyze and adjust the influence of emotional fluctuations on vital sign data. The emotional correction time is usually set based on the duration of emotional changes, the influence range, and the interference degree on vital signs. The purpose is to capture the influence of emotions on vital sign signals within this specific time period and correct or adjust the analysis result. Specifically, the emotional correction time can also be calibrated as a constant time of 5s. Optionally, in some embodiments, extracting emotional features from the multimodal physiological data based on a preset emotional correction time to obtain multiple emotional features corresponding to the multimodal physiological data can be implemented by the following steps:

[0047] Obtain the multi-modal physiological data, segment the multi-modal physiological data based on a preset emotion correction time to obtain multiple multi-modal physiological data segments. For any one of the multi-modal physiological data segments, perform data fitting on this multi-modal physiological data segment to obtain the fitting data of each modality.

[0048] Determine the baseline level features corresponding to the fitting data of each modality, and remove the baseline level differences from the fitting data of each modality according to the baseline level features corresponding to the fitting data of each modality to obtain the physiological characteristic waves corresponding to each modality.

[0049] Perform Fourier transform on the physiological characteristic waves corresponding to each modality to obtain the energy spectra corresponding to each modality. Extract feature vectors based on the physiological characteristic waves and energy spectra corresponding to each modality to obtain the emotion features corresponding to this multi-modal physiological data segment. Among them, the energy spectrum can be obtained by squaring the Fourier transform result.

[0050] Use the same method to extract emotion features from other multi-modal physiological data segments until the emotion features corresponding to each multi-modal data segment are obtained and used as the multiple emotion features corresponding to the multi-modal physiological data.

[0051] It should be noted that in this application, the data time length of each multi-modal physiological data segment is the same as the emotion correction time, and the start time of the multi-modal physiological data segment is used as its uniquely corresponding time label.

[0052] In some embodiments, in the process of performing data fitting on the multi-modal physiological data to obtain the fitting data of each modality, polynomial interpolation can be used for data fitting based on the discrete data values and corresponding time labels of the physiological data of different modalities to obtain the continuous fitting data of each modality. Among them, for the facial expression data in the multi-modal physiological data, facial expression scoring can be first performed through machine learning, and then data fitting can be performed based on the scoring data to obtain the fitting data corresponding to the facial expression modality.

[0053] Optionally, in some embodiments, the data mean values corresponding to the fitting data of each modality are used as the baseline level features, and then the fitting data of each modality are respectively subtracted from the corresponding baseline level features to eliminate the interference of the baseline level differences on the fitting data and obtain the physiological characteristic waves corresponding to each modality.

[0054] Optionally, in some embodiments, the emotion features corresponding to the multi-modal physiological data segments include multiple feature vectors, and each feature vector corresponds to the physiological data of one modality. Among them, the element values in the feature vector include: the mean and standard deviation of the peak rise time, adjacent peak interval time, peak height, and valley height of the physiological characteristic wave corresponding to the modality of this feature vector, a total of 8 time-domain features, and the mean, median, standard deviation, maximum value, minimum value, and range of the energy spectrum, a total of 6 frequency-domain features.

[0055] It should be noted that multi-modal physiological data contains comprehensive information about the user's physiological state and indirectly reflects the user's emotional state. By extracting emotion features from multi-modal data, the impact of emotional fluctuations on physiological signals can be captured more accurately, rather than relying on a single signal, avoiding significant interference from abnormal individual signals in the analysis results.

[0056] In step S102, label classification is performed on each emotion feature corresponding to the multi-modal physiological data to obtain multiple emotion labels and corresponding label times.

[0057] It should be noted that physiological signals play a key role in recognizing emotional expressions. For example, changes in heart rate are often related to stress or excitement, changes in skin conductance levels can reflect anxiety or relaxation states, and signals of facial muscle activities (such as smiling or frowning) directly show emotions such as happiness or anger. Therefore, using multiple different physiological signals for comprehensive analysis can understand and recognize an individual's emotional state more accurately than using only a single type of physiological signal. The technology of emotion recognition based on multi-modal physiological data is relatively mature in the prior art.

[0058] Among them, the first and most mature way in the emotion recognition process based on multi-modal physiological data is machine learning. The machine learning method first extracts manual features from the preprocessed physiological data, then simply fuses the extracted different physiological data features, and finally classifies and recognizes the emotions of the fused physiological data features through a feature classifier. For example, the paper "Review of Emotion Recognition Based on Multi-modal Physiological Data" published by the University of Electronic Science and Technology of China in September 2024 discloses various methods for feature extraction and classification recognition of multi-modal physiological data based on machine learning. In this application, label classification is performed on each emotion feature corresponding to the multi-modal physiological data to obtain multiple emotion labels and corresponding label times, which can adopt the prior art. Optionally, in some embodiments, a single-hidden layer neural network can also be used to perform label classification on the emotion features corresponding to each multi-modal physiological data segment to obtain multiple emotion labels and corresponding label times.

[0059] In some specific embodiments of the present application, several activation functions are included as neurons in the hidden layer of the single-hidden-layer neural network, which are used to perform non-linear mapping on the input data. Then, the emotional features are input into the trained single-hidden-layer neural network model. The output of this neural network is the emotional category label and the emotional fluctuation intensity value. For classification tasks, the number of neurons in the output layer will be the same as the number of emotional categories (for example, if there are three categories: fear, pain, anger, and happiness, then there are three nodes in the output layer). Specifically, historical emotional feature samples, corresponding manually-assessed emotional labels and label values can be used as training samples to train the single-hidden-layer neural network. Among them, when each emotional feature is used as the training sample input of the neural network, the number of input layer nodes corresponds to the dimension of the emotional feature, and the emotional label corresponding to each emotional feature is obtained. The acquisition moment corresponding to the emotional feature is used as the label time corresponding to this emotional label.

[0060] Optionally, in some embodiments, the emotional labels include four label categories: fear, pain, desire, and happiness. Each emotional label has a corresponding label value. It should be noted that by classifying the emotional fluctuations with labels in the present application, the time point and intensity of the emotional fluctuations are clarified, which lays a foundation for introducing an emotional correction coefficient in subsequent analysis.

[0061] In step S103, the vital sign information in the multi-modal physiological data is extracted, windowed according to the emotional correction time to obtain multiple information windows, and the vital sign information in each information window is given a health score.

[0062] It should be noted that the vital sign information in the multi-modal physiological data in the present application is part of the multi-modal physiological data that can be used for the physiological health judgment of the user. In some embodiments, the vital sign information includes the user's electrocardiogram data, respiratory rate data, body temperature data, blood oxygen saturation data, etc. The obtained vital sign data is divided into multiple time information windows according to the emotional correction time. The window length of the time information window is the same as the emotional correction time. The data in each time information window will be regarded as an independent sample for subsequent analysis. The size and sliding step of the information window can also be adjusted according to application requirements.

[0063] Optionally, in some embodiments, obtaining multiple information windows and giving a health score to the vital sign information in each information window can be implemented by the following steps:

[0064] Feature extraction is performed on the vital sign information in each information window to obtain the key health score features corresponding to each information window;

[0065] Obtain a health score sample, and perform clustering on each key health score feature based on the health score sample to obtain the health score corresponding to each information window.

[0066] In specific implementation, the same extraction method as that of the emotion feature can be adopted to extract feature vectors from the data of each category of vital sign information in the information window respectively and form a feature matrix as the key health score feature corresponding to each information window. Then, perform singular value decomposition on the feature matrix to determine the eigenvalue composition classification vector of the feature matrix, and perform clustering in the sample space based on the classification vector. The sample space has multiple clustering centers, and the clustering centers are determined based on historical different category vital sign information and corresponding health score values. Each clustering center has a uniquely corresponding health score value. In some embodiments, the health score value corresponding to the clustering center closest to the classification vector is used as the health score corresponding to the information window. In some other embodiments, the reciprocal of the distance value between the information window and each clustering center can also be obtained as a weight value, and the health score values corresponding to each clustering center are weighted and fused to obtain the health score corresponding to the information window.

[0067] In step S104, perform an analysis of the correlation degree of the health scores of each information window based on each emotion label and the corresponding label time to determine the emotion relevance corresponding to the health score;

[0068] Optionally, in some embodiments, the analysis of the correlation degree of the health scores of each information window based on each emotion label and the corresponding label time to determine the emotion relevance corresponding to the health score can be implemented by the following steps:

[0069] Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model;

[0070] Extract the emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence;

[0071] Perform an analysis of the correlation degree based on the emotion interference quantization sequence and the health scores of each information window to determine the emotion relevance corresponding to the vital sign information.

[0072] It should be noted that the two-dimensional emotion mapping model is a set of data points obtained by mapping two-dimensional data points based on the label categories and label values corresponding to the emotion labels. In the prior art, there are mainly two different theoretical models for emotion research. One is the classification model, which represents different emotions with discrete categories; the other is the dimensional model, which maps different types of emotions into a multi-dimensional space, and each axis represents a continuous variable. Among them, Cicero's classification of emotions into four basic categories: fear, pain, desire, and happiness has been widely recognized. In the specific implementation of this application, the two-dimensional emotion mapping model has a fear-desire emotion axis and a pain-happiness emotion axis. Among them, the desire label value is the positive coordinate of the fear-desire emotion axis, the fear label value is the negative coordinate of the fear-desire emotion axis, the pain label value is the negative coordinate of the pain-happiness emotion axis, and the happiness label value is the positive coordinate of the pain-happiness emotion axis. Based on the label time, the label categories and label values corresponding to all emotion labels within the information window are superimposed to determine the emotion mapping data points corresponding to different information windows, and the two-dimensional emotion mapping model is formed according to the time sequence.

[0073] It should be noted that in this application, the emotion interference quantization value is the quantization value of the intensity of emotion during multi-modal physiological data acquisition, which is used to reflect the degree of interference of emotion fluctuations on the process of vital sign analysis. Optionally, in some embodiments, the emotion interference quantization sequence is a data sequence composed of emotion interference quantization values. Among them, the emotion interference quantization value is the deviation distance value between each emotion mapping data point in the two-dimensional emotion mapping model and the origin of the two-dimensional coordinate axes of the two-dimensional emotion mapping model. Since the emotion mapping data points are determined based on the emotion labels in the information window, the sequence length of the emotion interference quantization sequence is the same as the number of information windows. Furthermore, the degree of correlation between the change trend of the emotion interference quantization value and the health score is determined as the emotion correlation degree. The emotion correlation degree is the overall degree of interference of emotion fluctuations on the health score. Specifically, the more similar the trend changes of the emotion interference quantization value and the health score, the greater the influence of emotion fluctuations on the health score, and the greater the emotion correlation degree.

[0074] Optionally, in some embodiments, the Pearson correlation coefficient between each emotion interference quantization value in the emotion interference quantization sequence and the health score of the corresponding information window is determined and used as the emotion correlation degree corresponding to the vital sign information.

[0075] In step S105, when the emotion correlation degree is higher than the preset threshold, the emotion correction coefficient corresponding to each information window is determined according to the emotion correlation degree, each emotion label, and the corresponding label time, and the vital sign analysis result is generated based on the health score and the emotion correction coefficient corresponding to each information window of the vital sign information.

[0076] Optionally, in some embodiments, when the emotion relevance is lower than a preset threshold, a vital sign analysis result is generated based on the health scores respectively corresponding to each information window. Specifically, in implementation, according to the threshold interval where the average score of the health scores respectively corresponding to each information window is located, a corresponding vital sign analysis result is obtained by mapping based on a preset mapping table. For example, when 90 ≤ average score ≤ 100, the vital sign status is good; when 70 ≤ average score < 90, the vital sign status is mildly abnormal; when 50 ≤ average score < 70, the vital sign status is moderately abnormal; when average score < 50, the vital sign status is severely abnormal.

[0077] Optionally, in some embodiments, determining the emotion correction coefficients respectively corresponding to each information window according to the emotion relevance, each emotion label, and the corresponding label time can be implemented by the following steps:

[0078] Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model;

[0079] Extract the emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence;

[0080] After normalizing the emotion interference quantization sequence, use the emotion relevance as an adjustment coefficient to perform proportional adjustment on each emotion interference quantization in the emotion interference quantization sequence, and respectively use the adjusted emotion interference amounts corresponding to the corresponding label times as the emotion correction coefficients respectively corresponding to each information window.

[0081] Specifically, in implementation, each adjusted emotion interference amount can be used as an emotion correction coefficient to perform emotion correction on the health scores of each information window. Specifically, in implementation, a proportional correction method can be adopted. For example, multiply the health scores respectively corresponding to each information window by the emotion correction coefficient to obtain the corrected health scores respectively corresponding to each information window.

[0082] Optionally, in some embodiments, generating a vital sign analysis result based on the health scores and emotion correction coefficients respectively corresponding to each information window can be implemented by the following steps: perform weighted fusion on the health scores respectively corresponding to each information window based on the emotion correction coefficients respectively corresponding to each information window, and generate a vital sign analysis result according to the corrected health scores respectively corresponding to each information window.

[0083] It should be noted that the present application generates a vital sign analysis result based on the health scores and emotion correction coefficients respectively corresponding to each information window of the vital sign information. Thus, an emotion correction coefficient is generated according to the emotion characteristics, and the health scores corresponding to each information window are corrected based on the emotion correction coefficient, reducing the interference of emotional fluctuations on the vital sign signal. Moreover, the emotion correction coefficient is generated based on real-time data, adapting to the instantaneous emotional changes of individuals, making the acquisition and analysis process more comprehensive and accurate, reducing the negative interference of emotional fluctuations on the vital sign analysis result, and improving the scientificity and reliability of the overall health assessment.

[0084] In addition, on the other hand of the present application, in some embodiments, the present application provides a vital sign acquisition and analysis system based on multimodal physiological data. Referring to Figure 2 , which is a schematic structural diagram of the vital sign acquisition and analysis system shown in some embodiments of the present application. The vital sign acquisition and analysis system 200 includes: a physiological data acquisition module 201, a physiological data processing module 202, and a vital sign result analysis module 203, which are described as follows:

[0085] The physiological data acquisition module 201 is configured to acquire multimodal physiological data, extract emotion characteristics from the multimodal physiological data based on a preset emotion correction time, and obtain a plurality of emotion characteristics corresponding to the multimodal physiological data;

[0086] The physiological data processing module 202 is configured to perform label classification on each emotion characteristic corresponding to the multimodal physiological data to obtain a plurality of emotion labels and corresponding label times;

[0087] The physiological data processing module 202 is further configured to extract the vital sign information from the multimodal physiological data, window the vital sign information according to the emotion correction time to obtain a plurality of information windows, and perform a health score on the vital sign information in each information window;

[0088] The physiological data processing module 202 is further configured to analyze the degree of association of the health scores of each information window based on each emotion label and the corresponding label time to determine the emotion relevance corresponding to the health score;

[0089] The vital sign result analysis module 203 is configured to, when the emotion relevance is higher than a preset threshold, determine the emotion correction coefficients respectively corresponding to each information window according to the emotion relevance, each emotion label, and the corresponding label time, and generate a vital sign analysis result based on the health scores and emotion correction coefficients respectively corresponding to each information window of the vital sign information.

[0090] The above text has introduced in detail an example of a vital sign acquisition and analysis system and method based on multimodal physiological data provided by the embodiments of the present application. It can be understood that, in order to implement the above functions, the corresponding device includes the corresponding hardware structure and / or software module for executing each function.

[0091] Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function in the application is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Therefore, professionals can use different methods to implement the described functions for each specific application, but this kind of implementation should not be considered to exceed the scope of the present application.

[0092] In addition, the present application also provides a computer terminal device, the computer terminal device includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned vital sign acquisition and analysis method based on multimodal physiological data.

[0093] In some embodiments, refer to Figure 3 , this figure is a schematic structural diagram of a computer terminal device for implementing the vital sign acquisition and analysis method based on multimodal physiological data according to some embodiments of the present application. The above-mentioned vital sign acquisition and analysis method based on multimodal physiological data in the embodiments can be implemented by Figure 3 the computer terminal device shown. The computer terminal device 300 includes at least one communication bus 301, a communication interface 302, a processor 303, and a memory 304.

[0094] The processor 303 can be a general-purpose central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more for controlling the execution of a vital sign acquisition and analysis method based on multimodal physiological data in the present application.

[0095] The communication bus 301 may include a path for transmitting information between the above components.

[0096] The memory 304 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 304 can exist independently and be connected to the processor 303 through the communication bus 301. The memory 304 can also be integrated with the processor 303.

[0097] Among them, the memory 304 is used to store the program code for executing the solution of this application and is controlled by the processor 303 for execution. The processor 303 is used to execute the program code stored in the memory 304. The program code can include one or more software modules. The determination of the emotion relevance described in the above embodiments can be implemented by one or more software modules in the processor 303 and the program code in the memory 304.

[0098] The communication interface 302 uses any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0099] Optionally, the above computer terminal device 300 can further include a power supply 305 for supplying power to various components or circuits in the real-time computer terminal device.

[0100] In a specific implementation, as an embodiment, the computer terminal device can include multiple processors, and each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, the processor can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0101] The above computer terminal device can be a general computer terminal device or a dedicated computer terminal device. In specific implementations, the computer terminal device can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of the computer terminal device.

[0102] In addition, in other aspects of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above method for collecting and analyzing vital signs based on multimodal physiological data.

[0103] In summary, in a vital sign collection and analysis system and method based on multimodal physiological data disclosed in the embodiments of the present application, by collecting multimodal physiological data, extracting emotional features of the multimodal physiological data based on a preset emotion correction time, multiple emotional features corresponding to the multimodal physiological data are obtained; performing label classification on each emotional feature corresponding to the multimodal physiological data to obtain multiple emotional labels and corresponding label times; extracting vital sign information from the multimodal physiological data, windowing the vital sign information according to the emotion correction time to obtain multiple information windows, and performing a health score on the vital sign information in each information window; when the emotion relevance is higher than a preset threshold, determining the emotion correction coefficient corresponding to each information window according to the emotion relevance, each emotional label, and the corresponding label time, and generating a vital sign analysis result based on the health score and the emotion correction coefficient corresponding to each information window of the vital sign information, the vital sign analysis result can be corrected based on the emotional features of the multimodal physiological data, reducing the interference of emotional fluctuations of the user during the collection of vital sign information on the vital sign analysis result.

[0104] The above are only the embodiments of the present application. Specific technical solutions or common knowledge such as well-known features are not described in detail herein. It should be noted that for those skilled in the art, without departing from the technical solutions of the present application, several modifications and improvements can be made, which should also be regarded as the protection scope of the present application, and these will not affect the implementation effect of the present application and the practicality of the patent.

[0105] The scope of protection claimed in this application shall be subject to the content of its claims, and the specific implementation manners and other records in the description may be used to interpret the content of the claims. Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to cover these modifications and variations.

Claims

1. A method for collecting and analyzing vital signs based on multimodal physiological data, characterized in that: The method includes: Collecting multimodal physiological data, and extracting emotional features from the multimodal physiological data based on a preset emotion correction time to obtain a plurality of emotional features corresponding to the multimodal physiological data; Classifying each emotion feature corresponding to the multimodal physiological data by labeling to obtain a plurality of emotion labels and corresponding labeling times; Extracting vital sign information from the multimodal physiological data, windowing the vital sign information according to the emotion correction time to obtain a plurality of information windows and performing health scoring on the vital sign information in each information window; Performing correlation analysis on the health scores of each information window according to each emotion tag and the corresponding tag time to determine the emotion correlation corresponding to the health score; When the emotion correlation is higher than a preset threshold, the emotion correction coefficients corresponding to the information windows are determined according to the emotion correlation, each emotion label and the corresponding label time, and the vital sign analysis results are generated based on the health scores and emotion correction coefficients corresponding to the information windows of the vital sign information; Among them, the health score of each information window is analyzed for correlation according to each emotion tag and the corresponding tag time, and the emotion correlation corresponding to the health score is determined to specifically include: Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model; Extracting emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence; Performing a correlation analysis based on the emotional interference quantification sequence and the health scores of each information window to determine the emotional relevance corresponding to the vital sign information; The emotion correction coefficients corresponding to each information window are determined according to the emotion relevance, each emotion label and the corresponding label time by the following steps: After normalizing the emotion interference quantization sequence, the emotion correlation is used as an adjustment coefficient to proportionally adjust each emotion interference quantization in the emotion interference quantization sequence, and each adjusted emotion interference amount is used as an emotion correction coefficient corresponding to each information window according to the corresponding label time; Among them, the emotion interference quantization sequence is a data sequence composed of emotion interference quantization values, wherein the emotion interference quantization value is the deviation distance value between each emotion mapping data point in the two-dimensional emotion mapping model and the origin of the two-dimensional coordinate axis of the two-dimensional emotion mapping model, and the emotion correlation is the overall interference degree of emotion fluctuations on the health score, specifically the correlation degree of the changing trend between the emotion interference quantization value and the health score.

2. The method according to claim 1, characterized in that Extracting emotion features from the multimodal physiological data based on the preset emotion correction time to obtain a plurality of emotion features corresponding to the multimodal physiological data specifically includes: Acquire the multimodal physiological data, segment the multimodal physiological data based on a preset emotion correction time to obtain a plurality of multimodal physiological data segments, and perform data fitting on any multimodal physiological data segment to obtain fitting data of each modality; Determine the basic level features corresponding to the fitting data of each modality, and remove the basic level differences of the fitting data of each modality according to the basic level features corresponding to the fitting data of each modality, so as to obtain the physiological characteristic waves corresponding to each modality; Performing Fourier transform on the physiological characteristic waves corresponding to each modality to obtain the energy spectrum corresponding to each modality, extracting feature vectors based on the physiological characteristic waves and energy spectrum corresponding to each modality to obtain the emotional features corresponding to the multimodal physiological data segment; The same method is adopted to extract emotion features from other multimodal physiological data segments until emotion features corresponding to each multimodal data segment are obtained and used as multiple emotion features corresponding to the multimodal physiological data.

3. The method according to claim 1, characterized in that Obtaining multiple information windows and performing health scoring on vital signs information in each information window specifically includes: Extract features of vital signs information in each information window to obtain key features of health scores corresponding to each information window; A health score sample is obtained, and each health score key feature is clustered based on the health score sample to obtain a health score corresponding to each information window.

4. The method according to claim 1, characterized in that A single hidden layer neural network is used to classify the emotion features corresponding to each multimodal physiological data segment, and multiple emotion labels and corresponding label times are obtained.

5. The method according to claim 1, characterized in that When the emotion relevance is lower than a preset threshold, a vital sign analysis result is generated based on the health scores corresponding to each information window.

6. The method according to claim 1, characterized in that The multimodal physiological data is collected through a wearable device worn by a user.

7. A vital sign collection and analysis system based on multimodal physiological data, which uses the method according to any one of claims 1 to 6 to perform vital sign collection and analysis, characterized in that: The vital signs collection and analysis system comprises: A physiological data acquisition module, used for acquiring multimodal physiological data, performing emotional feature extraction on the multimodal physiological data based on a preset emotion correction time, and obtaining a plurality of emotional features corresponding to the multimodal physiological data; A physiological data processing module, used for labeling and classifying each emotion feature corresponding to the multimodal physiological data to obtain a plurality of emotion labels and corresponding label times; The physiological data processing module is further used to extract vital sign information from the multimodal physiological data, window the vital sign information according to the emotion correction time, obtain multiple information windows, and perform health scoring on the vital sign information in each information window; The physiological data processing module is further used to perform correlation analysis on the health score of each information window according to each emotion tag and the corresponding tag time, and determine the emotion correlation corresponding to the health score; The vital signs result analysis module is used to determine the emotion correction coefficient corresponding to each information window according to the emotion correlation, each emotion label and the corresponding label time when the emotion correlation is higher than a preset threshold, and generate the vital signs analysis result based on the health score and emotion correction coefficient corresponding to each information window of the vital signs information.

8. A computer terminal device, characterized in that: The computer terminal device includes a memory and a processor, the memory stores codes, and the processor is configured to obtain the codes and execute the vital sign collection and analysis method based on multimodal physiological data as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing at least one computer program, characterized in that: The computer program is loaded and executed by a processor to implement the operations performed by the method for collecting and analyzing vital signs based on multimodal physiological data as described in any one of claims 1 to 6.

Images