Vital sign acquisition and analysis system and method based on multi-modal physiological data

Abstract

The invention provides a vital sign acquisition and analysis system and method based on multi-modal physiological data, and the method comprises the steps: carrying out the tag classification of each emotion feature corresponding to the multi-modal physiological data, and obtaining a plurality of emotion tags and corresponding tag time; after extracting the vital sign information, windowing the vital sign information to obtain a plurality of information windows, and performing health scoring on the vital sign information in each information window; when the emotion relevancy is higher than a preset threshold value, according to the emotion relevancy, each emotion label and the corresponding label time, determining an emotion correction coefficient corresponding to each information window, and generating a vital sign analysis result based on the health score and the emotion correction coefficient corresponding to each information window of the vital sign information. The vital sign analysis result can be corrected based on the emotional characteristics of the multi-modal physiological data, and interference of emotional fluctuation on the vital sign analysis result when a user collects vital sign information is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of vital sign analysis, and more specifically, to a vital sign collection and analysis system and method based on multimodal physiological data. Background Art

[0002] Vital signs refer to physiological indicators used to judge the severity and criticality of a patient's condition, including body temperature, pulse, respiration, blood pressure, heart rate, etc. With the rapid development of wearable devices, the public has generally improved their ability to manage their own health. They monitor a variety of physiological parameters through wearable devices and understand vital signs data. Changes in these vital signs data may contain early clues to chronic diseases.

[0003] In the prior art, the process of collecting users' vital signs information is relatively complicated. In particular, users often face a series of emotions such as tension and anxiety when collecting vital signs information, which causes large fluctuations in the obtained vital signs information, making the analysis results of the users' vital signs inaccurate, which greatly restricts the development of vital signs collection and analysis systems. Summary of the invention

[0004] The present application provides a vital sign collection and analysis system and method based on multimodal physiological data, which can correct the vital sign analysis results based on the emotional characteristics of the multimodal physiological data, thereby reducing the interference of the user's emotional fluctuations on the vital sign analysis results when collecting vital sign information.

[0005] In a first aspect, the present application provides a method for collecting and analyzing vital signs based on multimodal physiological data. The method can be executed by a network device, or can also be executed by a chip configured in the network device, and the present application does not limit this.

[0006] Specifically, the method includes: Collecting multimodal physiological data, and extracting emotional features from the multimodal physiological data based on a preset emotion correction time to obtain a plurality of emotional features corresponding to the multimodal physiological data; Classifying each emotion feature corresponding to the multimodal physiological data by labeling to obtain a plurality of emotion labels and corresponding labeling times; Extracting vital sign information from the multimodal physiological data, windowing the vital sign information according to the emotion correction time to obtain a plurality of information windows and performing health scoring on the vital sign information in each information window; Performing correlation analysis on the health scores of each information window according to each emotion tag and the corresponding tag time to determine the emotion correlation corresponding to the health score; When the emotion correlation is higher than a preset threshold, the emotion correction coefficient corresponding to each information window is determined according to the emotion correlation, each emotion label and the corresponding label time, and the vital sign analysis result is generated based on the health score and emotion correction coefficient corresponding to each information window of the vital sign information.

[0007] In combination with the first aspect, in certain implementations of the first aspect, performing emotion feature extraction on the multimodal physiological data based on a preset emotion correction time to obtain a plurality of emotion features corresponding to the multimodal physiological data specifically includes: Acquire the multimodal physiological data, segment the multimodal physiological data based on a preset emotion correction time to obtain a plurality of multimodal physiological data segments, and perform data fitting on any multimodal physiological data segment to obtain fitting data of each modality; Determine the basic level features corresponding to the fitting data of each modality, and remove the basic level differences of the fitting data of each modality according to the basic level features corresponding to the fitting data of each modality, so as to obtain the physiological characteristic waves corresponding to each modality; Performing Fourier transform on the physiological characteristic waves corresponding to each modality to obtain the energy spectrum corresponding to each modality, extracting feature vectors based on the physiological characteristic waves and energy spectrum corresponding to each modality to obtain the emotional features corresponding to the multimodal physiological data segment; The same method is adopted to extract emotion features from other multimodal physiological data segments until emotion features corresponding to each multimodal data segment are obtained and used as multiple emotion features corresponding to the multimodal physiological data.

[0008] In conjunction with the first aspect, in certain implementations of the first aspect, obtaining multiple information windows and performing health scoring on vital sign information in each information window specifically includes: Extract features of vital signs information in each information window to obtain key features of health scores corresponding to each information window; A health score sample is obtained, and each health score key feature is clustered based on the health score sample to obtain a health score corresponding to each information window.

[0009] In combination with the first aspect, in some implementations of the first aspect, performing a correlation analysis on the health score of each information window according to each emotion tag and the corresponding tag time, and determining the emotion relevance corresponding to the health score specifically includes: Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model; Extracting emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence; A correlation degree analysis is performed based on the emotion interference quantification sequence and the health score of each information window to determine the emotion relevance corresponding to the vital sign information.

[0010] In combination with the first aspect, in certain implementations of the first aspect, a single hidden layer neural network is used to perform label classification on the emotion features corresponding to each multimodal physiological data segment to obtain multiple emotion labels and corresponding label times.

[0011] In combination with the first aspect, in certain implementations of the first aspect, when the emotion relevance is lower than a preset threshold, a vital sign analysis result is generated based on the health scores corresponding to each information window.

[0012] In combination with the first aspect, in some implementations of the first aspect, the multimodal physiological data is collected through a wearable device worn by a user.

[0013] In a second aspect, the present application provides a vital sign collection and analysis system based on multimodal physiological data, the vital sign collection and analysis system comprising: A physiological data acquisition module, used for acquiring multimodal physiological data, performing emotional feature extraction on the multimodal physiological data based on a preset emotion correction time, and obtaining a plurality of emotional features corresponding to the multimodal physiological data; A physiological data processing module, used for labeling and classifying each emotion feature corresponding to the multimodal physiological data to obtain a plurality of emotion labels and corresponding label times; The physiological data processing module is further used to extract vital sign information from the multimodal physiological data, window the vital sign information according to the emotion correction time, obtain multiple information windows, and perform health scoring on the vital sign information in each information window; The physiological data processing module is further used to perform correlation analysis on the health score of each information window according to each emotion tag and the corresponding tag time, and determine the emotion correlation corresponding to the health score; The vital signs result analysis module is used to determine the emotion correction coefficient corresponding to each information window according to the emotion correlation, each emotion label and the corresponding label time when the emotion correlation is higher than a preset threshold, and generate the vital signs analysis result based on the health score and emotion correction coefficient corresponding to each information window of the vital signs information.

[0014] In a third aspect, the present application provides a computer terminal device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned vital signs collection and analysis method based on multimodal physiological data.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above-mentioned vital signs collection and analysis method based on multimodal physiological data.

[0016] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: In a vital sign collection and analysis system and method based on multimodal physiological data provided by the present application, multimodal physiological data are first collected, and emotion features are extracted from the multimodal physiological data based on a preset emotion correction time to obtain multiple emotion features corresponding to the multimodal physiological data; each emotion feature corresponding to the multimodal physiological data is labeled and classified to obtain multiple emotion labels and corresponding label times; vital sign information in the multimodal physiological data is extracted, the vital sign information is windowed according to the emotion correction time to obtain multiple information windows and a health score is given to the vital sign information in each information window; when the emotion correlation is higher than a preset threshold, the emotion correction coefficient corresponding to each information window is determined according to the emotion correlation, each emotion label and the corresponding label time, and the vital sign analysis result is generated based on the health score and emotion correction coefficient corresponding to each information window of the vital sign information.

[0017] It can be seen that the present application segments and analyzes multimodal physiological data based on the preset emotion correction time, thereby extracting features reflecting emotion changes, and classifying the extracted emotion features to generate quantitative emotion labels and their corresponding times. That is, the emotion fluctuations are classified by labels to clarify the time point and intensity of the emotion fluctuations, which lays the foundation for introducing the emotion correction coefficient in the subsequent analysis, and then the vital sign information is processed in a windowed manner, so that the health score can correspond to the emotion fluctuations in a short period of time, so as to perform a separate correction on the impact of the vital sign information in the local time period. Based on the preset emotion correction time, the specific time range of the emotion fluctuations can be accurately captured, so that the system can focus on the vital sign data of the time period. When the emotion correlation of the emotion fluctuation is higher than the preset threshold, that is, the emotion fluctuation significantly interferes with the stability of the vital sign signal, the emotion correction coefficient is determined according to the intensity, type and occurrence time of the emotion fluctuation, and the health score is dynamically adjusted during the analysis process according to the emotion correction coefficient to eliminate or reduce the interference of the emotion fluctuations, thereby improving the accuracy of the vital sign analysis results.

[0018] To summarize, the present application determines the emotion correction coefficient based on the emotion characteristics, and generates the vital signs analysis results according to the health scores and emotion correction coefficients corresponding to each information window of the vital signs information, thereby being able to correct the vital signs analysis results based on the emotion characteristics of the multimodal physiological data, thereby reducing the interference of the user's emotional fluctuations on the vital signs analysis results when collecting vital signs information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is an exemplary flow chart of a method for collecting and analyzing vital signs based on multimodal physiological data according to some embodiments of the present application; Figure 2 is a structural schematic diagram of a vital sign collection and analysis system according to some embodiments of the present application; Figure 3 It is a structural schematic diagram of a computer terminal device for implementing a method for collecting and analyzing vital signs based on multimodal physiological data according to some embodiments of the present application. DETAILED DESCRIPTION

[0020] The present application collects multimodal physiological data, extracts emotion features from the multimodal physiological data based on a preset emotion correction time, and obtains a plurality of emotion features corresponding to the multimodal physiological data; performs label classification on each emotion feature corresponding to the multimodal physiological data, and obtains a plurality of emotion labels and corresponding label times; extracts vital sign information from the multimodal physiological data, window-codes the vital sign information according to the emotion correction time, obtains a plurality of information windows, and performs health scoring on the vital sign information in each information window; when the emotion correlation is higher than a preset threshold, determines the emotion correction coefficient corresponding to each information window according to the emotion correlation, each emotion label, and the corresponding label time, and generates vital sign analysis results based on the health scores and emotion correction coefficients corresponding to each information window of the vital sign information, so as to correct the vital sign analysis results based on the emotion features of the multimodal physiological data, thereby reducing the interference of the user's emotion fluctuations on the vital sign analysis results when collecting vital sign information.

[0021] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a method for collecting and analyzing vital signs based on multimodal physiological data according to some embodiments of the present application. The method 100 for collecting and analyzing vital signs based on multimodal physiological data mainly includes the following steps: In step S101, multimodal physiological data is collected, and emotion features are extracted from the multimodal physiological data based on a preset emotion correction time to obtain a plurality of emotion features corresponding to the multimodal physiological data.

[0022] It should be noted that the multimodal physiological data described in the present application specifically includes: facial expression data, electrocardiogram data, skin conductance data, heart rate data and breathing data, etc. The multimodal physiological data can be collected through wearable devices worn by users and corresponding other types of sensors.

[0023] It should be noted that the emotion correction time described in the present application refers to a time range or window length used to analyze and adjust the impact of emotion fluctuations on vital sign data. The emotion correction time is usually set based on the duration of emotion changes, the scope of influence, and the degree of interference with vital signs. The purpose is to capture the impact of emotions on vital sign signals within this specific time period and correct or adjust the analysis results. In specific implementation, the emotion correction time can also be calibrated as a constant time of 5s. Optionally, in some embodiments, emotion feature extraction is performed on the multimodal physiological data based on the preset emotion correction time, and multiple emotion features corresponding to the multimodal physiological data can be obtained by the following steps: Acquire the multimodal physiological data, segment the multimodal physiological data based on a preset emotion correction time to obtain a plurality of multimodal physiological data segments, and perform data fitting on any multimodal physiological data segment to obtain fitting data of each modality; Determine the basic level features corresponding to the fitting data of each modality, and remove the basic level differences of the fitting data of each modality according to the basic level features corresponding to the fitting data of each modality, so as to obtain the physiological characteristic waves corresponding to each modality; Performing Fourier transform on the physiological characteristic waves corresponding to each modality to obtain the energy spectrum corresponding to each modality, performing feature vector extraction based on the physiological characteristic waves and energy spectrum corresponding to each modality to obtain the emotion feature corresponding to the multimodal physiological data segment, wherein the energy spectrum can be obtained by squaring the Fourier transform result; The same method is adopted to extract emotion features from other multimodal physiological data segments until emotion features corresponding to each multimodal data segment are obtained and used as multiple emotion features corresponding to the multimodal physiological data.

[0024] It should be noted that the data time length of each multimodal physiological data segment in the present application is consistent with the emotion correction time, and the start time of the multimodal physiological data segment is used as its unique corresponding time label.

[0025] In some embodiments, in the process of performing data fitting based on the multimodal physiological data to obtain fitting data for each modality, data fitting can be performed using polynomial interpolation based on discrete data values ​​of physiological data of different modalities and corresponding time labels to obtain continuous fitting data for each modality, wherein for facial expression data in the multimodal physiological data, facial expression scoring can be first performed through machine learning, and then data fitting can be performed based on the scoring data to obtain fitting data corresponding to the facial expression modality.

[0026] Optionally, in some embodiments, the data mean corresponding to the fitting data of each modality is used as the basic level feature, and then the corresponding basic level feature is subtracted from the fitting data of each modality, thereby eliminating the interference of the basic level difference on the fitting data and obtaining the physiological characteristic wave corresponding to each modality.

[0027] Optionally, in some embodiments, the emotional features corresponding to the multimodal physiological data segments include multiple feature vectors, each feature vector corresponding to the physiological data of a modality, wherein the element values ​​in the feature vector include: eight time domain features, namely the mean and standard deviation of the peak rise time, adjacent peak interval time, peak height, and trough height of the physiological characteristic wave of the modality corresponding to the feature vector, and six frequency domain features, namely the mean, median, standard deviation, maximum value, minimum value, and range of the energy spectrum.

[0028] It should be noted that multimodal physiological data contains comprehensive information about the user's physiological state and also indirectly reflects the user's emotional state. By extracting emotional features from multimodal data, the impact of emotional fluctuations on physiological signals can be more accurately captured, rather than relying on a single signal, thus avoiding significant interference of individual signal abnormalities in the analysis results.

[0029] In step S102, label classification is performed on each emotion feature corresponding to the multimodal physiological data to obtain a plurality of emotion labels and corresponding label times.

[0030] It should be noted that physiological signals play a key role in identifying emotional expressions. For example, changes in heart rate are often related to stress or excitement, changes in skin conductance levels can reflect anxiety or relaxation, and facial muscle activity (such as smiling or frowning) signals directly express emotions such as happiness or anger. Therefore, using a variety of different physiological signals for comprehensive analysis can more accurately understand and identify an individual's emotional state than using only a single type of physiological signal. In the existing technology, the technology for emotion recognition based on multimodal physiological data is relatively mature.

[0031] Among them, the first and most mature method in the emotion recognition process based on multimodal physiological data is machine learning. The machine learning method first performs manual feature extraction on the preprocessed physiological data, and then performs simple feature fusion on the extracted different physiological data features. Finally, the fused physiological data features are classified by feature classifier for emotion recognition. For example, the paper "A Review of Emotion Recognition Based on Multimodal Physiological Data" published by Chengdu University of Electronic Science and Technology in September 2024 discloses a variety of methods for feature extraction and classification of multimodal physiological data based on machine learning. In this application, the emotion features corresponding to the multimodal physiological data are labeled and classified to obtain multiple emotion labels and corresponding label times. The existing technology can be used. Optionally, in some embodiments, a single hidden layer neural network can also be used to label the emotion features corresponding to each multimodal physiological data segment to obtain multiple emotion labels and corresponding label times.

[0032] In some specific embodiments of the present application, the hidden layer of the single hidden layer neural network includes several activation functions as neurons, which are used to perform nonlinear mapping on the input data, and then input the emotional features into the trained single hidden layer neural network model. The output of the neural network is the emotional category label and the emotional fluctuation intensity value. For the classification task, the number of neurons in the output layer will be the same as the number of emotional categories (for example, if there are three categories of fear, pain, anger and happiness, then the output layer has three nodes). In specific implementation, historical emotional feature samples and corresponding manually assessed emotional labels and label values ​​can be used as training samples to train the single hidden layer neural network. When each emotional feature is input as a training sample of the neural network, the number of nodes in the input layer corresponds to the dimension of the emotional feature, and the emotional label corresponding to each emotional feature is obtained, and the collection time corresponding to the emotional feature is used as the label time corresponding to the emotional label.

[0033] Optionally, in some embodiments, the emotion labels include four label categories: fear, pain, desire and happiness, and each emotion label has a corresponding label value. It should be noted that the present application clarifies the time point and intensity of emotion fluctuations by labeling emotion fluctuations, laying the foundation for introducing emotion correction coefficients in subsequent analysis.

[0034] In step S103, vital sign information is extracted from the multimodal physiological data, the vital sign information is windowed according to the emotion correction time, a plurality of information windows are obtained, and a health score is performed on the vital sign information in each information window.

[0035] It should be noted that the vital signs information in the multimodal physiological data described in the present application is part of the multimodal physiological data that can be used to judge the user's physiological health. In some embodiments, the vital signs information includes the user's electrocardiogram data, respiratory rate data, body temperature data, blood oxygen saturation data, etc. The acquired vital signs data is divided into multiple time information windows according to the emotion correction time. The window length of the time information window is consistent with the emotion correction time. The data in each time information window will be regarded as an independent sample for subsequent analysis. The size of the information window and the sliding step size can also be adjusted according to application requirements.

[0036] Optionally, in some embodiments, obtaining multiple information windows and performing health scoring on the vital sign information in each information window may be implemented by the following steps: Extract features of vital signs information in each information window to obtain key features of health scores corresponding to each information window; A health score sample is obtained, and each health score key feature is clustered based on the health score sample to obtain a health score corresponding to each information window.

[0037] In specific implementation, the same extraction method as the emotional feature can be used to extract feature vectors from data of each category of vital sign information in the information window and form a feature matrix as the key feature of the health score corresponding to each information window, and then singular value decomposition is performed on the feature matrix to determine the eigenvalues ​​of the feature matrix to form a classification vector, and clustering is performed in the sample space based on the classification vector. The sample space has multiple clustering centers, and the clustering centers are determined based on historical vital sign information of different categories and corresponding health score values. Each clustering center has a unique corresponding health score value. In some embodiments, the health score value corresponding to the cluster center closest to the classification vector is used as the health score corresponding to the information window. In some other embodiments, the inverse of the distance value between the information window and each clustering center can also be obtained as a weight value, and the health score values ​​corresponding to each clustering center are weighted fused to obtain the health score corresponding to the information window.

[0038] In step S104, the health scores of each information window are analyzed for correlation according to each emotion tag and the corresponding tag time to determine the emotion correlation corresponding to the health score; Optionally, in some embodiments, the health scores of each information window are analyzed for correlation according to each emotion tag and the corresponding tag time, and the emotion correlation corresponding to the health score is determined by the following steps: Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model; Extracting emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence; A correlation degree analysis is performed based on the emotion interference quantification sequence and the health score of each information window to determine the emotion relevance corresponding to the vital sign information.

[0039] It should be noted that the two-dimensional emotion mapping model is a set of data points obtained by mapping two-dimensional data points based on the label categories and label values ​​corresponding to the emotion labels. In the prior art, there are mainly two different theoretical models for emotion research. One is a classification model, which represents different emotions with discrete categories; the other is a dimensional model, which maps different types of emotions into a multidimensional space, where each axis represents a continuous variable. Among them, Cicero divides emotions into four basic categories: fear, pain, desire and happiness, which has been generally recognized. In the specific implementation of this application, the two-dimensional emotion mapping model has a fear-desire emotion axis and a pain-happiness emotion axis, wherein the desire label value is the positive coordinate of the fear-desire emotion axis, the fear label value is the secondary coordinate of the fear-desire emotion axis, the pain label value is the negative coordinate of the pain-happiness emotion axis, and the happiness label value is the positive coordinate of the pain-happiness emotion axis. Based on the label time, the label categories and label values ​​corresponding to all emotion labels in the information window are superimposed to determine the emotion mapping data points corresponding to different information windows, and the two-dimensional emotion mapping model is composed according to the time series.

[0040] It should be noted that the emotion interference quantification value in the present application is a quantified value of the intensity of emotion when performing multimodal physiological data collection, which is used to reflect the degree of interference of emotion fluctuations on the vital signs analysis process. Optionally, in some embodiments, the emotion interference quantification sequence is a data sequence composed of emotion interference quantification values, wherein the emotion interference quantification value is the deviation distance value of each emotion mapping data point in the two-dimensional emotion mapping model and the origin of the two-dimensional coordinate axis of the two-dimensional emotion mapping model. Since the emotion mapping data point is determined based on the emotion label in the information window, the sequence length of the emotion interference quantification sequence is equal to the number of information windows, and then the correlation degree of the change trend between the emotion interference quantification value and the health score is determined as the emotion correlation. The emotion correlation is the overall interference degree of emotion fluctuations on the health score. Specifically, the more similar the trend changes of the emotion interference quantification value and the health score are, the greater the health score is affected by the emotion fluctuation, and the greater the emotion correlation.

[0041] Optionally, in some embodiments, the Pearson correlation coefficient between each emotion interference quantification value in the emotion interference quantification sequence and the health score of the corresponding information window is determined and used as the emotion correlation corresponding to the vital sign information.

[0042] In step S105, when the emotion relevance is higher than a preset threshold, the emotion correction coefficient corresponding to each information window is determined according to the emotion relevance, each emotion label and the corresponding label time, and the vital sign analysis result is generated based on the health score and emotion correction coefficient corresponding to each information window of the vital sign information.

[0043] Optionally, in some embodiments, when the emotion relevance is lower than a preset threshold, a vital signs analysis result is generated based on the health scores corresponding to each information window. In specific implementation, the threshold interval of the mean score of the health score corresponding to each information window can be mapped based on a preset mapping table to obtain the corresponding vital signs analysis result. For example, 90≤mean score≤100 indicates a good vital signs status; 70≤mean score<90 indicates a slightly abnormal vital signs status; 50≤mean score<70 indicates a moderately abnormal vital signs status; and mean score<50 indicates a severely abnormal vital signs status.

[0044] Optionally, in some embodiments, determining the emotion correction coefficients corresponding to the respective information windows according to the emotion relevance, the respective emotion tags and the corresponding tag time may be implemented by the following steps: Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model; Extracting emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence; After normalizing the emotion interference quantization sequence, the emotion correlation is used as an adjustment coefficient to proportionally adjust each emotion interference quantization in the emotion interference quantization sequence, and each adjusted emotion interference amount is used as the emotion correction coefficient corresponding to each information window according to the corresponding label time.

[0045] In specific implementation, the adjusted emotional interference amounts can be used as emotion correction coefficients to perform emotion correction on the health scores of each information window. In specific implementation, a proportional correction method can be used, for example, multiplying the health scores corresponding to each information window by the emotion correction coefficient to obtain the corrected health scores corresponding to each information window.

[0046] Optionally, in some embodiments, generating vital signs analysis results based on the health scores and emotion correction coefficients corresponding to each information window can be achieved by using the following steps: weighted fusion of the health scores corresponding to each information window based on the emotion correction coefficients corresponding to each information window, and generating vital signs analysis results based on the corrected health scores corresponding to each information window.

[0047] It should be noted that the present application generates vital signs analysis results based on the health scores and emotion correction coefficients corresponding to each information window of the vital signs information, thereby generating an emotion correction coefficient based on the emotional characteristics, and correcting the health scores corresponding to each information window based on the emotion correction coefficient, thereby reducing the interference of emotional fluctuations on vital signs signals. In addition, the emotion correction coefficient is generated based on real-time data to adapt to the individual's instantaneous emotional changes, making the collection and analysis process more comprehensive and accurate, reducing the negative interference of emotional fluctuations on the vital signs analysis results, and improving the scientificity and reliability of the overall health assessment.

[0048] In addition, in another aspect of the present application, in some embodiments, the present application provides a vital sign collection and analysis system based on multimodal physiological data, referring to Figure 2 , which is a structural schematic diagram of a vital sign collection and analysis system according to some embodiments of the present application, the vital sign collection and analysis system 200 includes: a physiological data collection module 201, a physiological data processing module 202 and a vital sign result analysis module 203, which are respectively described as follows: A physiological data acquisition module 201, wherein the physiological data acquisition module 201 is used to acquire multimodal physiological data, and extract emotional features from the multimodal physiological data based on a preset emotion correction time to obtain a plurality of emotional features corresponding to the multimodal physiological data; A physiological data processing module 202, the physiological data processing module 202 is used to classify labels of various emotion features corresponding to the multimodal physiological data to obtain multiple emotion labels and corresponding label times; The physiological data processing module 202 is further used to extract vital sign information from the multimodal physiological data, window the vital sign information according to the emotion correction time, obtain multiple information windows, and perform health scoring on the vital sign information in each information window; The physiological data processing module 202 is further used to perform correlation analysis on the health score of each information window according to each emotion tag and the corresponding tag time, and determine the emotion correlation corresponding to the health score; The vital signs result analysis module 203 is used to determine the emotion correction coefficient corresponding to each information window according to the emotion relevance, each emotion label and the corresponding label time when the emotion relevance is higher than a preset threshold, and generate the vital signs analysis result based on the health score and emotion correction coefficient corresponding to each information window of the vital signs information.

[0049] The above describes in detail an example of a vital signs collection and analysis system and method based on multimodal physiological data provided in an embodiment of the present application. It can be understood that in order to achieve the above functions, the corresponding device includes a hardware structure and / or software module corresponding to executing each function.

[0050] Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this document, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function in the application is executed in hardware or in a computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Therefore, professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0051] In addition, the present application also provides a computer terminal device, which includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned vital signs collection and analysis method based on multimodal physiological data.

[0052] In some embodiments, reference Figure 3 , which is a schematic diagram of the structure of a computer terminal device for implementing a method for collecting and analyzing vital signs based on multimodal physiological data according to some embodiments of the present application. The method for collecting and analyzing vital signs based on multimodal physiological data in the above embodiments can be performed by Figure 3 The computer terminal device 300 shown in the figure is implemented, and the computer terminal device 300 includes at least one communication bus 301, a communication interface 302, a processor 303 and a memory 304.

[0053] The processor 303 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more for controlling the execution of a vital sign collection and analysis method based on multimodal physiological data in the present application.

[0054] The communication bus 301 may include a path for transmitting information between the above-mentioned components.

[0055] The memory 304 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 304 may exist independently and be connected to the processor 303 via the communication bus 301. The memory 304 may also be integrated with the processor 303.

[0056] The memory 304 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 303. The processor 303 is used to execute the program code stored in the memory 304. The program code may include one or more software modules. The determination of the emotion relevance described in the above embodiment can be implemented by the processor 303 and one or more software modules in the program code in the memory 304.

[0057] The communication interface 302 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0058] Optionally, the computer terminal device 300 may further include a power supply 305 for providing power to various devices or circuits in the real-time computer terminal device.

[0059] In a specific implementation, as an embodiment, a computer terminal device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0060] The above-mentioned computer terminal device may be a general-purpose computer terminal device or a dedicated computer terminal device. In a specific implementation, the computer terminal device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer terminal device.

[0061] In addition, in other aspects of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above-mentioned vital signs collection and analysis method based on multimodal physiological data.

[0062] In summary, in a vital sign collection and analysis system and method based on multimodal physiological data disclosed in an embodiment of the present application, multimodal physiological data is collected, and emotion features are extracted from the multimodal physiological data based on a preset emotion correction time to obtain multiple emotion features corresponding to the multimodal physiological data; each emotion feature corresponding to the multimodal physiological data is labeled and classified to obtain multiple emotion labels and corresponding label times; vital sign information in the multimodal physiological data is extracted, the vital sign information is windowed according to the emotion correction time to obtain multiple information windows and a health score is performed on the vital sign information in each information window; when the emotion relevance is higher than a preset threshold, the emotion correction coefficient corresponding to each information window is determined according to the emotion relevance, each emotion label and the corresponding label time, and the vital sign analysis results are generated based on the health score and emotion correction coefficient corresponding to each information window of the vital sign information. This method can correct the vital sign analysis results based on the emotion features of the multimodal physiological data, thereby reducing the interference of the user's emotional fluctuations on the vital sign analysis results when collecting vital sign information.

[0063] The above is only an embodiment of the present application, and the common knowledge such as the specific technical scheme or characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical scheme of the present application, several modifications and improvements can be made, which should also be regarded as the scope of protection of the present application, and these will not affect the effect of the implementation of the present application and the practicality of the patent.

[0064] The scope of protection claimed by this application shall be based on the content of its claims. The specific implementation methods and other records in the specification can be used to interpret the content of the claims. Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include these modifications and variations.

Claims

1. A method for collecting and analyzing vital signs based on multimodal physiological data, characterized in that: The method includes: Collecting multimodal physiological data, and extracting emotional features from the multimodal physiological data based on a preset emotion correction time to obtain a plurality of emotional features corresponding to the multimodal physiological data; Classifying each emotion feature corresponding to the multimodal physiological data by labeling to obtain a plurality of emotion labels and corresponding labeling times; Extracting vital sign information from the multimodal physiological data, windowing the vital sign information according to the emotion correction time to obtain a plurality of information windows and performing health scoring on the vital sign information in each information window; Performing correlation analysis on the health scores of each information window according to each emotion tag and the corresponding tag time to determine the emotion correlation corresponding to the health score; When the emotion correlation is higher than a preset threshold, the emotion correction coefficient corresponding to each information window is determined according to the emotion correlation, each emotion label and the corresponding label time, and the vital sign analysis result is generated based on the health score and emotion correction coefficient corresponding to each information window of the vital sign information.

2. The method according to claim 1, characterized in that Extracting emotion features from the multimodal physiological data based on the preset emotion correction time to obtain a plurality of emotion features corresponding to the multimodal physiological data specifically includes: Acquire the multimodal physiological data, segment the multimodal physiological data based on a preset emotion correction time to obtain a plurality of multimodal physiological data segments, and perform data fitting on any multimodal physiological data segment to obtain fitting data of each modality; Determine the basic level features corresponding to the fitting data of each modality, and remove the basic level differences of the fitting data of each modality according to the basic level features corresponding to the fitting data of each modality, so as to obtain the physiological characteristic waves corresponding to each modality; Performing Fourier transform on the physiological characteristic waves corresponding to each modality to obtain the energy spectrum corresponding to each modality, extracting feature vectors based on the physiological characteristic waves and energy spectrum corresponding to each modality to obtain the emotional features corresponding to the multimodal physiological data segment; The same method is adopted to extract emotion features from other multimodal physiological data segments until emotion features corresponding to each multimodal data segment are obtained and used as multiple emotion features corresponding to the multimodal physiological data.

3. The method according to claim 1, characterized in that Obtaining multiple information windows and performing health scoring on vital signs information in each information window specifically includes: Extract features of vital signs information in each information window to obtain key features of health scores corresponding to each information window; A health score sample is obtained, and each health score key feature is clustered based on the health score sample to obtain a health score corresponding to each information window.

4. The method according to claim 1, characterized in that The health scores of each information window are analyzed for correlation according to each emotion tag and the corresponding tag time, and the emotion correlation corresponding to the health score is determined to specifically include: Perform two-dimensional mapping based on each emotion label and the corresponding label time to obtain a two-dimensional emotion mapping model; Extracting emotion deviation distance from the two-dimensional emotion mapping model to obtain an emotion interference quantization sequence; A correlation degree analysis is performed based on the emotion interference quantification sequence and the health score of each information window to determine the emotion relevance corresponding to the vital sign information.

5. The method according to claim 1, characterized in that A single hidden layer neural network is used to classify the emotion features corresponding to each multimodal physiological data segment, and multiple emotion labels and corresponding label times are obtained.

6. The method according to claim 1, characterized in that When the emotion relevance is lower than a preset threshold, a vital sign analysis result is generated based on the health scores corresponding to each information window.

7. The method according to claim 1, characterized in that The multimodal physiological data is collected through a wearable device worn by a user.

8. A vital sign collection and analysis system based on multimodal physiological data, characterized in that: The vital signs collection and analysis system comprises: A physiological data acquisition module, used for acquiring multimodal physiological data, performing emotional feature extraction on the multimodal physiological data based on a preset emotion correction time, and obtaining a plurality of emotional features corresponding to the multimodal physiological data; A physiological data processing module, used for labeling and classifying each emotion feature corresponding to the multimodal physiological data to obtain a plurality of emotion labels and corresponding label times; The physiological data processing module is further used to extract vital sign information from the multimodal physiological data, window the vital sign information according to the emotion correction time, obtain multiple information windows, and perform health scoring on the vital sign information in each information window; The physiological data processing module is further used to perform correlation analysis on the health score of each information window according to each emotion tag and the corresponding tag time, and determine the emotion correlation corresponding to the health score; The vital signs result analysis module is used to determine the emotion correction coefficient corresponding to each information window according to the emotion correlation, each emotion label and the corresponding label time when the emotion correlation is higher than a preset threshold, and generate the vital signs analysis result based on the health score and emotion correction coefficient corresponding to each information window of the vital signs information.

9. A computer terminal device, characterized in that: The computer terminal device includes a memory and a processor, the memory stores codes, and the processor is configured to obtain the codes and execute the vital sign collection and analysis method based on multimodal physiological data as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing at least one computer program, characterized in that: The computer program is loaded and executed by a processor to implement the operations performed by the method for collecting and analyzing vital signs based on multimodal physiological data as described in any one of claims 1 to 7.

Images