Physical and psychological index dynamic characteristic multi-modal identification and analysis system based on artificial intelligence

Abstract

The invention discloses a physical and mental index dynamic characteristic multi-modal identification analysis system based on artificial intelligence, and relates to the technical field of biomedical signal processing, and the system comprises a multi-source data acquisition module, a dynamic risk assessment module, a blood flow abnormity analysis module, a multi-modal risk layering module and an intelligent early warning module; the multi-source data collection module collects respiration, heart rate and blood oxygen data, the dynamic risk assessment module predicts risks through a long-short-term memory network, the blood flow abnormity analysis module identifies blood flow abnormity through a convolutional neural network, the multi-mode risk layering module assesses the risk level, and the intelligent early warning module generates a report and takes early warning measures. Through multi-modal identification analysis, comprehensive, real-time and accurate monitoring of physical and mental indexes is realized, the risk assessment accuracy is improved, potential health risks are early warned in time, powerful support is provided for clinical decision making, the public health management level is improved, the medical cost is reduced, and the life quality is improved.

Description

Technical Field

[0001] This invention relates to the field of biomedical signal processing technology, specifically to an artificial intelligence-based multimodal recognition and analysis system for dynamic features of psychosomatic indicators. Background Technology

[0002] The dynamic characteristics of physical and mental indicators are mainly reflected in the differences, continuity, and variability of an individual's physical and mental state as it changes with time and context. First, differences refer to the variations in physical and mental indicators among different individuals. These differences may stem from various factors such as heredity, environment, and lifestyle. Therefore, each person's physical and mental state is unique and requires personalized attention and assessment. Second, continuity means that physical and mental state is a continuously changing process, rather than a static state. This continuity requires us to pay attention to the development trends and patterns of change when assessing physical and mental indicators in order to identify problems and take timely measures. Finally, variability means that physical and mental state can be changed. By adjusting lifestyle, improving the environment, and engaging in psychotherapy, we can actively influence physical and mental state and promote its development in a healthier and more positive direction. In summary, the dynamic characteristics of physical and mental indicators can remind us to view an individual's physical and mental state from a dynamic and comprehensive perspective, thereby better promoting their healthy development.

[0003] To address the issues of insufficient accuracy and comprehensiveness in physical and mental health monitoring and risk assessment, existing technologies primarily rely on regular physical examinations and questionnaires. However, these methods often only capture static physiological parameters and psychological states, failing to comprehensively and in real-time reflect an individual's dynamic physical and mental characteristics. Consequently, in practical applications, there are frequent instances of untimely and inaccurate warnings of potential health risks, leading to delayed diagnosis and poor treatment outcomes. To overcome these limitations, an AI-based multimodal recognition and analysis system for dynamic characteristics of physical and mental indicators is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide an artificial intelligence-based multimodal recognition and analysis system for dynamic features of physical and mental indicators, in order to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multimodal recognition and analysis system for dynamic characteristics of physical and mental indicators based on artificial intelligence, including a multi-source data acquisition module, a dynamic risk assessment module, a blood flow abnormality analysis module, a multimodal risk stratification module, and an intelligent early warning module; The multi-source data acquisition module collects the dynamic characteristics of the physical and mental indicators of the subject, including respiratory rate data, heart rate data, and blood oxygen saturation data. The dynamic risk assessment module, based on the respiratory rate and heart rate data of the subject, uses a long short-term memory network algorithm to construct a risk assessment model, capture the dynamic change patterns of respiratory rate and heart rate, and output a preliminary risk score. The blood flow abnormality analysis module, based on blood oxygen saturation data, uses a convolutional neural network algorithm to construct a blood flow abnormality analysis model and extracts abnormal blood flow distribution patterns of the tested subject from the time-series features of blood oxygen saturation. The multimodal risk stratification module constructs a dynamic risk stratification model based on the risk assessment model and the blood flow abnormality analysis model. It imports the latest collected preliminary risk scores and blood flow distribution abnormality pattern datasets into the dynamic risk stratification model and outputs the risk level. The intelligent early warning module generates a risk heat map and individual trend report based on the preliminary risk score and risk level, and takes corresponding early warning mechanisms according to the risk level.

[0006] A further improvement to the technical solution of this invention lies in that: the multi-source data acquisition module, in the process of acquiring the respiratory rate data, heart rate data, and blood oxygen saturation data of the subject, includes: An integrated millimeter-wave radar sensor, employing the Doppler effect principle, is deployed at the front end of the detection equipment, maintaining a distance of 0.5 to 1.5 meters from the subject and facing the chest area. The radar's operating frequency is set to 60 to 64 GHz, with a transmission power not exceeding 10mW. By emitting millimeter waves and receiving the micro-motion signals reflected from the chest surface, the peak frequency of the spectrum is calculated as the respiratory frequency. The millimeter-wave radar reuses the respiratory detection channel and collects heart rate data through signal time-division multiplexing technology. A high frame rate infrared camera is deployed above the screen of the detection device and aimed at the face area of ​​the subject. The infrared camera integrates a dual-wavelength light source and collects the subcutaneous blood flow signal of the subject through reflective photoplethysmography. The absorption ratio of red light to infrared light is calculated, and the blood oxygen saturation is derived based on the empirical calibration curve.

[0007] A further improvement to the technical solution of the present invention lies in that: the preprocessing process of the multi-source data acquisition module for the acquired respiratory rate data, heart rate data, and blood oxygen saturation data includes: The thoracic cavity micro-motion signal received by millimeter-wave radar is bandpass filtered, and the signal spectrum is calculated by Fourier transform. The frequency corresponding to the peak value of the spectrum is extracted as the respiratory fundamental frequency. The respiratory detection channel of millimeter-wave radar is reused, and the heartbeat signal is separated by time-division multiplexing technology. Based on the chest cavity micro-movement caused by the heartbeat, the heartbeat cycle signal is extracted by time-frequency analysis, and the heartbeat cycle signal is segmented by a sliding window with a length of 5 seconds. The dual-wavelength reflection signals acquired by the infrared camera are normalized, and the arterial blood pulsation component of the photoplethysmography wave is extracted by filtering. The AC and DC components of red and infrared light are separated, and the blood oxygen saturation data are analyzed using a sliding window with a length of 10 seconds. Micro-vibrations on the skin surface are detected by phase changes of the heartbeat cycle signal using millimeter-wave radar, and the light absorption characteristics of the dual-wavelength reflected signal from an infrared camera are combined to collect the skin conductivity and body temperature data of the subject.

[0008] A further improvement to the technical solution of this invention lies in the fact that the multi-source data acquisition module performs feature extraction on the preprocessed respiratory rate data, heart rate data, and blood oxygen saturation data, including: Based on the fundamental frequency of respiration, harmonic signals in its integer multiples are extracted, and the ratio of the sum of harmonic energy to the fundamental frequency energy is calculated to obtain the respiratory harmonic energy ratio. The standard deviation of adjacent heartbeat intervals is calculated, and the heartbeat signal is decomposed into low-frequency and high-frequency components, and their power ratio is calculated. The standard deviation of blood oxygen saturation within the sliding window is calculated to quantify the dynamic change amplitude of blood oxygen saturation. Based on the ratio of the AC to DC components of the photoplethysmography pulse wave, the local blood flow inhomogeneity of the subject is calculated. The standard deviation of skin conductivity data is extracted, and combined with the dynamic trend slope of body temperature data, the extroversion index of the subject is obtained. Based on the Pearson correlation coefficient between blood oxygen saturation and heart rate variability, the logical thinking potential of the subject is assessed.

[0009] A further improvement to the technical solution of this invention lies in the following: the dynamic risk assessment module, in constructing a risk assessment model and outputting a preliminary risk score, includes the following process: The architecture of a long short-term memory network model is designed to construct a risk assessment model. The long short-term memory network model includes an input layer, a hidden layer, and an output layer. The hidden layer includes a forget gate, an input gate, a memory unit, and an output gate. The time-series feature sequence, including respiratory fundamental frequency, respiratory harmonic energy ratio, standard deviation of adjacent heartbeat interval, and low-frequency to high-frequency power ratio, is input into the input layer of the risk assessment model. The hidden layer explores the dynamic correlation between respiratory and heart rate features. The forget gate dynamically adjusts the weight of the long-term dependence of respiratory and heart rate based on the historical memory state. The input gate filters short-term features in the current input that are sensitive to emotional state and updates the memory unit. The output gate outputs the hidden state at the current moment to the output layer of the long short-term memory network model. The output layer maps the latent state to a four-dimensional emotion probability distribution and a three-dimensional personality potential distribution. It calculates the percentage difference between the standard deviation of adjacent heartbeat intervals and the corresponding baseline value of healthy individuals to obtain the heart rate variability deviation. The normalized results of heart rate variability deviation, respiratory harmonic energy ratio, and blood oxygen fluctuation range are weighted and summed to generate a preliminary risk score of 0-100. The dominant emotion label and dominant personality label are output through a normalization function. The emotion label includes calmness, anxiety, depression, and stress, and the personality label includes extraversion, logical thinking, and leadership. The Pearson correlation coefficient between respiratory and heart rate signals is calculated, and this Pearson correlation coefficient is used as an additional feature input into the risk assessment model. Based on the historical data of the subjects, the threshold for judging respiratory harmonic energy ratio and heart rate variability is adaptively adjusted. The newly collected data is input into the trained risk assessment model, which outputs a preliminary risk score, emotion label, and personality label within 5 seconds. The risk assessment model predicts the deterioration trend of the subject's emotion and personality based on the rate of change of the score within a continuous window.

[0010] A further improvement to the technical solution of this invention lies in the fact that the blood flow abnormality analysis module, using a convolutional neural network algorithm, constructs a blood flow abnormality analysis model, comprising: Employing a convolutional neural network architecture, the system receives time-series blood oxygen saturation data and converts it into a two-dimensional feature map of time step × blood oxygen value. The blood oxygen saturation time-series data is segmented into 10-second time windows, and a spectrogram is generated through short-time Fourier transform. This spectrogram is then mapped to a 128×128 pixel grayscale image, constructing a blood flow anomaly analysis model. The convolutional kernels in the convolutional neural network extract local blood flow fluctuation patterns, identify high-frequency abnormal signals, and fuse spatiotemporal information to capture the spatial correlation of blood flow distribution. Based on the abnormal ratio of AC to DC components of the photoplethysmography pulse wave, combined with the feature map output by the risk assessment model, the system locates the abnormal blood flow region on the subject's face and outputs the probability of blood flow anomaly through a fully connected network. Calculate the Pearson correlation coefficient between blood oxygen fluctuation and heart rate variability. If the Pearson correlation coefficient between blood oxygen fluctuation and heart rate variability is less than 0.3, it is determined that there is a synergistic effect between blood flow abnormality and autonomic nervous system disorder, thereby enhancing the sensitivity of abnormal area in the blood flow abnormality analysis model. Based on the historical data of the subjects, the threshold for blood flow non-uniformity is adaptively set. The newly collected blood oxygen data is preprocessed and then input into the trained blood flow abnormality analysis model. Within 5 seconds, the model outputs a blood flow abnormality score and a blood flow distribution heatmap, and marks the blood flow abnormality areas of the tested subject.

[0011] A further improvement to the technical solution of this invention lies in the fact that the process of constructing a dynamic risk stratification model by the multimodal risk stratification module includes: The preliminary risk score, blood flow abnormality characteristics, and personality potential index are concatenated into a fusion feature vector. Based on the Pearson correlation coefficient between respiratory harmonic energy ratio and blood oxygen fluctuation standard deviation, the weights of the preliminary risk score, blood flow abnormality characteristics, and personality potential index are adjusted to construct a dynamic risk stratification fusion model. The fused feature vector is input into the dynamic risk stratification model. If the Pearson correlation coefficient is below 0.3, it is determined that respiratory and blood flow abnormalities have a synergistic effect, and the weight of blood flow abnormality features is increased. If the Pearson correlation coefficient is above 0.3, the time series weight is assigned to the baseline as 0.6, the weight of blood flow abnormality features as 0.3, and the weight of personality potential index as 0.1. Based on the fused feature vector and the weight assignment results, the comprehensive risk score is calculated as follows: (preliminary risk score × corresponding weight) + (blood flow abnormality score × corresponding weight) - (personality potential index × corresponding weight). Among them, when the extraversion score is >70, the comprehensive risk score is reduced by 10%, and when the logical thinking score is >75, the risk level is automatically downgraded by one level.

[0012] A further improvement to the technical solution of this invention lies in the fact that the process of outputting the risk level by the multimodal risk stratification module includes: The preliminary risk score, which includes respiratory harmonic energy ratio, personality potential index, heart rate variability deviation, and blood oxygen fluctuation range, and the abnormal area annotation, which includes blood oxygen standard deviation, local blood flow inhomogeneity, and blood flow distribution heatmap, are input into the dynamic risk stratification model. The output is a comprehensive risk score. Based on the test subject's historical data, the comprehensive risk score is divided into red warning (a score higher than 75), orange warning (a score between 50 and 75), yellow warning (a score between 30 and 50), and green warning (a score lower than 30).

[0013] A further improvement to the technical solution of this invention lies in the fact that the intelligent early warning module generates a risk heatmap and an individual trend report by comprising the following steps: The comprehensive risk score output by the dynamic risk stratification model is integrated with the geographical location data of the tested individuals in the region. Based on the time-series data of respiration, heart rate and blood oxygen aligned with hardware timestamps, the proportion of people at risk in each region is statistically analyzed according to time windows to generate a risk distribution matrix. According to the color coding rules, the risk distribution matrix is ​​mapped to a heat map through the geographic information system of the cloud platform. The abnormal high-incidence areas are updated in real time. The average Pearson correlation coefficient of respiration-heart rate-blood oxygen in the region is calculated. If the average Pearson correlation coefficient is lower than 0.3, it is marked as physiological synergy abnormality and superimposed on the risk heat map. If the average Pearson correlation coefficient is higher than 0.3, the risk level is increased and data resampling verification is performed. Plot time-series index curves and line graphs of comprehensive risk scores to show the dynamic changes of respiratory harmonic energy ratio, heart rate variability deviation, blood oxygen fluctuation standard deviation, and extraversion index. Mark event nodes, customize time spans, and analyze the rate of change of comprehensive risk scores based on sliding windows. If the comprehensive risk score rises by 15 points in a single day, it is determined to be a period of emotional deterioration and marked with red shading in the individual trend report. Combined with the physiological correlation between respiratory rate and heart rate, data points that exceed the range during abnormal periods are marked with yellow warning symbols. If the logical thinking score in the personality potential distribution is >75, then the report will indicate high logical thinking potential and recommend logical reasoning training tools. If the leadership score is >70, then the test subject is advised to participate in team collaboration simulation training.

[0014] A further improvement to the technical solution of the present invention is that the intelligent early warning module, in the process of adopting a corresponding early warning mechanism, includes: If the comprehensive risk score exceeds 75 points and there are cases where the respiratory harmonic energy ratio is >35%, the standard deviation of blood oxygen fluctuation is >3%, and the respiratory-heart rate Pearson correlation coefficient is <0.2, it is judged as a red alert, the dominant risk factor is marked, and the hardware device is automatically locked to lock the data during the abnormal period. If the comprehensive risk score is between 50 and 75 points and the heart rate variability deviation is >60%, it is judged as an orange warning, and the threshold is adjusted based on the test subject's historical data; If the comprehensive risk score is between 30 and 50, the standard deviation of blood oxygen fluctuation is between 1% and 2%, and the correlation coefficient between respiration and heart rate is between 0.2 and 0.4, it is judged as a yellow warning, and the individual trend report of the subject is marked with an observation label. If the overall risk score is below 30 points, and the respiratory harmonic energy ratio is <15% and the standard deviation of blood oxygen fluctuation is <1%, it is judged as green and normal, and the baseline file is updated monthly.

[0015] Due to the adoption of the above technical solution, the technical progress achieved by this invention compared to the prior art is as follows: This invention provides an AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators. It can collect and analyze multi-dimensional physical and mental indicator data such as respiratory rate, heart rate, and blood oxygen saturation of the tested subject in real time and accurately. By introducing long short-term memory network model and convolutional neural network model, the system can capture the dynamic change patterns of these indicators, thereby achieving early warning and accurate assessment of potential health risks, and effectively improving the accuracy and reliability of physical and mental health monitoring.

[0016] This invention provides an AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators. Through a multimodal risk stratification module, it effectively integrates various information such as preliminary risk scores and abnormal blood flow distribution patterns to construct a dynamic risk stratification fusion model. This innovative design enables the system to more comprehensively assess the physical and mental health status of the tested individuals, output more accurate risk levels, provide strong support for clinical decision-making, and reduce the risk of misdiagnosis and missed diagnosis.

[0017] This invention provides an AI-based multimodal recognition and analysis system for dynamic characteristics of physical and mental health indicators. The intelligent early warning module can generate intuitive risk heat maps and individual trend reports based on risk scores and risk levels, helping doctors and patients to better understand the changing trends of their physical and mental health. At the same time, the system can automatically trigger corresponding early warning mechanisms based on risk levels, take timely intervention measures, effectively protect the physical and mental health of the tested individuals, and improve their quality of life. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a block diagram of the present invention; Figure 2 This is a flowchart illustrating the preprocessing of respiratory rate and heart rate data in the multi-source data acquisition module of this invention. Figure 3 This is a flowchart illustrating the preprocessing of blood oxygen saturation data in the multi-source data acquisition module of this invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Examples, such as Figures 1 to 3 As shown, this invention provides a multimodal recognition and analysis system for dynamic features of physical and mental indicators based on artificial intelligence, including a multi-source data acquisition module, a dynamic risk assessment module, a blood flow abnormality analysis module, a multimodal risk stratification module, and an intelligent early warning module.

[0022] The multi-source data acquisition module collects dynamic characteristics of the subject's physical and mental indicators, including respiratory rate, heart rate, and blood oxygen saturation. It integrates a millimeter-wave radar sensor, employing the Doppler effect principle, deployed at the front end of the detection device at a distance of 0.5 to 1.5 meters from the subject, directly facing the chest area. The radar's operating frequency is set to 60 to 64 MHz. With a transmission power not exceeding 10mW, the millimeter-wave radar transmits millimeter waves and receives micro-motion signals reflected from the chest cavity surface. The peak frequency of the spectrum is calculated as the respiratory frequency. The millimeter-wave radar reuses the respiratory detection channel and acquires heart rate data using time-division multiplexing technology. A high-frame-rate infrared camera is deployed above the detection device screen, aimed at the subject's face. The infrared camera integrates a dual-wavelength light source and acquires subcutaneous blood flow signals using reflective photoplethysmography. The absorption ratio of red and infrared light is calculated, and blood oxygen saturation is derived based on an empirical calibration curve. The chest cavity micro-motion signals received by the millimeter-wave radar are bandpass filtered, and the signal spectrum is calculated using Fourier transform. The frequency corresponding to the peak frequency is extracted as the respiratory fundamental frequency. The millimeter-wave radar's respiratory detection channel is reused, and the heartbeat signal is separated using time-division multiplexing technology. Based on the chest cavity micro-motion caused by the heartbeat, the heartbeat cycle signal is extracted through time-frequency analysis. A 5-second sliding window is used to segment the heartbeat cycle signal. The dual-wavelength reflected signals acquired by the infrared camera are normalized, and the photoplethysmography pulse wave is extracted through filtering. Arterial blood pulsation components were analyzed, separating the AC and DC components of red and infrared light. A 10-second sliding window was used to analyze blood oxygen saturation data. Micro-vibrations on the skin surface were detected by phase changes in the heartbeat cycle signal from millimeter-wave radar. Combined with the light absorption characteristics of the dual-wavelength reflected signal from an infrared camera, skin conductivity and body temperature data were collected. Based on the respiratory fundamental frequency, harmonic signals in integer multiples of the fundamental frequency were extracted, and the ratio of the total harmonic energy to the fundamental frequency energy was calculated to obtain the respiratory harmonic energy ratio. The standard deviation of adjacent heartbeat intervals was calculated, and the heartbeat signal was decomposed into low-frequency and high-frequency components, calculating their power ratio. The standard deviation of blood oxygen saturation within the sliding window was calculated to quantify the dynamic change amplitude of blood oxygen saturation. Based on the ratio of the AC to DC components of the photoplethysmography pulse wave, the local blood flow inhomogeneity of the subject was calculated. The fluctuation standard deviation of skin conductivity data was extracted, and combined with the dynamic trend slope of body temperature data, the extroversion index of the subject was obtained. Based on the Pearson correlation coefficient between blood oxygen saturation and heart rate variability, the logical thinking potential of the subject was assessed.

[0023] The dynamic risk assessment module, based on the respiratory rate and heart rate data of the subjects, uses a Long Short-Term Memory (LSTM) network algorithm to construct a risk assessment model, capturing the dynamic changes in respiratory rate and heart rate, and outputting a preliminary risk score. The architecture of the LTM network model is designed to construct the risk assessment model, which includes an input layer, a hidden layer, and an output layer. The hidden layer includes a forget gate, an input gate, memory units, and an output gate. A temporal feature sequence containing the respiratory fundamental frequency, respiratory harmonic energy ratio, standard deviation of adjacent heartbeat intervals, and low-frequency to high-frequency power ratio is input to the input layer of the risk assessment model. The hidden layer explores the dynamic correlation between respiratory and heart rate features. The forget gate dynamically adjusts the weights of the long-term dependence of respiratory and heart rate based on historical memory states. The input gate filters short-term features sensitive to emotional states in the current input and updates the memory units. The output gate outputs the current hidden state to the output layer of the LTM network model. The output layer stores the hidden state... The model maps to a four-dimensional emotion probability distribution and a three-dimensional personality potential distribution. It calculates the percentage difference between the standard deviation of adjacent heart rate intervals and the corresponding baseline value in healthy individuals to obtain the heart rate variability deviation. The normalized results of heart rate variability deviation, respiratory harmonic energy ratio, and blood oxygen fluctuation range are weighted and summed to generate a preliminary risk score of 0-100. A dominant emotion label and a dominant personality label are output through a normalization function. The emotion labels include calmness, anxiety, depression, and stress; the personality labels include extraversion, logical thinking, and leadership. The Pearson correlation coefficient between respiratory and heart rate signals is calculated and used as an additional feature input to the risk assessment model. Based on the subject's historical data, the model adaptively adjusts the thresholds for judging respiratory harmonic energy ratio and heart rate variability. Newly collected data is input into the trained risk assessment model, which outputs a preliminary risk score, emotion label, and personality label within 5 seconds. The risk assessment model predicts the deterioration trend of the subject's emotions and personality based on the rate of change of scores within a continuous window.

[0024] The blood flow anomaly analysis module, based on blood oxygen saturation data, utilizes a convolutional neural network algorithm to construct a blood flow anomaly analysis model. It extracts abnormal blood flow distribution patterns from the temporal features of blood oxygen saturation. Employing a convolutional neural network architecture, it receives and converts the blood oxygen saturation temporal data into a two-dimensional feature map of time step × blood oxygen value. The blood oxygen saturation temporal data is segmented into 10-second time windows, and a spectrum is generated through short-time Fourier transform, mapped to a 128×128 pixel grayscale image. This constructs the blood flow anomaly analysis model. The convolutional kernels in the convolutional neural network extract local blood flow fluctuation patterns, identify high-frequency abnormal signals, and fuse spatiotemporal information to capture the spatial correlation of blood flow distribution. This is based on the intersection of photoplethysmography (PPG) pulse waves. The ratio of abnormal flow to abnormal direct flow components, combined with the feature map output by the risk assessment model, locates the abnormal blood flow areas on the subject's face. The probability of abnormal blood flow is output through a fully connected network. The Pearson correlation coefficient between blood oxygen fluctuation and heart rate variability is calculated. If the Pearson correlation coefficient between blood oxygen fluctuation and heart rate variability is less than 0.3, it is determined that there is a synergistic effect between abnormal blood flow and autonomic nervous system disorder, thus enhancing the sensitivity of the abnormal area of ​​the blood flow abnormality analysis model. Based on the subject's historical data, the blood flow non-uniformity threshold is adaptively set. Newly acquired blood oxygen data is preprocessed and input into the trained blood flow abnormality analysis model. Within 5 seconds, the blood flow abnormality score and blood flow distribution heatmap are output, and the abnormal blood flow areas of the subject are marked.

[0025] The multimodal risk stratification module, based on a risk assessment model and a blood flow anomaly analysis model, constructs a dynamic risk stratification model. It imports the latest collected preliminary risk scores and blood flow distribution anomaly pattern datasets into the dynamic risk stratification model, outputting risk levels. The preliminary risk score, blood flow anomaly features, and personality potential index are concatenated into a fusion feature vector. Based on the Pearson correlation coefficient between respiratory harmonic energy ratio and blood oxygen fluctuation standard deviation, the weights of the preliminary risk score, blood flow anomaly features, and personality potential index are adjusted to construct a dynamic risk stratification fusion model. The fusion feature vector is input into the dynamic risk stratification model. If the Pearson correlation coefficient is below 0.3, it is determined that respiratory and blood flow anomalies are synergistic, and the weight of blood flow anomaly features is increased. If the Pearson correlation coefficient is above 0.3, the time-series weight is assigned as 0.6, the weight of blood flow anomaly features as 0.3, and the weight of personality potential index as 0.1, based on the baseline. Based on the fusion feature vector and weight allocation results, the comprehensive risk score is calculated as follows: (Preliminary risk score × corresponding weight) + (Blood flow abnormality score × corresponding weight) - (Personality potential index × corresponding weight). When the extraversion score is >70, the comprehensive risk score decreases by 10%. When the logical thinking score is >75, the risk level is automatically downgraded by one level. The preliminary risk score, which includes respiratory harmonic energy ratio, personality potential index, heart rate variability deviation, and blood oxygen fluctuation range, and the abnormal area labels, which include blood oxygen standard deviation, local blood flow inhomogeneity, and blood flow distribution heatmap, are input into the dynamic risk stratification model to output the comprehensive risk score. Based on the test subject's historical data, the comprehensive risk score is divided into four categories: a comprehensive risk score above 75 is a red warning, a comprehensive risk score between 50 and 75 is an orange warning, a comprehensive risk score between 30 and 50 is a yellow warning, and a comprehensive risk score below 30 is a green warning.

[0026] The intelligent early warning module generates a risk heatmap and individual trend reports based on preliminary risk scores and risk levels. It implements corresponding early warning mechanisms according to risk levels, integrating the comprehensive risk score output by the dynamic risk stratification model. Combined with the geographic location data of those being monitored in the region, and based on hardware timestamp-aligned respiratory, heart rate, and blood oxygen time-series data, it calculates the proportion of at-risk individuals in each region according to time windows, generating a risk distribution matrix. Following color coding rules, the risk distribution matrix is ​​mapped to a heatmap via the cloud platform's geographic information system, updating abnormally high-incidence areas in real time and calculating the average Pearson risk level for respiratory, heart rate, and blood oxygen levels within the region. The correlation coefficient is used to indicate physiological synergy abnormalities. If the average Pearson correlation coefficient is below 0.3, it is marked as abnormal and superimposed on the risk heatmap. If the average Pearson correlation coefficient is above 0.3, the risk level is increased, and data resampling and verification are performed. Time-series index curves and a comprehensive risk score line graph showing the dynamic changes of respiratory harmonic energy ratio, heart rate variability deviation, blood oxygen fluctuation standard deviation, and extraversion index are plotted, event nodes are marked, and custom time spans are defined. The rate of change of the comprehensive risk score is analyzed based on a sliding window. If the comprehensive risk score rises by 15 points in a single day, it is determined to be a period of emotional deterioration and is marked in red in the individual trend report. Color-coded shading, combined with the physiological correlation between respiratory rate and heart rate, marks data points exceeding the range during abnormal periods with yellow warning symbols. If the logical thinking score in the personality potential distribution is >75, high logical thinking potential is indicated in the report, and logical reasoning training tools are recommended. If the leadership score is >70, the test subject is advised to participate in team collaboration simulation training. If the comprehensive risk score exceeds 75 and there are conditions such as respiratory harmonic energy ratio >35%, blood oxygen fluctuation standard deviation >3%, and respiratory-heart rate Pearson correlation coefficient <0.2, it is judged as a red warning, the dominant risk factor is marked, and hardware devices are triggered. The system automatically locks data from abnormal periods. If the comprehensive risk score is between 50 and 75 and the heart rate variability deviation is >60%, it is considered an orange alert, and the threshold is adjusted based on the subject's historical data. If the comprehensive risk score is between 30 and 50 and the blood oxygen fluctuation standard deviation is between 1% and 2%, and the respiratory-heart rate correlation coefficient is between 0.2 and 0.4, it is considered a yellow alert, and an observation label is marked in the subject's individual trend report. If the comprehensive risk score is below 30 and the respiratory harmonic energy ratio is <15%, and the blood oxygen fluctuation standard deviation is <1%, it is considered green and normal, and the baseline file is updated monthly.

[0027] First, the multi-source data acquisition module collects real-time data on the subjects' physical and mental indicators, such as respiratory rate, heart rate, and blood oxygen saturation. Then, the dynamic risk assessment module uses a long short-term memory network model to analyze these data, capture the dynamic changes in respiratory rate and heart rate, and output a preliminary risk score to predict risk trends. Simultaneously, the blood flow abnormality analysis module uses a convolutional neural network model to extract abnormal blood flow distribution patterns from the blood oxygen saturation data. Next, the multimodal risk stratification module combines the preliminary risk score and blood flow abnormality features to construct a dynamic risk stratification model and imports the latest dataset to output accurate risk levels. Finally, the intelligent early warning module generates an intuitive risk heatmap and individual trend report based on the risk score and risk level, and triggers corresponding early warning mechanisms according to the risk level to ensure timely implementation of necessary health intervention measures.

[0028] Furthermore, traditional and existing psychological testing methods primarily screen for negative emotional tags to identify psychological problems such as depression and low mood, facilitating timely intervention or treatment. However, the technical solution presented in this application not only filters for negative emotional tags for timely intervention but also identifies individuals with strong extroversion, logical thinking, focus, and leadership qualities based on personality tags. For example, when analyzing and identifying minors, the results can be used to define their personality traits and discover their strengths, such as high self-confidence, outgoing or extroverted personality, strong leadership, or strong logical thinking. This allows for targeted suggestions and advice based on personality tags and characteristics, such as providing psychological support, potential identification, and potential stimulation.

[0029] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal recognition and analysis system for dynamic features of physical and mental indicators based on artificial intelligence, characterized in that: It includes a multi-source data acquisition module, a dynamic risk assessment module, a blood flow abnormality analysis module, a multimodal risk stratification module, and an intelligent early warning module; The multi-source data acquisition module collects the dynamic characteristics of the physical and mental indicators of the subject, including respiratory rate data, heart rate data, and blood oxygen saturation data. The dynamic risk assessment module, based on the respiratory rate and heart rate data of the subject, uses a long short-term memory network algorithm to construct a risk assessment model, capture the dynamic change patterns of respiratory rate and heart rate, and output a preliminary risk score. The blood flow abnormality analysis module, based on blood oxygen saturation data, uses a convolutional neural network algorithm to construct a blood flow abnormality analysis model and extracts abnormal blood flow distribution patterns of the tested subject from the time-series features of blood oxygen saturation. The multimodal risk stratification module constructs a dynamic risk stratification model based on the risk assessment model and the blood flow abnormality analysis model. It imports the latest collected preliminary risk scores and blood flow distribution abnormality pattern datasets into the dynamic risk stratification model and outputs the risk level. The intelligent early warning module generates a risk heat map and individual trend report based on the preliminary risk score and risk level, and takes corresponding early warning mechanisms according to the risk level.

2. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 1, characterized in that: The multi-source data acquisition module collects respiratory rate data, heart rate data, and blood oxygen saturation data of the subject, including the following processes: An integrated millimeter-wave radar sensor, employing the Doppler effect principle, is deployed at the front end of the detection equipment, maintaining a distance of 0.5 to 1.5 meters from the subject and facing the chest area. The radar's operating frequency is set to 60 to 64 GHz, with a transmission power not exceeding 10mW. By emitting millimeter waves and receiving the micro-motion signals reflected from the chest surface, the peak frequency of the spectrum is calculated as the respiratory frequency. The millimeter-wave radar reuses the respiratory detection channel and collects heart rate data through signal time-division multiplexing technology. A high frame rate infrared camera is deployed above the screen of the detection device and aimed at the face area of ​​the subject. The infrared camera integrates a dual-wavelength light source and collects the subcutaneous blood flow signal of the subject through reflective photoplethysmography. The absorption ratio of red light to infrared light is calculated, and the blood oxygen saturation is derived based on the empirical calibration curve.

3. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 2, characterized in that: The multi-source data acquisition module performs the following preprocessing steps on the acquired respiratory rate data, heart rate data, and blood oxygen saturation data: The thoracic cavity micro-motion signal received by millimeter-wave radar is bandpass filtered, and the signal spectrum is calculated by Fourier transform. The frequency corresponding to the peak value of the spectrum is extracted as the respiratory fundamental frequency. The respiratory detection channel of millimeter-wave radar is reused, and the heartbeat signal is separated by time-division multiplexing technology. Based on the chest cavity micro-movement caused by the heartbeat, the heartbeat cycle signal is extracted by time-frequency analysis, and the heartbeat cycle signal is segmented by a sliding window with a length of 5 seconds. The dual-wavelength reflection signals acquired by the infrared camera are normalized, and the arterial blood pulsation component of the photoplethysmography wave is extracted by filtering. The AC and DC components of red and infrared light are separated, and the blood oxygen saturation data are analyzed using a sliding window with a length of 10 seconds. Micro-vibrations on the skin surface are detected by phase changes of the heartbeat cycle signal using millimeter-wave radar, and the light absorption characteristics of the dual-wavelength reflected signal from an infrared camera are combined to collect the skin conductivity and body temperature data of the subject.

4. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 3, characterized in that: The multi-source data acquisition module performs feature extraction on the preprocessed respiratory rate data, heart rate data, and blood oxygen saturation data, including the following steps: Based on the fundamental frequency of respiration, harmonic signals in its integer multiples are extracted, and the ratio of the sum of harmonic energy to the fundamental frequency energy is calculated to obtain the respiratory harmonic energy ratio. The standard deviation of adjacent heartbeat intervals is calculated, and the heartbeat signal is decomposed into low-frequency and high-frequency components, and their power ratio is calculated. The standard deviation of blood oxygen saturation within the sliding window is calculated to quantify the dynamic change amplitude of blood oxygen saturation. Based on the ratio of the AC to DC components of the photoplethysmography pulse wave, the local blood flow inhomogeneity of the subject is calculated. The standard deviation of skin conductivity data is extracted, and combined with the dynamic trend slope of body temperature data, the extroversion index of the subject is obtained. Based on the Pearson correlation coefficient between blood oxygen saturation and heart rate variability, the logical thinking potential of the subject is assessed.

5. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 4, characterized in that: The dynamic risk assessment module, in its process of constructing a risk assessment model and outputting a preliminary risk score, includes: The architecture of a long short-term memory network model is designed to construct a risk assessment model. The long short-term memory network model includes an input layer, a hidden layer, and an output layer. The hidden layer includes a forget gate, an input gate, a memory unit, and an output gate. The time-series feature sequence, including respiratory fundamental frequency, respiratory harmonic energy ratio, standard deviation of adjacent heartbeat interval, and low-frequency to high-frequency power ratio, is input into the input layer of the risk assessment model. The hidden layer explores the dynamic correlation between respiratory and heart rate features. The forget gate dynamically adjusts the weight of the long-term dependence of respiratory and heart rate based on the historical memory state. The input gate filters short-term features in the current input that are sensitive to emotional state and updates the memory unit. The output gate outputs the hidden state at the current moment to the output layer of the long short-term memory network model. The output layer maps the latent state to a four-dimensional emotion probability distribution and a three-dimensional personality potential distribution. It calculates the percentage difference between the standard deviation of adjacent heartbeat intervals and the corresponding baseline value of healthy individuals to obtain the heart rate variability deviation. The normalized results of heart rate variability deviation, respiratory harmonic energy ratio, and blood oxygen fluctuation range are weighted and summed to generate a preliminary risk score of 0-100. The dominant emotion label and dominant personality label are output through a normalization function. The emotion label includes calmness, anxiety, depression, and stress, and the personality label includes extraversion, logical thinking, and leadership. The Pearson correlation coefficient between respiratory and heart rate signals is calculated, and this Pearson correlation coefficient is used as an additional feature input into the risk assessment model. Based on the historical data of the subjects, the threshold for judging respiratory harmonic energy ratio and heart rate variability is adaptively adjusted. The newly collected data is input into the trained risk assessment model, which outputs a preliminary risk score, emotion label, and personality label within 5 seconds. The risk assessment model predicts the deterioration trend of the subject's emotion and personality based on the rate of change of the score within a continuous window.

6. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 5, characterized in that: The blood flow anomaly analysis module, using a convolutional neural network algorithm, constructs a blood flow anomaly analysis model, including the following steps: Employing a convolutional neural network architecture, the system receives time-series blood oxygen saturation data and converts it into a two-dimensional feature map of time step × blood oxygen value. The blood oxygen saturation time-series data is segmented into 10-second time windows, and a spectrogram is generated through short-time Fourier transform. This spectrogram is then mapped to a 128×128 pixel grayscale image, constructing a blood flow anomaly analysis model. The convolutional kernels in the convolutional neural network extract local blood flow fluctuation patterns, identify high-frequency abnormal signals, and fuse spatiotemporal information to capture the spatial correlation of blood flow distribution. Based on the abnormal ratio of AC to DC components of the photoplethysmography pulse wave, combined with the feature map output by the risk assessment model, the system locates the abnormal blood flow region on the subject's face and outputs the probability of blood flow anomaly through a fully connected network. Calculate the Pearson correlation coefficient between blood oxygen fluctuation and heart rate variability. If the Pearson correlation coefficient between blood oxygen fluctuation and heart rate variability is less than 0.3, it is determined that there is a synergistic effect between blood flow abnormality and autonomic nervous system disorder, thereby enhancing the sensitivity of abnormal area in the blood flow abnormality analysis model. Based on the historical data of the subjects, the threshold for blood flow non-uniformity is adaptively set. The newly collected blood oxygen data is preprocessed and then input into the trained blood flow abnormality analysis model. Within 5 seconds, the model outputs a blood flow abnormality score and a blood flow distribution heatmap, and marks the blood flow abnormality areas of the tested subject.

7. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 6, characterized in that: The process of constructing a dynamic risk stratification model by the multimodal risk stratification module includes: The preliminary risk score, blood flow abnormality characteristics, and personality potential index are concatenated into a fusion feature vector. Based on the Pearson correlation coefficient between respiratory harmonic energy ratio and blood oxygen fluctuation standard deviation, the weights of the preliminary risk score, blood flow abnormality characteristics, and personality potential index are adjusted to construct a dynamic risk stratification fusion model. The fused feature vector is input into the dynamic risk stratification model. If the Pearson correlation coefficient is below 0.3, it is determined that respiratory and blood flow abnormalities have a synergistic effect, and the weight of blood flow abnormality features is increased. If the Pearson correlation coefficient is above 0.3, the time series weight is assigned to the baseline as 0.6, the weight of blood flow abnormality features as 0.3, and the weight of personality potential index as 0.

1. Based on the fused feature vector and the weight assignment results, the comprehensive risk score is calculated as follows: (preliminary risk score × corresponding weight) + (blood flow abnormality score × corresponding weight) - (personality potential index × corresponding weight). Among them, when the extraversion score is >70, the comprehensive risk score is reduced by 10%, and when the logical thinking score is >75, the risk level is automatically downgraded by one level.

8. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 7, characterized in that: The process by which the multimodal risk stratification module outputs the risk level includes: The preliminary risk score, which includes respiratory harmonic energy ratio, personality potential index, heart rate variability deviation, and blood oxygen fluctuation range, and the abnormal area annotation, which includes blood oxygen standard deviation, local blood flow inhomogeneity, and blood flow distribution heatmap, are input into the dynamic risk stratification model. The output is a comprehensive risk score. Based on the test subject's historical data, the comprehensive risk score is divided into red warning (a score higher than 75), orange warning (a score between 50 and 75), yellow warning (a score between 30 and 50), and green warning (a score lower than 30).

9. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 8, characterized in that: The intelligent early warning module generates risk heatmaps and individual trend reports through the following process: The comprehensive risk score output by the dynamic risk stratification model is integrated with the geographical location data of the tested individuals in the region. Based on the time-series data of respiration, heart rate and blood oxygen aligned with hardware timestamps, the proportion of people at risk in each region is statistically analyzed according to time windows to generate a risk distribution matrix. According to the color coding rules, the risk distribution matrix is ​​mapped to a heat map through the geographic information system of the cloud platform. The abnormal high-incidence areas are updated in real time. The average Pearson correlation coefficient of respiration-heart rate-blood oxygen in the region is calculated. If the average Pearson correlation coefficient is lower than 0.3, it is marked as physiological synergy abnormality and superimposed on the risk heat map. If the average Pearson correlation coefficient is higher than 0.3, the risk level is increased and data resampling verification is performed. Plot time-series index curves and line graphs of comprehensive risk scores to show the dynamic changes of respiratory harmonic energy ratio, heart rate variability deviation, blood oxygen fluctuation standard deviation, and extraversion index. Mark event nodes, customize time spans, and analyze the rate of change of comprehensive risk scores based on sliding windows. If the comprehensive risk score rises by 15 points in a single day, it is determined to be a period of emotional deterioration and marked with red shading in the individual trend report. Combined with the physiological correlation between respiratory rate and heart rate, data points that exceed the range during abnormal periods are marked with yellow warning symbols. If the logical thinking score in the personality potential distribution is >75, then the report will indicate high logical thinking potential and recommend logical reasoning training tools. If the leadership score is >70, then the test subject is advised to participate in team collaboration simulation training.

10. The AI-based multimodal recognition and analysis system for dynamic features of physical and mental indicators according to claim 9, characterized in that: The intelligent early warning module's process of implementing the corresponding early warning mechanism includes: If the comprehensive risk score exceeds 75 points and there are cases where the respiratory harmonic energy ratio is >35%, the standard deviation of blood oxygen fluctuation is >3%, and the respiratory-heart rate Pearson correlation coefficient is <0.2, it is judged as a red alert, the dominant risk factor is marked, and the hardware device is automatically locked to lock the data during the abnormal period. If the comprehensive risk score is between 50 and 75 points and the heart rate variability deviation is >60%, it is judged as an orange warning, and the threshold is adjusted based on the test subject's historical data; If the comprehensive risk score is between 30 and 50, the standard deviation of blood oxygen fluctuation is between 1% and 2%, and the correlation coefficient between respiration and heart rate is between 0.2 and 0.4, it is judged as a yellow warning, and the individual trend report of the subject is marked with an observation label. If the overall risk score is below 30 points, and the respiratory harmonic energy ratio is <15% and the standard deviation of blood oxygen fluctuation is <1%, it is judged as green and normal, and the baseline file is updated monthly.

Images