State detection method and device

Abstract

The invention is suitable for the technical field of physiological detection, and provides a state detection method and device, and the method comprises the steps: obtaining a face image of a to-be-detected person; acquiring a physiological signal of the to-be-tested person; determining physiological probability information according to the physiological signal; determining visual probability information according to the face image; carrying out fusion processing on the physiological probability information and the visual probability information to obtain fusion probability information; and determining a state detection result according to the fusion probability information, the state detection result representing a state category to which the to-be-detected person belongs, and the state category to which the to-be-detected person belongs being one of at least one state category. According to the state detection method and device, the state category detection accuracy and detection efficiency can be improved.

Description

Technical Field

[0001] This application belongs to the field of physiological detection technology, specifically relating to a state detection method and device. Background Technology

[0002] With the development of artificial intelligence, computer vision, and physiological sensing technologies, real-time monitoring of individual states is becoming increasingly important in fields such as industrial safety, transportation, medical monitoring, and educational assessment. Especially in high-risk scenarios such as fatigued driving and high-pressure work, timely identification of individual states such as fatigue, stress, or decreased attention is crucial for accident prevention and ensuring personal safety. Currently, existing technologies still have many limitations. On the one hand, state detection based on facial images is highly dependent on lighting conditions, camera angle, and background environment. On the other hand, while physiological signal monitoring methods based on wearable devices can provide relatively stable physiological indicators, this approach cannot directly reflect individual behavioral performance or, for example, changes in facial expressions, lacking intuitive behavioral semantic information and making it difficult to determine what specific behavior triggered the state change. Therefore, there is an urgent need for a state detection method with higher accuracy and detection efficiency. Summary of the Invention

[0003] This application provides a state detection method and apparatus that can improve the accuracy and efficiency of state category detection.

[0004] In a first aspect, embodiments of this application provide a state detection method, comprising: acquiring a facial image of a person to be tested; acquiring physiological signals of the person to be tested; determining physiological probability information based on the physiological signals, the physiological probability information being used to characterize the state category of the person to be tested identified based on physiological parameters: each being the probability of at least one state category, the physiological parameters being determined based on the physiological signals; determining visual probability information based on the facial image, the visual probability information being used to characterize the state category of the person to be tested identified based on facial features: each being the probability of at least one state category, the facial features being determined based on the facial image; fusing the physiological probability information and the visual probability information to obtain fused probability information, the fused probability information characterizing the state category of the person to be tested jointly determined based on the physiological parameters and facial features: each being the probability of at least one state category; and determining a state detection result based on the fused probability information, the state detection result characterizing the state category to which the person to be tested belongs, the state category to which the person to be tested belongs being one of at least one state category.

[0005] In this embodiment, the detection device initially determines the state category of the person being tested based on physiological signals, providing physiological probability information for each state category. This physiological probability information represents the probability information along the physiological dimension. The detection device also initially determines the state category of the person being tested based on facial images, providing visual probability information for each state category. This visual probability information represents the probability information along the visual dimension. By fusing the physiological and visual probability information, the detection device can overcome the limitations of single probability information. The resulting fused probability information is more accurate. Therefore, the state detection result determined based on the fused probability information is also more accurate.

[0006] In one possible implementation of the first aspect, after determining the state detection result based on the fusion probability information, the state detection method further includes: acquiring feature information of the person to be tested within a first preset time period, the first preset time period being determined based on a first moment, the first moment including the acquisition time of the face image and / or the acquisition time of the physiological signal, the feature information including one or more of the following: physiological parameters, face features, fusion probability information; classifying the feature information to obtain a state verification result, the state verification result including the state category to which the person to be tested belongs, the state verification result being used to verify whether the state detection result is accurate; if the state detection result and the state verification result are the same, the state detection result or the state verification result is used as the state category to which the person to be tested belongs; if the state detection result and the state verification result are different, the state verification result is used as the state category to which the person to be tested belongs.

[0007] In this embodiment, compared to acquiring facial images and physiological information based on a single moment, the detection device obtains more feature information over a first preset duration. Therefore, the state verification result obtained based on this feature information is more accurate and stable. Furthermore, the feature information represents the "fingerprint-like" characteristics of the person being tested, making the state category determined based on this feature information more accurate. The state detection result is then verified using the feature information to obtain the state verification result. If the state detection result and the state verification result are the same, it indicates that the state detection result determined by combining facial images and physiological signals is accurate. If they differ, it indicates that the state detection result determined by combining facial images and physiological signals has lower accuracy. In this case, the detection device uses the more accurate state verification result as the state category of the person being tested, resulting in a more accurate state verification result.

[0008] In one possible implementation of the first aspect, obtaining the feature information for the first preset duration includes: when the state detection result indicates that the state category of the person to be tested is an abnormal state category, obtaining the feature information of the person to be tested within the first preset duration.

[0009] In this embodiment, the detection device obtains the characteristic information of the person under test within a first preset time period when the state detection result indicates that the state category of the person under test belongs to the abnormal state category, and verifies it based on the characteristic information, so as to accurately detect whether the physiological state of the person under test belongs to the abnormal state category.

[0010] In one possible implementation of the first aspect, fusing physiological probability information and visual probability information to obtain fused probability information includes: weighting physiological probability information and visual probability information according to a first weight corresponding to physiological probability information and a second weight corresponding to visual probability information to obtain fused probability information, wherein the first weight is less than or equal to the second weight.

[0011] In this embodiment, the physiological probability information is determined primarily based on facial images, and a secondary probability is determined based on physiological signals. Combining both methods to determine the state category of the person being tested is more accurate and has better real-time performance and stability.

[0012] In one possible implementation of the first aspect, the fusion probability information includes X fusion probability values, where X is an integer greater than or equal to 3. The state detection method further includes: performing temporal smoothing on the X fusion probability values ​​to obtain smoothed fusion probability information; wherein, determining the state detection result based on the fusion probability information includes: if the smoothed probability fusion information exceeds a probability threshold, determining the state category corresponding to the smoothed fusion probability information as the state detection result.

[0013] In this embodiment, the detection device obtains fused probability information after fusing physiological and visual probability information. Furthermore, it performs temporal smoothing on the fused probability information, thereby removing abrupt noise. The state detection result determined based on the smoothed fused probability information is more accurate.

[0014] In one possible implementation of the first aspect, temporal smoothing of the X fusion probability values ​​includes: performing Kalman filtering on the X fusion probability values ​​when they are linearly correlated; and / or performing unscented Kalman filtering on the X fusion probability values ​​when they are non-linearly correlated.

[0015] In this embodiment, the detection device performs different temporal smoothing processes on X fusion probability values ​​with different correlations based on their correlation, in order to accurately remove burst noise in the fusion probability information and improve the accuracy of the state detection results. Using Kalman filtering or unscented Kalman filtering for temporal smoothing of multiple fusion probability values ​​involves less computation and offers higher real-time performance and accuracy. Kalman filtering or unscented Kalman filtering also places lower resource requirements on the detection device, thus providing superior performance for temporal smoothing and making it suitable for scenarios with a large number of people to be tested or high real-time requirements.

[0016] In one possible implementation of the first aspect, the state detection method further includes: when the duration of the test subject's state category being an abnormal state category reaches a second preset duration, outputting alarm information, the alarm information indicating that the test subject's state category is an abnormal state category. This can improve alarm accuracy and reduce false alarms.

[0017] In one possible implementation of the first aspect, the physiological signal includes a heart rate signal, and the physiological parameters include a heart rate variability index; determining physiological probability information based on the physiological signal includes: performing a first preprocessing on the heart rate signal to determine the heart rate variability index, the first preprocessing including a bandpass filtering operation; determining the physiological probability information based on the degree of difference between the heart rate variability index and the reference physiological information of the subject, the reference physiological information of the subject including: the subject's historical heart rate variability index and / or the state category corresponding to the subject's historical heart rate variability index.

[0018] In this embodiment, the detection device improves state detection efficiency by determining a heart rate variability index that is more correlated with state category detection based on the heart rate signal. The reference physiological information of the person being tested serves as reference data representing their physiological state. Based on the heart rate variability index and the reference physiological information, physiological probability information can be accurately determined.

[0019] In one possible implementation of the first aspect, at least one state category includes N state categories, where N is an integer greater than or equal to 2; determining physiological probability information based on the degree of difference between heart rate variability indicators and the reference physiological information of the subject includes: extracting N frequency domain components of the heart rate signal, with each of the N frequency domain components corresponding to one of the N state categories; determining the N heart rate variability indicators corresponding to the N frequency domain components; and performing a probability mapping between the degree of difference between the N heart rate variability indicators and the reference physiological information of the subject to obtain the physiological probability information.

[0020] In this embodiment, by mining the correlation between each state category and the frequency domain components of the heart rate signal, and matching each state category with the frequency domain components of the heart rate signal one by one, it is possible to accurately distinguish each state category and facilitate fine-grained state detection. Separating the frequency domain components corresponding to each state category from the heart rate signal first involves differentiating the relevant heart rate signals for each state category. Based on the known frequency domain components corresponding to specific state categories, probability mapping is performed according to the degree of difference between the heart rate variability index of the frequency domain components and reference physiological information, thereby accurately distinguishing each state category and accurately determining visual probability information.

[0021] In one possible implementation of the first aspect, before determining the physiological probability information based on the degree of difference between the heart rate variability index and the reference physiological information of the test subject, the state detection method further includes: acquiring a sample set of the test subject, the sample set including multiple historical heart rate variability index samples, each historical heart rate variability index sample having a state category label; performing Gaussian mixture fitting on the sample set according to the state category label to obtain a Gaussian distribution corresponding to each state category; and determining the reference physiological information state category of the test subject according to the Gaussian distribution corresponding to each state category.

[0022] In this embodiment, each subject establishes their own reference physiological information to achieve "person-specific" reference physiological information and avoid state detection errors caused by natural physiological differences between different subjects.

[0023] In one possible implementation of the first aspect, determining visual probability information based on a face image includes: performing a second preprocessing on the face image to extract facial features, which include facial depth features and / or action unit features; wherein the action unit features include facial expression features, and the facial depth features include one or more of the following: facial texture features, facial muscle features, facial skin features, facial angle features, and facial motion features; the second preprocessing includes a face alignment operation; processing the facial features using an image processing model to obtain N face category features corresponding to N state categories, wherein the image processing model includes long short-term memory networks and / or attention mechanism network state categories; and determining visual probability information based on the N face category features.

[0024] In this embodiment, the facial features extracted through the second preprocessing are more representative for determining the state category of the person being tested. Facial depth features are a high-dimensional representation of the facial features of the person being tested. These facial depth features can accurately distinguish which of the at least one state category the person being tested belongs to. Action unit features represent the emotions of the person being tested, and these action units can accurately distinguish the emotions of the person being tested. Subsequently, by using both facial depth features and action unit features, the emotions of the person being tested can be avoided from being confused with the state category, thereby accurately determining the visual probability information.

[0025] In one possible implementation of the first aspect, the state detection method further includes: Update the reference physiological information of the person being tested based on the physiological parameters corresponding to the state detection results and / or the physiological parameters corresponding to the state verification results; and / or update the image processing model based on the face image corresponding to the state detection results and / or the face image corresponding to the state verification results.

[0026] In this embodiment, the detection device first updates the sample set based on the physiological parameters corresponding to the state detection results and / or the physiological parameters corresponding to the state verification results, and then updates the reference physiological information. This makes the reference physiological information highly real-time and accurate. The physiological probability information determined by the detection device based on the physiological parameters and the reference physiological information of the person being tested is more accurate.

[0027] In one possible implementation of the first aspect, at least one state category includes at least one normal state category and / or at least one abnormal state category, wherein the abnormal state category includes a fatigue state category and a stress state category.

[0028] In one possible implementation of the first aspect, the fused probability information and / or smoothed fused probability information includes a normal probability value, a fatigue probability value, and a stress probability value. The normal probability value represents the probability that the test subject belongs to the normal state category; the fatigue probability value represents the probability that the test subject belongs to the fatigue state category; and the stress probability value represents the probability that the test subject belongs to the stress state category. Based on the fused probability information, determining the state detection result includes: determining the test subject's state category as the normal state category when the normal probability value exceeds a probability threshold; determining the test subject's state category as the fatigue state category when the fatigue probability value exceeds a probability threshold; and determining the test subject's state category as the stress state category when the stress probability value exceeds a probability threshold.

[0029] In this embodiment, the fused probability information includes the normal probability value, fatigue probability value, and stress probability value corresponding to the normal state category, fatigue state category, and stress state category, respectively. By comparing each value with the probability threshold, the state category to which the person being tested belongs can be determined.

[0030] In one possible implementation of the first aspect, the state detection method is applied to a state detection system, which includes an image acquisition device and a physiological signal acquisition device; the image acquisition device is used to acquire facial images of the person to be tested; and the physiological signal acquisition device is used to acquire physiological signals of the person to be tested.

[0031] Secondly, embodiments of this application provide a state detection device, comprising: a first acquisition module for acquiring a facial image of a person to be tested; a second acquisition module for acquiring physiological signals of the person to be tested; a processing module for determining physiological probability information based on the physiological signals, wherein the physiological probability information characterizes the state category of the person to be tested identified based on physiological parameters: each being the probability of at least one state category, and the physiological parameters are determined based on the physiological signals; the processing module is further configured to determine visual probability information based on the facial image, wherein the visual probability information characterizes the state category of the person to be tested identified based on facial features: each being the probability of at least one state category, and the facial features are determined based on the facial image; the processing module is further configured to fuse the physiological probability information and the visual probability information to obtain fused probability information, wherein the fused probability information characterizes the state category of the person to be tested determined jointly by the physiological parameters and the facial features: each being the probability of at least one state category; and the processing module is further configured to determine a state detection result based on the fused probability information, wherein the state detection result characterizes the state category to which the person to be tested belongs, and the state category to which the person to be tested belongs is one of at least one state category.

[0032] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the state detection method as described in any of the first aspects above.

[0033] Fourthly, embodiments of this application provide a state detection system, including: an image acquisition device, a physiological signal acquisition device, and an electronic device. The electronic device acquires a facial image of a person to be tested through the image acquisition device, and acquires physiological signals of the person to be tested through the physiological signal acquisition device. The electronic device is used to perform the state detection method as described in any of the first aspects above.

[0034] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the state detection method as described in any of the first aspects above.

[0035] In a sixth aspect, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute any of the state detection methods described in the first aspect above.

[0036] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0037] The beneficial effects of the embodiments of this application compared with the prior art include at least one of the following: 1) The detection device determines physiological probability information based on facial images as a visual detection method, and determines visual probability information based on physiological signals of different modalities than facial images as a physiological detection method. By fusing physiological and visual probability information, the limitations of relying solely on visual or physiological detection can be avoided, improving the accuracy and efficiency of state detection. In this embodiment, through visual facial images and physiological signals, the facial images can directly reflect the behavior of the person being tested, such as changes in facial expressions, possessing intuitive behavioral semantic information. This behavioral semantic information indicates which behavior of the person being tested triggered the state change, thereby improving the accuracy and stability of state detection.

[0038] 2) The detection equipment constructs individual reference physiological information for each individual based on their physiological differences, avoiding the use of the same reference physiological information to compare different individuals. This allows for accurate detection of the state category of each individual.

[0039] 3) The detection equipment can not only detect whether the state category of the person being tested is normal or abnormal, but also mine the facial features and physiological signal characteristics of different abnormal state categories. Through high-dimensional features such as facial features and physiological parameters, the state of the person being tested can be accurately and personally evaluated, so as to achieve accurate differentiation of each state category and fine-grained state detection. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of a state detection system provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a state detection method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a state detection method provided in another embodiment of this application; Figure 4 This is a schematic diagram of the sample distribution corresponding to a Gaussian mixture model provided in an embodiment of this application; Figure 5 This is a schematic diagram of a state detection method provided in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of a state detection device according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0042] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0043] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0044] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0046] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0048] The following will describe in detail the relevant terminology and background of the state detection method and apparatus in the embodiments of this application.

[0049] State detection: The process of collecting data from subjects for state category detection and analyzing this data to determine the subject's state category. The subject's data includes physiological signals and image data. Data collection includes acquiring physiological signals through contact methods and acquiring image data through non-contact methods. State categories include normal state categories and abnormal state categories. Abnormal state categories include fatigue state categories, stress state categories, etc.

[0050] Contact-based data acquisition: This method collects physiological signals using wearable physiological signal acquisition devices. These devices include, for example, heart rate sensors, which can collect the body's heart rate signals. Heart rate sensors can be installed inside wristbands or helmets.

[0051] Non-contact data acquisition method: Image data of the person being tested is acquired through remote acquisition devices. These devices include, for example, image acquisition equipment such as cameras. The image data of the person being tested includes their facial image.

[0052] Physiological signals: Raw data acquired by physiological signal acquisition equipment. Physiological signals include photoplethysmography (PPG) signals and blood volume pulse (BVP) signals. Physiological parameters can be determined through physiological signals.

[0053] PPG signal: When green or infrared light irradiates the skin, the amount of light absorbed by the blood changes periodically with the heartbeat. Based on the blood's absorption and reflection characteristics of light, it is a signal that reflects changes in blood vessel volume.

[0054] BVP signal: a signal that reflects the periodic changes in blood volume within blood vessels as the heart beats.

[0055] Physiological parameters are quantitative indicators that reflect the physiological state, function, and metabolic activity of a subject. The values ​​of physiological parameters can be used to assess the subject's health status, emotional state, etc. Physiological parameters include one or more of the following: heart rate (HR), heart rate variability (HRV), peripheral oxygen saturation (SpO2), respiratory rate (RR), and blood pressure (BP).

[0056] Gaussian distribution: also known as normal distribution, Gaussian distribution is a probability distribution of continuous random variables. Its probability density function is shaped like a bell curve, symmetrical and unimodal.

[0057] Gaussian mixture model (GMM): A probabilistic model consisting of a linear combination of multiple Gaussian distributions (each of which is called a Gaussian component), used to describe complex multimodal distribution data.

[0058] Facial image: An image showing the face of the person being tested. Facial features can be determined from the facial image.

[0059] Region of Interest (ROI): A specific region in an image, either manually or automatically defined. This region is used for feature extraction or targeted analysis and processing.

[0060] Facial features: The unique shape and structure of a face as revealed by a facial image. Facial features include macroscopic facial features visible to the naked eye, as well as microscopic facial features that require technical analysis.

[0061] Action units (AUs) are the basic units in a facial action coding system (FACS) that describe facial muscle movements. The intensity vector of an action unit is used to quantify the activation level of each action unit. Action units are used for facial behavior analysis in computer vision and emotion computing.

[0062] Feature concatenation: The process of merging feature vectors from different sources or types into a single feature vector.

[0063] Normal state category: The physiological state of the subject when at rest, without external interference, and in a state of mental and physical balance. Under the normal state category, the physiological signals of the subject show that various physiological indicators are within a stable range; the facial image of the subject shows relaxed facial muscles, uniform skin color, stable eye movements, normal blinking frequency, and natural facial expression, without facial features such as continuous tension or stiffness.

[0064] Fatigue State Category: A physiological state characterized by a decline in the physiological functions of the test subjects. Under the fatigue state category, the physiological signals of the test subjects show features such as a low heart rate and autonomic nervous system dysfunction; the facial images of the test subjects show facial features such as muscle relaxation and sagging, dark skin tone, constricted pupils, and slow micro-expression changes.

[0065] Stress State Category: Physiological states in which the subject exhibits non-specific responses. Under the stress state category, the subject's physiological signals show sympathetic hyperactivity, such as higher heart rate and higher blood pressure; the subject's facial images show facial features such as muscle tension, pale skin, enlarged pupils, increased blinking frequency, raised eyebrows corresponding to micro-expressions, and flared nostrils.

[0066] Kalman filter (KF): A recursive filtering algorithm used to estimate the state of a system in noisy measurement environments.

[0067] Nonlinear augmentation: Algorithms used to address state estimation problems in nonlinear systems. Nonlinear augmentation includes unscented Kalman filtering (UKF).

[0068] Unscented Kalman filter: a recursive algorithm for estimating the state of a nonlinear system.

[0069] Status detection can be applied in fields such as healthcare, health management, motion detection, and security. Taking security as an example, in industries such as nuclear power, chemical engineering, medical care, and construction, status detection of workers can promptly identify abnormal states and ensure worker safety through alarms and other means.

[0070] Taking nuclear power plants as an example, through the analysis of numerous human-caused events on-site, the factors inducing these events can be divided into two parts: psychological factors and physiological factors. Psychological factors are categorized into seven states: habitual psychology, tension psychology, boredom psychology, desire for speed, arrogance psychology, wishful thinking psychology, and rebellious and pessimistic psychology. Physiological factors include states such as fatigue, inattention, low work motivation, memory confusion, influence of biological rhythms, lack of technical proficiency, poor reasoning and judgment ability, and insufficient knowledge.

[0071] To prevent human-caused incidents, it is necessary to conduct job suitability tests on employees. The status detection method in this application can be applied to job suitability testing, specifically to detect whether an employee's status is abnormal. In some embodiments, job suitability testing is conducted through periodic occupational health examinations and questionnaires. However, this method has the following drawbacks: The timing of periodic occupational health examinations and questionnaire completion for staff is not synchronized with their actual working hours. Therefore, status monitoring through these methods is lagging, leading to lower accuracy. Secondly, the status indicators obtained from occupational health examinations or questionnaires are not accurate for job suitability assessment; for example, they cannot accurately detect and differentiate between the aforementioned physiological and psychological states. Furthermore, in some scenarios, nuclear power plants have large numbers of personnel with frequent movement, posing a threat to plant safety. Therefore, real-time, rapid, and accurate status monitoring is necessary.

[0072] Currently, there is no complete system or device for monitoring the status of workers that can collect and analyze multimodal data such as physiological, psychological, and behavioral states, while also being accurate and scientifically sound. Furthermore, there is a lack of comprehensive status monitoring methods and evaluation standards for personnel status.

[0073] In this embodiment, intelligent detection and early warning of the suitability status of nuclear power plant personnel based on high-dimensional feature occupational profiles are implemented. Building upon this, some embodiments also incorporate technologies for pre-job suitability diagnosis and assessment, rapid screening and identification of pre-job suitability, and real-time detection and early warning of on-the-job suitability based on high-dimensional feature occupational profiles of nuclear power plant personnel. This achieves a comprehensive evaluation of suitability status throughout the entire operational process—pre-job, pre-job, and on-the-job—and completes the research and development and integration of an enterprise-level nuclear power plant key personnel suitability status monitoring and early warning system.

[0074] Specifically, taking intelligent state detection and early warning as an example, state detection has the following drawbacks: Contact-based acquisition requires the subject to wear a physiological signal acquisition device. Improper wearing or prolonged wear causing displacement of the device can reduce the quality of the acquired physiological signals. Furthermore, when the subject's actions or facial expressions drive their state category, the physiological parameter values ​​obtained from the physiological signals cannot accurately determine the subject's state category. For example, a decreased blinking frequency indicates fatigue, but the heart rate determined from the physiological signals does not show a decreasing trend; therefore, relying on physiological parameters like heart rate cannot accurately detect fatigue. Moreover, state detection based on physiological signals, relying on physiological parameter thresholds to trigger abnormal state alarms, may miss abnormal states before the physiological parameters reach the threshold. All of these factors affect the efficiency and accuracy of state detection based on physiological signals.

[0075] In non-contact data acquisition methods, state detection based on the acquired facial images of the subject requires high image quality. For example, lighting, occlusion, and background complexity in the facial image can all affect the accuracy of state detection. Furthermore, if facial features are missing, the subject's state category cannot be detected. Moreover, the subject's facial expressions can be influenced by state category or emotion; if the subject's facial expressions are affected by emotion, the state category cannot be accurately determined based on facial expressions alone. Additionally, while physiological signal monitoring methods based on wearable devices can provide relatively stable physiological indicators, this approach cannot directly reflect the subject's behavioral performance or, for example, changes in facial expressions. It lacks intuitive behavioral semantic information, making it difficult to determine which specific behavior triggered the state change.

[0076] Based on this, this application provides a state category detection method. The detection device combines physiological signals and visual facial images to perform state detection. The facial images can directly reflect the behavior of the person being tested, such as changes in facial expressions, and have intuitive behavioral semantic information. This behavioral semantic information indicates which behavior of the person being tested triggered the state change, thereby improving the accuracy and stability of state detection.

[0077] Furthermore, in this embodiment of the application, the detection device constructs its own reference data for each individual subject based on the physiological differences of each individual subject, avoiding the use of the same reference data to target different individuals subject, thereby accurately detecting the state category of each individual subject.

[0078] Furthermore, in this embodiment of the application, the detection device can not only detect whether the state category of the person being tested is normal or abnormal, but also accurately distinguish different abnormal state categories, thereby achieving fine-grained state detection.

[0079] The system architecture of the state detection method and apparatus applied in the embodiments of this application will be described in detail below, which can also be understood as a state detection system. (Reference) Figure 1 This is a schematic diagram of a state detection system according to an embodiment of this application. Figure 1 As shown, the state detection system includes: an image acquisition device 101, a physiological signal acquisition device 102, and a detection device 103.

[0080] Image acquisition device 101 is used to acquire images of the person being tested. For example, image acquisition device 101 may include a camera or other device or camera module with image acquisition capabilities. Image acquisition device 101 acquires facial images of the person being tested, for example, at 15 frames per second (fps), providing continuous facial image input for subsequent state detection. Physiological signal acquisition device 102 is used to acquire physiological signals of the person being tested. For example, physiological signal acquisition device 102 may include a heart rate sensor. The detection device 103 is communicatively connected to the image acquisition device 101 and the physiological signal acquisition device 102. The detection device 103 receives the facial image of the subject acquired by the image acquisition device 101 and the physiological signals acquired by the physiological signal acquisition device 102. The detection device 103 also performs state detection based on the facial image and physiological signals of the subject. The detection device 101 can also control the physiological signal acquisition device 102 to synchronize with the facial image acquisition device 101 via a clock circuit or other means. Any one of the image acquisition device 101, the physiological signal acquisition device 102, and the detection device 103 can perform the functions of one or two of the others.

[0081] It should be noted that, Figure 1 This is merely an example of a state detection system and is not intended to limit the scope of the application. In this embodiment, the collection, storage, use, processing, transmission, provision, and disclosure of personal information of the person being tested, such as physiological signals and facial images, all comply with relevant laws and regulations and do not violate public order and good morals. In this embodiment, the authorization or consent of the person being tested is obtained before acquiring or collecting their personal information.

[0082] The following will combine Figure 1 The system architecture shown illustrates the state detection method of this application embodiment. This state detection method can be executed by a detection device. (See reference...) Figure 2This is a flowchart of a state detection method according to an embodiment of this application. Figure 2 As shown, the state detection method includes steps S21 to S26.

[0083] S21. Obtain the facial image of the person to be tested.

[0084] S22. Obtain the physiological signals of the subject. Combined with... Figure 1 The system architecture is as follows: facial images are acquired by an image acquisition device, which then sends the acquired facial images to a detection device. Physiological signals are acquired by a physiological signal acquisition device, which then sends the acquired physiological signals to the detection device.

[0085] S23. Determine physiological probability information based on physiological signals. Physiological probability information is used to characterize the state category of the test subject identified based on physiological parameters: it represents the probability of each state category in at least one state category. The physiological parameters are determined based on physiological signals.

[0086] In one possible embodiment, the detection device performs preprocessing, physiological parameter calculation, and probability calculation on the physiological signal to obtain physiological probability information. The preprocessing operation reduces noise in the physiological signal to improve its quality. The physiological parameter calculation operation uses relevant algorithms to calculate physiological parameters representing the physiological state of the subject. The probability calculation operation, for example, compares the physiological parameters with those corresponding to each state category and determines the probability of the subject's state category being any of those categories based on the comparison results.

[0087] In step S23, the detection device performs preliminary state detection based on physiological signals to preliminarily determine the probability of each state category of the person being tested.

[0088] S24. Determine visual probability information based on the facial image. Visual probability information is used to characterize the state category of the person being tested, identified based on facial features: it represents the probability of each state category in at least one category. The facial features are determined based on the facial image.

[0089] In one possible embodiment, the detection device inputs a face image into a first neural network, such as a Convolutional Neural Network (CNN) or a Long Short-Term Memory Network (LSTM), which outputs visual probability information. This first neural network is pre-trained using face image samples with state category labels.

[0090] In step S24, the detection device performs preliminary state detection based on the face image to preliminarily determine the probability of each state category of the person being tested.

[0091] S25. The physiological probability information and visual probability information are fused to obtain fused probability information. The fused probability information represents the state category of the test subject determined jointly by physiological parameters and facial features: it is the probability of each state category in at least one state category.

[0092] By fusing physiological and visual probability information, the detection device can overcome the limitations of either single physiological or visual probability information, resulting in more accurate fused probability information. For example, the detection device obtains fused probability information by weighted fusion of physiological and visual probability information.

[0093] S26. Determine the state detection result based on the fusion probability information. The state detection result represents the state category to which the person being tested belongs, and the state category to which the person being tested belongs is one of at least one state category.

[0094] In this embodiment, the detection device initially determines the state category of the person being tested based on physiological signals, providing physiological probability information for each state category. This physiological probability information represents the probability information of the physiological dimension. The detection device also initially determines the state category of the person being tested based on facial images, providing visual probability information for each state category. This visual probability information represents the probability information of the visual dimension. By fusing the physiological and visual probability information, the detection device can overcome the limitations of single probability information. The resulting fused probability information is more accurate. The state detection result determined based on the fused probability information is also more accurate. In this embodiment, state detection is performed by combining facial images and physiological signals. The facial images can directly reflect the behavior of the person being tested, such as changes in facial expressions, and possess intuitive behavioral semantic information. This behavioral semantic information indicates which behavior of the person being tested triggered the state change, thereby improving the accuracy and stability of state detection.

[0095] The detection equipment, by fusing physiological and visual probability information, can overcome the limitations of single probability information in at least one of the following aspects: Improper wearing or prolonged wear of the physiological signal acquisition device can cause displacement, reducing the quality of the physiological signals acquired and thus decreasing the accuracy of physiological probability information. Alternatively, changes in the subject's state category due to actions or facial expressions can also lower the accuracy of the initial probability value obtained from the physiological signals. Furthermore, relying on physiological parameter thresholds for state detection may miss abnormal physiological states before the threshold is reached, further reducing the accuracy of the physiological probability information determined from the physiological signals. Therefore, the accuracy of state detection results is low when determined solely by physiological probability information. However, fusing physiological probability values ​​with visual probability information yields more accurate fused probability information compared to purely physiological probability information. State detection results determined based on this fused probability information are also more accurate.

[0096] State detection based on collected facial images of the subject requires high image quality. Low-quality facial images result in low accuracy of the visual probability information determined from them. Alternatively, missing facial expression features may prevent the detection of the subject's state category, leading to either inaccurate or incomplete visual probability information. Furthermore, facial expressions can be influenced by state category or emotion; emotionally driven expressions can also hinder accurate state category determination, further reducing the accuracy of visual probability information. Therefore, relying solely on visual probability information results in low accuracy of state verification. By fusing physiological and visual probability values, the resulting fused probability information is more accurate than purely visual probability information. State detection results determined based on this fused probability information are also more accurate.

[0097] In one possible embodiment, at least one state category includes at least one normal state category and / or at least one abnormal state category, whereby the abnormal state category includes a fatigue state category and a stress state category. This allows the state detection in this application embodiment to be not limited to distinguishing whether the state category to which the test subject belongs is a normal state category or an abnormal state category, but can also distinguish multiple normal state categories and / or multiple abnormal state categories with finer granularity. It should be noted that, in addition to the fatigue state category and the stress state category, the abnormal state category may also include the aforementioned habitual psychological state category, tense psychological state category, boredom psychological state category, speed-seeking psychological state category, arrogance psychological state category, wishful thinking psychological state category, rebellious and pessimistic psychological state category, distracted state category, low work motivation state category, memory confusion state category, biorhythm-affected state category, technically unskilled state category, poor reasoning and judgment ability state category, and knowledge-deficient state category, etc.

[0098] In one possible embodiment, any one or more of the physiological probability information, visual probability information, and fusion probability information can be in the form of a vector, which includes probability values ​​equal to the number of state categories. Taking N (N is an integer greater than or equal to 1) state categories as an example, the physiological probability information includes N physiological probability values ​​corresponding to the N state categories, where the i-th physiological probability value represents the probability that the test subject's state category is the i-th state category among the N state categories. Similarly, the visual probability information includes N visual probability values, where the i-th visual probability value represents the probability that the test subject's state category is the i-th state category among the N state categories. The fusion probability information includes N third probability values, where the i-th third probability value represents the probability that the test subject's state category is the i-th state category among the N state categories.

[0099] The following will describe in detail a specific example of state detection based on face images to determine visual probability information according to an embodiment of this application. (Reference) Figure 3 This is a schematic diagram of a state detection method according to an embodiment of this application. Figure 3 As shown, step S24 above includes steps S241 to S243.

[0100] S241. Perform a second preprocessing on the face image to extract facial features. Facial features include facial depth features and / or action unit features.

[0101] In one possible embodiment, the second preprocessing includes one or more of the following: operations to determine the region of interest, and face alignment operations.

[0102] For the operation of determining the region of interest (ROI), the detection device can use a facial geometry processing model to detect multiple key facial feature points displayed in a face image and construct a facial mesh based on these key feature points. The detection device can also use triangulation algorithms to divide the facial network into ROIs to obtain the region of interest. Facial geometry processing models are used to detect and track key facial feature points, including active shape models (ASM) and mediapipe face meshes. Triangulation algorithms include Delaunay triangulation and greedy triangulation. By determining the ROI, the detection device can subsequently extract facial depth features and / or action unit features only from the ROI, thereby reducing the amount of data required for extracting facial depth features and / or action unit features.

[0103] For face alignment operations, the detection device can use a facial geometry processing model to detect multiple key facial feature points displayed in a face image. Based on these key feature points, it performs affine transformations or affine cropping to ensure that the face of the person being tested, regardless of its pose or position, maintains consistent dimensions within the face image. These dimensions include, for example, the angle and size of the person's face displayed in the image. This reduces interference caused by facial rotation, translation, and scaling, ensuring that the face image displays the person from a "standard perspective," thus facilitating improved accuracy in subsequent facial feature extraction.

[0104] In one possible embodiment, in step S241 above, the detection device uses a second neural network to determine facial depth features. And / or, the detection device detects the region where each action unit in the face image is located based on a Facial Action Coding System (FACS), and determines the action unit features based on the optical flow difference between the regions where the action units are located in two adjacent face images. The second neural network includes one or more of the following: a convolutional neural network (CNN), a residual neural network (ResNet), etc.

[0105] In one possible embodiment, the detection device concatenates facial depth features and action unit features to obtain facial features. Feature concatenation includes methods such as concatenating facial depth features and action unit features based on their respective weights, or superimposing both features. Facial features combine facial depth features and action unit features, facilitating subsequent determination of visual probability information.

[0106] In one possible embodiment, the facial depth features include at least one of the following: facial texture features, facial muscle features, facial skin features, facial angle features, and facial behavior features. Action unit features can be characterized as intensity vectors of action units, which are used to characterize the intensity of the micro-expressions of the subject. Action unit features include at least facial expression features.

[0107] Facial texture features: a quantitative representation of the microstructure of the skin surface, reflecting the state category of the subject. For example, when the subject is in a fatigued state, facial texture is coarser. When the subject is in a stressful state, facial texture is disordered.

[0108] Facial muscle characteristics: The dynamic characteristics of muscle contraction and movement, reflecting the state category of the test subject. For example, when the test subject is in a fatigue state, the orbicularis oculi muscle is weaker. When the test subject is in a stress state, the corrugator supercilii and / or platysma muscles are stronger.

[0109] Facial skin characteristics: Dynamic changes in the physical properties of the skin reflect the state of autonomic nervous system regulation, and thus reflect the subject's state category. For example, when a subject is in a fatigued state, the secretion of microperspiration on the skin surface will change, and the skin's luster will decrease.

[0110] Facial angular features: Kinematic parameters of three-dimensional facial posture, reflecting the state category of the subject. For example, when the subject is in a fatigued state, the head will droop involuntarily, causing changes in the facial pitch angle.

[0111] Facial behavioral characteristics: Dynamic information of facial three-dimensional posture, reflecting the state category of the subject. For example, when the subject is in a fatigued state, the blinking frequency decreases.

[0112] Facial expression features: transient muscle movement patterns related to emotions reflect the emotions of the test subject, and thus reflect the test subject's state category. For example, when the test subject is in a stress state category, the eyebrows will be lowered and the mouth will tend to turn downwards.

[0113] In this embodiment, facial depth features are a high-dimensional representation of the facial features of the test subject, including facial texture features, facial muscle features, facial skin features, and facial angle features. These facial depth features can accurately distinguish which of at least one state category the test subject belongs to. Action unit features represent the test subject's emotions, and these features can accurately distinguish the test subject's emotions. Subsequently, using both facial depth features and action unit features can prevent confusion between the test subject's emotions and state categories, thereby accurately determining visual probability information.

[0114] S242. Using an image processing model, process the facial features to obtain N facial category features corresponding to N state categories. In one possible embodiment, the image processing model includes: a Long Short-Term Memory (LSTM) network and / or an attention mechanism network.

[0115] The state categories of the test subjects exhibit a pattern of change over time. For example, the state category of the test subjects changes from a normal state category to a mild fatigue state category over time, and then further to a severe fatigue state category. Alternatively, the state category of the test subjects changes from a normal state category to a stress state category over time.

[0116] In one possible embodiment, based on the characteristic that the state category of the person being tested changes regularly over time, in step S242, the detection device can process the facial features using a time-series-focused neural network such as LSTM to obtain encoded features. These encoded features can capture and learn the time dependence of the changes in the state category of the person being tested. In another possible embodiment, to accurately detect each state category, the detection device can set up N processing branches corresponding to N state categories to process the facial features and obtain N facial category features corresponding to the N state categories. Figure 3 In the example, taking the detection of fatigue and stress states as examples, the fatigue processing branch is used, for instance, to process facial features and obtain fatigue-related facial features corresponding to the fatigue state category. Similarly, the stress processing branch is used to process facial features and obtain stress-related facial features corresponding to the stress state category. These processing branches allow for more granular state detection of facial features.

[0117] Furthermore, facial features combine facial depth features and action unit features. Some state categories possess their own unique facial characteristics. For example, features more relevant to the fatigue state category include one or more of the following: eyelid opening / closing corresponding to facial expression features, blinking frequency corresponding to facial behavior features, and head drooping angle corresponding to facial angle features. Features relevant to the stress state category include one or more of the following: facial muscle tension corresponding to facial muscle features, and microvascular congestion in the cheekbone area corresponding to facial skin features.

[0118] Based on this, the processing branch can be an attention mechanism network. This allows the processing branch to focus on features corresponding to the corresponding state category, achieving accurate differentiation of each state category and fine-grained state detection. The aforementioned encoded features are used as inputs to each processing branch to obtain N face category features corresponding to the N state categories. In this embodiment, the image processing model includes an LSTM and an attention network corresponding to each processing branch. S243. Determine visual probability information based on the N face category features.

[0119] like Figure 3 As shown, in one possible embodiment, the detection device can integrate the fatigue face category features and the stress face category features using a fully connected layer to obtain integrated features. Furthermore, the integrated features are processed by a probability mapping layer to obtain visual probability information. The probability mapping layer includes one or more of the following: a softmax function and a sigmoid function. The following formula (1) characterizes the visual probability information. .

[0120] (1) in, Visual probability values ​​representing fatigue state categories. . Visual probability values ​​representing stress state categories. .

[0121] The following will describe in detail a specific example of state detection based on physiological signals and the determination of physiological probability information according to an embodiment of this application. In step S23 above, the physiological signal includes a heart rate signal, and the physiological parameters include a heart rate variability index. Determining physiological probability information based on the physiological signal includes: performing a first preprocessing on the heart rate signal to determine the heart rate variability index; and determining the physiological probability information based on the degree of difference between the heart rate variability index and the reference physiological information of the person being tested.

[0122] In one possible embodiment, the first preprocessing includes at least a bandpass filter (BP) operation. For example, the detection device uses a bandpass filter to pass the heart rate signal in the 0.5 Hz to 5 Hz frequency band to detect the heart rate variability index of the subject, which covers the main energy range of the heart rate. The bandpass filter operation can reduce at least one or more of the following interferences: power frequency interference, baseline drift interference, and motion artifact interference caused by changes in the position of the subject or the status category acquisition device. Among them, power frequency interference refers to electromagnetic radiation interference generated by the heart rate signal acquisition device. Baseline drift interference refers to low-frequency slow fluctuation interference of the heart rate signal, which manifests as drift or fluctuation of the signal baseline (zero line). The bandpass filter operation improves the quality of the heart rate signal. The following formula (2) characterizes the bandpass filter operation.

[0123] (2) in, Characterizes the heart rate signal after bandpass filtering (retaining the heart rate signal in the 0.5Hz to 5Hz frequency band and removing noise in other frequency bands). The input raw heart rate signal is characterized by the raw heart rate signal acquired by the heart rate signal acquisition device. Characterizes bandpass filtering operation, specifically indicating the passage of heart rate signals in the 0.5Hz to 5Hz frequency band.

[0124] When a triaxial accelerometer is integrated into a physiological signal acquisition device, the physiological signal acquisition device can detect the motion state of the subject by measuring the acceleration change of the subject through triaxial acceleration measurement, assist in identifying motion artifacts in the heart rate signal, and improve the accuracy of the heart rate signal. However, the heart rate signal acquired after the heart rate signal acquisition device integrates a triaxial accelerometer will introduce motion noise. The detection device can use adaptive noise cancellation (ANC) technology to suppress motion noise. Adaptive noise cancellation is a signal processing technology that suppresses noise by dynamically adjusting filter parameters (filter parameters include, for example, the coefficients of the above-mentioned bandpass filter). Adaptive noise cancellation technology uses the reference signal of noise to separate the target signal from the mixed signal. The following formula (3) is used to calculate the error signal. The following formula (4) is used to update the filter coefficient vector. .

[0125] (3) In formula (3), The error signal characterizes the deviation between the current filtering effect and the ideal situation. Characterizing the raw heart rate signal containing noise, i.e. The constantly received mixed signal containing heart rate signals and noise. Characterization Filter coefficients at time 1 transpose, Characterize the reference signal, Characterizing filter coefficients With reference signal The weighted inner product.

[0126] (4) In formula (4), Characterization The filter coefficients are updated at each time step. Characterization The filter coefficients at time 1. The step size is a parameter that controls the rate at which the filter coefficients are updated.

[0127] The detection device extracts the adjacent heartbeat interval sequence based on the heart rate signal after the first preprocessing. Taking this physiological parameter as an example, M represents the number of intervals (RR) between heartbeats. The detection device performs peak detection on the preprocessed heart rate signal to obtain the sequence of adjacent heartbeat intervals. Heart rate variability indices can be determined based on the sequence of adjacent heartbeat intervals. These indices include one or more of the following: overall variability and short-term variability. Overall variability is characterized by the standard deviation of normal-to-normal intervals (SDNN), while short-term variability is characterized by the root mean square of successive differences (RMSSD) of the differences between adjacent heartbeat intervals.

[0128] The following formulas (5) and (6) respectively characterize the standard deviation of the normal heartbeat interval (SDNN) and the root mean square of the difference between adjacent heartbeat intervals (RMSSD).

[0129] (5) (6) in, Characterizing the first RR interval between heartbeats, Characterizing the first RR intervals between heartbeats. It represents the average of the RR intervals of N heartbeat intervals, that is, the average heartbeat interval time of all records.

[0130] The reference physiological information for the test subject includes the subject's historical heart rate variability (HRV) index and the corresponding state category. This reference physiological information has a state category label. By comparing the currently obtained HRV index with the reference physiological information, the degree of similarity or difference between the two can be determined. This degree of similarity or difference indicates the probability that the HRV index corresponds to the state category of the reference physiological information. This allows for the acquisition of accurate physiological probability information.

[0131] In one possible embodiment, before determining the physiological probability information based on the degree of difference between the heart rate variability index and the reference physiological information of the test subject, the reference physiological information of the test subject can be obtained, for example, through the following steps: obtaining a sample set of the test subject; performing Gaussian mixture fitting on the sample set according to the state category label to obtain a Gaussian distribution corresponding to each state category; and determining the reference physiological information based on the Gaussian distribution corresponding to each state category.

[0132] The sample set includes multiple historical heart rate variability (HRV) samples, each with a status category label. For example, the historical HRV sample set may include multiple HRV samples from the subject over a historical period, ranging from 3 to 15 days, specifically 7 days.

[0133] The historical heart rate variability (HRV) samples in the aforementioned sample set exhibit a probability distribution pattern. For example, the historical HRV samples labeled with a normal state category follow a Gaussian distribution, as do the historical HRV samples labeled with any one of the abnormal state categories.

[0134] In this embodiment, the number of state categories is greater than or equal to 1. When there are multiple state categories, the historical heart rate variability index samples of each state category all follow their corresponding Gaussian distribution. This makes it impossible to fit the probability distribution of historical heart rate variability index samples of multiple state categories using a single Gaussian distribution. Therefore, in this embodiment, the Gaussian distribution corresponding to each state category can be obtained by performing Gaussian mixture fitting on the sample set using a Gaussian mixture model. The following formula (7) characterizes the Gaussian mixture model.

[0135] (7) in, Characterizing the difference between adjacent heartbeats The probability density function, which describes The probability distribution when taking different values. The number of Gaussian distributions in a Gaussian mixture model is used as an example, taking at least three state categories: normal state, fatigue state, and stress state. A value of 3 means that a combination of three Gaussian distributions is used to fit the data. The distribution of . Characterizing the first The weight coefficients of the Gaussian distribution reflect the weights of the first Gaussian distribution. The proportion of each Gaussian distribution in the entire mixture model. Characterizing the first The probability density functions of a Gaussian distribution, where: Characterizing the difference between adjacent heartbeats, Characterizing the first The mean of a Gaussian distribution, The center position of the k-th Gaussian distribution is, i.e. In the The average value of a Gaussian distribution. The variance characterizes the k-th Gaussian distribution, measuring its dispersion. A larger variance indicates... The more dispersed the values ​​are under this Gaussian distribution.

[0136] Using the Gaussian mixture model described above, the detection device can fit Gaussian distributions for multiple state categories, with each Gaussian distribution associated with a corresponding state category. This allows the Gaussian mixture model to better explain the probability distribution of historical heart rate variability index samples for each state category.

[0137] refer to Figure 4 This is a schematic diagram of the sample distribution corresponding to the Gaussian mixture model. For example... Figure 4 As shown, the Gaussian mixture model includes Gaussian distribution 1, Gaussian distribution 2, and Gaussian distribution 3. Among them, Gaussian distribution 1 is obtained from historical heart rate variability index samples with normal state category labels, Gaussian distribution 2 is obtained from historical heart rate variability index samples with fatigue state category labels, and Gaussian distribution 3 is obtained from historical heart rate variability index samples with stress state category labels.

[0138] Understandably, the number of samples in the normal state category is much larger than that in any other abnormal state category. From a probability distribution perspective, the probability that the state category of the person being tested is normal is relatively high. In one possible embodiment, the detection device can take the mean of the Gaussian distribution corresponding to the normal state category. and variance As reference physiological information.

[0139] In one possible embodiment, at least one state category includes N state categories, where N is an integer greater than or equal to 2. Determining physiological probability information based on the degree of difference between heart rate variability indicators and the reference physiological information of the subject includes: extracting N frequency domain components of the heart rate signal, with each N frequency domain component corresponding to one of the N state categories; determining the N heart rate variability indicators corresponding to the N frequency domain components; and performing a probability mapping between the degree of difference between the N heart rate variability indicators and the reference physiological information to obtain the physiological probability information.

[0140] Taking at least one of the aforementioned state categories—normal state, fatigue state, and stress state—as an example, in the fatigue state category, the subject's heart rate signal exhibits low-frequency characteristics, reflecting a decreasing trend in heart rate variability. In the stress state category, the subject's heart rate signal exhibits high-frequency characteristics, or the ratio of high-frequency to low-frequency components in the heart rate signal is high, reflecting the degree of sympathetic nerve activation. At least for the fatigue and stress state categories, the detection device can extract the low-frequency component (e.g., including the 0.04 Hz to 0.15 Hz frequency band) and the high-frequency component (e.g., including the 0.15 Hz to 0.4 Hz frequency band) of the heart rate signal.

[0141] In one possible embodiment, the detection device can compare each of N heart rate variability indicators with the reference physiological information of the subject to obtain N degrees of difference. Each degree of difference characterizes the degree of deviation between the corresponding heart rate variability indicator and the reference physiological information. The detection device performs probability mapping based on the N degrees of difference to obtain physiological probability information.

[0142] Specifically, the following formulas (8) and (9) respectively characterize the degree of difference between the heart rate variability index based on the low-frequency domain component and the reference physiological information, and the degree of difference between the heart rate variability index based on the high-frequency domain component and the reference physiological information.

[0143] (8) (9) in, l characterization Heart rate variability index of low-frequency frequency domain components (here, heart rate variability index is the standard deviation of normal heartbeat interval SDNN). Heart rate variability indicators and reference physiological information characterizing low-frequency components The degree of difference between them. h characterization Heart rate variability index of high frequency domain components at time (here, heart rate variability index is the standard deviation of normal heartbeat interval SDNN). Heart rate variability indicators and reference physiological information characterizing high-frequency components The degree of difference between them.

[0144] The following formula (10) characterizes the degree of difference between the heart rate variability index based on the low-frequency domain component and the reference physiological information. Physiological probability values ​​mapping fatigue state categories Specifically, the logistic function is used for probability mapping. The following formula (11) characterizes the degree of difference between the heart rate variability index based on the high-frequency domain components and the reference physiological information. Physiological probability values ​​for mapping stress state categories Specifically, the logistic function is used for probability mapping.

[0145] (10) (11) in, The slope parameter characterizes the steepness of the logistic function curve. The parameter representing the center point of the logistic function determines the position of the logistic function curve in the horizontal direction. exp represents the exponential function. The logistic function can map the degree of difference to probability information with the same dimensions as visual probability information, which is convenient for subsequent comprehensive determination of fusion probability information with visual probability values. The following formula (12) represents the heart rate variability index. Visual probability information .

[0146] (12) In this embodiment, each subject establishes their own reference physiological information to achieve "person-specific" reference physiological information, avoiding state detection errors caused by natural physiological differences between different subjects. In one possible embodiment, the identification information of each subject can be associated with heart rate signals, facial images, and reference physiological information, so that each subject is matched one-to-one with the reference physiological information.

[0147] In this embodiment, heart rate variability indicators, which are more correlated with state category detection, are determined based on the heart rate signal to improve state detection efficiency. Furthermore, by mining the correlation between each state category and its frequency domain components, and matching each state category with its frequency domain component, the efficiency of state category detection can be accurately distinguished, facilitating fine-grained state detection. Separating the frequency domain components corresponding to each state category from the heart rate signal involves first distinguishing the relevant heart rate signals for each state category. Based on the known specific state categories corresponding to the frequency domain components, the degree of difference between the heart rate variability indicators of the frequency domain components and reference physiological information accurately distinguishes each state category and accurately determines visual probability information.

[0148] Through the above embodiments, the detection device accurately determines physiological probability information based on physiological signals and visual probability information based on facial images. Next, the detection device fuses these two probabilistic information to obtain fused probability information, and determines the first physiological detection result based on the fused probability information.

[0149] In step S25 above, the process of fusing physiological probability information and visual probability information to obtain fused probability information includes: weighting physiological probability information and visual probability information according to a first weight corresponding to physiological probability information and a second weight corresponding to visual probability information to obtain fused probability information, wherein the first weight is less than or equal to the second weight.

[0150] The fatigue state category and the stress state category will be used as examples for explanation. Formula (13) below is the calculation method for the fusion probability value corresponding to the fatigue state category, and formula (14) is the calculation method for the fusion probability value corresponding to the stress state category. Formula (15) below represents the fusion probability information S. .

[0151] (13) in, This represents the fusion probability value for the fatigue state category. Characterize the second weight. For example, . Visual probability values ​​representing fatigue state categories. Physiological probability values ​​that characterize fatigue state categories.

[0152] (14) in, This represents the fusion probability value for the stress state category. Characterize the second weight. For example, . Visual probability values ​​representing stress state categories. Physiological probability values ​​that characterize the category of stress state.

[0153] S (15) The first and second weights can be pre-configured or adjusted based on the performance metrics of the state detection system. When discussing multiple state categories, the fusion probability values ​​for all state categories use both the first and second weights. The first weights used for the fusion probability values ​​of different state categories can be the same or different, and the second weights used for the fusion probability values ​​of different state categories can also be the same or different.

[0154] In scenarios with high real-time requirements and limited physiological signal acquisition, the detection device primarily uses facial image-based state detection, supplemented by physiological signal-based state detection. The first weight is less than or equal to the second weight, or the first weight is not greater than the second weight, or the first weight does not exceed the second weight. Conversely, in scenarios requiring accurate reflection of the subject's internal state category and where facial image acquisition is limited, the detection device primarily uses physiological signal-based state detection, supplemented by facial image-based state detection. The first weight is greater than or equal to the second weight, or the first weight is not less than the second weight, or the second weight does not exceed the first weight.

[0155] In the above embodiments, one physiological signal and one face image are used as examples. In some cases, there are multiple physiological signals and multiple face images, and the corresponding fusion probability information may include multiple fusion probability values. Based on the temporal relationship between the multiple physiological signals and multiple face images, the multiple fusion probability values ​​also exhibit a temporal relationship. In this case, when an abnormal fusion probability value appears among the multiple fusion probability values, the abnormal fusion probability value will affect the accuracy of the subsequent determination of the physiological detection result. Therefore, another embodiment of this application also performs temporal smoothing processing on the multiple fusion probability values.

[0156] Specifically, in one possible embodiment, the fusion probability information includes X fusion probability values, where X is an integer greater than or equal to 3. The state detection method further includes: performing temporal smoothing processing on the X fusion probability values ​​to obtain smoothed fusion probability information; and determining the state detection result based on the fusion probability information, including: if the smoothed probability fusion information exceeds a probability threshold, determining the state category corresponding to the smoothed fusion probability information as the state detection result.

[0157] In this embodiment, the detection device performs time-series smoothing on the X fusion probability values ​​to remove abnormal fusion probability values, resulting in more accurate smoothed fusion probability information. Determining the state detection result based on the smoothed fusion probability information is also more accurate.

[0158] Furthermore, the X fusion probability values ​​may be linearly correlated or non-linearly correlated. Based on different correlations, the detection device can perform different temporal smoothing processes on the X fusion probability values.

[0159] Specifically, in one possible embodiment, time-series smoothing of the X fusion probability values ​​includes: performing Kalman filtering on the X fusion probability values ​​when they are linearly correlated; and / or performing unscented Kalman filtering on the X fusion probability values ​​when they are non-linearly correlated.

[0160] For example, Kalman filtering is used to estimate the state of a dynamic system from a series of noisy data. In time-smoothing scenarios, Kalman filtering utilizes the system's state equation and observation equation, and through continuous prediction and updating, gradually filters out noise from the measurement data to obtain a more accurate and smoother state estimate. For multiple time-related fusion probability values ​​(which can be analogous to observations),... First, the current state is predicted based on the state at the previous moment, and then the predicted value is corrected by combining the actual observation value at the current moment, so as to achieve low-latency and highly robust estimation of the system state and filter out random jitter.

[0161] Specifically, fusing probabilistic information The fusion probability value in the observation is used as the fusion probability value. One-dimensional state Following a linear model, the following formula (16) characterizes the state equation and the observation equation: (16) Formula (16) describes the state transition of the system from time k-1 to time k, using the state at the previous time step. To predict the state at the current moment .in, The state transition matrix represents how the state at the previous time step transitions to the state at the current time step, and reflects the dynamic change law of the system state. Characterize the true state of the system at time k-1 (the true state of the system at time k-1 is known). The true state of the system at time k is represented (the true state of the system at time k is unknown). Characterization process noise, It is a random variable that represents the noise generated by various unknown factors during the state transition process. It is assumed that the random variable follows a Gaussian distribution with a mean of 0.

[0162] (17) Formula (17) is used to establish the true state of the system. Compared with actual observations The relationship between them. Among them, Characterization Observations at any given time, such as fusion probability values It is data obtained from actual measurements, which includes noise. The observation matrix represents how the real state is mapped to the observation space. Characterization The actual state of the system at any given moment. The observation noise is a random variable that represents the noise introduced during the measurement process. It is assumed that the random variable follows a Gaussian distribution with a mean of 0.

[0163] Based on the above state equation and observation equation, the detection equipment uses the following formulas (18) to (22) to make predictions and updates.

[0164] (18) Formula (18) is used for state prediction. Formula (18) specifically represents the state estimate based on the previous time step. Using the state transition matrix predict state of time The predictions here include process noise. .

[0165] (19) Formula (19) is used for prediction Covariance of state estimates at time points This reflects the degree of uncertainty in the predicted state. It is the covariance matrix of the process noise. yes The transpose of .

[0166] (20) Formula (20) is used to calculate the Kalman gain. Kalman gain It determines how to combine forecasts and observations to update the state estimate, balancing the weights of forecasts and observations. The covariance matrix characterizing observation noise, It is the observation matrix The transpose of .

[0167] (twenty one) Formula (21) is used to utilize Kalman gain Observation and predicted state Update the state estimate to obtain a more accurate result. Time-state estimate . Characterization Observations of time The error between the observed and predicted values.

[0168] (twenty two) Formula (22) is used to update Covariance of state estimates at time points This reflects the degree of uncertainty in the updated state estimate. Represents the identity matrix.

[0169] For example, the unscented Kalman filter (UKF). The unscented Kalman filter works by determining a set of... The sampling points (sigma) approximate the mean and covariance of the nonlinear state distribution. Compared with Kalman filtering, unscented Kalman filtering avoids linearization error problems and provides more accurate state estimation when dealing with nonlinear systems. Specifically, the following formula (23) is used to generate point.

[0170] (twenty three) Formula (23) is used to determine the source of the first σ point, that is, to use the filtered state estimate of the previous time step as the initial point in the current set of σ points. The first σ point is represented at time k-1, and the first σ point serves as the set of σ points and the reference point. The state estimate after Kalman filtering at time k-1 represents the best estimate of the system state at the previous time step.

[0171] The next step is to test the equipment according to formula (23). Click to proceed Point prediction. Specifically, the detection device propagates all generated σ points through a nonlinear state transition function. This simulates the nonlinear evolution of the system state over time, with each σ point updated to the current time (i.e., time k) based on the system's dynamic characteristics. Subsequently, based on these propagated σ points, the detection device calculates the predicted mean and covariance. The predicted mean represents a predicted estimate of the system state at the current time, while the covariance reflects the degree of uncertainty in the predicted state.

[0172] Next, the detection device maps the predicted σ point through a nonlinear observation function h to obtain the corresponding observed prediction value. These observed prediction values ​​reflect the theoretical observation based on the predicted state. The detection device compares the observed prediction values ​​with the actual observations (which are multiple fused probability values ​​with temporal relationships) and recalculates the Kalman gain. The Kalman gain determines how to balance the predicted values ​​and actual observations to update the state estimate and covariance matrix, making the estimate of the system state closer to reality.

[0173] When dealing with nonlinear problems involving multiple fused probability values ​​with temporal relationships, unscented Kalman filtering, compared to linear Kalman filtering, more accurately captures the statistical characteristics of nonlinear systems through the σ point, effectively improving the accuracy of state estimation.

[0174] In this embodiment, the detection device obtains fused probability information after fusing physiological and visual probability information. It also performs temporal smoothing on the fused probability information to remove sudden noise. This embodiment employs Kalman filtering or unscented Kalman filtering for temporal smoothing of multiple fused probability values, resulting in lower computational complexity and higher real-time performance and accuracy. Kalman filtering or unscented Kalman filtering places lower resource requirements on the detection device, thus offering superior performance in temporal smoothing and making it suitable for scenarios with a large number of participants or high real-time requirements.

[0175] Through the above embodiments, fusion probability information can be obtained. The state detection result can be determined based on the fusion probability information. For example, in one possible embodiment, the fusion probability information and / or smoothed fusion probability information include a normal probability value, a fatigue probability value, and a stress probability value (here, the normal probability value, fatigue probability value, and stress probability value are all fusion probability values ​​mentioned above). The normal probability value represents the probability that the state category of the person being tested is the normal state category; the fatigue probability value represents the probability that the state category of the person being tested is the fatigue state category; the stress probability value represents the probability that the state category of the person being tested is the stress state category. Determining the state detection result based on the fusion probability information includes: if the normal probability value exceeds a probability threshold, determining that the state category of the person being tested is the normal state category; if the fatigue probability value exceeds a probability threshold, determining that the state category of the person being tested is the fatigue state category; if the stress probability value exceeds a probability threshold, determining that the state category of the person being tested is the stress state category. Alternatively, the above-mentioned normal probability value, fatigue probability value, and stress probability value can also be compared among themselves, and the state category corresponding to the maximum value can be determined as the state category of the person being tested.

[0176] To illustrate with a specific example, when the probability information is fused to characterize the state category of the test subject determined jointly by physiological parameters and facial features: the probability of being in the normal state category is 30% (i.e., the probability value of normal is 30%), the probability of being in the fatigue state category is 60% (i.e., the probability value of fatigue is 60%), and the probability of being in the stress state category is 10% (i.e., the probability value of stress is 30%), if the probability of the fatigue state category is the highest or the probability of the fatigue state category is greater than the probability threshold of 50% (the probability threshold of 50% is only an example and not a limitation), the detection device can determine that the state detection result characterizes the state category to which the test subject belongs to the fatigue state category.

[0177] To improve the accuracy of the first physiological detection result, the detection device can also verify the first physiological detection result. Specifically, in one possible embodiment, after determining the state detection result based on the fusion probability information, the state detection method includes: acquiring feature information of the person to be tested within a first preset time period; classifying the feature information to obtain a state verification result, the state verification result including the state category to which the person to be tested belongs; determining whether the state detection result and the state verification result are the same; if the state detection result and the state verification result are the same, using the state detection result or the state verification result as the state category to which the person to be tested belongs; if the state detection result and the state verification result are different, using the state verification result as the state category to which the person to be tested belongs.

[0178] The first preset duration is determined based on a first moment, which includes the acquisition time of the face image and / or the acquisition time of the physiological signal. The feature information includes one or more of the following: physiological parameters, facial features, and fusion probability information. The state verification result is used to verify whether the state detection result is accurate.

[0179] refer to Figure 5 This is a schematic diagram of a state detection method according to an embodiment of this application. Figure 5 As shown, the first preset duration is a period of time before the first moment and ends at the first moment. Of course, the first preset moment can also be a period of time before the moment, which is not specifically limited here. For example, the first preset duration can be in the range of 1 second to 10 seconds, and the first preset duration can specifically be 5 seconds. Figure 5 In the example, the first preset duration includes a total of three times from time t0 to time t2, and each time corresponds to the time when the face image and / or physiological signal are collected.

[0180] Figure 5 In the example, the current state detection targets the face image 2 and physiological signal 2 at the first time t2. Using the state detection method described above, the detection device can obtain the face image 2, physiological parameters 2, and fusion probability information 2 based on the face image 2 and physiological signal 2. Time points t0 and t1 before the first time t2 are historical times relative to the first time t2. The detection device has already executed the state detection method based on the face image and physiological signal at historical time t0, obtaining the face feature 0, physiological parameters 0, fusion probability information 0, and the state category of the person being tested corresponding to historical time t0. The state category of the person being tested serves as the state category label 0 for historical time t0. Feature information 1 can include one or more of the face feature 0, physiological parameters 0, fusion probability information 0, and state category label 0. Similarly, the detection device obtains feature information 1 corresponding to historical time t1 based on the face image and physiological signal at historical time t1. Compared to the current first time t2, the detection device cannot obtain the corresponding state category label at first time t2; therefore, feature information 2, including face image 2, physiological parameters 2, and fusion probability information 2, can be obtained at first time t2. Therefore, the feature information of the first preset duration includes feature information 0, feature information 1 and feature information 2.

[0181] In one possible embodiment, the detection device uses a classifier to classify the feature information to obtain a state verification result. The classifier outputs the state category of the person being tested: the probability of each state category among at least one state category. Alternatively, the classifier outputs the state category of the person being tested. Specifically, to accurately distinguish between fatigue and stress states, the classifier can output the state category of the person being tested as either fatigue or stress. The classifier can include one or more of the following: support vector machine (SVM), random forest, or lightweight fourth neural network. The classifier can be pre-trained using feature information samples.

[0182] In one possible embodiment, the termination time of the first preset duration can also be after the first moment. In this case, the detection device can also acquire multiple physiological signals and multiple face images after the first moment, and execute the state detection method of this application embodiment, thereby obtaining the detection result after the first moment. The detection result after the first moment can be used to verify the state detection result of the first moment.

[0183] In this embodiment, compared to acquiring facial images and physiological information based on a single moment, the detection device obtains more feature information over a first preset duration. Therefore, the state verification result obtained based on this feature information is more accurate and stable. Furthermore, the feature information represents the "fingerprint-like" characteristics of the person being tested, making the state category determined based on this feature information more accurate. The state detection result is then verified using the feature information to obtain the state verification result. If the state detection result and the state verification result are the same, it indicates that the state detection result determined by combining facial images and physiological signals is accurate. If they differ, it indicates that the state detection result determined by combining facial images and physiological signals has lower accuracy. In this case, the detection device uses the more accurate state verification result as the state category of the person being tested, resulting in a more accurate state verification result.

[0184] The state detection method of this embodiment is applied to scenarios such as security, medical care, and motion detection. In these scenarios, state detection is used to determine whether the state category of the person being tested is an abnormal state category. To accurately detect whether the physiological state of the person being tested is an abnormal state category, in one possible embodiment, if the state detection result indicates that the state category of the person being tested is an abnormal state category, the detection device can acquire the characteristic information of the person being tested within a first preset time period and verify the state detection result based on the characteristic information. The verification process is similar to the previous embodiment and will not be described again here.

[0185] In the above embodiments, the detection device can accurately detect the state based on facial images and physiological signals. When the detection result indicates that the state category of the person being tested is an abnormal state category, it indicates that there is a safety hazard in the person's work. At this time, the detection device can issue an alarm message to indicate that the state category of the person being tested is an abnormal state category and improve the safety of the person's work. In order to improve the accuracy of the alarm and reduce false alarms, in this embodiment, when the duration of the person being tested being in an abnormal state category reaches a second preset duration, the detection device outputs an alarm message. The alarm message indicates that the person being tested is in an abnormal state category. For example, the second preset duration can be in the range of 2s to 10s. Specifically, the second preset duration can be 5s.

[0186] In one possible embodiment, if the duration of the tested person's state category being an abnormal state reaches a second preset duration, the detection device outputs an alarm message. The detection device can perform a performance re-evaluation (PF) of the state detection method (and the corresponding state detection system). This performance re-evaluation is used to reassess the performance metrics of the state detection method and system based on newly collected alarm information. These performance metrics include, for example, accuracy and latency. Based on the performance metrics, the detection device can adjust one or more of the following mentioned in the above steps: algorithm parameters, threshold range, preset duration, and weights, to optimize the performance of the state detection method and system.

[0187] Through the above embodiments, the detection device integrates facial images and physiological signals to determine, for example, state detection results and / or state verification results (hereinafter collectively referred to as state category detection results), which indicate the state category of the person being tested. The state detection results obtained through the above embodiments have higher accuracy and stability.

[0188] In one embodiment of this application, state category labels can be added to the state detection data obtained in the above steps using the state category detection results to obtain state detection samples. The state detection data includes one or more of the following: face images, face features, physiological signals, physiological parameters, physiological probability information, visual probability information, and fusion probability information. Corresponding to the state detection data, the state detection samples include one or more of the following: face image samples, face feature samples, physiological signal samples, physiological parameter samples, physiological probability information samples, visual probability information samples, and fusion probability information samples. The state detection samples can be used to update one or more of the following: neural networks, reference physiological information, first weights, second weights, and feature information, thereby enabling closed-loop adaptive optimization of the state detection method. The neural network here can include one or more of the above-mentioned components: a first neural network, a second neural network, and processing branches. This ensures that the detection device maintains high stability and accuracy even when using data such as neural networks, reference physiological information, first weights, second weights, and feature information for state detection over a long period.

[0189] For example, the detection device updates the sample set and thus the reference physiological information of the target object based on the physiological parameters corresponding to the state detection results and / or the physiological parameters corresponding to the state verification results. And / or it updates the image processing model based on the face images corresponding to the state detection results and / or the face images corresponding to the state verification results. This makes the reference physiological information highly real-time and accurate. The physiological probability information determined by the detection device based on the physiological parameters and the reference physiological information of the person being tested is more accurate. The image processing model has better generalization ability and robustness, and the visual probability information obtained using the image processing model is more accurate and robust.

[0190] In summary, the state detection method according to the embodiments of this application has at least one of the following technical effects: 1) The detection device determines physiological probability information based on facial images as a visual detection method, and determines visual probability information based on physiological signals of different modalities than facial images as a physiological detection method. By fusing physiological and visual probability information, the shortcomings of relying solely on visual or physiological detection can be avoided, improving the accuracy and efficiency of state detection. In this embodiment, state detection is performed by combining facial images and physiological signals. The facial images directly reflect the behavior of the person being tested, such as changes in facial expressions, and possess intuitive behavioral semantic information. This behavioral semantic information indicates which behavior of the person being tested triggered the state change, thereby improving the accuracy and stability of state detection.

[0191] 2) The detection equipment constructs individual reference physiological information for each individual based on their physiological differences, avoiding the use of the same reference physiological information to compare different individuals. This allows for accurate detection of the state category of each individual.

[0192] 3) The detection equipment can not only detect whether the state category of the person being tested is normal or abnormal, but also mine the facial features and physiological signal characteristics of different abnormal state categories. Through high-dimensional features such as facial features and physiological parameters, the state of the person being tested can be accurately and personally evaluated, so as to achieve accurate differentiation of each state category and fine-grained state detection.

[0193] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0194] Corresponding to the state detection method in the above embodiments, Figure 6 A structural block diagram of the state detection device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0195] Reference Figure 6 The status detection device includes: a first acquisition module 310, a second acquisition module 320, and a processing module 330.

[0196] The first acquisition module 310 is used to acquire the facial image of the person to be tested. The second acquisition module 320 is used to acquire the physiological signals of the person to be tested. The processing module 330 is used to determine physiological probability information based on the physiological signals. The physiological probability information is used to characterize the state category of the person to be tested identified based on physiological parameters: each of the at least one state category has a probability of being one of at least one state category. The physiological parameters are determined based on the physiological signals. The processing module 330 is also used to determine visual probability information based on the facial image. The visual probability information is used to characterize the state category of the person to be tested identified based on facial features: each of the at least one state category has a probability of being one of at least one state category. The facial features are determined based on the facial image. The processing module 330 is also used to fuse the physiological probability information and the visual probability information to obtain fused probability information. The fused probability information characterizes the state category of the person to be tested, which is determined based on both the physiological parameters and the facial features: each of the at least one state category has a probability of being one of at least one state category. The processing module 330 is also used to determine a state detection result based on the fused probability information. The state detection result characterizes the state category to which the person to be tested belongs, and the state category to which the person to be tested belongs is one of at least one state category.

[0197] In one possible embodiment, the processing module 330 is further configured to acquire feature information of the person to be tested within a first preset time period, the first preset time period being determined based on a first moment, the first moment including the acquisition time of the face image and / or the acquisition time of the physiological signal, the feature information including one or more of the following: physiological parameters, face features, fusion probability information; the processing module 330 is further configured to classify the feature information to obtain a state verification result, the state verification result including the state category to which the person to be tested belongs, the state verification result being used to verify whether the state detection result is accurate; the processing module 330 is further configured to, when the state detection result and the state verification result are the same, use the state detection result or the state verification result as the state category to which the person to be tested belongs; the processing module 330 is further configured to, when the state detection result and the state verification result are different, use the state verification result as the state category to which the person to be tested belongs.

[0198] In one possible embodiment, the processing module 330 is specifically used to obtain the characteristic information of the person to be tested within a first preset time period when the state detection result indicates that the state category to which the person to be tested belongs is an abnormal state category.

[0199] In one possible embodiment, the processing module 330 is specifically used to weight the physiological probability information and the visual probability information according to a first weight corresponding to the physiological probability information and a second weight corresponding to the visual probability information to obtain fused probability information, wherein the first weight is less than or equal to the second weight.

[0200] In one possible embodiment, the fusion probability information includes X fusion probability values, where X is an integer greater than or equal to 3. The processing module 330 is further used to perform temporal smoothing processing on the X fusion probability values ​​to obtain smoothed fusion probability information. Specifically, the processing module 330 is used to determine the state category corresponding to the smoothed fusion probability information as the state detection result when the smoothed probability fusion information exceeds the probability threshold.

[0201] In one possible embodiment, the processing module 330 is specifically configured to perform Kalman filtering on the X fusion probability values ​​when the X fusion probability values ​​are linearly correlated; and / or perform unscented Kalman filtering on the X fusion probability values ​​when the X fusion probability values ​​are non-linearly correlated.

[0202] In one possible embodiment, the processing module 330 is further configured to output alarm information when the duration of the test subject's status category being an abnormal status category reaches a second preset duration. The alarm information is used to indicate that the test subject's status category is an abnormal status category.

[0203] In one possible embodiment, the physiological signal includes a heart rate signal, and the physiological parameters include a heart rate variability index; the processing module 330 is specifically used to perform a first preprocessing on the heart rate signal to determine the heart rate variability index, the first preprocessing including a bandpass filtering operation; and to determine physiological probability information based on the degree of difference between the heart rate variability index and the reference physiological information of the person being tested, wherein the reference physiological information of the person being tested includes: the historical heart rate variability index of the person being tested and / or the state category corresponding to the historical heart rate variability index of the person being tested.

[0204] In one possible embodiment, at least one state category includes N state categories, where N is an integer greater than or equal to 2; the processing module 330 is further specifically used to extract N frequency domain components of the heart rate signal, the N frequency domain components corresponding to the N state categories; determine N heart rate variability indices corresponding to the N frequency domain components; and perform probability mapping between the degree of difference between the N heart rate variability indices and the reference physiological information of the person being tested to obtain physiological probability information.

[0205] In one possible embodiment, the processing module 330 is further configured to acquire a sample set of the person to be tested, the sample set including multiple historical heart rate variability index samples, each historical heart rate variability index sample having a state category label; perform Gaussian mixture fitting processing on the sample set according to the state category label to obtain a Gaussian distribution corresponding to each state category; and determine the reference physiological information of the person to be tested according to the Gaussian distribution corresponding to each state category.

[0206] In one possible embodiment, the processing module 330 is specifically configured to perform a second preprocessing on the face image, extracting face features from the face image, the face features including face depth features and / or action unit features; wherein, the action unit features include facial expression features, and the face depth features include one or more of the following: facial texture features, facial muscle features, facial skin features, facial angle features, and facial motion features; the second preprocessing includes a face alignment operation; processing the face features using an image processing model to obtain N face category features corresponding to N state categories, the image processing model including a long short-term memory network and / or an attention mechanism network; and determining visual probability information based on the N face category features.

[0207] In one possible embodiment, the processing module 330 is further configured to update the reference physiological information of the person to be tested based on the physiological parameters corresponding to the state detection result and / or the physiological parameters corresponding to the state verification result; and / or update the image processing model based on the face image corresponding to the state detection result and / or the face image corresponding to the state verification result.

[0208] In one possible embodiment, at least one state category includes at least one normal state category and / or at least one abnormal state category, wherein the abnormal state category includes a fatigue state category and a stress state category.

[0209] In one possible embodiment, the fused probability information and / or the smoothed fused probability information includes a normal probability value, a fatigue probability value, and a stress probability value. The normal probability value represents the probability that the test subject belongs to the normal state category; the fatigue probability value represents the probability that the test subject belongs to the fatigue state category; and the stress probability value represents the probability that the test subject belongs to the stress state category. The processing module 330 is specifically configured to: determine that the test subject belongs to the normal state category when the normal probability value exceeds a probability threshold; determine that the test subject belongs to the fatigue state category when the fatigue probability value exceeds a probability threshold; and determine that the test subject belongs to the stress state category when the stress probability value exceeds a probability threshold.

[0210] In one possible embodiment, the state detection device is applied to a state detection system, which includes an image acquisition device and a physiological signal acquisition device; the image acquisition device is used to acquire facial images of the person being tested; and the physiological signal acquisition device is used to acquire physiological signals of the person being tested.

[0211] In applications, the modules and units in the state detection device can be software program modules, or they can be implemented by different logic circuits integrated in the processor, or they can be implemented by multiple distributed processors.

[0212] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0213] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0214] This application also provides an electronic device, which includes the aforementioned detection device. (See reference...) Figure 7 This is a schematic diagram of an electronic device. For example... Figure 7 As shown, the electronic device includes: at least one processor 41, a memory 42, and a computer program 43 stored in the memory 42 and executable on at least one processor 41, wherein the processor 41 executes the computer program 43 to implement the steps in any of the above method embodiments.

[0215] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0216] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0217] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0218] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0219] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0220] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0221] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0222] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A state detection method, characterized in that, include: Obtain facial images of the individuals to be tested; Acquire the physiological signals of the person being tested; Based on the physiological signals, physiological probability information is determined, which is used to characterize the state category of the person under test identified based on physiological parameters: the probability of each state category in at least one state category, and the physiological parameters are determined based on the physiological signals. Based on the facial image, visual probability information is determined, which is used to characterize the state category of the person under test identified based on facial features: the probability of each state category in the at least one state category, and the facial features are determined based on the facial image; The physiological probability information and the visual probability information are fused to obtain fused probability information, which represents the state category of the person under test determined jointly by the physiological parameters and the facial features: the probability of each state category in the at least one state category; Based on the fusion probability information, a state detection result is determined, wherein the state detection result represents the state category to which the person being tested belongs, and the state category to which the person being tested belongs is one of the at least one state category.

2. The method according to claim 1, characterized in that, After determining the state detection result based on the fusion probability information, the method further includes: The feature information of the person to be tested is obtained within a first preset time period, the first preset time period is determined based on a first moment, the first moment includes the acquisition time of the face image and / or the acquisition time of the physiological signal, and the feature information includes one or more of the following: the physiological parameters, the face features, and the fusion probability information; The feature information is classified to obtain a state verification result, which includes the state category to which the person being tested belongs. The state verification result is used to verify whether the state detection result is accurate. If the state detection result is the same as the state verification result, the state detection result or the state verification result shall be used as the state category to which the person being tested belongs. If the state detection result differs from the state verification result, the state verification result shall be used as the state category to which the person being tested belongs.

3. The method of claim 2, wherein, The feature information for obtaining the first preset duration includes: If the state detection result indicates that the state category of the person being tested is an abnormal state category, the characteristic information of the person being tested within the first preset time period is obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The process of fusing the physiological probability information and the visual probability information to obtain the fused probability information includes: Based on a first weight corresponding to the physiological probability information and a second weight corresponding to the visual probability information, the physiological probability information and the visual probability information are weighted to obtain fusion probability information, wherein the first weight is less than or equal to the second weight.

5. The method according to any one of claims 1-3, characterized in that, The fusion probability information includes X fusion probability values, where X is an integer greater than or equal to 3. The method further includes: The X fusion probability values ​​are subjected to time-series smoothing to obtain smoothed fusion probability information; The step of determining the state detection result based on the fusion probability information includes: If the smoothed probability fusion information exceeds the probability threshold, the state category corresponding to the smoothed fusion probability information is determined as the state detection result.

6. The method of claim 5, wherein, The time-series smoothing process for the X fusion probability values ​​includes: If the X fusion probability values ​​are linearly correlated, Kalman filtering is applied to the X fusion probability values; and / or When the X fusion probability values ​​are nonlinearly correlated, the X fusion probability values ​​are subjected to unscented Kalman filtering.

7. The method of claim 5, wherein, The method further includes: If the duration of the test subject's status category being an abnormal status category reaches a second preset duration, an alarm message is output, which is used to indicate that the test subject's status category is an abnormal status category.

8. The method of any one of claims 1-3, wherein, The physiological signal includes a heart rate signal, and the physiological parameters include a heart rate variability index; determining physiological probability information based on the physiological signal includes: The heart rate signal is subjected to a first preprocessing to determine the heart rate variability index. The first preprocessing includes a bandpass filtering operation. The physiological probability information is determined based on the degree of difference between the heart rate variability index and the reference physiological information of the test subject. The reference physiological information of the test subject includes: the test subject's historical heart rate variability index and / or the state category corresponding to the test subject's historical heart rate variability index.

9. The method of claim 8, wherein, The at least one state category includes N state categories, where N is an integer greater than or equal to 2; determining the physiological probability information based on the degree of difference between the heart rate variability index and the reference physiological information of the subject includes: N frequency domain components of the heart rate signal are extracted, and the N frequency domain components correspond to the N state categories; Determine the N heart rate variability indices corresponding to the N frequency domain components; The physiological probability information is obtained by probabilistically mapping the degree of difference between the N heart rate variability indicators and the reference physiological information of the test subject.

10. The method of claim 8, wherein, Before determining the physiological probability information based on the degree of difference between the heart rate variability index and the reference physiological information of the subject, the method further includes: Obtain a sample set of the person to be tested, the sample set including multiple historical heart rate variability index samples, each historical heart rate variability index sample having a status category label; Based on the state category labels, the sample set is subjected to Gaussian mixture fitting to obtain the Gaussian distribution corresponding to each state category; The reference physiological information of the person being tested is determined based on the Gaussian distribution corresponding to each state category.

11. The method of any one of claims 1-3, wherein, The step of determining visual probability information based on the face image includes: The face image is subjected to a second preprocessing step to extract facial features, which include facial depth features and / or action unit features; wherein the action unit features include facial expression features, and the facial depth features include one or more of the following: facial texture features, facial muscle features, facial skin features, facial angle features, and facial motion features; the second preprocessing step includes a face alignment operation. The facial features are processed using an image processing model to obtain N facial category features corresponding to N state categories. The image processing model includes a long short-term memory network and / or an attention mechanism network. The visual probability information is determined based on the N individual face category features.

12. The method of any one of claims 2, 9, 11, wherein, The method further includes: Update the reference physiological information of the person being tested based on the physiological parameters corresponding to the state detection results and / or the physiological parameters corresponding to the state verification results; and / or The image processing model is updated based on the face image corresponding to the state detection result and / or the face image corresponding to the state verification result.

13. The method according to any one of claims 1-3, characterized in that, The at least one state category includes at least one normal state category and / or at least one abnormal state category, wherein the abnormal state category includes a fatigue state category and a stress state category.

14. The method of any one of claims 1-3, wherein, The fusion probability information includes a normal probability value, a fatigue probability value, and a stress probability value. The normal probability value represents the probability that the test subject belongs to the normal state category; the fatigue probability value represents the probability that the test subject belongs to the fatigue state category; and the stress probability value represents the probability that the test subject belongs to the stress state category. Determining the first detection result based on the fusion probability information includes: If the normal probability value exceeds the probability threshold, the state category to which the person being tested belongs is determined to be the normal state category; If the fatigue probability value exceeds the probability threshold, the state category to which the person being tested belongs is determined to be the fatigue state category. If the stress probability value exceeds the probability threshold, the state category to which the person being tested belongs is determined as the stress state category.

15. The method of any one of claims 1-3, wherein, The method is applied to a state detection system, which includes an image acquisition device and a physiological signal acquisition device; the image acquisition device is used to acquire a facial image of the person being tested; and the physiological signal acquisition device is used to acquire the physiological signals of the person being tested.

16. A state detection device characterized by comprising: include: The first acquisition module is used to acquire the facial image of the person to be tested; The second acquisition module is used to acquire the physiological signals of the person to be tested; The processing module is used to determine physiological probability information based on the physiological signal. The physiological probability information is used to characterize the state category of the person under test identified based on physiological parameters: the probability of each state category in at least one state category. The physiological parameters are determined based on the physiological signal. The processing module is further configured to determine visual probability information based on the face image, wherein the visual probability information is used to characterize the state category of the person under test identified based on the face features: the probability of each state category in the at least one state category, and the face features are determined based on the face image; The processing module is further configured to fuse the physiological probability information and the visual probability information to obtain fused probability information, wherein the fused probability information represents the state category of the person under test determined jointly based on the physiological parameters and the facial features: the probability of each state category in the at least one state category; The processing module is further configured to determine a state detection result based on the fusion probability information, wherein the state detection result characterizes the state category to which the person to be tested belongs, and the state category to which the person to be tested belongs is one of the at least one state category.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 15.

18. A condition detection system, characterized by It includes an image acquisition device, a physiological signal acquisition device, and an electronic device. The electronic device acquires a facial image of the person being tested through the image acquisition device; the electronic device acquires the physiological signals of the person being tested through the physiological signal acquisition device. The electronic device is used to perform the method as described in any one of claims 1 to 15. 19.A computer readable storage medium, storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 15.

Images