Non-diagnosis and treatment type pain state assessment method and system based on facial time sequence sign fusion

Abstract

The application provides a non-diagnosis and treatment type pain state evaluation method and system based on facial time sequence sign fusion, and the method comprises the following steps: collecting a front facial dynamic video stream of a person to be measured, and respectively extracting dynamic micro-expression time sequence sign data and facial skin color time sequence sign data; performing time sequence quantization on the micro-expression data to obtain action unit time sequence change quantity, and performing multi-dimensional quantization on the skin color data to obtain a skin color time sequence quantization numerical value sequence; building an exclusive pathological correlation knowledge graph, mapping the two types of quantized data to a knowledge graph semantic space to complete feature alignment and fusion; introducing a time sequence causal correlation analysis mechanism, mining the time sequence influence relationship between the action unit change and the skin color change, and deducing the continuous evolution trend of the physical and mental discomfort state; and outputting an objective sign evaluation result. Through the fusion of facial dynamic and static signs, the combination of knowledge graph constraints and time sequence causal analysis, the objective and continuous evaluation of the physical and mental discomfort state is realized.

Description

Technical Field

[0001] This invention relates to the field of pain assessment, and in particular to a non-diagnostic pain state assessment method and system based on facial temporal sign fusion. Background Technology

[0002] Pain assessment is a fundamental part of clinical diagnosis and health monitoring. Currently, clinical pain assessment mainly relies on patient self-report tools and observer-dependent tools, but existing technologies have the following problems: Human assessment is highly subjective and suffers from low consistency among assessors; intermittent human assessment cannot achieve continuous dynamic monitoring over 24 hours, and is prone to missing fluctuating pain episodes; existing automated solutions mostly focus on the analysis of facial action units in a single modality, ignoring static signs such as facial skin color; existing methods are mostly single-frame classification or short-term window recognition, lacking modeling of the temporal causal relationship between micro-expressions and skin color changes; significant differences exist among individuals in facial structure and pain expression habits, resulting in poor model generalization ability across populations; in addition, environmental factors such as lighting, angle, and occlusion seriously affect the stability of the model in real clinical environments.

[0003] Therefore, there is a need to provide a non-diagnostic pain state assessment method and system based on facial temporal sign fusion to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a non-diagnostic pain state assessment method and system based on facial temporal sign fusion. By simultaneously collecting and analyzing dynamic features of facial micro-expressions and static features of skin color, combined with knowledge graph constraints and temporal causal analysis, it automatically outputs an objective and continuous trend of the evolution of physical and mental discomfort state, solving the problems of strong subjectivity, inability to continuously monitor, insufficient information of a single modality, and poor cross-individual generalization ability of existing technologies.

[0005] To achieve the above objectives, this invention provides a non-therapeutic pain state assessment method based on facial temporal sign fusion, implemented by a processor executing a computer program stored on a readable storage medium, specifically including the following steps: S1: Collect dynamic video stream of the frontal face of the person to be tested within a preset continuous time window; S2: Perform face localization, pose correction and pixel region segmentation on the dynamic facial video stream, and extract dynamic micro-expression temporal characteristic data and facial skin color temporal characteristic data respectively; S3: The dynamic micro-expression temporal characteristic data extracted in S2 is used to perform temporal quantification of the micro-expression temporal characteristic data based on the preset facial action unit evaluation rules, so as to obtain the temporal change of human physiological pain-related action units. S4: The facial skin color temporal characteristic data extracted in S2 is used to complete the full-domain pixel statistics of facial skin color based on the CIE-Lab uniform color space. The skin color brightness, saturation and color difference distribution are quantified in multiple dimensions, and the skin color temporal quantization numerical sequence is output. S5: Construct a dedicated pathological association knowledge graph that integrates the mechanism of facial motion units with the logic of somatic discomfort signs. The entity nodes of the dedicated pathological association knowledge graph include facial motion units, skin color representation parameters, somatic discomfort symptoms and physical and mental state levels. The association weights between entity nodes are set according to the correspondence of clinical signs. S6: Map the temporal change of the action unit obtained in S3 and the temporal quantization numerical sequence of skin color obtained in S4 to the semantic space of the exclusive pathological association knowledge graph built in S5, complete feature alignment and fusion, and obtain an integrated physical sign fusion representation that takes into account dynamic expression changes and facial skin color status. S7: Using the temporal change of the action unit obtained in S3 and the temporal quantitative numerical sequence of skin color obtained in S4 as input, a temporal causal correlation analysis mechanism is introduced to explore the temporal influence relationship between the two. Combined with the integrated physical sign fusion representation obtained in S6, the continuous evolution trend of the physical and mental discomfort state of the test subject is deduced. S8: Output the continuous evolution trend of physical and mental discomfort obtained in S7 as the objective physical sign assessment result.

[0006] Preferably, the dynamic micro-expression temporal characteristic data extracted in S2 includes: displacement trajectory data of multiple key facial points over time and temporal data of duration and fluctuation amplitude of various pain-related basic facial movement units; The key points include the core points in the eyebrow and eye area, the mouth and nose area, and the jaw area; The pain-related facial basic movement unit covers typical facial movement types that indicate discomfort in the human body, such as frowning, tightening of the eyebrows and eyes, downturned corners of the lips, and tense facial muscles.

[0007] Preferably, the facial skin color temporal characteristic data extracted in S2 includes lip color level classification data, facial skin brightness and luster temporal data, and facial local color difference distribution data.

[0008] Preferably, the micro-expression quantification process in S3 and the facial skin color quantification process in S4 share the same front-end video spatiotemporal feature extraction backbone structure, and complete the specific quantification of vital signs through the back-end branch structure.

[0009] Preferably, the construction of the pathological association knowledge graph in S5 and the feature alignment method in S6 are as follows: By integrating observational data of facial signs of daily physical and mental discomfort and statistical data on the external manifestations of human pain, we can sort out the relationships between entity categories and entities in terms of positive influence, negative constraints, and synchronous accompaniment. The weights of the association edges between different entities are determined based on the actual frequency of physical signs matching; each entity node in the pathological association knowledge graph is mapped to a low-dimensional embedding vector using a graph neural network to obtain the embedding vector of each entity node. The temporal change of the action unit obtained in S3 and the temporal quantization numerical sequence of skin color obtained in S4 are mapped to the unified semantic dimension of the exclusive pathological association knowledge graph built in S5 to obtain the feature vector to be fused. The semantic similarity between the feature vector to be fused and the corresponding entity node embedding vector in the pathological association knowledge graph is calculated using the following formula: Match( v , e )= α CosSim( v , e )+ β wedge ; In the formula, v The feature vector to be fused e Let e ​​be the embedding vector of the corresponding entity node in the knowledge graph, CosSim be the cosine similarity, and e be the embedding vector of the entity node in the knowledge graph. wedge For the weight of the associated edge, α and β All are preset balance coefficients; With the goal of maximizing semantic matching similarity, feature calibration and fusion are performed on the temporal change of action units and the temporal quantization numerical sequence of skin color. Irrelevant environmental light and shadow interference features are removed to obtain an integrated physical feature fusion representation after alignment and fusion.

[0010] Preferably, the temporal causal association analysis in S7 adopts the temporal residual comparison judgment method, which specifically means using a structural vector autoregressive model or a dynamic Bayesian network as the temporal causal model; The influence relationship between the temporal variation of action units and the temporal quantization numerical sequence of skin color is calculated using the Granger causality strength formula as follows: F X→Y =ln ; In the formula, X This represents the timing change of the action unit. Y This represents a time-series quantized numerical sequence of skin color. For use only Y Historical value prediction Y The residual, For simultaneous use X and Y Historical value prediction Y The residual.

[0011] Preferably, it also includes a cross-population lightweighting step for adapting the assessment method, specifically expressed as: Step 1: Collect facial dynamic signs samples from various age groups and physical conditions, and divide them into a general basic sample domain and a personalized fine-tuning sample domain. Step 2: Build an integrated basic evaluation model that integrates facial micro-expression feature extraction, facial skin color quantification, knowledge graph association and fusion, and temporal state inference. Temporal state inference includes temporal causal model operation. Step 3: Use the obtained general basic sample domain as training data to pre-train the built integrated basic assessment model on basic capabilities. Adopt a multi-task batch training mode to enable the integrated basic assessment model to master the basic correspondence between general facial temporal signs and physical and mental discomfort states, and obtain the pre-trained basic assessment model. Step 4: Collect a small number of short-term facial dynamic samples from a single individual to form a dedicated fine-tuning sample set. Use the pre-trained basic assessment model as the initial model and perform rapid fine-tuning with few parameters on the dedicated fine-tuning sample set to complete the adaptation of individual physical characteristics differences in the population. After obtaining the individual-adapted assessment model, directly perform facial temporal physical characteristic fusion analysis on the subsequent facial dynamic videos of the test subjects to output continuous physical and mental discomfort state trend assessment results.

[0012] Preferably, an environmental and individual difference elimination constraint mechanism is added during the basic ability pre-training in step three and the rapid fine-tuning in step four. Through the feature domain distribution alignment method, the evaluation bias caused by different shooting light, shooting angle and the basic facial features of the person himself is offset.

[0013] A non-diagnostic pain state assessment system based on facial temporal sign fusion includes a video acquisition unit, a pre-processing unit, an expression and sign quantification unit, a skin color quantification unit, a pathology knowledge graph storage and retrieval unit, an individual rapid adaptation unit, a temporal feature fusion unit, a state temporal deduction unit, and a result output unit; these are the steps used to implement a non-diagnostic pain state assessment method based on facial temporal sign fusion. The input of the preprocessing unit is connected to the output of the video acquisition unit, and the output of the preprocessing unit is connected to the input of the facial expression quantification unit and the input of the skin color quantification unit, respectively. The first input of the temporal feature fusion unit is connected to the output of the facial expression quantification unit, the second input of the temporal feature fusion unit is connected to the output of the skin color quantification unit, and the third input of the temporal feature fusion unit is connected to the output of the pathological knowledge graph storage and retrieval unit. The first input of the state-time sequence deduction unit is connected to the output of the facial expression quantification unit; the second input of the state-time sequence deduction unit is connected to the output of the skin color quantification unit; the third input of the state-time sequence deduction unit is connected to the output of the temporal feature fusion unit; and the output of the state-time sequence deduction unit is connected to the input of the result output unit. The input terminal of the individual fast adaptation is connected to the output terminal of the preprocessing unit; the output terminal of the individual fast adaptation is connected to the control terminal of the facial expression quantification unit, skin color quantification unit, temporal feature fusion unit, and state temporal inference unit, and the adapted model parameters are loaded into the above units.

[0014] Therefore, the present invention employs the above-mentioned non-diagnostic pain state assessment method and system based on facial temporal sign fusion, and the technical effects are as follows: (1) This invention integrates the dynamic features of facial micro-expressions and the static features of skin color, and combines prior constraints of knowledge graphs with temporal causal analysis to achieve the comprehensive utilization of multi-dimensional vital signs information.

[0015] (2) In the process of model pre-training and fine-tuning, the present invention adds an environmental and individual difference elimination constraint mechanism, and achieves feature domain distribution alignment through gradient inversion layer and maximum mean difference constraint.

[0016] (3) This invention introduces Granger causality test to quantify the temporal driving relationship between micro-expression changes and skin color changes, and to deduce the continuous evolution trend of the pain state. It can help medical staff judge the effect of analgesia treatment, optimize the timing of medication, and realize the whole process of tracking the patient's condition.

[0017] (4) All steps of the present invention are automatically executed by a computer device, and the output is an objective physical sign assessment index, which is used as an auxiliary reference for the degree of physical and mental discomfort and the trend of changes in the state.

[0018] (5) This invention automatically extracts the activation intensity of facial action units and skin color quantification parameters through a deep learning model, and outputs standardized and repeatable objective evaluation indicators, eliminating the subjective bias of manual interpretation. Attached Figure Description

[0019] Figure 1 This is a flowchart of the non-diagnostic pain state assessment method based on facial temporal sign fusion according to the present invention; Figure 2 This is a framework diagram of the non-diagnostic pain state assessment system based on facial temporal sign fusion according to the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0022] Example 1 like Figure 1 As shown, this invention provides a non-therapeutic pain state assessment method based on facial temporal sign fusion, implemented by a processor executing a computer program stored on a readable storage medium, specifically including the following steps: S1: Acquire dynamic video stream of the subject's frontal face within a preset continuous time window; use an RGB camera with a resolution of 1920×1080 and a frame rate of 30fps, with a shooting distance of 40-60cm and an ambient illuminance of 200-500 lux. The acquisition time is 30 seconds, with the subject maintaining a frontal posture and no obvious facial obstruction.

[0023] S2: Perform face localization, pose correction and pixel region segmentation on the dynamic facial video stream, and extract dynamic micro-expression temporal characteristic data and facial skin color temporal characteristic data respectively; In this embodiment, the MediaPipe Face Mesh model is used for face detection and key point localization, extracting 468 facial key points. An affine transformation is performed on each frame to align the face to a standard template (224×224 pixels). RetinaFace is used for facial region cropping, removing the background and hair areas. A sliding window sampling is performed with a step size of 16 frames (approximately 0.53 seconds) across 64 consecutive frames (approximately 2.13 seconds) to obtain dynamic micro-expression temporal data and facial skin color temporal data.

[0024] The dynamic micro-expression temporal data extracted from S2 includes: displacement trajectory data of multiple key facial points over time and temporal data of duration and fluctuation amplitude of various pain-related basic facial movement units. The key points include the core points in the eyebrow and eye area, the mouth and nose area, and the jaw area; The pain-related facial basic movement unit covers typical facial movement types that indicate discomfort in the human body, such as frowning, tightening of the eyebrows and eyes, downturned corners of the lips, and tense facial muscles.

[0025] The temporal data of facial skin color extracted from S2 include lip color grading data, temporal data of facial skin brightness and hydration, and local color difference distribution data of the face.

[0026] S3: The dynamic micro-expression temporal characteristic data extracted in S2 is used to perform temporal quantification of the micro-expression temporal characteristic data based on the preset facial action unit evaluation rules, so as to obtain the temporal change of human physiological pain-related action units. An improved 3D-ResNet18 architecture is used as the first neural network, which outputs a 20-dimensional temporal distribution of action unit activation intensity, including the activation probabilities of 12 action units (AU1, AU2, AU3, AU4, AU6, AU7, AU9, AU10, AU12, AU15, AU17, and AU20), as well as 8 geometric features including eyebrow spacing change, mouth corner offset, eye opening, nasal wing expansion, mandibular displacement, eyebrow height change, cheek lifting, and lip tension.

[0027] S4: The facial skin color temporal characteristic data extracted in S2 is used to complete the full-domain pixel statistics of facial skin color based on the CIE-Lab uniform color space. The skin color brightness, saturation and color difference distribution are quantified in multiple dimensions, and the skin color temporal quantization numerical sequence is output. An improved Inception-ResNet-V2 architecture is used as the second neural network, and an attention guidance module (AGPM) is introduced before the final fully connected layer. The facial skin region is segmented into 10 Regions of Interest (ROIs) (forehead, left cheek, right cheek, nose, chin, left periorbital area, right periorbital area, left nasolabial fold, right nasolabial fold, and perioral area). The L, a, and b values ​​of each ROI in the CIE-Lab color space are calculated, and the temporal quantization sequence of skin color is output.

[0028] The micro-expression quantification process in S3 and the facial skin color quantification process in S4 share the same front-end video spatiotemporal feature extraction backbone structure, and complete the specific quantification of vital signs through the back-end branch structure.

[0029] S5: Construct a dedicated pathological association knowledge graph that integrates the mechanism of facial movement units with the correlation logic of physical discomfort signs. The dedicated pathological association knowledge graph adopts a heterogeneous graph structure and includes the following entity node types and relationship types. The node types include 12 facial motion unit nodes, skin color representation parameter nodes (brightness, saturation, color difference), physical discomfort syndrome nodes (pain, fatigue, tension, etc.), and physical and mental state level nodes (no pain, mild, moderate, severe). The association weights between entity nodes are set according to the correspondence of clinical signs.

[0030] Relationship types include cause, accompany, manifest, relieve, and exacerbate.

[0031] S6: Map the temporal change of the action unit obtained in S3 and the temporal quantization numerical sequence of skin color obtained in S4 to the semantic space of the exclusive pathological association knowledge graph built in S5, complete feature alignment and fusion, and obtain an integrated physical sign fusion representation that takes into account dynamic expression changes and facial skin color status. The specific construction of the pathological association knowledge graph in S5 and the feature alignment method in S6 are as follows: By integrating observational data of facial signs of daily physical and mental discomfort and statistical data on the external manifestations of human pain, we can sort out the relationships between entity categories and entities in terms of positive influence, negative constraints, and synchronous accompaniment. The weights of the association edges between different entities are determined based on the actual frequency of physical signs matching; each entity node in the pathological association knowledge graph is mapped to a low-dimensional embedding vector with a dimension of 128 using the graph neural network R-GCN, thus obtaining the embedding vector of each entity node. The temporal change of the action unit obtained in S3 and the temporal quantization numerical sequence of skin color obtained in S4 are mapped to the unified semantic dimension of the exclusive pathological association knowledge graph built in S5 to obtain the feature vector to be fused. The semantic similarity between the feature vector to be fused and the corresponding entity node embedding vector in the pathological association knowledge graph is calculated using the following formula: Match( v , e )= α CosSim( v , e )+ β wedge ; In the formula, v The feature vector to be fused e Let e ​​be the embedding vector of the corresponding entity node in the knowledge graph, CosSim be the cosine similarity, and e be the embedding vector of the entity node in the knowledge graph. wedge For the weight of the associated edge, α and β All are preset balance coefficients; in this embodiment α =0.6, β =0.4.

[0032] With the goal of maximizing semantic matching similarity, feature calibration and fusion are performed on the temporal change of action units and the temporal quantization numerical sequence of skin color. Irrelevant environmental light and shadow interference features are removed to obtain an integrated physical feature fusion representation after alignment and fusion.

[0033] S7: Using the temporal change of the action unit obtained in S3 and the temporal quantitative numerical sequence of skin color obtained in S4 as input, a temporal causal correlation analysis mechanism is introduced to explore the temporal influence relationship between the two. Combined with the integrated physical sign fusion representation obtained in S6, the continuous evolution trend of the physical and mental discomfort state of the test subject is deduced. In S7, the temporal causal association analysis adopts the temporal residual comparison judgment method, which is specifically expressed as using a structural vector autoregressive model or a dynamic Bayesian network as the temporal causal model. In this embodiment, the Structure Vector Autoregression (SVAR) model is used as the time-series causal model: Y t = A k Y t-k + ε t ; In the formula, Y t Indicates time t The multidimensional vital sign observation vector, in this embodiment, represents the concatenation vector of the 20-dimensional action unit temporal change quantity and the 5-dimensional skin color temporal quantization numerical sequence; t This is a discrete-time index, corresponding to the temporal order of video frames. t =1,2,…, T ,in T This represents the total number of frames within the time window. Indicates the lag order; This represents a lagged index variable, with a value of =1,2,…, ; A k Indicates the first k The coefficient matrix of the first-order lag term; ε t Indicates time t The structural impact vector; Y t-k Indicates time t-k Multidimensional vital sign observation vector.

[0034] The influence relationship between the temporal variation of action units and the temporal quantization numerical sequence of skin color is calculated using the Granger causality strength formula as follows: F X→Y =ln ; In the formula, X This represents the timing change of the action unit. Y This represents a time-series quantized numerical sequence of skin color. For use only Y Historical value prediction Y The residual, For simultaneous use X and Y Historical value prediction Y The residual.

[0035] S8: Output the continuous evolution trend of physical and mental discomfort obtained in S7 as the objective physical sign assessment result. In this embodiment, the output content includes: the probability distribution of the current degree of discomfort, the state evolution trend graph, and key characteristic indicators. The assessment result serves as an auxiliary reference for the degree of physical and mental discomfort and the trend of state change, and does not form a disease diagnosis, symptom classification, or clinical treatment opinion.

[0036] It also includes a lightweight cross-population step for adapting assessment methods, specifically expressed as follows: Step 1: Collect facial dynamic signs samples from various age groups and physical conditions, and divide them into a general basic sample domain and a personalized fine-tuning sample domain. In this embodiment, data were collected from 2,500 subjects from 3 medical centers, distributed as follows: postoperative pain (40%), chronic low back pain (30%), and other types of pain (30%); age range: 18-85 years (mean 52.3 years); gender ratio: 48% male and 52% female. Each subject had 10 minutes of facial video recorded, and pain intensity (NRS score 0-10) was marked every 30 seconds.

[0037] Step 2: Build an integrated basic evaluation model that integrates facial micro-expression feature extraction (based on 3D-ResNet18), facial skin color quantification (based on Inception-ResNet-V2), knowledge graph association fusion (based on R-GCN), and temporal state inference (based on SVAR model). Temporal state inference includes temporal causal model operation. Step 3: In this embodiment, a general basic sample domain (2000 people, 80%) is used as training data to pre-train the integrated basic assessment model. A multi-task batch training mode is adopted, with each batch containing 8 tasks, and each task containing 5-20 support samples and 15 query samples. The inner loop learning rate is 0.01, the outer loop learning rate is 0.001, the total number of iterations is 50000, and the optimizer is Adam. The pre-trained basic assessment model is obtained.

[0038] Step 4: In this embodiment, a 5-10 second short-term facial dynamic sample is collected from a single individual to form a dedicated fine-tuning sample set (5-shot). The pre-trained basic evaluation model is used as the initial model, and 5 gradient updates (learning rate 0.001) are performed on the dedicated fine-tuning sample set to complete the adaptation to individual physiological differences in the population. After obtaining the individual-adapted evaluation model, the facial temporal physiological fusion analysis is directly performed on the subsequent facial dynamic videos of the individual to output a continuous assessment result of the trend of physical and mental discomfort.

[0039] In the basic ability pre-training in step three and the rapid fine-tuning in step four, an environmental and individual difference elimination constraint mechanism is added. Through the feature domain distribution alignment method, the evaluation bias caused by different shooting light, shooting angle and the basic facial features of the person is offset.

[0040] A non-diagnostic pain state assessment system based on facial temporal sign fusion includes a video acquisition unit, a pre-processing unit, an expression and sign quantification unit, a skin color quantification unit, a pathology knowledge graph storage and retrieval unit, an individual rapid adaptation unit, a temporal feature fusion unit, a state temporal deduction unit, and a result output unit; these are the steps used to implement a non-diagnostic pain state assessment method based on facial temporal sign fusion. like Figure 2 As shown, the video acquisition unit acquires a dynamic video stream of the frontal face of the person under test for a preset continuous duration and outputs the dynamic video stream of the frontal face. The input end of the preprocessing unit is connected to the output end of the video acquisition unit, and performs face localization, posture correction, and pixel region segmentation on the dynamic video stream of the frontal face, and outputs dynamic micro-expression temporal characteristic data and facial skin color temporal characteristic data respectively. The output end of the preprocessing unit is connected to the input end of the expression characteristic quantization unit and the input end of the skin color quantization unit respectively. The expression characteristic quantization unit performs temporal quantization of the dynamic micro-expression temporal characteristic data based on the preset facial action unit evaluation rules, and outputs the temporal change of the human physiological pain-related action unit. The skin color quantization unit performs full-domain pixel statistics of facial skin color based on the CIE-Lab uniform color space, and performs multi-dimensional quantization of skin color brightness, saturation, and color difference distribution, and outputs the skin color temporal quantization numerical sequence. The first input of the temporal feature fusion unit is connected to the output of the facial expression quantification unit, the second input of the temporal feature fusion unit is connected to the output of the skin color quantification unit, and the third input of the temporal feature fusion unit is connected to the output of the pathological knowledge graph storage and retrieval unit. The pathological knowledge graph storage and retrieval unit stores a preset exclusive pathological association knowledge graph. The entity nodes of the exclusive pathological association knowledge graph include: facial action units, skin color representation parameters, physical discomfort symptoms, and physical and mental state levels. The association edges between entities are set with association weights based on the correspondence of clinical signs, and the system outputs retrievable knowledge graph data. The first input of the state-time sequence deduction unit is connected to the output of the facial expression quantification unit, the second input of the state-time sequence deduction unit is connected to the output of the skin color quantification unit, and the third input of the state-time sequence deduction unit is connected to the output of the temporal feature fusion unit. The state-time sequence deduction unit takes the temporal change of the action unit and the temporal quantification numerical sequence of skin color as input, introduces a temporal causal correlation analysis mechanism, explores the temporal influence relationship between the two, and combines the integrated physical sign fusion representation to output the continuous evolution trend of the physical and mental discomfort state. The output of the state-time sequence deduction unit is connected to the input of the result output unit. The continuous evolution trend of the physical and mental discomfort state is output as the objective physical sign assessment result. The objective physical sign assessment result is only used as an auxiliary reference for the degree of physical and mental discomfort and the trend of state change, and does not form a disease diagnosis, symptom classification, or clinical treatment opinion.

[0041] The input of the individual fast adaptation is connected to the output of the preprocessing unit. The individual fast adaptation collects a small number of short-term facial dynamic samples of a single individual to be tested to form a dedicated fine-tuning sample set. The pre-trained basic evaluation model is used as the initial model, and a few parameters are quickly fine-tuned on the dedicated fine-tuning sample set to output the evaluation model after individual adaptation. The output of the individual fast adaptation is connected to the control of the facial expression quantification unit, skin color quantification unit, temporal feature fusion unit, and state temporal inference unit, and the adapted model parameters are loaded into the above units.

[0042] The non-diagnostic pain state assessment system based on facial temporal sign fusion can be integrated into any physical hardware carrier, such as civilian intelligent shooting equipment, home health monitoring terminals, outpatient auxiliary observation equipment, or portable physical and mental state screening equipment.

[0043] Validation was performed on a multicenter dataset containing 2000 participants, divided into a training set (1400 participants), a validation set (300 participants), and a test set (300 participants) at a ratio of 70%, 15%, and 15%, respectively. Data acquisition was performed using a 1920×1080 resolution, 30fps RGB camera at a shooting distance of 40-60cm and an ambient illumination of 200-500 lux. Each participant had 10 minutes of facial video recorded, and pain intensity (NRS score 0-10) was labeled every 30 seconds using a four-level pain classification system (no pain, mild, moderate, severe). The validation results for each module are shown in Table 1. Table 1

[0044] Therefore, the present invention adopts the above-mentioned non-diagnostic pain state assessment method and system based on facial temporal sign fusion. Through multimodal temporal sign fusion, knowledge graph prior constraints, temporal causal analysis and meta-learning rapid adaptation, it realizes objective, continuous and generalizable non-diagnostic pain state assessment.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A non-diagnostic pain state assessment method based on facial temporal sign fusion, characterized in that, The process involves a processor executing a computer program stored on a readable storage medium, specifically including the following steps: S1: Collect dynamic video stream of the frontal face of the person to be tested within a preset continuous time window; S2: Perform face localization, pose correction and pixel region segmentation on the dynamic facial video stream, and extract dynamic micro-expression temporal characteristic data and facial skin color temporal characteristic data respectively; S3: The dynamic micro-expression temporal characteristic data extracted in S2 is used to perform temporal quantification of the micro-expression temporal characteristic data based on the preset facial action unit evaluation rules, so as to obtain the temporal change of human physiological pain-related action units. S4: The facial skin color temporal characteristic data extracted in S2 is used to complete the full-domain pixel statistics of facial skin color based on the CIE-Lab uniform color space. The skin color brightness, saturation and color difference distribution are quantified in multiple dimensions, and the skin color temporal quantization numerical sequence is output. S5: Construct a dedicated pathological association knowledge graph that integrates the mechanism of facial motion units with the logic of somatic discomfort signs. The entity nodes of the dedicated pathological association knowledge graph include facial motion units, skin color representation parameters, somatic discomfort symptoms and physical and mental state levels. The association weights between entity nodes are set according to the correspondence of clinical signs. S6: Map the temporal change of the action unit obtained in S3 and the temporal quantization numerical sequence of skin color obtained in S4 to the semantic space of the exclusive pathological association knowledge graph built in S5, complete feature alignment and fusion, and obtain an integrated physical sign fusion representation that takes into account dynamic expression changes and facial skin color status. S7: Using the temporal change of the action unit obtained in S3 and the temporal quantitative numerical sequence of skin color obtained in S4 as input, a temporal causal correlation analysis mechanism is introduced to explore the temporal influence relationship between the two. Combined with the integrated physical sign fusion representation obtained in S6, the continuous evolution trend of the physical and mental discomfort state of the test subject is deduced. S8: Output the continuous evolution trend of physical and mental discomfort obtained in S7 as the objective physical sign assessment result.

2. The non-diagnostic pain state assessment method based on facial temporal sign fusion according to claim 1, characterized in that, The dynamic micro-expression temporal data extracted from S2 includes: displacement trajectory data of multiple key facial points over time and temporal data of duration and fluctuation amplitude of various pain-related basic facial movement units. The key points include the core points in the eyebrow and eye area, the mouth and nose area, and the jaw area; The pain-related facial basic movement unit covers typical facial movement types that indicate discomfort in the human body, such as frowning, tightening of the eyebrows and eyes, downturned corners of the lips, and tense facial muscles.

3. The non-diagnostic pain state assessment method based on facial temporal sign fusion according to claim 1, characterized in that, The temporal data of facial skin color extracted from S2 include lip color grading data, temporal data of facial skin brightness and hydration, and data on local color difference distribution on the face.

4. The non-diagnostic pain state assessment method based on facial temporal sign fusion according to claim 1, characterized in that, The micro-expression quantification process in S3 and the facial skin color quantification process in S4 share the same front-end video spatiotemporal feature extraction backbone structure, and complete the specific quantification of vital signs through the back-end branch structure.

5. The non-diagnostic pain state assessment method based on facial temporal sign fusion according to claim 1, characterized in that, The specific details of the pathological association knowledge graph construction in S5 and the feature alignment method in S6 are as follows: By integrating observational data of facial signs of daily physical and mental discomfort and statistical data on the external manifestations of human pain, we can sort out the relationships between entity categories and entities in terms of positive influence, negative constraints, and synchronous accompaniment. The weights of the association edges between different entities are determined based on the actual frequency of physical signs matching; each entity node in the pathological association knowledge graph is mapped to a low-dimensional embedding vector using a graph neural network to obtain the embedding vector of each entity node. The temporal change of the action unit obtained in S3 and the temporal quantization numerical sequence of skin color obtained in S4 are mapped to the unified semantic dimension of the exclusive pathological association knowledge graph built in S5 to obtain the feature vector to be fused. The semantic similarity between the feature vector to be fused and the corresponding entity node embedding vector in the pathological association knowledge graph is calculated using the following formula: Match( v , e )= α CosSim( v , e )+ β wedge ; In the formula, v The feature vector to be fused e Let e ​​be the embedding vector of the corresponding entity node in the knowledge graph, CosSim be the cosine similarity, and e be the embedding vector of the entity node in the knowledge graph. wedge For the weight of the associated edge, α and β All are preset balance coefficients; With the goal of maximizing semantic matching similarity, feature calibration and fusion are performed on the temporal change of action units and the temporal quantization numerical sequence of skin color. Irrelevant environmental light and shadow interference features are removed to obtain an integrated physical feature fusion representation after alignment and fusion.

6. The non-diagnostic pain state assessment method based on facial temporal sign fusion according to claim 1, characterized in that: In S7, the temporal causal association analysis adopts the temporal residual comparison judgment method, which is specifically expressed as using a structural vector autoregressive model or a dynamic Bayesian network as the temporal causal model. The influence relationship between the temporal variation of action units and the temporal quantization numerical sequence of skin color is calculated using the Granger causality strength formula as follows: F X→Y =ln ； In the formula, X This represents the timing change of the action unit. Y This represents a time-series quantized numerical sequence of skin color. For use only Y Historical value prediction Y The residual, For simultaneous use X and Y Historical value prediction Y The residual.

7. The non-diagnostic pain state assessment method based on facial temporal sign fusion according to claim 1, characterized in that: It also includes a lightweight cross-population step for adapting assessment methods, specifically expressed as follows: Step 1: Collect facial dynamic signs samples from various age groups and physical conditions, and divide them into a general basic sample domain and a personalized fine-tuning sample domain. Step 2: Build an integrated basic evaluation model that integrates facial micro-expression feature extraction, facial skin color quantification, knowledge graph association and fusion, and temporal state inference. Temporal state inference includes temporal causal model operation. Step 3: Use the obtained general basic sample domain as training data to pre-train the built integrated basic assessment model on basic capabilities. Adopt a multi-task batch training mode to enable the integrated basic assessment model to master the basic correspondence between general facial temporal signs and physical and mental discomfort states, and obtain the pre-trained basic assessment model. Step 4: Collect a small number of short-term facial dynamic samples from a single individual to form a dedicated fine-tuning sample set. Use the pre-trained basic assessment model as the initial model and perform rapid fine-tuning with few parameters on the dedicated fine-tuning sample set to complete the adaptation of individual physical characteristics differences in the population. After obtaining the individual-adapted assessment model, directly perform facial temporal physical characteristic fusion analysis on the subsequent facial dynamic videos of the test subjects to output continuous physical and mental discomfort state trend assessment results.

8. The non-diagnostic pain state assessment method based on facial temporal sign fusion according to claim 7, characterized in that: In the basic ability pre-training in step three and the rapid fine-tuning in step four, an environmental and individual difference elimination constraint mechanism is added. Through the feature domain distribution alignment method, the evaluation bias caused by different shooting light, shooting angle and the basic facial features of the person is offset.

9. A non-diagnostic pain state assessment system based on facial temporal sign fusion, characterized in that it includes a video acquisition unit, a pre-processing unit, an expression sign quantification unit, a skin color quantification unit, a pathological knowledge graph storage and retrieval unit, an individual rapid adaptation unit, a temporal feature fusion unit, a state temporal deduction unit, and a result output unit; used to implement the steps of the non-diagnostic pain state assessment method based on facial temporal sign fusion as described in any one of claims 1-8; The input of the preprocessing unit is connected to the output of the video acquisition unit, and the output of the preprocessing unit is connected to the input of the facial expression quantification unit and the input of the skin color quantification unit, respectively. The first input of the temporal feature fusion unit is connected to the output of the facial expression quantification unit, the second input of the temporal feature fusion unit is connected to the output of the skin color quantification unit, and the third input of the temporal feature fusion unit is connected to the output of the pathological knowledge graph storage and retrieval unit. The first input of the state-time sequence deduction unit is connected to the output of the facial expression quantification unit; the second input of the state-time sequence deduction unit is connected to the output of the skin color quantification unit; the third input of the state-time sequence deduction unit is connected to the output of the temporal feature fusion unit; and the output of the state-time sequence deduction unit is connected to the input of the result output unit. The input terminal of the individual fast adaptation is connected to the output terminal of the preprocessing unit; the output terminal of the individual fast adaptation is connected to the control terminal of the facial expression quantification unit, skin color quantification unit, temporal feature fusion unit, and state temporal inference unit, and the adapted model parameters are loaded into the above units.

Images