Facial image-based pain assessment method and system

Abstract

The application relates to the technical field of computer vision, artificial intelligence and medical information processing, in particular to a pain assessment method and system based on facial images; the method comprises the following steps: acquiring an original facial video stream, performing position correction and scale normalization based on facial key points to obtain a standardized facial image sequence; performing occlusion detection and action unit region feature extraction to obtain a visible proportion and an original feature vector; determining a region validity coefficient according to boundary optical flow difference, edge gradient fluctuation and the visible proportion, and performing adjacent area compensation on a low-validity region; inputting a region feature to be processed into a feature decoupling model to output pain representation features and occlusion interference features and regress to obtain an instantaneous pain score; performing weighted smoothing according to the overall validity coefficient to output a continuous pain score sequence; and the application can improve the continuity and stability of pain scoring under a dynamic occlusion scenario.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence medical information processing technology, specifically to a pain assessment method and system based on facial images. Background Technology

[0002] Pain assessment based on facial images refers to the quantitative analysis of the pain level of subjects based on the changes in facial expressions presented in acquired facial images or videos. Current methods for pain assessment based on facial images include analysis methods based on changes in facial key points, recognition methods based on facial texture features, and regression assessment methods based on deep learning models.

[0003] However, when using existing technologies for pain assessment, the representation of key facial areas is easily distorted when there are local obstructions such as masks, catheters, or tape, as well as changes in head posture, resulting in inaccurate pain feature extraction. Existing methods often have difficulty effectively distinguishing between real pain expression changes and interference factors such as slippage of obstruction edges and local brightness fluctuations. Furthermore, the output results are prone to jitter during continuous video scoring, all of which reduce the continuity, stability, and accuracy of pain assessment. Summary of the Invention

[0004] The purpose of this invention is to provide a pain assessment method and system based on facial images, avoiding the problem of inaccurate pain feature extraction caused by partial occlusion and changes in head posture, and effectively distinguishing between real facial expression changes and occlusion interference, thereby improving the continuity, stability, and accuracy of pain assessment; specifically, the technical solution of this invention is as follows:

[0005] The pain assessment method based on facial images includes: acquiring raw facial video stream data, and performing position correction and scale normalization processing on the raw facial video stream data based on facial key points to obtain a standardized facial image sequence.

[0006] Occlusion region detection and feature extraction are performed on standardized facial image sequences to obtain the visible proportion and original feature vector of the action unit region;

[0007] The artifact intensity is determined based on the difference in boundary optical flow and edge gradient fluctuations in the action unit region, and the region effectiveness coefficient is determined in combination with the visible ratio.

[0008] When the region effectiveness coefficient is lower than the preset reference value, the original feature vector is locally compensated based on the action unit region association relationship established based on the facial muscle coordination relationship and the neighbor region transmission coefficient determined based on the region distance, linkage parameters and region effectiveness coefficient, so as to obtain the compensated region features.

[0009] When the regional effectiveness coefficient is not lower than the preset reference value, the original feature vector is used as the feature of the region to be processed.

[0010] The features of the region to be processed are input into a pre-trained feature decoupling model, which outputs pain representation features and occlusion interference features.

[0011] Regression calculations are performed based on pain representation features to obtain an instantaneous pain score for the current frame;

[0012] The overall effectiveness coefficient is determined based on the regional effectiveness coefficient of each action unit region. The instantaneous pain score of the current frame and the output score of the previous moment are then weighted and smoothed based on the overall effectiveness coefficient to obtain the current moment score, and a continuous pain score sequence is output.

[0013] Optionally, occlusion region detection and feature extraction are performed on the standardized facial image sequence to obtain the visible proportion and original feature vector of the action unit region, including:

[0014] Occlusion detection is performed on standardized facial image sequences to generate pixel-level occlusion masks;

[0015] The standardized facial image sequence is divided into multiple action unit regions, and the occluded pixel area of ​​each action unit region is determined according to the pixel-level occlusion mask; the visible ratio of the action unit region is determined according to the occluded pixel area and the total pixel area of ​​the region.

[0016] Local texture change features, local displacement increment features based on keypoint displacement, and short-term difference features based on grayscale differences between adjacent frames are extracted from the action unit region. The extracted features are then concatenated to obtain the original feature vector.

[0017] Optionally, the artifact intensity is determined based on the difference in boundary optical flow and edge gradient fluctuations of the action unit region, and the region effectiveness coefficient is determined in conjunction with the visible proportion, including:

[0018] A boundary detection band is formed by extending a preset pixel width inward and outward from the boundary of each action unit region, and the local optical flow difference and edge gradient fluctuation of adjacent frames within the boundary detection band are extracted.

[0019] The artifact intensity is determined based on local optical flow differences and edge gradient fluctuations.

[0020] The regional effectiveness coefficient is determined based on the visibility ratio and artifact intensity.

[0021] Optionally, based on the action unit region association relationship established based on facial muscle coordination and the neighbor region transfer coefficient determined based on region distance, linkage parameters, and region effectiveness coefficient, local feature compensation is performed on the original feature vector to obtain compensated region features, including:

[0022] The relative distance change between adjacent action unit regions is obtained, the average pain deformation reference value obtained in advance based on training samples is obtained, and the relative deformation deviation between the relative distance change and the average pain deformation reference value is calculated.

[0023] Based on the adjacency relationship of the action unit region association, the muscle linkage parameters obtained from pre-training, the regional effectiveness coefficient of the adjacent region, and the relative deformation deviation, the transfer weight of each adjacent region is normalized to obtain the neighbor region transfer coefficient.

[0024] The original feature vectors of adjacent regions are weighted and summed according to the neighbor region transfer coefficient, and then linearly fused with the original feature vector of the current region according to the predetermined fusion weight to obtain the compensated regional features.

[0025] Optionally, after obtaining the compensated region features, the method further includes:

[0026] Historical frames whose regional effectiveness coefficient is not lower than the preset reference value are identified as historical stable frames.

[0027] Calculate the compensation deviation between the compensated regional features and the mean value of the corresponding features in the historical stable frames; when the compensation deviation exceeds the preset deviation threshold, reduce the feature weight of the compensated regional features when inputting them into the feature decoupling model.

[0028] Optionally, the training steps of the pre-trained feature decoupling model include:

[0029] Acquire synchronously collected multi-source physiological signal data and extract physiological reference features;

[0030] Align the pain representation features output by the feature decoupling model to be trained with the physiological reference features to construct alignment constraints;

[0031] An orthogonal constraint error is constructed based on the pain representation features and occlusion interference features output by the feature decoupling model;

[0032] A pain score supervision error is constructed based on the real pain score labels corresponding to the training samples;

[0033] Based on alignment constraints, orthogonal constraint errors, and pain score supervision errors, the feature decoupling model is jointly trained to obtain the trained feature decoupling model.

[0034] Optionally, the overall effectiveness coefficient is determined based on the regional effectiveness coefficient of each action unit region, and the instantaneous pain score of the current frame and the output score of the previous moment are weighted and smoothed based on the overall effectiveness coefficient to obtain the current moment score, including:

[0035] The overall effectiveness coefficient is obtained by averaging the regional effectiveness coefficients of all action unit regions.

[0036] When the overall effectiveness coefficient is greater than or equal to the preset overall effectiveness reference value, the time smoothing weight is set to the preset upper limit weight.

[0037] When the overall effectiveness coefficient is less than the preset overall effectiveness reference value, the time smoothing weight is adaptively determined based on the overall effectiveness coefficient.

[0038] For non-first frames, the instantaneous pain score of the current frame and the output score of the previous moment are weighted and combined according to the time smoothing weight to obtain the score of the current moment;

[0039] For the first frame, the instantaneous pain score of the current frame is used as the first frame score in the continuous pain score sequence.

[0040] The pain assessment system based on facial images includes: a data acquisition and standardization module, an occlusion detection and extraction module, an effectiveness calculation module, a feature compensation module, a feature processing module, a score regression module, and a time smoothing and output module.

[0041] Each module is implemented by the processor executing program instructions from memory and is connected in communication.

[0042] The data acquisition and standardization module is used to acquire raw facial video stream data and perform position correction and scale normalization on the raw facial video stream data based on facial key points to obtain a standardized facial image sequence.

[0043] The occlusion detection and extraction module is used to detect occlusion regions and extract features from standardized facial image sequences to obtain the visible proportion and original feature vector of the action unit region.

[0044] The effectiveness calculation module is used to determine the artifact intensity based on the difference in boundary optical flow and edge gradient fluctuation of the action unit region, and to determine the regional effectiveness coefficient in combination with the visible ratio. It is also used to determine the overall effectiveness coefficient based on the regional effectiveness coefficient of each action unit region.

[0045] The feature compensation module is used to perform local feature compensation on the original feature vector when the region effectiveness coefficient is lower than the preset reference value, based on the action unit region association relationship established based on the facial muscle coordination relationship and the neighbor region transmission coefficient determined based on the region distance, linkage parameters and region effectiveness coefficient, to obtain the compensated region features.

[0046] When the regional effectiveness coefficient is not lower than the preset reference value, the original feature vector is used as the feature of the region to be processed;

[0047] The feature processing module is used to input the features of the region to be processed into a pre-trained feature decoupling model and output pain representation features and occlusion interference features.

[0048] The score regression module is used to perform regression calculations based on pain representation features to obtain the instantaneous pain score for the current frame;

[0049] The time smoothing and output module is used to perform weighted smoothing on the instantaneous pain score of the current frame and the output score of the previous moment based on the overall effectiveness coefficient, so as to obtain the current moment score and output a continuous pain score sequence.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. This invention establishes a unified standardized facial image sequence by acquiring the original facial video stream and performing position correction and scale normalization processing based on facial key points. This can effectively reduce the interference of head translation, rotation and distance changes on the observation of weak pain expressions, thereby improving the recognition stability of pain-related action unit regions.

[0052] 2. This invention achieves refined quantification of the visible proportion and original feature vector of local key regions by performing pixel-level occlusion detection, action unit region segmentation, and joint extraction of local texture change features, key point displacement increment features, and short-term difference features on standardized facial image sequences. This can avoid local misjudgment caused by rough judgment of the whole face and enhance the ability to perceive subtle pain movements.

[0053] 3. This invention determines artifact intensity by combining the difference in optical flow at the boundary of the action unit region and the fluctuation of the edge gradient, and further constructs a region effectiveness coefficient in conjunction with the visible ratio. This can effectively distinguish between real pain expression changes and interference factors such as mask edge slippage, catheter accessory movement, tape traction, and local brightness fluctuations, thereby improving the accuracy of feature credibility determination under occlusion scenarios. Attached Figure Description

[0054] Figure 1 A schematic flowchart illustrating the pain assessment method based on facial images provided in this application embodiment;

[0055] Figure 2 This is a schematic diagram of the modules of a pain assessment system based on facial images provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0057] Please see Figure 1 A pain assessment method based on facial images includes: acquiring raw facial video stream data, and performing position correction and scale normalization processing on the raw facial video stream data based on facial key points to obtain a standardized facial image sequence.

[0058] Occlusion region detection and feature extraction are performed on standardized facial image sequences to obtain the visible proportion and original feature vector of the action unit region; the artifact intensity is determined based on the boundary optical flow difference and edge gradient fluctuation of the action unit region, and the region effectiveness coefficient is determined in combination with the visible proportion.

[0059] If the region effectiveness coefficient is lower than the preset reference value, then the original feature vector is locally compensated based on the action unit region association relationship established based on the facial muscle coordination relationship and the neighbor region transfer coefficient determined based on the region distance, linkage parameters and region effectiveness coefficient, so as to obtain the compensated region features.

[0060] If the region validity coefficient is not lower than the preset reference value, the original feature vector is used as the feature of the region to be processed; the feature of the region to be processed is input into the pre-trained feature decoupling model, and the pain representation feature and occlusion interference feature are output.

[0061] Regression calculation is performed based on pain representation features to obtain the instantaneous pain score for the current frame; the overall effectiveness coefficient is determined based on the regional effectiveness coefficient of each action unit region, and the instantaneous pain score of the current frame and the output score of the previous moment are weighted and smoothed based on the overall effectiveness coefficient to obtain the score at the current moment, and a continuous pain score sequence is output.

[0062] Occlusion region detection and feature extraction are performed on standardized facial image sequences to obtain the visible proportion and original feature vector of the action unit region, including: occlusion detection of standardized facial image sequences and generation of pixel-level occlusion mask;

[0063] The standardized facial image sequence is divided into multiple action unit regions, and the occluded pixel area of ​​each action unit region is determined according to the pixel-level occlusion mask; the visible ratio of the action unit region is determined according to the occluded pixel area and the total pixel area of ​​the region.

[0064] Local texture change features, local displacement increment features based on keypoint displacement, and short-term difference features based on grayscale differences between adjacent frames are extracted from the action unit region. The extracted features are then concatenated to obtain the original feature vector.

[0065] The artifact intensity is determined based on the boundary optical flow difference and edge gradient fluctuation of the action unit region, and the region effectiveness coefficient is determined in combination with the visible ratio. This includes: expanding the boundary of each action unit region outward and inward by a preset pixel width to form a boundary detection band, and extracting the local optical flow difference and edge gradient fluctuation of adjacent frames within the boundary detection band; determining the artifact intensity based on the local optical flow difference and edge gradient fluctuation; and determining the region effectiveness coefficient based on the visible ratio and artifact intensity.

[0066] This embodiment provides a pain assessment method based on facial images. The method takes the original facial video stream as input, first completes the alignment and scale unification of facial key points, and then identifies the occluded area and extracts the action unit region features on the standardized image.

[0067] The region effectiveness coefficient, determined by the visible proportion and boundary artifacts, is calculated for each action unit region. Neighbor compensation is performed on low-effective regions at the feature layer. Then, the features of the region to be processed are input into a pre-trained feature decoupling model to obtain pain representation features and occlusion interference features. The instantaneous pain score of the current frame is obtained by regression based on these features. The instantaneous pain score is smoothed over time based on the overall effectiveness coefficient, and a continuous pain score sequence is output.

[0068] The purpose of this processing link is to maintain the continuity and stability of pain score output and reduce misjudgment caused by occlusion slippage in scenarios with dynamic occlusion such as breathing masks, catheter fixation devices, and patches.

[0069] In practice, the raw facial video stream can be periodically written to the processor's local frame buffer by the image acquisition unit. The frame buffer stores image matrix objects arranged in the acquisition order.

[0070] After the processor reads the current period image matrix, it establishes the corresponding key point coordinate array and region feature tensor in memory. The read frame object can be released after subsequent processing, thus avoiding long-term occupation of storage space. Instead of using a floating module direct transmission method between subsequent steps, they are sequentially unfolded around the standardized image matrix, occlusion mask matrix, region feature vector array and score result cache within the same processing cycle.

[0071] The raw facial video stream data is acquired, and the position correction and scale normalization processing are performed on the raw facial video stream data based on facial key points to generate a standardized facial image sequence. Among them, facial key points refer to the set of positioning points that can characterize the position of the eyebrows, eye area, nose, mouth area and jaw contour, such as the eyebrow peak, eye corner, nose wing, mouth corner and jaw contour turning point. Their role is not simply to provide geometric coordinates, but to provide a reference benchmark for the unified division of subsequent action unit areas.

[0072] Standardization can be achieved by using affine alignment, which allows the main facial structures in different frames to fall within a unified coordinate range, thereby reducing the interference of head translation, rotation, and changes in distance on weak expression recognition.

[0073] This implementation method first establishes a unified observation benchmark through key point constraints, which can separate posture changes from actual muscle deformation as much as possible, and avoids the subtle pain movements being drowned out by large head movements or changes in perspective.

[0074] Furthermore, based on the above embodiments, occlusion region detection and feature extraction are performed on the standardized facial image sequence; for ease of explanation, let the standardized current processing frame be . ,in Indicates the processing cycle index;

[0075] After performing occlusion detection on the frame, a pixel-level occlusion mask is generated. Its pixel position The value at point is denoted as ,in This indicates that the corresponding pixel is occluded. This indicates that the corresponding pixel is visible; the mask matrix is ​​stored in memory as a two-dimensional array of the same size as the image matrix, and serves as the direct basis for subsequent visible ratio calculation; a standardized facial image sequence is obtained, and based on the pixel-level occlusion mask, the standardized facial image sequence is divided into action unit regions and the visible ratio is calculated to generate the visible ratio of each action unit region;

[0076] Among them, the action unit area refers to the observation area set up in advance around the pain-related facial muscle activities, such as the eyebrow lifting area, periorbital contraction area, nasal wing traction area, upper lip lifting area, corner of mouth pressing area, and jaw tension area. The number of areas can be twelve to twenty, preferably seventeen, in order to take into account the details of muscle coordination and the computational burden of edge devices.

[0077] Let the first Each action unit area is The total pixel area of ​​its region is The area of ​​the obscured pixels is The visible ratio of that area It can be represented as:

[0078]

[0079] in, This indicates the percentage of the area in the action unit region that can be reliably observed within the current cycle. When this value is low, it means that the area is largely covered by the edge of the mask, fixing tape, or catheter accessories, and directly using the original observation of this area can easily lead to incorrect judgments.

[0080] The visible ratio threshold can be set between 0.35 and 0.45 as a reference for judging severe defects. This implementation refines the occlusion to the action unit area scale, which can distinguish the difference between the whole face being generally visible but the local key areas are distorted and the local area is still usable, transforming the whole image masking into precise area filtering.

[0081] Obtain a standardized facial image sequence, and extract local features from each action unit region according to the action unit region definition to generate the original feature vector;

[0082] The original feature vector refers to the region representation after combining local texture changes, key point displacement increments and short-term grayscale changes in adjacent frames within a single region. For example, the enhancement of fine wrinkles in the eyebrows, displacement compression caused by eyelid tightening, and short-term brightness and darkness fluctuations around the mouth. Its purpose is to simultaneously preserve information in three aspects: texture, geometry and short-term dynamics.

[0083] Let the first The action unit region in the first Of the three types of features extracted from the frame, local texture response features are extracted to characterize the subtle texture changes produced by the contraction of surface muscles in the region. ;

[0084] Extract local displacement increment features to reflect changes in the direction and amplitude of facial tissue contraction within this region. Extracting short-time difference features for capturing transient facial expression perturbations of short duration. In specific implementation, for local texture change features The texture histogram vector of the pixels in this region is extracted by using the local binary mode operator or the directional gradient histogram algorithm.

[0085] For local displacement increment characteristics Select several preset key points within the region, calculate the difference between their horizontal and vertical coordinates between the current frame and the neutral reference frame, and arrange them sequentially to form a displacement vector;

[0086] For short time difference features The region is divided into several equally sized sub-blocks. The average absolute difference in grayscale between the current frame and the previous frame is calculated for each sub-block to form a difference vector. These three sets of vectors are then concatenated along the feature dimension to construct the original feature vector of the corresponding region with a fixed dimension. ;

[0087] The three types of features can be written into the feature array of the corresponding region in the same processing cycle, and form a fixed-dimensional vector for subsequent steps to call;

[0088] This implementation describes the state of the action unit by coordinating changes of multiple local features, which can reduce the impact of single texture flicker or local lighting fluctuations on the judgment result.

[0089] Furthermore, based on the above embodiments, the artifact intensity is determined according to the difference in boundary optical flow and edge gradient fluctuation of the action unit region, and the region effectiveness coefficient is determined in combination with the visible ratio;

[0090] The purpose of this step is to identify situations where the surface of the region appears visible but has been obscured or contaminated, thus providing a basis for subsequent compensation; obtain the boundary of the action unit region, and expand the boundary of the action unit region inward and outward according to the preset pixel width to generate a boundary detection band;

[0091] The boundary detection band refers to a narrow band formed around the region's outline, such as a ring-shaped region formed by extending inward and outward by several pixels. Its function is to focus on observing whether the sliding of the occlusion edge is consistent with the muscle movement inside the region. The preset pixel width is used to control the coverage of the boundary band, and its value can be a constant at several pixels, as long as it can cover the occlusion contact edge and the transition area inside the region. The boundary detection band is not an additional independent image, but a set of local pixels dynamically indexed on the standardized image matrix and the region coordinate array.

[0092] Information of adjacent frames within the boundary detection band is obtained. Based on local optical flow differences and edge gradient fluctuations, artifact analysis is performed on the boundary detection band to determine the artifact intensity.

[0093] Among them, the local optical flow difference reflects the degree of inconsistency between the pixel movement direction of the boundary zone and the average movement direction inside the region; the edge gradient fluctuation reflects the degree of abrupt change in the brightness of the occluded edge, such as the high gradient jump caused by the movement of the mask edge, and the alternation of brightness caused by the stretching of the tape boundary.

[0094] Let the first The boundary detection band corresponding to each action unit region is The boundary pixel position is Pixels The local optical flow vector in the current period is The average optical flow vector at non-boundary locations within the region is Pixels The gradient magnitudes in the current cycle and the adjacent previous cycle are respectively and Then the artifact intensity It can be represented as:

[0095]

[0096] Among them, the weighting coefficients used to balance the contribution of boundary motion deviation to artifact judgment. And weighting coefficients used to balance the contribution of edge brightness transitions to artifact detection. Both satisfy ;

[0097] It should be noted that, since the local optical flow difference, which is measured in terms of pixel displacement or velocity, and the edge gradient fluctuation, which is measured in terms of gray level difference, have significant differences in physical dimensions and numerical distribution range, directly applying linear weighting will result in the larger value of the term dominating the artifact intensity calculation.

[0098] Therefore, before actually substituting into the above formula for calculation, the extracted... and All are processed using a preset extreme value or historical verification window for maximum and minimum normalization, and mapped to... The dimensionless interval ensures that the weighting coefficients can truly reflect the contribution ratio of the two factors;

[0099] When the mask or tubing accessories slip relative to each other, the movement trend in the boundary zone is usually inconsistent with the muscle contraction trend within the region, and the edge gradient fluctuates significantly. It will increase;

[0100] In actual muscle activity, the movement between the boundary zone and the interior of the region is more continuous. Relatively small;

[0101] This implementation quantifies boundary slip, a common disturbance in clinical scenarios, separately, and can distinguish visible but unreliable areas from ordinary visible areas;

[0102] The visibility ratio and artifact intensity of the action unit region are obtained. Based on the relationship between visibility ratio and artifact penalty, the effectiveness of the action unit region is evaluated, and the region effectiveness coefficient is determined.

[0103] Among them, the regional effectiveness coefficient refers to the reliability of the current action unit region for pain determination within the current processing cycle; let the first... The regional effectiveness coefficient of each action unit region is: It can be in the following form:

[0104]

[0105] Among them, the artifact penalty coefficient Used to adjust the impact of boundary artifacts on the rate at which the credibility of the region decreases; Adjustments can be made through verification data; specific calibration can be performed... The value range is set to By comparing the variance of scores under different artifact intensities on the validation set, a critical value that causes the regional effectiveness coefficient to decrease by more than 50% under typical occlusion and slip conditions is selected as the final parameter, so that the contamination caused by boundary slip is suppressed first, while not excessively suppressing the contribution of real weak expression regions.

[0106] The smaller the coefficient, the more likely the observation within the region is to be contaminated by occlusion artifacts, even if the region still has a certain visible area. For subsequent compensation processing, a preset reference value can be set as a judgment threshold. The preferred range of this preset reference value is 0.5 to 0.6, which can be quantitatively determined by shifting the mean of the regional effectiveness coefficient of unoccluded action units downward by one standard deviation in historical statistics. When the value is below this threshold, the area will be used for compensation.

[0107] In this embodiment, after obtaining the regional effectiveness coefficient of each action unit region, regions with regional effectiveness coefficients lower than the preset reference value can enter the neighbor feature compensation process; regions with regional effectiveness coefficients not lower than the preset reference value can directly use the original feature vector as the feature of the region to be processed; the feature of the region to be processed is written into the local feature tensor and input into the pre-trained feature decoupling model, and outputs pain representation features and occlusion interference features.

[0108] The instantaneous pain score of the current frame is then obtained by regression calculation, and weighted and smoothed with the previous output score in the score buffer area to finally form a continuous pain score sequence.

[0109] The continuous pain score sequence can be further encapsulated into structured monitoring results and written into the monitoring interface cache for the upper monitoring terminal to read and display the current continuous score results and low confidence warning markers;

[0110] In terms of comparative analysis, for continuous scoring scenarios of medical images, the recognized difficulty is not whether a single frame of face is successfully detected, but whether the score remains stable when dynamic occlusion, low-amplitude facial expressions and local high-frequency perturbations coexist.

[0111] This implementation method uses the action unit region as the basic processing unit, quantifies the visible area, then quantifies the boundary artifacts, and then decides whether to perform subsequent compensation. Compared with the traditional direct modeling of the whole face, it is less likely to produce single-frame peak scores under the conditions of repeated sliding of occlusion edges and frequent changes in local instrument contact.

[0112] Especially in pain-sensitive areas such as around the eyes, nose, and mouth, this implementation can preferentially suppress pseudo-high-frequency changes caused by the edge of the device, preserving an input basis that is closer to real muscle activity.

[0113] In this application example, the method can be deployed in a ward-side processing terminal; the image acquisition unit continuously outputs facial video frames, and the processor reads the image matrix from the local frame buffer according to the processing cycle and performs the above-mentioned standardization, occlusion detection, feature extraction and effectiveness evaluation;

[0114] The generated features of the region to be processed are formed into a region feature tensor in local memory and then fed into the feature decoupling model; the output pain score results and the confidence-related parameters are encapsulated into structured data and written into the result cache area corresponding to the monitoring display interface; the monitoring terminal polls the result cache area and updates the continuous pain score curve in the interface, and triggers a low confidence prompt when the overall effectiveness is low, thereby forming a practical and usable continuous pain assessment process;

[0115] In some possible implementations, the original feature vector is locally compensated based on the relationship between the action unit regions established based on the synergistic relationship of facial muscles and the neighbor transfer coefficient determined based on the region distance, linkage parameters and region effectiveness coefficient, to obtain the compensated region features, including: obtaining the relative distance change between adjacent action unit regions, obtaining the average pain deformation reference value obtained in advance based on training samples, and calculating the relative deformation deviation between the relative distance change and the average pain deformation reference value.

[0116] Based on the adjacency relationship of the action unit region association, the muscle linkage parameters obtained from pre-training, the regional effectiveness coefficient of the adjacent region, and the relative deformation deviation, the transfer weight of each adjacent region is normalized to obtain the neighbor region transfer coefficient; the original feature vector of the adjacent region is weighted and summed according to the neighbor region transfer coefficient, and linearly fused with the original feature vector of the current region according to the predetermined fusion weight to obtain the compensated region features.

[0117] After obtaining the compensated regional features, the method further includes: determining historical frames with a regional effectiveness coefficient not lower than a preset reference value as historical stable frames; and calculating the compensation deviation between the compensated regional features and the mean value of the corresponding features of the current region in the historical stable frames.

[0118] If the compensation deviation exceeds the preset deviation threshold, the feature weights of the compensated region feature input feature decoupling model will be reduced.

[0119] This embodiment is used to illustrate the local feature compensation process of low-efficiency action unit regions and the constraint method after compensation; the input of this embodiment is the original feature vector of each action unit region, the region effectiveness coefficient, the action unit region correlation relationship, and the muscle linkage parameters and pain average deformation reference value formed during the training phase;

[0120] The processing involves compensating for low-efficiency regions with neighboring information within the feature space, and constraining the compensation results using historical stable frames. The output is the compensated region features and their adjusted input weights.

[0121] The purpose of this approach is to correct missing information only at the action unit region level, without performing pixel-level image filling and repair on the face, thereby reducing the propagation of errors caused by image artifacts generated without basis; to obtain low-efficiency action unit regions, and to filter neighbor regions of low-efficiency regions based on the action unit region association relationship established by facial muscle coordination, and to determine the set of neighbor regions of the region to be compensated.

[0122] Among them, the action unit region association relationship refers to the region connection relationship established based on the linkage law of facial muscle chain, tissue traction and pain expression, such as the linkage between the eyebrow region and the periorbital region, the linkage between the nasal wing region and the upper lip region, and the linkage between the corner of the mouth region and the mandibular region. Its processing logic is not limited to the closest in geometric space, that is, direct transmission, but prioritizes the adjacent relationship that has physiological synergistic significance in pain expression.

[0123] Let the action unit topology be as follows: ,in For a set of region nodes, For the set of connecting edges, It is an adjacency matrix. For the muscle linkage parameter matrix; when When, it indicates the region. Can be used as a region Candidate sources of compensation;

[0124] This implementation method first uses the action unit coordination relationship to filter the information source, which can avoid regions that have no obvious linkage with the current region from being incorrectly introduced into compensation;

[0125] Furthermore, based on the above embodiments, the relative distance change between adjacent action unit regions is obtained, the average pain deformation reference value obtained in advance based on training samples is obtained, and the relative deformation deviation is calculated. The purpose of this step is to determine whether the deformation relationship between the current neighboring regions still conforms to the normal linkage trend under pain expression, so as to suppress the abnormal transmission caused by device slippage.

[0126] Obtain the key point positions of adjacent action unit regions, calculate the relative distance change of adjacent regions based on the relative distance change of key points between the current cycle and the reference cycle, and generate the relative distance change between region pairs.

[0127] Among them, the relative distance change refers to the degree of change in the relative distance between two adjacent action unit areas in the current cycle, such as the tightening trend between the eyebrows and the eye area, and the traction trend between the nostrils and the upper lip, which is used to reflect whether the local muscle chain changes in synergy according to the direction of the pain expression; let the area be... With the region The change in relative distance in the current period is ;

[0128] To ensure consistency between the dimensions and physical meaning, the change in relative distance here... Specifically, the calculation is performed for the region within the current period. With the region The ratio of the geometric distance between the center points to the corresponding geometric distance between the center points in the neutral state reference frame is expressed as a dimensionless relative scaling ratio.

[0129] Among them, the average pain deformation reference value obtained from the training samples. and deformation standard deviation The specific acquisition logic is as follows: Extract all data frames of pain states from a pre-built, unobstructed historical database of pain expressions, and calculate the region for each. With the region The relative scaling factor of the geometric distance of the center point with respect to the neutral state is used, and the mean and standard deviation of this scaling factor are calculated for all samples, respectively, as... and ;

[0130] because and They have been uniformly defined on a dimensionless scaling scale, and the two can be directly added or subtracted; the aforementioned muscle linkage parameters The Pearson correlation coefficient between the activation intensities of corresponding regional features in the historical sample set is calculated and normalized to form a preset structured static parameter, thus avoiding the lack of transmission basis;

[0131] Relative deformation deviation The absolute value of the difference between the relative distance change in the current cycle and the average pain deformation reference value is calculated and then divided by the standard deviation of deformation and the minimum constant. The sum is obtained, where the smallest constant is... This is used to ensure that calculations can still be performed stably when the reference fluctuations are small; The larger the value, the more the linkage between the current regions deviates from the common pain deformation patterns observed in training, and the more likely it is to be affected by occlusion slippage or local traction artifacts.

[0132] This implementation incorporates relative deformation deviation constraints, which can reduce the weight of neighboring regions that are adjacent in position but have abnormal deformation relationships during propagation.

[0133] The association relationships of action unit regions, muscle linkage parameters, regional effectiveness coefficients of adjacent regions, and relative deformation deviations are obtained. Based on the normalization processing rules, the transmission weights of each adjacent region are calculated to determine the neighbor region transmission coefficients.

[0134] Among them, the neighboring cell transmission coefficient refers to the proportion of each adjacent cell in compensating for the current low-efficiency cell. Its role is to dynamically balance the combined effects of five factors: structural adjacency, physiological linkage, current reliability of neighboring cells, current deformation consistency, and regional distance.

[0135] Obtain the geometric distance between the center points of the region to be compensated and each adjacent region, and use this distance as the region distance; let the region... With the region The regional distance is ,area To the region The unnormalized transitive weights can be expressed as:

[0136]

[0137] And for the region All adjacent regions set The unnormalized propagation weights of all adjacent regions within a given area are summed. The unnormalized propagation weight of a single adjacent region is then divided by this summed value to perform a normalization operation, yielding the neighbor region propagation coefficient. ;in, , , and This is the preset adjustment coefficient;

[0138] Used to control the strength of the influence of muscle linkage parameters in compensation to highlight regions known to be significantly synergistic in pain expressions;

[0139] The influence intensity of the current validity of adjacent regions is used to prioritize the use of neighboring regions with better current observation quality as the source of compensation;

[0140] Used to control the intensity of inhibition of abnormal deformation in order to target and suppress neighboring areas that deviate from the normal pain linkage pattern; The attenuation intensity is used to control the distance between regions in order to limit the spread of invalid information caused by excessively distant regions;

[0141] The specific values ​​of each adjustment coefficient were determined by optimization on the validation set using a grid search method. The recommended value ranges are as follows: Desirable This is to ensure the priority and dominant role of muscle physiological linkage; Desirable To avoid introducing features from neighboring areas that are of low quality;

[0142] Desirable By using a larger penalty weight, the spread of errors in abnormal neighboring regions can be blocked; Desirable To achieve a constraint that gradually decreases with distance, the coefficients are adjusted in conjunction with actual occlusion scenario data;

[0143] This implementation method achieves the distribution of transmission strength through multi-factor joint normalization, which can make the compensation information closer to the actual facial muscle linkage.

[0144] Obtain the original feature vectors of adjacent regions, and perform linear fusion of the original feature vectors of adjacent regions and the original feature vectors of the current region based on the neighboring region transfer coefficient and the predetermined fusion weight to generate compensated region features;

[0145] The predetermined fusion weight is used to adjust the proportional relationship between the remaining original observations in the current region and the compensated information from adjacent regions; let the current low-efficiency region be... Its original feature vector is The original feature vector of the adjacent region is The feature mapping matrix is The fusion weight is Then the compensated regional features It can be represented as:

[0146]

[0147] Among them, the fusion weight Used to control the proportion of original features in the current region when some credible observations are still retained; when the regional effectiveness is low. A relatively small value can be chosen to increase the compensation share of neighboring areas; when the regional effectiveness is close to the reference threshold, A relatively large value can be chosen to avoid over-rewriting the current observations;

[0148] Feature mapping matrix This method is used to map the features of adjacent regions to a comparable feature space of the current region, so that features from different regions can participate in fusion under the same scale. This implementation only performs compensation at the region feature layer, which can reduce the uncertainty caused by high-computational reconstruction and non-physical artifacts.

[0149] Furthermore, based on the above embodiments, after obtaining the compensated regional characteristics, a historical stability constraint is applied to the compensation result; the purpose of this constraint is to prevent the region from gradually deviating from the normal characteristic range under reliable observation conditions during continuous compensation.

[0150] The regional effectiveness coefficients in the historical processing period are obtained. Based on the condition that the regional effectiveness coefficients are not lower than the preset reference value, the historical frames are screened to determine the historical stable frames. The historical stable frames refer to the set of historical frames with high observation quality and minimal impact from occlusion and boundary artifacts in the region. Their role is to provide a comparable reference feature benchmark for the current compensation results. The historical stable frames do not need to cover the entire historical period. Only frames that meet the conditions need to be retained within a predetermined historical window.

[0151] The compensated regional features are obtained, and the deviation of the compensation result is calculated based on the mean of the corresponding regional features in the historical stable frames to determine the compensation deviation. The compensation deviation refers to the distance between the current compensated regional features and the typical features of the region in the historical stable state.

[0152] Let the mean feature value of the corresponding region in the historical stable frame be . Then compensate for the deviation It is obtained by calculating the Euclidean distance between the feature vector of the current compensated region and the mean vector of the corresponding region; Used to measure whether the current compensation has deviated from the common characteristic distribution of the region under reliable observation conditions;

[0153] If the deviation increases significantly, it often means that the current compensation result has been affected by the propagation of abnormal neighboring regions or the indirect spread of occlusion artifacts. The compensation deviation is obtained, and the feature weights of the input feature decoupling model of the compensated region feature are adjusted according to the preset deviation threshold to generate constrained input region features.

[0154] Among them, the preset deviation threshold can be set according to the average compensation deviation and its fluctuation range in the training samples, for example, by using the superposition of the mean and standard deviation to determine the reference interval;

[0155] Used to define the boundary between acceptable compensation and excessive deviation from compensation; when When the deviation exceeds a preset threshold, the input feature weight of that region is reduced. Specifically, the original region feature weight can be multiplied by a decay coefficient such as a fixed percentage decay of 0.5. This directly reduces its influence in subsequent networks by quantizing the step size, which is equivalent to weakening the influence of that region on pain representation extraction in subsequent decoupled model processing, rather than forcibly accepting suspicious compensation results. This implementation adds historical stable frame verification, which can intercept abnormal drift in continuous compensation before it is input into the model.

[0156] For facial analysis tasks in clinical occlusion scenarios, common solutions either discard low-quality areas directly or perform image repair on the occluded areas; the former is prone to causing long-term missing tests of key action units, while the latter is prone to generating texture changes that do not exist.

[0157] This implementation uses regional feature compensation and limits the propagation source to adjacent areas that have muscle linkage significance and are currently reliable in observation. At the same time, the average value of historical stable frames is used to constrain the compensation result a second time. Therefore, under extreme conditions such as repeated movement of the mask edge and pulling of local skin by the cannula fixation device, the consistency between the compensation result and the actual facial movement changes can be better maintained.

[0158] In this application embodiment, this implementation can be deployed as a feature compensation program in the processor's local inference process; within the current processing cycle, the low-efficiency region number, the distance between key points in adjacent regions, the original feature vector, and the region validity coefficient are all stored in the local memory array; the processor first retrieves adjacent regions according to the region association relationship, and then calculates the neighbor region transfer coefficient and the compensated region features in memory;

[0159] The mean value of stable features for the corresponding region is read from the historical stable frame cache to complete the compensation bias judgment; if it exceeds the threshold, the reduced region weight is written into the input feature tensor; finally, the weighted region feature tensor is sent to the subsequent feature decoupling model to perform pain scoring, ensuring that the compensation process can truly be implemented in the actual inference link.

[0160] In some possible implementations, the training steps of the pre-trained feature decoupling model include: acquiring synchronously collected multi-source physiological signal data and extracting physiological reference features; aligning the pain representation features output by the feature decoupling model to be trained with the physiological reference features to construct alignment constraints;

[0161] An orthogonal constraint error is constructed based on the pain representation features and occlusion interference features output by the feature decoupling model; a pain score supervision error is constructed based on the real pain score labels corresponding to the training samples; the feature decoupling model is jointly trained based on the alignment constraint, orthogonal constraint error, and pain score supervision error to obtain the trained feature decoupling model.

[0162] The overall effectiveness coefficient is determined based on the regional effectiveness coefficient of each action unit region. The instantaneous pain score of the current frame and the output score of the previous moment are then weighted and smoothed based on the overall effectiveness coefficient to obtain the score at the current moment. This includes: calculating the average of the regional effectiveness coefficients of all action unit regions to obtain the overall effectiveness coefficient.

[0163] When the overall effectiveness coefficient is greater than or equal to the preset overall effectiveness reference value, the time smoothing weight is set to the preset upper limit weight.

[0164] When the overall effectiveness coefficient is less than the preset overall effectiveness reference value, the time smoothing weight is adaptively determined based on the overall effectiveness coefficient; for non-first frames, the instantaneous pain score of the current frame and the output score of the previous moment are weighted and combined according to the time smoothing weight to obtain the score of the current moment; for the first frame, the instantaneous pain score of the current frame is used as the score of the first frame in the continuous pain score sequence.

[0165] Please see Figure 2 The pain assessment system based on facial images includes: a data acquisition and standardization module, an occlusion detection and extraction module, an effectiveness calculation module, a feature compensation module, a feature processing module, a score regression module, and a time smoothing and output module; each module is implemented by the processor executing program instructions in the memory and is interconnected.

[0166] The data acquisition and standardization module is used to acquire raw facial video stream data and perform position correction and scale normalization on the raw facial video stream data based on facial key points to obtain a standardized facial image sequence.

[0167] The occlusion detection and extraction module is used to detect occlusion regions and extract features from standardized facial image sequences to obtain the visible proportion and original feature vector of the action unit region.

[0168] The effectiveness calculation module is used to determine the artifact intensity based on the difference in boundary optical flow and edge gradient fluctuation of the action unit region, and to determine the regional effectiveness coefficient in combination with the visible ratio. It is also used to determine the overall effectiveness coefficient based on the regional effectiveness coefficient of each action unit region.

[0169] The feature compensation module is used to perform local feature compensation on the original feature vector when the region effectiveness coefficient is lower than the preset reference value, based on the action unit region association relationship established based on the facial muscle coordination relationship and the neighbor region transmission coefficient determined based on the region distance, linkage parameters and region effectiveness coefficient, to obtain the compensated region features.

[0170] When the regional effectiveness coefficient is not lower than the preset reference value, the original feature vector is used as the feature of the region to be processed; the feature processing module is used to input the feature of the region to be processed into the pre-trained feature decoupling model and output pain representation features and occlusion interference features;

[0171] The score regression module is used to perform regression calculations based on pain representation features to obtain the instantaneous pain score for the current frame.

[0172] The time smoothing and output module is used to perform weighted smoothing of the instantaneous pain score of the current frame and the output score of the previous moment based on the overall effectiveness coefficient, so as to obtain the current moment score and output a continuous pain score sequence.

[0173] This embodiment illustrates the training method of the feature decoupling model, the time smoothing method of continuous scores, and the deployment structure of the corresponding system. The training input of this embodiment includes facial video samples, real pain score labels corresponding to the training samples, and synchronously collected multi-source physiological signal data. The training output is a feature decoupling model that can receive only facial video input during the inference stage.

[0174] During the reasoning phase, an overall effectiveness coefficient is formed based on the effectiveness coefficients of each action unit region, and the instantaneous pain score is smoothed over time accordingly; at the system level, the processor executes program instructions in the memory to enable the coordinated work of each functional module.

[0175] Acquire synchronously collected multi-source physiological signal data, extract features from the multi-source physiological signal data based on the synchronous correspondence of physiological signals, and generate physiological reference features;

[0176] Among them, multi-source physiological signal data refers to auxiliary physiological observation data collected synchronously with facial video, such as one or more of surface electromyography, skin conductance, and pulse wave. Its role is not to continue to participate in the input during the inference stage, but to provide the visual model with a reference that is closer to the real physiological changes of pain during the training stage.

[0177] In terms of specific feature quantification and extraction, taking surface electromyography and skin conductance as examples, the root mean square and median frequency of surface electromyography signals in the same cycle can be extracted as quantitative indicators of muscle tension, and the baseline skin conductance level and skin conductance response amplitude of skin conductance signals can be extracted as quantitative indicators of sympathetic nerve activity. The extracted one-dimensional numerical indicators are linearly spliced ​​and normalized to form physiological reference features with clear dimensions.

[0178] Physiological reference characteristics can be expressed as ,in This represents the index of the processing cycle in the training sample sequence; this feature is used to summarize the strength of the patient's physiological response to pain changes within that cycle, providing an alignment target for the subsequent learning of pain representation features; this implementation introduces physiological reference information during training, which can reduce the model's dependence on purely visual representation noise.

[0179] The output of the feature decoupling model to be trained is obtained. Based on the correspondence between pain representation features and physiological reference features, feature alignment is performed between the two to construct alignment constraints. In the design of the transparent network topology, the feature decoupling model adopts a multi-layer network architecture that includes a shared coding layer and a dual-branch output layer.

[0180] Its input end receives the spliced ​​one-dimensional high-dimensional features of the region to be processed, reduces the dimensionality through a fully connected shared coding layer with at least one layer and generates fused hidden layer features, and distributes them to two parallel independent fully connected branches at the decoding end, respectively mapping and outputting pain representation features and occlusion interference features of the same preset dimension, such as 64-dimensional or 128-dimensional, to ensure that the algorithm operation path is traceable and the feature flow boundary is clear.

[0181] Among them, pain representation features refer to the pain-related main channel features output from one branch of the feature decoupling model, which can be represented as follows: This is used to carry the main information needed for subsequent pain regression; to enable To more closely approximate real physiological feedback, a dimension alignment matrix can be introduced to project physiological reference features onto a space comparable to pain representation features. Establish the following alignment constraint error :

[0182]

[0183] The smaller the error, the closer the pain representation extracted by the model is to the physiological reference. This implementation adopts a training-phase alignment and inference-phase single-modal operation approach, which retains the guiding role of the physiological reference without increasing the real-time deployment burden on the clinical end.

[0184] Obtain the pain representation features and occlusion interference features output by the feature decoupling model, and perform orthogonal constraint training on the model based on the correlation between the two types of features to construct the orthogonal constraint error.

[0185] Among them, the occlusion interference feature refers to the characteristic channel used to absorb non-pain factors such as mask edge displacement, adhesive tape traction, and local bright / dark flickering, which can be represented as: To separate the pain channel from the artifact channel as much as possible, the following orthogonal constraint error can be established. :

[0186]

[0187] Among them, the minimum constant This is used to prevent the numerical value from being too small due to an excessively small denominator; the smaller the error, the less likely the pain representation features and occlusion interference features are to be mixed with each other; this implementation can control the influence of artifacts on the final pain score to a low level by separating pain-related changes and occlusion interference changes.

[0188] Obtain the true pain score labels corresponding to the training samples, and perform regression supervision on the model based on the mapping relationship between pain representation features and true pain scores to construct pain score supervision error;

[0189] The actual pain score label can be a scale value or a continuous rating value that is synchronously labeled with the training samples, and the regression output... By multiplying the pain representation features by a regression weight matrix used to map each dimension of the feature to its score contribution. It also includes a bias term used to adjust the overall score baseline. get;

[0190] Calculate regression output Compared with the true pain score label The sum of squares of the differences between them is used as the supervision error of the pain score. This supervision process ensures that the pain representations extracted by the model can ultimately be translated into specific scoring outputs, rather than remaining in an abstract feature space.

[0191] The alignment constraint, orthogonal constraint error, and pain score supervision error are obtained. The feature decoupling model is trained according to the joint training rules to obtain the trained feature decoupling model.

[0192] The joint training employs a weighted summation method to organize the overall objective, and monitors the error of pain scores. Multiplied by the first balance coefficient Alignment constraint error And multiplied by the second balance coefficient Orthogonal constraint error The total training loss is obtained by linear summation. When it is necessary to combine regional compensation constraints and temporal smoothing constraints, corresponding loss terms can also be superimposed; parameters and These are used to balance the contribution ratios of physiological alignment targets and feature separation targets during training;

[0193] In the specific model training phase and A dynamic adjustment strategy can be adopted: In the early stages of training, [the following steps can be taken] Set between 0.5 and 1.0 to quickly establish a physiological alignment baseline, while also... Set it to around 0.1 to avoid early interference;

[0194] In the later stages of model convergence, gradually... The resolution is increased to between 0.5 and 1.0 to refine the separation of occlusion artifact features; overall, when greater emphasis is needed on physiological consistency, the overall resolution can be appropriately increased. ;

[0195] When a greater emphasis is needed on separating occlusion artifacts, the overall level can be appropriately increased. The trained model parameters are written to the model file area in the memory and are directly loaded by the feature processing module during the inference phase, no longer relying on additional physiological sensor input.

[0196] Furthermore, based on the above embodiments, the overall effectiveness coefficient is determined according to the regional effectiveness coefficient of each action unit region, and the instantaneous pain score of the current frame and the output score of the previous moment are weighted and smoothed according to the overall effectiveness coefficient;

[0197] Improve the stability of continuous scoring when the quality of single-frame observations deteriorates, while retaining instantaneous changes when the observation quality is high; obtain the regional effectiveness coefficients of all action unit regions, summarize the regional effectiveness coefficients according to the mean calculation rules, and determine the overall effectiveness coefficient;

[0198] The overall effectiveness coefficient refers to the comprehensive usability of the entire face at the action unit level in the current processing cycle, calculated by analyzing all data within the current cycle. Regional effectiveness coefficient of each action unit region Sum the results and divide by the total number of action unit regions. The overall effectiveness coefficient is obtained in the form of an arithmetic mean. ;

[0199] Overall effectiveness coefficient It provides an overall overview of the reliability of key regions within the current cycle; when multiple regions are simultaneously affected by occlusion or boundary artifacts, The score will decrease, serving as a reminder that subsequent outputs should not rely entirely on the instantaneous score of the current frame; this implementation method aggregates the quality information of each region into an overall effectiveness coefficient, providing a quantitative basis for time smoothing.

[0200] Obtain the overall effectiveness coefficient, determine the time smoothing weight based on the preset overall effectiveness reference value, and generate the time smoothing weight for the current processing cycle;

[0201] The preset overall effective reference value can be between 0.55 and 0.70, which is used to distinguish between two cases: the overall quality of the current frame is good and the overall quality of the current frame is insufficient. The time smoothing weight represents the contribution ratio of the instantaneous score of the current frame to the final output score.

[0202] like If the value is greater than or equal to the preset overall effective reference value, then the time smoothing weight will be applied. Set to a preset upper limit weight; if If it is less than this reference value, then according to Adaptive determination ;

[0203] Adaptive time smoothing weights can be calculated by establishing a linear mapping relationship between the overall effectiveness coefficient and the time smoothing weights. In the interval The values ​​on are mapped proportionally to the interval. This ensures that its value is between 0 and 1;

[0204] The preset lower limit weight can be set to 0.1 to 0.2, which is used to maximize the inheritance of historical stable output under extremely low confidence conditions.

[0205] The preset upper limit weight can be set to 0.8 to 0.95 to maintain a high sensitivity response to the current instantaneous state when the facial observation is completely clear; time smoothing weight. Weighting is used to dynamically balance the immediate pain response in the current frame and the continuity of historical output: the more reliable the current frame, the better. The larger;

[0206] The less reliable the current frame, The smaller the value; obtain the instantaneous pain score of the current frame and the output score of the previous moment, and combine them in a weighted manner according to the time smoothing weight to obtain the score of the current moment; where, for non-first frames, let the instantaneous pain score of the current frame be . The output score at the previous time step was The current output score is It can be in the following form:

[0207]

[0208] in, This represents the historical output inheritance share, used to suppress occasional spikes when the current frame is unreliable; for the first frame, the instantaneous pain score of the current frame is directly used as the first frame score in the continuous pain score sequence; this implementation binds the smoothing intensity to the current observation quality, which can achieve a more reasonable trade-off between preserving the true instantaneous pain changes and suppressing occlusion false alarms;

[0209] Furthermore, this embodiment also provides a corresponding pain assessment system based on facial images; the system includes a data acquisition and standardization module, an occlusion detection and extraction module, an effectiveness calculation module, a feature compensation module, a feature processing module, a score regression module, and a time smoothing and output module. Each module is implemented by the processor executing program instructions in the memory, and they work together within the same device through memory object sharing.

[0210] The communication connection is not an abstract description, but refers to the fact that the output of the previous module is written to local memory in the form of a standardized image matrix, an occlusion mask matrix, a region feature vector array, a region validity array, an overall validity scalar, and a score cache object; the next module reads the corresponding data through object references, and releases the temporary intermediate objects after reading to control memory usage.

[0211] The data acquisition and standardization module writes the acquired raw facial video stream data into the local frame buffer and outputs a standardized facial image sequence.

[0212] The occlusion detection and extraction module reads the standardized image matrix and generates an occlusion mask and an array of original feature vectors; the validity calculation module calculates the artifact intensity, regional validity coefficient, and overall validity coefficient within the same processing cycle; the feature compensation module decides whether to generate compensated regional features or retain the original feature vectors as the features of the region to be processed, depending on whether the values ​​are lower than the preset reference values.

[0213] The feature processing module loads the trained feature decoupling model, infers the features of the region to be processed, and outputs pain representation features and occlusion interference features; the score regression module generates the instantaneous pain score of the current frame based on the pain representation features; the time smoothing and output module combines the overall effectiveness coefficient and the output score of the previous time step to calculate the score of the current time step, and writes the continuous pain score sequence into the result cache area.

[0214] In terms of comparative analysis, for clinical continuous pain quantification tasks, the evaluation focus is usually on two types of abilities: whether the score can still be reliable when faced with occlusion, postural fluctuations and weak facial expressions; whether the output curve is neither too jittery nor too sluggish; if traditional methods directly smooth the visual scores, they are prone to spikes when the occlusion is severe or excessively flattened when there are real transients.

[0215] This implementation introduces physiological reference alignment and orthogonal separation during the training phase, enabling the model to first learn to distinguish between pain representation patterns and occlusion interference patterns; then, during the inference phase, the overall effectiveness coefficient is used to determine whether the instantaneous score should be amplified and adopted. Therefore, it is more suitable for long-term operation in scenarios with frequent dynamic occlusion, such as intensive care, anesthesia recovery, and postoperative observation.

[0216] In this application example, the system can be deployed on a bedside computing device with a processor and memory; the image acquisition unit continuously writes the acquired raw facial video stream data into the local frame buffer;

[0217] The processor calls program instructions in memory to complete standardization, occlusion detection, validity calculation, feature compensation, feature decoupling, score regression, and time smoothing; the final continuous pain score sequence and corresponding low confidence prompts are encapsulated as structured output data and written to the monitoring interface buffer.

[0218] After the upper-level monitoring system reads the buffer, it updates the continuous pain score curve on the display interface and synchronously displays a review prompt when the overall effectiveness is lower than the set conditions, so that the output of this system can be directly used for continuous clinical observation.

[0219] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A pain assessment method based on facial images, characterized in that, include: The raw facial video stream data is acquired, and the raw facial video stream data is subjected to position correction and scale normalization based on facial key points to obtain a standardized facial image sequence. The standardized facial image sequence is subjected to occlusion region detection and feature extraction to obtain the visible proportion and original feature vector of the action unit region; The artifact intensity is determined based on the difference in boundary optical flow and edge gradient fluctuations in the action unit region, and the region effectiveness coefficient is determined in combination with the visible ratio. If the region effectiveness coefficient is lower than the preset reference value, then the original feature vector is locally compensated based on the action unit region association relationship established based on the facial muscle coordination relationship and the neighbor region transmission coefficient determined based on the region distance, linkage parameters and region effectiveness coefficient, to obtain the compensated region features. If the region validity coefficient is not lower than the preset reference value, then the original feature vector is used as the feature of the region to be processed. The features of the region to be processed are input into a pre-trained feature decoupling model, which outputs pain representation features and occlusion interference features. Regression calculation is performed based on the pain representation features to obtain the instantaneous pain score for the current frame; The overall effectiveness coefficient is determined based on the regional effectiveness coefficient of each action unit region, and the instantaneous pain score of the current frame and the output score of the previous moment are weighted and smoothed based on the overall effectiveness coefficient to obtain the current moment score, and a continuous pain score sequence is output.

2. The pain assessment method based on facial images according to claim 1, characterized in that, The standardized facial image sequence is subjected to occlusion region detection and feature extraction to obtain the visible proportion and original feature vector of the action unit region, including: Occlusion detection is performed on the standardized facial image sequence to generate a pixel-level occlusion mask; The standardized facial image sequence is divided into multiple action unit regions, and the area of ​​occluded pixels in each action unit region is determined according to the pixel-level occlusion mask. The visibility ratio of the action unit region is determined based on the area of ​​the occluded pixels and the total pixel area of ​​the region. Local texture change features, local displacement increment features extracted based on key point displacement, and short-time difference features extracted based on grayscale differences between adjacent frames are extracted from the action unit region. The extracted features are then concatenated to obtain the original feature vector.

3. The pain assessment method based on facial images according to claim 1, characterized in that, The artifact intensity is determined based on the difference in boundary optical flow and edge gradient fluctuations of the action unit region, and the region effectiveness coefficient is determined in conjunction with the visible ratio, including: A boundary detection band is formed by extending a preset pixel width inward and outward from the boundary of each action unit region, and the local optical flow difference and edge gradient fluctuation of adjacent frames within the boundary detection band are extracted. The artifact intensity is determined based on the local optical flow difference and the edge gradient fluctuation; The region effectiveness coefficient is determined based on the visibility ratio and the artifact intensity.

4. The pain assessment method based on facial images according to claim 1, characterized in that, Based on the action unit region association relationship established based on facial muscle coordination and the neighbor region transfer coefficient determined based on region distance, linkage parameters, and region effectiveness coefficient, local feature compensation is performed on the original feature vector to obtain compensated region features, including: The relative distance change between adjacent action unit regions is obtained, the average pain deformation reference value obtained in advance based on training samples is obtained, and the relative deformation deviation between the relative distance change and the average pain deformation reference value is calculated. Based on the adjacency relationship of the action unit region association, the muscle linkage parameters obtained from pre-training, the regional effectiveness coefficient of the adjacent region, and the relative deformation deviation, the transmission weight of each adjacent region is normalized to obtain the neighbor region transmission coefficient. The original feature vectors of adjacent regions are weighted and summed according to the neighbor transfer coefficient, and then linearly fused with the original feature vector of the current region according to a predetermined fusion weight to obtain the compensated region features.

5. The pain assessment method based on facial images according to claim 4, characterized in that, After obtaining the compensated region features, the method further includes: Historical frames whose regional validity coefficient is not lower than the preset reference value are identified as historical stable frames. Calculate the compensation deviation between the compensated regional features and the mean value of the corresponding features of the current region in the historical stable frames; If the compensation deviation exceeds a preset deviation threshold, the feature weights of the compensated region features when inputting them into the feature decoupling model are reduced.

6. The pain assessment method based on facial images according to claim 1, characterized in that, The training steps of the pre-trained feature decoupling model include: Acquire synchronously collected multi-source physiological signal data and extract physiological reference features; Align the pain representation features output by the feature decoupling model to be trained with the physiological reference features to construct alignment constraints; An orthogonal constraint error is constructed based on the pain representation features and occlusion interference features output by the feature decoupling model. A pain score supervision error is constructed based on the real pain score labels corresponding to the training samples; Based on the alignment constraint, the orthogonal constraint error, and the pain score supervision error, the feature decoupling model is jointly trained to obtain the trained feature decoupling model.

7. The pain assessment method based on facial images according to claim 1, characterized in that, The overall effectiveness coefficient is determined based on the regional effectiveness coefficient of each action unit region, and the instantaneous pain score of the current frame and the output score of the previous moment are weighted and smoothed based on the overall effectiveness coefficient to obtain the current moment score, including: The average of the regional effectiveness coefficients of all the aforementioned action unit regions is calculated to obtain the overall effectiveness coefficient. When the overall effectiveness coefficient is greater than or equal to the preset overall effectiveness reference value, the time smoothing weight is set to the preset upper limit weight. When the overall effectiveness coefficient is less than the preset overall effectiveness reference value, the time smoothing weight is adaptively determined based on the overall effectiveness coefficient. For non-first frames, the instantaneous pain score of the current frame and the output score of the previous moment are weighted and combined according to the time smoothing weight to obtain the score at the current moment; For the first frame, the instantaneous pain score of the current frame is used as the first frame score in the continuous pain score sequence.

8. A facial image-based pain assessment system for performing the facial image-based pain assessment method as described in any one of claims 1-7, characterized in that, It includes modules for data acquisition and standardization, occlusion detection and extraction, validity calculation, feature compensation, feature processing, score regression, and time smoothing and output. Each module is implemented by the processor executing program instructions in the memory and is connected in communication. The data acquisition and standardization module is used to acquire raw facial video stream data and perform position correction and scale normalization processing on the raw facial video stream data based on facial key points to obtain a standardized facial image sequence. The occlusion detection and extraction module is used to perform occlusion region detection and feature extraction on the standardized facial image sequence to obtain the visible proportion and original feature vector of the action unit region. The effectiveness calculation module is used to determine the artifact intensity based on the boundary optical flow difference and edge gradient fluctuation of the action unit region, and to determine the region effectiveness coefficient in combination with the visible ratio. It is also used to determine the overall effectiveness coefficient based on the region effectiveness coefficient of each action unit region. The feature compensation module is used to perform local feature compensation on the original feature vector when the region effectiveness coefficient is lower than a preset reference value, based on the action unit region association relationship established based on the facial muscle coordination relationship and the neighbor region transmission coefficient determined based on the region distance, linkage parameters and region effectiveness coefficient, to obtain the compensated region features. When the region validity coefficient is not lower than a preset reference value, the original feature vector is used as the feature of the region to be processed; The feature processing module is used to input the features of the region to be processed into a pre-trained feature decoupling model and output pain representation features and occlusion interference features. The score regression module is used to perform regression calculations based on the pain representation features to obtain the instantaneous pain score for the current frame. The time smoothing and output module is used to perform weighted smoothing on the instantaneous pain score of the current frame and the output score of the previous moment according to the overall effectiveness coefficient, so as to obtain the current moment score and output a continuous pain score sequence.

Images