A facial expression feature lightweight edge inference method for delivery room scenarios

Abstract

The present application provides a kind of face expression feature lightweight edge inference method for delivery room scene, belong to medical visual artificial intelligence, edge computing and privacy security protection technical field;The present application is adapted to the complex illumination environment of delivery room by RGB-IR binocular infrared imaging combined with adaptive light supplementing technology;Based on the confidence of KLT optical flow tracking, a hierarchical skip frame adaptive strategy is designed, the inference frequency of face detection is dynamically optimized, and the power consumption of edge device is reduced;Through facial key point affine alignment and pain-specific multi-region AOI cropping, the pain-related facial region is accurately extracted;Construct LiteAUNet lightweight parallel inference network, combined with adversarial training and privacy bottleneck layer to realize feature desensitization.The present application solves the technical problems of large power consumption, poor real-time performance, high privacy leakage risk, weak adaptability to complex delivery room environment, multi-target interference and other technical problems in the prior art, and realizes the lightweight, low delay, high security and high-precision edge inference of maternal facial pain features in the delivery room scene.

Description

Technical Field

[0001] This invention provides a lightweight edge inference method for facial expression features in delivery room scenarios, belonging to the fields of medical visual artificial intelligence, edge computing, and privacy protection technology. Background Technology

[0002] Labor pain is the most crucial physiological pain indicator during childbirth. Accurate and real-time assessment of labor pain is of critical guiding significance for obstetric clinical monitoring, labor intervention, and adjustment of analgesia protocols. Currently, clinical labor pain assessment relies heavily on subjective scoring by medical staff, the mother's self-report, and objective data monitoring such as uterine contractions and vital signs. This approach suffers from strong subjectivity, poor real-time accuracy, and limited data dimensions, failing to accurately reflect the mother's true pain status.

[0003] With the development of computer vision technology, pain recognition technology based on facial expressions is gradually being applied to medical pain assessment scenarios. The intensity changes of facial action units (AUs) can objectively represent the degree of human pain. However, existing technologies have several shortcomings when applied to delivery room scenarios: First, traditional facial expression recognition networks have a large number of parameters and high inference time, making it difficult to adapt to the low computing power and low latency real-time processing requirements of embedded edge devices in delivery rooms, and unable to support high-frequency real-time inference of delivery bed video streams; Second, the delivery room environment has complex lighting, multiple delivery beds with coexisting patients, and varied facial poses, making traditional fixed frame rate detection and inference methods redundant in computing power and with poor tracking stability, prone to detection failures and feature extraction deviations; Third, maternal facial images are highly sensitive medical privacy data, and existing inference solutions generally have privacy risks related to the storage, uploading, and leakage of original images, failing to meet medical data security compliance requirements; Fourth, traditional AU inference models have weak anti-interference capabilities, and noise interference in low-light and infrared acquisition scenarios can easily lead to AU misjudgments, and there is a lack of effective privacy protection and adversarial optimization mechanisms, resulting in insufficient model security and robustness; Fifth, in multi-delivery bed scenarios, devices are prone to facial target confusion and data cross-interference, making it impossible to achieve independent and accurate monitoring of a single delivery woman.

[0004] In summary, existing facial expression reasoning technologies cannot simultaneously address the real-time requirements, lightweight design, privacy security, and environmental adaptability of the delivery room scenario. There is an urgent need for a facial expression feature edge reasoning solution that is adapted to the specific delivery room environment, offers low latency, high security, and strong robustness. (Invention Content) The purpose of this invention is to overcome the shortcomings of the prior art and provide a lightweight edge inference method and device for facial expression features in delivery room scenarios. This invention solves the technical problems of high computing power consumption, poor real-time performance, high risk of privacy leakage, weak adaptability to complex delivery room environments, and multi-target interference in the prior art. It achieves lightweight, low-latency, high-security, and high-accuracy edge inference of facial pain features of mothers in delivery room scenarios.

[0005] To address the aforementioned problems, the present invention proposes the following technical solution: a lightweight edge inference method for facial expression features in delivery room scenarios, comprising the following steps: S1. Collect real-time video stream of the delivery bed area; S2. Perform face detection and tracking on each frame of the real-time video stream, and dynamically adjust the execution frequency of face detection based on the face tracking confidence using a skip frame adaptive strategy. S3. Based on the tracked face region, perform facial key point localization, and use the facial key points to complete face alignment through affine transformation. S4. Based on the region of interest (AOI) corresponding to the pain-related facial action unit (AU) in the delivery room scene, multiple AOI sub-images are cropped from the aligned facial image. S5. Input each of the AOI subgraphs into the lightweight AU intensity regression network for inference, and output the pain-related AU intensity vector and the low-dimensional feature vector output by the privacy bottleneck layer. S6. After obtaining the AU intensity vector and low-dimensional feature vector, perform instantaneous secure erasure on all image data of the original video frame, aligned face image, and AOI sub-image stored in the memory buffer. The original image data is not written to disk, uploaded, or leaked. S7. Output only the AU intensity vector and the low-dimensional feature vector output by the privacy bottleneck layer for subsequent multimodal assessment of labor pain.

[0006] Furthermore, the lightweight AU intensity regression network is LiteAUNet, which includes three parallel sub-graph input branches, a feature fusion module, a privacy bottleneck fully connected layer, and an AU intensity output head. The three parallel sub-graph input branches correspond to the upper orbital AOI region between the eyebrows, the middle nasal root AOI region, and the lower upper lip corner AOI region, respectively. Each parallel sub-graph input branch is composed of multiple depthwise separable convolutions. The output features of each parallel sub-graph input branch are concatenated and fused, and then input into the privacy bottleneck fully connected layer. The privacy bottleneck fully connected layer outputs a privacy bottleneck feature vector. The privacy bottleneck feature vector is input into the AU intensity output head, which outputs AU intensity values ​​related to pain.

[0007] Furthermore, the total number of parameters in LiteAUNet is less than 500K; the output of the privacy bottleneck fully connected layer is a 64-dimensional privacy bottleneck feature vector, and the AU strength output head outputs a 6-dimensional AU strength value.

[0008] Furthermore, adversarial training is performed on the LiteAUNet; during the training phase, an adversarial decoder network is introduced, which takes the privacy bottleneck feature vector as input and reconstructs the original facial image; a privacy penalty term is added to the total loss function of LiteAUNet, which is a negative weighting of the reconstruction error between the reconstructed image output by the adversarial decoder and the original facial image; during the training process, the weight coefficient corresponding to the privacy penalty term is gradually increased from 0.01 to 1.0.

[0009] Furthermore, the skip-frame adaptive strategy calculates the face tracking confidence based on the KLT optical flow method, and the hierarchical control rules are as follows: when the tracking confidence is greater than 0.9, the face detection step is skipped, and only key point alignment and AU inference are performed; when the tracking confidence is greater than 0.6 and less than or equal to 0.9, AU inference is performed, and face key point correction is performed once every preset fixed number of frames; when the tracking confidence is less than or equal to 0.6, the face detection, key point localization and alignment, and AU inference process are fully executed; through the above strategy, the average inference frequency of face detection is reduced to 25fps.

[0010] Furthermore, the instant secure erasure employs a three-pass overwrite mechanism, sequentially writing 0x00, 0xFF, and random bytes to the memory buffer storing the image data; after erasure is completed, a memory barrier instruction is invoked to confirm the overwrite operation is effective before releasing the memory buffer; all image data resides only in volatile RAM and is not written to the file system for persistent storage.

[0011] Furthermore, the real-time video stream is acquired by an RGBIR binocular camera; when the ambient light intensity is lower than the preset light threshold, the 940nm infrared fill light is automatically turned on, and the camera switches to the IR infrared channel to acquire images; IR noise data enhancement samples are added during the training phase of the lightweight AU intensity regression network; in the IR infrared acquisition mode, the AU regression confidence threshold is increased to reduce AU misjudgment caused by noise.

[0012] Furthermore, if the confidence of facial detection or AU inference for a consecutive preset number of frames is lower than a preset quality threshold, it is determined that the facial data quality is continuously declining; the weight of the subjective branch of facial expression in the multimodal assessment fusion model of labor pain is reduced or zeroed, and objective data of uterine contractions and vital signs are used to conduct pain assessment, and facial data quality alarm signals are pushed to the superior nursing station; in the scenario of labor room with multiple delivery beds, each set of acquisition and inference devices operates independently; the device locks the target face within a distance of 1.5-2.5m as the tracking object based on depth information, and filters out other irrelevant faces in the field of view; the devices do not exchange image data and feature data, and each device independently outputs AU intensity vector and privacy bottleneck feature vector.

[0013] Furthermore, the low-dimensional feature vector of the privacy bottleneck layer output is encrypted using AES128 and then transmitted to the edge server via the local area network.

[0014] A lightweight edge inference device for facial expression features in a delivery room setting includes an RGBIR binocular camera module, a 940nm infrared illumination module, an embedded NPU computing module, a memory secure erasure logic module, and an encrypted transmission module. The RGBIR binocular camera module is used to acquire RGB and IR video streams from the delivery room bed area. The 940nm infrared illumination module provides infrared illumination for the mother's face in low-light conditions. The embedded NPU computing module deploys a facial detection and tracking model, a keypoint detection model, and a lightweight AU intensity regression network to achieve facial detection and tracking, keypoint alignment, AOI cropping, and AU feature inference. The memory... The secure erase logic module performs a three-pass secure erase operation on various image buffers in RAM memory; the encrypted transmission module performs AES128 encryption on the feature vector output by the privacy bottleneck layer and outputs the encrypted feature vector to the outside via the local area network; the device does not perform persistent storage on any original facial image data; the embedded NPU computing module uses an RK3588 chip or a Huawei Ascend310 chip; the lightweight AU strength regression network deployed in the embedded NPU computing module undergoes quantization conversion; the single-frame AU inference time is less than 5ms; the overall end-to-end processing latency of the device is less than 12ms, supporting real-time processing of 30fps video streams.

[0015] Due to the adoption of the above technical solution, the beneficial effects of the lightweight edge inference method for facial expression features in the delivery room scenario of the present invention are as follows: 1. Achieves lightweight, high-speed inference at the edge, adaptable to embedded device deployment in delivery rooms. The LiteAUNet network designed in this invention has less than 500K total parameters, adopts a multi-branch depthwise separable convolutional structure, and combines model quantization optimization. The single-frame AU inference time is less than 5ms, the overall end-to-end latency is less than 12ms, and it supports real-time processing of 30fps high-definition video streams. It breaks through the bottleneck of high computing power requirements of traditional models and the inability of edge devices to perform real-time inference, and is suitable for embedded edge hardware deployment scenarios in delivery rooms.

[0016] 2. The adaptive frame skipping strategy significantly reduces device computing power and power consumption. This invention is based on the KLT optical flow method to design an inference strategy with confidence level, dynamically adjusting the face detection frequency and reducing the average inference frequency of face detection to 2-5 fps. While ensuring face tracking accuracy and feature extraction accuracy, it greatly reduces invalid computation, lowers the power consumption of edge devices, and adapts to the needs of long-term continuous operation of medical devices.

[0017] 3. Comprehensive protection of maternal facial privacy data, meeting medical compliance requirements. This invention designs a three-pass memory overwrite and instant erasure mechanism, ensuring that the original image data resides only in RAM, is not written to disk, uploaded, or leaked, thus preventing the leakage of original privacy data at the source. At the same time, through multiple mechanisms such as privacy bottleneck layer dimensionality reduction, adversarial training privacy constraints, and AES-128 encrypted transmission, the original image privacy information is completely stripped away, and only desensitized feature vectors are output, perfectly complying with medical data privacy protection standards.

[0018] 4. Extremely robust to complex delivery room scenarios. This invention employs an RGB-IR binocular acquisition + 940nm infrared illumination scheme, adapting to complex delivery room lighting environments such as low light, nighttime, and fluctuating light and shadow. Through IR noise data enhancement and infrared mode confidence adjustment strategies, it effectively reduces inference misjudgments caused by lighting noise. At the same time, it locks the target face through depth distance, achieving face isolation tracking in multi-delivery-bed scenarios, avoiding interference from irrelevant personnel, and adapting to the complex environment of real obstetric clinical settings.

[0019] 5. Precisely focusing on labor pain characteristics to improve pain assessment accuracy. This invention specifically selects three AOI regions with the highest correlation to labor pain: the glabella, the root and alar of the nose, and the corners of the lips and mouth. It discards invalid facial feature interference and combines adversarial training to optimize the network's feature extraction capabilities, accurately outputting a 6-dimensional pain-related AU intensity vector, providing high-precision feature support for subsequent multimodal fusion assessment of labor pain. At the same time, a data quality anomaly adaptation mechanism is designed to automatically avoid interference from low-quality data and ensure the stability of assessment results. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Real-time inference under normal lighting conditions The mother, Ms. Zhang, is 30 years old and is in the active stage of the first stage of labor.

[0022] The RGB-IR binocular camera directly in front of the delivery bed captures a video stream at 30fps with a resolution of 1920×1080 (the face area occupies about 40% of the screen).

[0023] The UltraFace detector ran at 2fps (detecting once every 15 frames) and detected a face bounding box in the first frame (confidence 0.97). The face was then tracked using KLT optical flow for the next 14 frames, with a tracking confidence consistently >0.9.

[0024] Frames 2-15: Skip face detection and directly use optical flow tracing face bounding boxes for key point localization (17-point simplified model) and alignment (affine transformation), computation time is about 2ms.

[0025] Cropping three AOI sub-images (64×64 each) from the aligned facial image takes <0.5ms.

[0026] LiteAUNet (quantized with TensorFlow Lite and converted to RKNN format, deployed on RK3588 NPU) infers three subgraphs in 4.2ms, outputting a 6-dimensional AU intensity vector: [AU4=2.1,AU6=1.8,AU7=3.5,AU9=0.7,AU10=2.3,AU43=0.2].

[0027] The 64-dimensional privacy bottleneck layer feature vector is encrypted with AES-128 and sent to the edge server via the local area network. The original image data is overwritten and erased three times in RAM.

[0028] The labor pain assessment model on the edge server receives an encrypted 64-dimensional feature vector, which is then decoded and used as input for the subjective branch. This input is then fused with objective branch data such as uterine contraction data and vital signs at the attention layer.

[0029] Example 2: IR Switching and Adaptive Noise Reduction in Low-Light Environments The woman, Ms. Li, gave birth at night, and the lights in the delivery room were dimmed.

[0030] When the ambient light sensor detects an illuminance of <10 lux, the system automatically switches to IR mode: turning on the 940nm infrared fill light and switching the camera to the IR channel.

[0031] The signal-to-noise ratio of IR images is low. LiteAUNet already includes 20% simulated IR noise data enhancement (Gaussian noise + salt and pepper noise) during training. The relative error of AU detection under IR input is about 15% higher than that of RGB, but it is still within the clinically acceptable range.

[0032] When the system detects an IR mode switch, it automatically raises the confidence threshold for AU intensity regression by 10% (from the default 0.7 to 0.77) to reduce false positives under low signal-to-noise ratio conditions.

[0033] When the IR image quality further deteriorates due to facial reflection caused by the mother's sweating, and the confidence level of three consecutive frames is <0.7, the system triggers a "deterioration in facial data quality" alarm, sending a notification to the nurses' station without shutting down the system. At this point, the system relies entirely on objective data channels (uterine contractions, vital signs) for pain assessment, and the weight of subjective channels is temporarily reduced to 0.

[0034] Example 3: Multi-device scenario – Two pregnant women sharing a delivery room In the twin-bed delivery room, an independent edge inference device is deployed directly in front of each delivery bed. Each device operates independently, processing the video stream within its own field of view. During initialization, the UltraFace detector measures the distance to faces using a 3D depth camera, automatically locking onto the nearest face (within a range of 1.5-2.5 meters) as the tracking target, ignoring other faces in the background. There is no data exchange between the two devices; each outputs data independently.

[0035] The present invention and its embodiments have been described above. This description is not restrictive. In short, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the present invention, such design should fall within the protection scope of the present invention.

Claims

1. A lightweight edge inference method based on facial expression features for delivery room scenarios, characterized in that, Includes the following steps: S1. Collect real-time video stream of the delivery bed area; S2. Perform face detection and tracking on each frame of the real-time video stream, and dynamically adjust the execution frequency of face detection based on the face tracking confidence using a skip frame adaptive strategy. S3. Based on the tracked face region, perform facial key point localization, and use the facial key points to complete face alignment through affine transformation. S4. Based on the region of interest (AOI) corresponding to the pain-related facial action unit (AU) in the delivery room scene, multiple AOI sub-images are cropped from the aligned facial image. S5. Input each of the AOI subgraphs into the lightweight AU intensity regression network for inference, and output the pain-related AU intensity vector and the low-dimensional feature vector output by the privacy bottleneck layer. S6. After obtaining the AU intensity vector and low-dimensional feature vector, perform instantaneous secure erasure on all image data of the original video frame, aligned face image, and AOI sub-image stored in the memory buffer. The original image data is not written to disk, uploaded, or leaked. S7. Output only the AU intensity vector and the low-dimensional feature vector output by the privacy bottleneck layer for subsequent multimodal assessment of labor pain.

2. The lightweight edge inference method for facial expression features in a delivery room scenario according to claim 1, characterized in that, The lightweight AU intensity regression network is LiteAUNet, which includes three parallel sub-graph input branches, a feature fusion module, a privacy bottleneck fully connected layer, and an AU intensity output head. The three parallel sub-graph input branches correspond to the upper orbital AOI region between the eyebrows, the middle nasal root AOI region, and the lower upper lip AOI region, respectively. Each parallel sub-graph input branch is composed of multiple depthwise separable convolutions. The output features of each parallel sub-graph input branch are concatenated and fused, and then input into the privacy bottleneck fully connected layer. The privacy bottleneck fully connected layer outputs a privacy bottleneck feature vector. The privacy bottleneck feature vector is input into the AU intensity output head, which outputs AU intensity values ​​related to pain.

3. The lightweight edge inference method for facial expression features in a delivery room scenario according to claim 2, characterized in that, The total number of parameters in LiteAUNet is less than 500K; the output of the privacy bottleneck fully connected layer is a 64-dimensional privacy bottleneck feature vector, and the AU strength output head outputs a 6-dimensional AU strength value.

4. The lightweight edge inference method for facial expression features in a delivery room scenario according to claim 2, characterized in that, Adversarial training is performed on the LiteAUNet; during the training phase, an adversarial decoder network is introduced, which takes the privacy bottleneck feature vector as input and reconstructs the original face image; a privacy penalty term is added to the total loss function of LiteAUNet, which is a negative weighted sum of the reconstruction error between the reconstructed image output by the adversarial decoder and the original face image; During training, the weight coefficient corresponding to the privacy penalty term was gradually increased from 0.01 to 1.

0.

5. A lightweight edge inference method for facial expression features in a delivery room setting according to claim 1, characterized in that, The skip-frame adaptive strategy calculates the face tracking confidence based on the KLT optical flow method, and the hierarchical control rules are as follows: when the tracking confidence is greater than 0.9, the face detection step is skipped, and only key point alignment and AU inference are performed; when the tracking confidence is greater than 0.6 and less than or equal to 0.9, AU inference is performed, and face key point correction is performed once every preset fixed number of frames; when the tracking confidence is less than or equal to 0.6, the face detection, key point localization and alignment, and AU inference process are fully executed; the above strategy reduces the average inference frequency of face detection to 25fps.

6. The lightweight edge inference method for facial expression features in a delivery room scenario according to claim 1, characterized in that, The instant secure erasure employs a three-pass overwrite mechanism, sequentially writing 0x00, 0xFF, and a random byte to the memory buffer storing the image data. After erasure is completed, a memory barrier instruction is invoked to confirm the overwrite operation is effective before releasing the memory buffer. All image data resides only in volatile RAM and is not written to the file system for persistent storage.

7. A lightweight edge inference method for facial expression features in a delivery room setting according to claim 1, characterized in that, The real-time video stream is captured by an RGBIR binocular camera; when the ambient light intensity is lower than the preset light threshold, the 940nm infrared fill light is automatically turned on and the camera switches to the IR infrared channel to capture images; IR noise data enhancement samples are added during the training phase of the lightweight AU intensity regression network; in the IR infrared acquisition mode, the AU regression confidence threshold is increased to reduce AU misjudgment caused by noise.

8. A lightweight edge inference method for facial expression features in a delivery room setting according to claim 1, characterized in that, If the confidence of facial detection or AU inference for a consecutive preset number of frames is lower than the preset quality threshold, it is determined that the facial data quality is continuously declining; the weight of the subjective branch of facial expression in the multimodal assessment fusion model of labor pain is reduced or zeroed, and objective data of uterine contractions and vital signs are used to conduct pain assessment, and facial data quality alarm signals are pushed to the superior nursing station; in the scenario of labor room with multiple delivery beds, each set of acquisition and inference equipment operates independently; the equipment locks the target face within a distance of 1.5-2.5m as the tracking object based on depth information, and filters out other irrelevant faces in the field of view; the equipment does not exchange image data and feature data with each other, and each equipment independently outputs AU intensity vector and privacy bottleneck feature vector.

9. A lightweight edge inference method for facial expression features in a delivery room setting according to claim 1, characterized in that, The low-dimensional feature vector of the privacy bottleneck layer output is encrypted using AES128 and then transmitted to the edge server via the local area network.

10. A lightweight edge inference device for facial expression features in a delivery room setting, used to execute the lightweight edge inference method for facial expression features in a delivery room setting as described in any one of claims 19, comprising an RGBIR binocular camera module, a 940nm infrared fill light module, an embedded NPU computing module, a memory secure erase logic module, and an encrypted transmission module; The RGBIR binocular camera module is used to acquire RGB and IR video streams of the delivery room bed area; the 940nm infrared fill light module is used to provide infrared fill light for the mother's face in low-light environments; the embedded NPU computing module is equipped with a face detection and tracking model, a key point detection model, and a lightweight AU intensity regression network to achieve face detection and tracking, key point alignment, AOI cropping, and AU feature inference. The memory-secure erasure logic module is used to perform a three-pass overwrite secure erasure operation on various image buffers in RAM memory; the encrypted transmission module is used to perform AES128 encryption on the feature vector output by the privacy bottleneck layer and output the encrypted feature vector to the outside via the local area network; the device does not perform persistent storage on any original facial image data; the embedded NPU computing module uses an RK3588 chip or a Huawei Ascend310 chip; the lightweight AU strength regression network deployed in the embedded NPU computing module is quantized; the single-frame AU inference time is less than 5ms; the overall end-to-end processing latency of the device is less than 12ms, and it supports real-time processing of 30fps video streams.