Privacy protection depression recognition method and system based on deep optical imaging, computer device and medium
By constructing optical and depth models using depth optical imaging technology, and combining feature extraction and information fusion, the privacy leakage problem in facial image depression recognition is solved, achieving efficient privacy protection and accurate depression recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH
- Filing Date
- 2024-02-08
- Publication Date
- 2026-07-21
AI Technical Summary
Existing facial image-based depression recognition methods pose a risk of privacy breaches during the initial data storage and transmission stages, failing to effectively protect user privacy and security.
We employ a depth-based optical imaging approach, constructing optical and depth models, using Zernike polynomials to represent lens surfaces, and combining spatial attention convolutional networks and multimodal Transformer models to extract features from facial images and identify depression. By fusing emotional and depression information, we achieve privacy protection.
While ensuring recognition accuracy, it effectively protects user privacy, prevents personal information in facial images from being identified and recovered, and provides the best privacy protection performance.
Smart Images
Figure CN118097745B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent medical technology, specifically to a privacy-preserving depression identification method, system, computer device, and medium based on depth optical imaging. Background Technology
[0002] Depression has become a major mental illness threatening human life and health, affecting people of all ages. Current diagnostic methods for depression primarily focus on interviews guided by professional physicians, where experienced doctors can glean diagnostic clues from abnormal facial expressions. Some research has already utilized facial expression signals for automated detection and identification of depression. Current mainstream research primarily uses visual signals (facial videos captured by cameras) and automated detection algorithms. These algorithms use well-designed image feature extractors to extract features from facial images, and then use statistical algorithms to map these features to a depression severity score, typically using assessment results such as the BDI or PHQ scale. However, facial images from subjects are highly sensitive data because they contain biometric identifiers, personal identification information, age, and other characteristics, posing a high risk of privacy breaches. To minimize these risks, previous research has introduced facial image anonymization techniques. This type of method does not use fully visible facial images, but instead uses various facial biometric features (such as facial landmarks) for depression identification.
[0003] In the field of depth optics, lenses can be parameterized to different layers of deep learning models, a process involving joint optimization of the optical system and downstream tasks. The basic idea of depth optics is to capture images that are beneficial to downstream tasks through lenses whose physical structure is specially optimized, thereby improving performance on specific tasks. Similarly, optical imaging systems can also be designed to intentionally reduce image quality or mask private information, but the imaging results can still support downstream tasks.
[0004] Currently, in the field of facial depression recognition, directly using subjects' facial images for identification involves the formation and transmission of facial privacy information, which cannot guarantee user privacy and security. Existing privacy protection methods mainly include using algorithms to generate privacy-free facial images to replace the original images for recognition, or extracting important facial features from facial images to avoid transmitting complete facial images for recognition. Although these methods can protect privacy information from being stored or transmitted to a certain extent, the first step of these methods is usually to acquire a complete and identifiable facial image through camera imaging, which still poses a risk of leakage in the initial data storage and transmission stages. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a privacy-preserving depression identification method, system, computer device, and medium based on depth optical imaging.
[0006] The present invention provides a privacy-preserving depression identification method based on depth optical imaging, the method comprising the following steps:
[0007] S1. Construct an optical model and pre-train a lens with privacy protection capabilities;
[0008] S2. Construct a deep model and incorporate sentiment information data;
[0009] S3. Introduce depression information data into optical and depth models;
[0010] S4. Integrate emotional information and depression information to predict the level of depression.
[0011] Furthermore, S1 specifically includes the following steps:
[0012] An optical model of the lens surface is constructed using Zernike polynomials, represented as follows:
[0013]
[0014] Among them, Z j Let α be the j-th factorization term of the Zernike polynomial. j The coefficients to be optimized;
[0015] Based on the optical model, the point spread function corresponding to the lens is obtained, and its imaging result is related to the point spread function. The point spread function at a distance z from the lens is represented as an infinitely distant point source with amplitude A and phase Φ, specifically calculated as follows:
[0016]
[0017] The image result received by the camera I c It can be calculated in the following ways:
[0018] I c (x,y)=∫(I λ *P λ (x,y)κ c (λ)dλ+η,
[0019] Among them, I λ This represents the light field corresponding to the actual scene, where λ corresponds to the wavelengths of R, G, and B light, which are 640nm, 550nm, and 460nm, respectively.
[0020] To achieve the goal of visual degradation from a lens, the mean square error between the image result and the real image is maximized:
[0021]
[0022] A spatial attention convolutional network model is used for feature extraction from images. This spatial attention convolutional network is constructed by stacking multiple layers of spatial attention convolutional layers. Each layer uses the following method for feature extraction:
[0023]
[0024]
[0025] in, This represents a convolution with K groups, where BN and ReLU are the normalization and activation functions, respectively; ⊙ represents the Hadamard product symbol; all layer parameters are initialized using the Kaiming initialization method during the initialization phase;
[0026] For the obtained identity features, the model is optimized through backpropagation using an inverse triplet loss function:
[0027]
[0028] in, These are the features extracted through the identity feature model, corresponding to a baseline image a, an image p with the same identity as the baseline image, and an image n with a different identity; m is a boundary distance, taken as m=1; the distance d(·) is calculated as follows:
[0029] Introducing the key point recognition loss function L lm The loss function is calculated by measuring the mean squared error between the actual keypoint coordinates and the predicted coordinates:
[0030]
[0031] Where k is the index of the facial key points, lm and These represent the predicted and actual keypoint coordinates, respectively. Keypoints are obtained by performing a shallow nonlinear mapping on the facial embedding extracted from the identity feature recognition model, resulting in the predicted keypoint coordinates in the imaging result. The overall loss function is:
[0032] L i =αL v +βL id +γL lm
[0033] The weights are set as α = -0.1, β = 1, and γ = 1, respectively.
[0034] Furthermore, S2 specifically includes the following steps:
[0035] By utilizing the identity feature recognition model obtained in step S1, and inheriting its model structure and parameters, as well as the learned privacy-de-stigmatized facial features, and using the lens obtained in the previous step, further fine-tuning of the parameters of the emotion feature recognition model and the optical lens is performed. The recognition of facial emotion categories is trained and learned through the following cross-entropy loss:
[0036]
[0037] The last feature layer of the emotion feature recognition model obtained after training is retained as the emotion feature.
[0038] Furthermore, S3 specifically includes the following steps:
[0039] The structure and parameters of the emotion feature recognition model trained in step S2 are copied and reused, and the parameters of this copied model are fine-tuned to serve as a depression feature recognition model. The parameters of the optical lens in step S2 are then fine-tuned to obtain information about the level of depression.
[0040] Convert the real-valued depression scores into a discrete probability distribution, represented as p = [p1, p2, ..., p...]. c ], where c represents the level of depression based on the depression score; during fine-tuning, the model's prediction p is minimized. d With reality The distribution similarity between positive and negative feature pairs is assessed, and histogram loss is used to constrain the label-perceived similarity between positive and negative feature pairs by constructing a histogram H of the distributions of positive and negative features. + H - To calculate this loss, the depression metric loss can be defined as:
[0041]
[0042] Where n represents the number of samples and r represents the index of the constructed histogram; the last feature layer of the depression feature recognition model obtained after training is retained as the depression feature.
[0043] Furthermore, step S4 specifically includes the following steps:
[0044] The fusion model, Multimodal Transformer, aggregates sentiment features and coarse-grained depression features to predict refined depression scores. The core of the fusion model is a cross-attention mechanism, which fuses features from two different information sources and performs automatic weighted decision-making. Its calculation method is as follows:
[0045] Z = softmax((W QS1)(W K S2))W V S2+softmax((W Q S2)(W K S1))W V S1,
[0046] Among them, W Q W K W V S1 and S2 are the attention vector weights, respectively, and the emotion feature and depression feature are the emotion feature and depression feature, respectively. For the obtained fusion feature Z, a multi-layer nonlinear mapping is used to obtain the final depression score output.
[0047] The complete model set includes the optical model, emotion feature recognition model, and depression feature recognition model obtained from step S3, along with their corresponding pre-trained weights. At this stage, these model weights are frozen, and only the parameters of the final fusion model are optimized. This model uses depression scores. The ground truth is used for training, enabling the multimodal Transformer model to learn the mapping from sentiment features and depression features to depression scores S; this step optimizes the fusion model using mean squared error loss.
[0048]
[0049] The present invention also relates to a privacy-preserving depression identification system based on depth optical imaging, the system comprising a computer module that applies a privacy-preserving depression identification method based on depth optical imaging.
[0050] The present invention also relates to a computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described privacy-preserving depression identification method based on depth optical imaging.
[0051] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described privacy-preserving depression identification method based on depth optical imaging.
[0052] Beneficial effects
[0053] This invention designs a depression recognition model based on depth optics, capable of identifying depression in facial images without generating any privacy information, while ensuring recognition accuracy. By comparing this model with other methods in terms of balancing recognition performance and privacy protection, the model designed in this invention exhibits the best performance. Compared with image blurring methods of varying degrees (Gaussian blur, defocus blur), the privacy protection strategy of this invention achieves optimal privacy protection performance while minimizing recognition error. Attached Figure Description
[0054] Figure 1 This is a diagram illustrating different frameworks for the privacy-preserving depression identification based on depth optical imaging, as described in this invention.
[0055] Figure 2 This is a diagram of the deep optical depression recognition model in this invention.
[0056] Figure 3 This diagram illustrates the identification error level and privacy protection performance under different privacy protection methods in this invention.
[0057] Figure 4 These are images generated by different privacy protection strategies in this invention.
[0058] Figure 5 These are images generated by different privacy protection strategies in this invention.
[0059] Figure 6 These are feature activation heatmaps for different models in this invention. Detailed Implementation
[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0061] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0062] The following is in conjunction with the appendix Figure 1 -6 and specific embodiments further illustrate the present invention, but are not intended to limit the present invention.
[0063] This invention uses a specially optimized lens to obtain images with personal information erased. This lens requires joint optimization with a depth recognition model, and the entire process is as follows: Figure 2As shown, the system comprises two key components: an optical model and a deep feature extraction model. In the optical model, a learnable optical device is parameterized to generate privacy-preserving images. The deep feature model is responsible for subsequently extracting depressive features and using these features to predict the final degree of depression.
[0064] Considering the difficulty in obtaining a complete dataset with identity, emotion, and depression labels simultaneously in practice, this invention introduces progressive learning to train the entire model. This involves progressively training the optical lens, the model for identity feature recognition, the model for emotion feature recognition, and the model for depression feature recognition. The final score is predicted by the fusion model, the multimodal Transformer.
[0065] S1. Construct an optical model and pre-train a lens with privacy protection capabilities.
[0066] To achieve privacy protection, images need to be visually difficult to distinguish and also impossible for deep face recognition algorithms to recognize.
[0067] The optical model used in this invention is modeled using Zernike polynomials, which are expressed as follows:
[0068]
[0069] Among them, Z j Let α be the j-th factorization term of the Zernike polynomial. j The coefficients to be optimized are , and the learnable parameters will be optimized together with the deep model. During the initialization phase, α is set to 0 for all terms except α3, which is set to -51, to improve the optimization convergence speed.
[0070] Based on the lens model, the point spread function corresponding to the lens can be obtained, and its imaging result is related to the point spread function. The point spread function at a distance z from the lens can be expressed as follows: The relationship between amplitude A and phase Φ of an infinitely distant point source is calculated as follows:
[0071]
[0072] The image result received by the camera I c It can be calculated in the following ways:
[0073] I c (x,y)=∫(I λ *P λ (x,y)κ c (λ)dλ+η
[0074] Among them, I λThis represents the light field corresponding to the actual scene, where λ corresponds to the wavelengths of R, G, and B light, which are 640nm, 550nm, and 460nm respectively. The impact of noise on the imaging model is additionally considered here, therefore... Gaussian noise.
[0075] To achieve the goal of visual degradation with a lens, this can be accomplished by maximizing the mean square error between the image result and the real image:
[0076]
[0077] To further obfuscate the identifiable deep identity features of faces, an identity feature recognition model is used to extract and process identity features from the obtained images. For this identity feature recognition model, a spatial attention convolutional network model is employed to extract features from the images. This spatial attention convolutional network is constructed by stacking multiple layers of spatial attention convolutional layers. Each layer uses the following method for feature extraction:
[0078]
[0079]
[0080] in, This represents a convolution with K groups, where BN and ReLU are the normalization and activation functions, respectively. ⊙ represents the Hadamard product symbol. All layer parameters are initialized using the Kaiming initialization method during the initialization phase.
[0081] For the obtained identity features, the model is optimized through backpropagation using an inverse triplet loss function:
[0082]
[0083] in, These are the features extracted through the identity feature model, corresponding to a baseline image 'a', an image 'p' with the same identity as the baseline image, and an image 'n' with a different identity. 'm' represents a boundary distance, which is set to 1 here. The distance 'd(·)' is calculated as follows.
[0084] Furthermore, facial landmarks have proven crucial in expressing fundamental features of depression. To ensure that blurred images still retain these important depression-related features, a landmark recognition loss function L needs to be introduced. lm This loss function is calculated by measuring the mean squared error between the actual keypoint coordinates and the predicted coordinates:
[0085]
[0086] Where k is the index of the facial key points, lm and These represent the predicted and actual keypoint coordinates, respectively. The keypoint prediction method involves performing a shallow nonlinear mapping on the facial embedding extracted from the identity feature recognition model to obtain the predicted keypoint coordinates in the imaging result.
[0087] The overall loss function in this step is
[0088] L i =αL v +βL id +γL lm
[0089] The weights are set as α = -0.1, β = 1, and γ = 1, respectively.
[0090] S2. Construct a deep model and incorporate sentiment information data.
[0091] In this step, emotional information is incorporated as a form of prior emotional knowledge, which aids in the identification of depression because research indicates that individuals with depression often exhibit abnormal levels of emotional arousal. Additionally, the lens should also be able to include information helpful for emotional feature recognition in the captured image. Utilizing the identity feature recognition model obtained in S1, and inheriting its model structure and parameters, as well as the learned de-privacy facial features, while using the lens obtained in the previous step, further fine-tuning of the emotional feature recognition model and optical lens parameters is performed. The recognition of facial emotion categories is trained using the following cross-entropy loss:
[0092]
[0093] The last feature layer of the emotion feature recognition model obtained after training is retained as the emotion feature.
[0094] S3. Introducing depression information data into optical and depth models.
[0095] To introduce coarse-grained features related to depression and establish a connection between emotion and depression features, the structure and parameters of the emotion feature recognition model trained in S2 are copied and reused, and the parameters of this copied model are fine-tuned as a depression feature recognition model. At the same time, the parameter fine-tuning steps of the optical lens in S2 are continued to obtain information about the level of depression.
[0096] This step converts the real-valued depression scores into a discrete probability distribution, represented as p = [p1, p2, ..., p...]. c ], where c represents the level of depression based on the depression score, as shown in Table 1.
[0097] During the fine-tuning process in this step, the model's prediction p is minimized. d With reality The distribution similarity between features is assessed, and histogram loss is used to constrain the label-perceived similarity between positive feature pairs (two samples from the same depression level) and negative feature pairs by constructing a histogram H of the distributions of positive and negative features. + H - To calculate this loss. Overall, the depression metric loss can be defined as:
[0098]
[0099] Where n represents the number of samples and r represents the index of the constructed histogram.
[0100] The last feature layer of the depression feature recognition model obtained after training is retained as a feature of depression.
[0101] Table 1. Mapping relationship between BDI-II Depression Scale scores and depression levels
[0102] level 0 (minimum) 1 (Mild) 2 (Moderate) 3 (Severe)
[0103] S4. Integrate emotional and depression information to predict depression levels.
[0104] The multimodal Transformer fusion model aggregates sentiment features and coarse-grained depression features to predict fine-grained depression scores. The core of this fusion model is the cross-attention mechanism, which fuses features from two different information sources and automatically weights them for decision-making. Its calculation method is as follows:
[0105] Z = softmax((W Q S1)(W K S2))W V S2+softmax((W Q S2)(W K S1))W V S1,
[0106] Among them, W Q W K W V S1 and S2 are the attention vector weights, respectively, and the emotion feature and depression feature are the emotion feature and depression feature, respectively. For the obtained fused feature Z, a multi-layer nonlinear mapping is used to obtain the final depression score output.
[0107] At this point, the complete model includes the optical model, emotion feature recognition model, and depression feature recognition model obtained from S3, along with their corresponding pre-trained weights. During this stage, these model weights are frozen, and optimization is performed only on the final fusion model parameters. This model uses depression scores. The data is used as ground truth for training, enabling the multimodal Transformer model to learn the mapping from sentiment features and depression features to depression scores S. This step optimizes the fusion model using mean squared error loss.
[0108]
[0109] The above four steps are executed sequentially, using the loss function corresponding to each step to optimize the corresponding part of the model, such as... Figure 2 As shown, the emotion feature recognition model is derived from the weights of the identity feature recognition model with fine-tuning, while the depression feature recognition model is derived from the weights of the emotion feature recognition model with fine-tuning, in order to integrate the prior information brought by each part.
[0110] Effect verification
[0111] The depression recognition model based on depth optics of this invention can identify depression in facial images without generating any privacy information, while ensuring accuracy. By comparing this model with other methods in terms of balancing recognition performance and privacy protection, the model designed in this invention exhibits the best performance.
[0112] like Figure 3 As shown, compared with image blurring methods of different degrees (Gaussian blur, out-of-focus blur), the privacy protection strategy of the present invention can achieve the best privacy protection performance while ensuring recognition error.
[0113] Compared to specific imaging results, the optical device-based imaging method designed in this invention also fully ensures the indistinguishability of information such as identity and age during image observation, such as... Figure 4 As shown.
[0114] Both inverse transformations using a known imaging model and inverse transformations using a deep learning model failed to recover the original information from the optical imaging results, further demonstrating the high level of security in terms of privacy protection of this invention. Figure 5 As shown.
[0115] The model designed in this invention also demonstrates superior overall recognition performance. As shown in Table 2, on the publicly available depression recognition datasets AVEC 2013 and AVEC 2014, compared with various existing privacy-preserving models, this invention achieves the lowest recognition error.
[0116] Table 2 shows the identification errors of various existing methods for identifying depression that support privacy protection.
[0117]
[0118] Furthermore, the activation graph analysis of the model features of this invention also shows that, compared with no-strategy optimization, the progressive optimization strategy designed in this invention has more reasonable model features and possesses sample specificity related to depression, such as... Figure 6 As shown.
[0119] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.
Claims
1. A privacy-preserving depression identification method based on depth optical imaging, the method comprising the following steps: S1. Construct an optical model and pre-train a lens with privacy protection capabilities; S2. Construct a deep model and incorporate sentiment information data; S3. Introduce depression information data into optical and depth models; S4. Integrate emotional information and depression information to predict the level of depression; S1 specifically includes the following steps: An optical model of the lens surface is constructed using Zernike polynomials, represented as follows: in, For the Zernike polynomial of the th Each decomposition term The coefficients to be optimized; Based on the optical model, the point spread function corresponding to the lens is obtained, and its imaging result is related to the point spread function; at the distance of the lens The point spread function at a given location is represented as an amplitude. With phase The calculation relationship for a point light source at infinity is as follows: The image received by the camera Calculated in the following way: in, This represents the light field corresponding to the actual scene. These correspond to the wavelengths of R, G, and B light, respectively, which are 640nm, 550nm, and 460nm. To achieve the goal of visual degradation from a lens, the mean square error between the image result and the real image is maximized: A spatial attention convolutional network model is used for image feature extraction. The spatial attention convolutional network is constructed by stacking multiple layers of spatial attention convolutional layers. Each layer uses the following method for feature extraction: in, Represents Convolution of groups, and These are the normalization function and the activation function, respectively. This is the symbol for Hadamard product; all layer parameters are initialized using the Kaiming initialization method during the initialization phase. For the obtained identity features, the model is optimized through backpropagation using an inverse triplet loss function: in, , , Each feature is extracted through an identity feature model, corresponding to a baseline image. An image with the same identity as the baseline image. and an image with a different identity. ; For a boundary distance, take ;distance The calculation method is as follows ; Introducing a key point recognition loss function The loss function is calculated by measuring the mean squared error between the actual keypoint coordinates and the predicted coordinates: in, An index for facial key points. and These represent the predicted and actual keypoint coordinates, respectively. Keypoints are obtained by performing a shallow nonlinear mapping on the facial embedding extracted from the identity feature recognition model, resulting in the predicted keypoint coordinates in the imaging result. The overall loss function is: The weights are respectively set as follows: .
2. The privacy-preserving depression identification method based on depth optical imaging according to claim 1, characterized in that, S2 specifically includes the following steps: By utilizing the identity feature recognition model obtained in step S1, and inheriting its model structure and parameters, as well as the learned privacy-de-stigmatized facial features, and using the lens obtained in the previous step, further fine-tuning of the parameters of the emotion feature recognition model and the optical lens is performed. The recognition of facial emotion categories is trained and learned through the following cross-entropy loss: The last feature layer of the emotion feature recognition model obtained after training is retained as the emotion feature.
3. The privacy-preserving depression identification method based on depth optical imaging according to claim 1, characterized in that, S3 specifically includes the following steps: The structure and parameters of the emotion feature recognition model trained in step S2 are copied and reused, and the parameters of this copied model are fine-tuned to serve as a depression feature recognition model. The parameters of the optical lens in step S2 are then fine-tuned to obtain information about the level of depression. Transform real-valued depression scores into discrete probability distributions, represented as follows: Where c represents the level of depression based on the depression score; during fine-tuning, the model's predictions are minimized. With reality The distribution similarity between positive and negative feature pairs is assessed, and histogram loss is used to constrain the label-aware similarity between positive and negative feature pairs by constructing histograms of the distributions of positive and negative features. To calculate this loss, the depression metric loss is defined as: in Represents the number of samples. The index of the constructed histogram is represented by the number of indexes. The last feature layer of the depression feature recognition model obtained after training is retained as a depression feature.
4. The privacy-preserving depression identification method based on depth optical imaging according to claim 1, characterized in that, Step S4 specifically includes the following steps: The fusion model, Multimodal Transformer, aggregates sentiment features and coarse-grained depression features to predict refined depression scores. The core of the fusion model is a cross-attention mechanism, which fuses features from two different information sources and performs automatic weighted decision-making. Its calculation method is as follows: in, , , These are the attention vector weights, and These are respectively affective features and depressive features; for the obtained fusion features Multi-layer nonlinear mapping is used to obtain the final depression score output; The complete model set includes the optical model, emotion feature recognition model, and depression feature recognition model obtained from step S3, along with their corresponding pre-trained weights; these model weights are frozen, and optimization is only performed on the parameters of the final fusion model; this model uses depression scores. The data is used as ground truth for training, enabling the multimodal Transformer model to learn from sentiment features and depression features to depression scores. The mapping; this step optimizes the fusion model using mean squared error loss: 。 5. A privacy-preserving depression identification system based on depth optical imaging, characterized in that, It includes a computer module that applies the privacy-preserving depression identification method based on depth optical imaging as described in any one of claims 1 to 4.
6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of any one of claims 1 to 4.