Sleep apnea risk assessment method and system based on face recognition

Through the sleep apnea risk assessment method based on face recognition, using mobile phones to collect and process images, combined with deep learning technology, a convenient and low-cost sleep apnea risk assessment at home is achieved, solving the inconvenience and high quality of traditional evaluation methods. Cost issues improve the convenience and accuracy of evaluation.

CN119092117AInactive Publication Date: 2024-12-06DAOBO MEDICAL TECH (BEIJING) CO LTD

Patent Information

Application Number
CN202411089671.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The risk assessment method of sleep apnea in the prior art requires patients to go to a hospital or sleep laboratory for overnight monitoring, and requires the collection of multi-angle facial and sign data, which leads to inconvenience and high cost.

Method used

Using the sleep apnea risk assessment method based on face recognition, user images are collected through the mobile phone, key points of the face and neck are detected and aligned, input images are generated, and inputted into the preset sleep apnea risk prediction model for prediction. Use deep residual networks and GradCAM technology to improve the accuracy and convenience of evaluation.

Benefits of technology

Users can use smartphones at home to complete sleep apnea risk assessment instantly, avoid the needs of traditional devices and complex data collection, improve the convenience and user experience of assessment, and improve the accuracy and credibility of assessment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119092117A_ABST
    Figure CN119092117A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence and medical health, in particular to a sleep apnea risk assessment method and system based on face recognition, and the method comprises the steps: collecting a user image which comprises the face and neck of a user; detecting key points in the user image, and performing face alignment processing on the user image based on the key points; identifying and cutting face and neck areas in the aligned user image to generate an input image; and inputting the input image into a preset sleep apnea risk prediction model, and outputting a prediction result by the model. According to the method, the requirements of traditional monitoring equipment and complex data acquisition can be avoided as far as possible, and the evaluation process of the patient is greatly simplified, so that the evaluation convenience and the user experience are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and medical health technology, and in particular to a sleep apnea risk assessment method and system based on face recognition. Background Art

[0002] Sleep apnea is a common sleep disorder characterized by repeated apnea and hypopnea during sleep. This disease not only causes frequent awakenings at night, seriously affecting sleep quality, but may also cause a series of health problems, such as high blood pressure, heart disease, diabetes, stroke, etc. Therefore, timely diagnosis and risk assessment of sleep apnea are of great significance, which can help patients detect problems early and take appropriate treatment measures to avoid serious complications.

[0003] At present, the traditional risk assessment method for sleep apnea requires patients to go to the hospital or sleep laboratory and use professional medical equipment for overnight monitoring to complete the risk assessment. In addition, with the continuous development of technology, some people have developed a sleep apnea prediction system for this disease, which collects facial data from multiple angles and multi-dimensional physical data of patients, conducts in-depth analysis of the data, and completes the risk prediction and assessment of sleep apnea.

[0004] However, both the traditional method and the subsequently developed sleep apnea prediction system may cause inconvenience to patients. In the traditional method, patients need to go to the hospital or sleep laboratory for overnight monitoring, which not only takes up a lot of patients' time, but also the cost of monitoring equipment is too high, causing inconvenience to patients. The sleep apnea prediction system needs to collect facial data from multiple angles and multi-dimensional physical sign data of patients. The collection of multi-angle facial data and multi-dimensional physical sign data also takes up a lot of patients' time, and still needs the assistance of certain equipment, which also causes inconvenience to patients. Therefore, how to simply, quickly and low-costly realize the assessment of sleep apnea risk is a difficult problem currently faced. Summary of the invention

[0005] This application provides a sleep apnea risk assessment method and system based on face recognition, which can avoid the need for traditional monitoring equipment and complex data collection as much as possible, and greatly simplify the patient assessment process, thereby improving the convenience of assessment and user experience. This application provides the following technical solutions:

[0006] In a first aspect, the present application provides a sleep apnea risk assessment method based on face recognition, the method comprising:

[0007] Collecting a user image, wherein the user image includes the user's face and neck;

[0008] Detecting key points in the user image, and performing face alignment processing on the user image based on the key points;

[0009] Identify and crop the face and neck area in the aligned user image to generate an input image;

[0010] The input image is input into a preset sleep apnea risk prediction model, and the model outputs a prediction result.

[0011] In a specific implementation scheme, collecting the user image includes:

[0012] Integrate an SDK with face detection function into the mobile phone;

[0013] The mobile phone calls the SDK of the face detection function to process the image or video frame captured by the mobile phone camera, recognize the face and calculate the head posture, and obtain the roll angle, pitch angle, and yaw angle corresponding to the head posture;

[0014] Based on the results of face detection and head posture evaluation, voice and text prompts are given to the user to adjust the position. When the user adjusts the face to the appropriate position, if the head posture angle meets the preset angle offset condition, the posture score corresponding to the photo is calculated. The posture score formula is as follows:

[0015]

[0016] Where N = 3, i = 1, 2, 3 represent the roll angle, pitch angle, and yaw angle respectively, θ i represents the actual angle of the i-th orientation, represents the expected angle of the i-th orientation, w i represents the weight of the i-th position;

[0017] When the posture score is less than the score of the previous photo, the current photo is updated to the corresponding target position to obtain multiple user images at different angles that meet the head posture angle requirements.

[0018] In a specific implementation scheme, detecting key points in the user image and performing face alignment processing on the user image based on the key points includes:

[0019] Perform face detection on user images and identify key points of the face;

[0020] Calculate the tilt angle of the face using the detected key points;

[0021] Face alignment is achieved through affine transformation algorithm.

[0022] In a specific possible implementation scheme, after detecting key points in the user image and performing face alignment processing on the user image based on the key points, the method further includes:

[0023] The verification method combining key point position and texture features is used to evaluate the effect of face alignment. The Euclidean distance between the key point position of the aligned face and the standard position is calculated using the following formula to obtain the key point position accuracy score D keypoints :

[0024]

[0025] Among them, d i is the Euclidean distance of the i-th key point, N is the number of key points;

[0026] The aligned face image is converted into a grayscale image, and its local binary pattern feature histogram is calculated. The chi-square distance of the LBP histogram of the image before and after alignment is calculated using the following formula to obtain the texture feature similarity score D texture :

[0027]

[0028] Among them, H o and H a They are the LBP histograms of the user images before and after;

[0029] The final score is calculated using the weighted average formula:

[0030] Score = α·D keypoints +β·(1-D texture )

[0031] Among them, α and β are weight coefficients. A threshold is set and the final score is compared with the threshold. If the final score is greater than or equal to the threshold, the image alignment effect is considered to be good. If the final score is less than the threshold, the image alignment operation needs to be performed again or the alignment algorithm or parameters need to be further optimized.

[0032] In a specific implementation scheme, the identifying and cropping of the face and neck area in the aligned user image to generate the input image includes:

[0033] Use InsightFace's 106-point precise positioning technology or Dlib's 68-point annotation technology to locate key points on the face and neck in the user's image.

[0034] Based on the detected key points, the boundary area of ​​the face and neck is determined, and the identified face and neck area is cropped in the image;

[0035] The cropped image only contains the key face and neck areas.

[0036] In a specific implementation scheme, the input image is input into a preset sleep apnea risk prediction model, and the model outputs a prediction result including:

[0037] The improved ResNet50 model is selected as the core recognition engine for learning and extracting features from face images. The last fully connected layer in the ResNet50 model architecture is removed, and the output of the second-to-last layer is taken as the feature. After combining the features of photos from multiple angles, the features are submitted to SVM for classification prediction.

[0038] Prepare a labeled face dataset, which includes normal and sleep apnea samples. The annotation ensures that each sample has the correct category label and the probability label of the risk level;

[0039] The improved ResNet50 model is trained using the prepared data set to obtain a sleep apnea risk prediction model.

[0040] In a specific implementation scheme, the step of inputting the input image into a preset sleep apnea risk prediction model, and after the model outputs a prediction result, further includes:

[0041] When the sleep apnea risk prediction model makes predictions on the input image, the GradCAM technology is used to generate a heat map of the face by analyzing the gradient information of the target category in the deep network.

[0042] In the second aspect, the present application provides a sleep apnea risk assessment system based on face recognition, which adopts the following technical solutions:

[0043] A sleep apnea risk assessment system based on face recognition, comprising:

[0044] An image acquisition module, used to acquire a user image, wherein the user image includes the user's face and neck;

[0045] A face alignment module, used to detect key points in the user image and perform face alignment processing on the user image based on the key points;

[0046] An image cropping module, used to identify and crop the face and neck area in the aligned user image to generate an input image;

[0047] The model prediction module is used to input the input image into a preset sleep apnea risk prediction model, and the model outputs a prediction result.

[0048] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement a sleep apnea risk assessment method based on face recognition as described in the first aspect.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the storage medium stores a program, and when the program is executed by a processor, the program is used to implement a sleep apnea risk assessment method based on face recognition as described in the first aspect.

[0050] In summary, the beneficial effects of this application include at least:

[0051] 1) Users can complete facial image collection and sleep apnea risk assessment through their mobile phones without the need for professional equipment and complex operations, greatly improving the convenience of the assessment process and user experience.

[0052] 2) Combining advanced face positioning technology, face alignment technology and regional cropping strategy, it effectively reduces photo differences and background interference, and uses the deep residual network ResNet50 model and N-fold cross-validation method to improve the accuracy of sleep apnea risk prediction.

[0053] 3) Users can obtain risk assessment results instantly and intuitively understand the key areas of concern of the model through GradCAM heat maps, which enhances users' understanding and trust in the assessment results.

[0054] Through the mobile application combined with deep learning technology, users can use their smartphones at home to collect facial and neck images and use the built-in face detection and alignment algorithm to ensure image accuracy. Subsequently, these images are analyzed by a preset deep learning model to assess the user's sleep apnea risk. The above method not only avoids the need for traditional monitoring equipment and complex data collection, but also greatly simplifies the patient's assessment process, thereby improving the convenience of assessment and user experience.

[0055] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flowchart of a sleep apnea risk assessment method based on face recognition in an embodiment of the present application.

[0057] Figure 2 It is a schematic diagram of the overall process of the sleep apnea risk assessment method based on face recognition in an embodiment of the present application.

[0058] Figure 3 It is a schematic diagram of a use case for collecting multiple images of a user in an embodiment of the present application.

[0059] Figure 4 It is a structural diagram of the ResNet50 model architecture in the embodiment of the present application.

[0060] Figure 5 It is a schematic diagram of the model structure of multi-angle image merging processing in an embodiment of the present application.

[0061] Figure 6 It is a structural block diagram of a sleep apnea risk assessment system based on face recognition in an embodiment of the present application.

[0062] Figure 7 It is a block diagram of an electronic device for sleep apnea risk assessment based on face recognition in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0064] Optionally, the present application takes the sleep apnea risk assessment method based on face recognition provided in each embodiment as an example of being used in an electronic device, where the electronic device is a terminal or a server, and the terminal may be a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0065] Reference Figure 1 , is a flowchart of a sleep apnea risk assessment method based on face recognition provided by an embodiment of the present application, the method comprising at least the following steps:

[0066] Step S101: Collect a user image, where the user image includes the user's face and neck.

[0067] Step S102: Detect key points in the user image, and perform face alignment processing on the user image based on the key points.

[0068] Step S103: Identify and crop the face and neck area in the aligned user image to generate an input image.

[0069] Step S104: input the input image to a preset sleep apnea risk prediction model, and the model outputs a prediction result.

[0070] In step S101, firstly, a user image is collected through a mobile phone, wherein the user image needs to include the entire face and neck of the user. Specifically, an SDK with a face detection function is integrated in the mobile phone, such as Apple ARKit, Google ML Kit or SDK provided by other third parties. When the user opens the mobile phone to perform image collection operations, the mobile phone calls the face detection function to process the image or video frame captured by the mobile phone camera to identify the position and size of the face. According to the result of the face detection, a prompt is given to the user to adjust the position. The prompt can be achieved by drawing guide lines, boxes or texts on the screen. When the user adjusts the face to a suitable position, the user is allowed to take a photo to confirm, thereby completing the collection of the user image.

[0071] Optionally, in another feasible embodiment, multiple user images of the user may also be collected. For details, refer to Figure 2 The mobile phone uses Mediapipe's face landmark technology to perform fast face recognition, and combines it with opencv to estimate the head posture, thereby calculating the three angles corresponding to the head posture, namely roll, pitch, and yaw. Then, from a clinical medical perspective, we focus on photos from multiple angles and make angle requirements for multiple user images. The specific requirements are shown in the following table:

[0072]

[0073]

[0074] At the same time, based on the results of face detection and head posture evaluation, voice and text prompts are given to the user to adjust the position. When the user adjusts the face to the appropriate position, if the head posture angle meets the preset angle offset condition, the posture score corresponding to the photo is calculated. The posture score formula is as follows

[0075]

[0076] Where N = 3, i = 1, 2, 3 represent the three attitudes of roll, pitch, and yaw respectively, θ i represents the actual angle of the i-th orientation, represents the expected angle of the i-th orientation, w i Indicates the weight of the i-th orientation. When the posture score is less than the score of the previous photo, the current photo is updated to the corresponding target position, so as to ensure that the user's head captures the best posture score photo as much as possible during dynamic movement, ensuring the quality of face photo collection. Finally, multiple user images with different angles that meet the head posture angle requirements are obtained.

[0077] In step S102, firstly, the SDK in step S101 is used to perform face detection on the user image submitted by the user, and the key points of the face are identified, such as the inner and outer corners of the eyes, the tip of the nose, the corners of the mouth, etc. Then, the tilt angle of the face is calculated using the detected key points, for example, the tilt angle of the line connecting the center points of the two eyes or the tilt angle of the line connecting the tip of the nose and the corners of the mouth is calculated, so as to determine the rotation angle of the face. Finally, the accurate alignment of the face is achieved through the affine transformation algorithm. Affine transformation is a linear transformation method that can achieve image alignment through operations such as rotation, translation, and scaling. The specific operation is as follows: According to the coordinates of the key points, the affine transformation matrix is ​​calculated. The affine transformation matrix is ​​used to transform the position of the detected key points to the standard position to achieve face alignment. Then, the calculated affine transformation matrix is ​​used to transform the photos submitted by the user. The specific operations include rotating the image to align the face vertically; translating the image to align the center of the face; scaling the image to make the face size meet the standard size.

[0078] Optionally, after the face is aligned, a verification method combining key point positions and texture features can be used to evaluate the effect of face alignment. Specifically, the Euclidean distance between the key point positions of the aligned face and the standard positions is first calculated using the following formula to obtain the key point position accuracy score D: keypoints :

[0079]

[0080] Among them, d i is the Euclidean distance of the i-th key point, and N is the number of key points. Then, the aligned face image is converted into a grayscale image, and its local binary pattern (LBP) feature histogram is calculated. The chi-square distance of the LBP histogram of the image before and after alignment is calculated using the following formula to obtain the texture feature similarity score D texture :

[0081]

[0082] Among them, H o and H a They are the LBP histograms of the user images before and after. Finally, in order to comprehensively consider the key point position accuracy and texture feature similarity, the final score is calculated using the weighted average calculation formula:

[0083] Score = α·D keypoints +β·(1-D texture )

[0084] Among them, α and β are weight coefficients, which are used to balance the importance of key point position accuracy and texture feature similarity, and α+β=1. Then set a threshold based on experience or experimental results, and compare the final score with the threshold. If the final score is greater than or equal to the threshold, it is considered that the image alignment effect is good. If the final score is less than the threshold, it is necessary to re-align the image or further optimize the alignment algorithm or parameters. The verification method combining key point position and texture features is used to evaluate the effect of face alignment. The key point position accuracy can directly reflect the matching degree of face geometry. By calculating the average Euclidean distance of the key point position, the alignment accuracy of the face in geometric position can be ensured to a certain extent. Texture feature similarity can reflect the consistency of image texture. Through the similarity evaluation of the local binary pattern (LBP) histogram, it can be ensured that the original texture information is retained in the aligned image and the loss of details is reduced. Combining the key point position accuracy and texture feature similarity can reduce the deviation caused by a single indicator, comprehensively consider geometric and texture information, and improve the reliability and accuracy of the alignment algorithm. In addition, in some cases, such as face recognition and expression analysis, accurate alignment of geometric positions may be the key. In other cases, such as surface texture analysis or cosmetic effect evaluation, maintaining the consistency of texture features may be more important. The comprehensive scoring method can adapt to different application needs. In summary, combining the key point position accuracy and texture feature similarity for alignment effect evaluation can provide more comprehensive and accurate evaluation results and improve the reliability and adaptability of the alignment algorithm. This method can not only ensure the accuracy of the geometric position, but also maintain the consistency of the texture features, so that satisfactory alignment effects can be obtained in different application scenarios.

[0085] In step S103, the key points of the face and neck in the user image are first located using the 106 punctuation point precise positioning technology provided by InsightFace or the 68-point annotation technology of Dlib. InsightFace provides 106 key points, covering the key positions of the face and neck, including the eyes, nose, mouth, cheekbones, chin and other positions. Dlib's 68-point annotation technology can also provide similar key point information. Although the number of points is less, it can also effectively identify facial and neck features. Then, based on the detected key points, the boundary area of ​​the face and neck is determined, and the identified face and neck areas are cropped in the image. The cropping operation is intended to remove irrelevant background information so that the image focuses on the analysis and processing of facial and neck features. The cropped image only contains the key face and neck areas, which is convenient for subsequent feature analysis and processing steps.

[0086] In step S104, the ResNet50 model is first selected as the core recognition engine for learning and extracting features from facial images to predict the risk of sleep apnea. ResNet50 is a deep residual network with a deep network structure and excellent feature learning capabilities. A large number of labeled face data sets are then prepared, including normal and sleep apnea samples. The annotation ensures that each sample has the correct category label (whether there is sleep apnea) and the probability label of the risk level. The ResNet50 model is trained using the prepared data set to obtain a sleep apnea risk prediction model. During the training process, the model optimizes the network parameters through the back propagation algorithm to minimize the error between the predicted output and the true label.

[0087] In addition, preferably, reference Figure 3 , is a schematic diagram of the structure of the ResNet50 model architecture in the embodiment of the present application. Since the present application obtains seven user images from different angles in step S101, the traditional ResNet50 model is improved by removing the last fully connected layer in the ResNet50 model architecture, taking the output of the second-to-last layer as the feature, and merging them and submitting them to SVM for classification. Figure 4 , which is a schematic diagram of the model structure of multi-angle image merging processing in the embodiment of the present application, that is, ResNet50 processes 7 pictures of the user at the same time, and stacks the 7 pictures into a 7x3x224x224 tensor during input, and hands it over to the deep learning model ResNet50 for processing; the ResNet50 model extracts 2048-dimensional vector features for each image; when the penultimate layer of ResNet50 outputs, the feature vectors of the 7 pictures are merged into a 7x2048-dimensional vector as the final feature; finally, it is handed over to the SVM classifier for prediction, and the corresponding risk level of the user is output.

[0088] In implementation, in order to evaluate the performance and generalization ability of the sleep apnea risk prediction model, an N-fold cross-validation method is used. Specifically, the data set is divided into N mutually exclusive subsets (usually 5 or 10), and each subset is used as a validation set in turn, and the rest are used as training sets. Each model is trained N times, and a different validation set is used for validation each time, and the overall performance of the model is evaluated by averaging the validation results. In addition, in the prediction stage, multiple trained ResNet50 models are used for prediction at the same time. For example, voting or averaging strategies can be used to fuse the prediction results of multiple models to improve the recognition accuracy of sleep apnea risk. The above integration strategy can make up for the possible limitations of a single model and improve the stability and accuracy of the overall prediction.

[0089] In addition, as a preferred method, after the input image is input into the sleep apnea risk prediction model, the GradCAM technology is used to generate a heat map of the face by analyzing the gradient information of the target category in the deep network. GradCAM is a technology used to explain the decision-making process of the deep learning model. The importance of each pixel to the final classification result is determined by calculating the gradient of the target category relative to the last convolutional layer. When the model predicts the input image, GradCAM will calculate and analyze which parts of the features contribute more to the model's judgment that the image belongs to the category of sleep apnea risk, and generate a heat map based on the calculation results of the GradCAM technology. The heat map will show areas with different color intensities on the face image, emphasizing the important parts that the model pays attention to when making decisions. These areas are usually related to specific feature points of the face (such as eyes, nose, mouth, etc.) or features that may be related to sleep apnea. The heat map can intuitively show the key areas that the model pays attention to during the prediction process. Users can quickly understand the decision-making basis of the model by observing the heat map, thereby increasing the trust and understanding of the prediction results.

[0090] In summary, refer to Figure 5 This application combines deep learning technology with a mobile phone application. Users can use their smartphones at home to collect facial and neck images and use built-in face detection and alignment algorithms to ensure image accuracy. Subsequently, these images are analyzed using a preset deep learning model to assess the user's sleep apnea risk. The above method not only avoids the need for traditional monitoring equipment and complex data collection, but also greatly simplifies the patient's assessment process, thereby improving the convenience of the assessment and user experience.

[0091] Figure 6 This is a structural block diagram of a sleep apnea risk assessment system based on face recognition provided by an embodiment of the present application. The device includes at least the following modules:

[0092] The image acquisition module is used to acquire user images, which include the user's face and neck.

[0093] The face alignment module is used to detect key points in the user image and perform face alignment processing on the user image based on the key points.

[0094] The image cropping module is used to identify and crop the face and neck area in the aligned user image to generate an input image.

[0095] The model prediction module is used to input the input image into a preset sleep apnea risk prediction model, and the model outputs the prediction result.

[0096] For relevant details, refer to the above method embodiment.

[0097] Figure 7 4 is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0098] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0099] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the sleep apnea risk assessment method based on face recognition provided in the method embodiment of the present application.

[0100] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0101] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0102] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the sleep apnea risk assessment method based on face recognition of the above method embodiment.

[0103] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the sleep apnea risk assessment method based on face recognition of the above method embodiment.

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A sleep apnea risk assessment method based on face recognition, characterized in that: The method comprises: Collecting a user image, wherein the user image includes the user's face and neck; Detecting key points in the user image, and performing face alignment processing on the user image based on the key points; Identify and crop the face and neck area in the aligned user image to generate an input image; The input image is input into a preset sleep apnea risk prediction model, and the model outputs a prediction result.

2. The sleep apnea risk assessment method based on face recognition according to claim 1, characterized in that: The collecting of user images comprises: Integrate an SDK with face detection function into the mobile phone; The mobile phone calls the SDK of the face detection function to process the image or video frame captured by the mobile phone camera, recognize the face and calculate the head posture, and obtain the roll angle, pitch angle, and yaw angle corresponding to the head posture; Based on the results of face detection and head posture evaluation, voice and text prompts are given to the user to adjust the position. When the user adjusts the face to the appropriate position, if the head posture angle meets the preset angle offset condition, the posture score corresponding to the photo is calculated. The posture score formula is as follows: Where N = 3, i = 1, 2, 3 represent the roll angle, pitch angle, and yaw angle respectively, θ i represents the actual angle of the i-th orientation, represents the expected angle of the i-th orientation, w i represents the weight of the i-th orientation; When the posture score is less than the score of the previous photo, the current photo is updated to the corresponding target position to obtain multiple user images at different angles that meet the head posture angle requirements.

3. The sleep apnea risk assessment method based on face recognition according to claim 1, characterized in that: The detecting key points in the user image and performing face alignment processing on the user image based on the key points comprises: Perform face detection on user images and identify key points of the face; Calculate the tilt angle of the face using the detected key points; Face alignment is achieved through affine transformation algorithm.

4. The sleep apnea risk assessment method based on face recognition according to claim 1, characterized in that: After detecting key points in the user image and performing face alignment processing on the user image based on the key points, the following further comprises: The verification method combining key point position and texture features is used to evaluate the effect of face alignment. The Euclidean distance between the key point position of the aligned face and the standard position is calculated using the following formula to obtain the key point position accuracy score D keypoints : Among them, d i is the Euclidean distance of the i-th key point, N is the number of key points; The aligned face image is converted into a grayscale image, and its local binary pattern feature histogram is calculated. The chi-square distance of the LBP histogram of the image before and after alignment is calculated using the following formula to obtain the texture feature similarity score t exture : Among them, H o and H a They are the LBP histograms of the user images before and after; The final score is calculated using the weighted average formula: Score=α·D keypoints +β·(1-D texture ) Among them, α and β are weight coefficients. A threshold is set and the final score is compared with the threshold. If the final score is greater than or equal to the threshold, the image alignment effect is considered to be good. If the final score is less than the threshold, the image alignment operation needs to be performed again or the alignment algorithm or parameters need to be further optimized.

5. The sleep apnea risk assessment method based on face recognition according to claim 1, characterized in that: The identifying and cropping of the face and neck area in the aligned user image to generate the input image comprises: Use InsightFace's 106-point precise positioning technology or Dlib's 68-point annotation technology to locate key points on the face and neck in the user's image; Based on the detected key points, the boundary area of ​​the face and neck is determined, and the identified face and neck area is cropped in the image; The cropped image only contains the key face and neck areas.

6. The sleep apnea risk assessment method based on face recognition according to claim 1, characterized in that: The input image is input into a preset sleep apnea risk prediction model, and the model outputs a prediction result including: The improved ResNet50 model is selected as the core recognition engine for learning and extracting features from face images. The last fully connected layer in the ResNet50 model architecture is removed, and the output of the second-to-last layer is taken as the feature. After combining the features of photos from multiple angles, the features are submitted to SVM for classification prediction. Prepare a labeled face dataset, which includes normal and sleep apnea samples. The annotation ensures that each sample has the correct category label and the probability label of the risk level; The improved ResNet50 model is trained using the prepared data set to obtain a sleep apnea risk prediction model.

7. The sleep apnea risk assessment method based on face recognition according to claim 1, characterized in that: The step of inputting the input image into a preset sleep apnea risk prediction model, and after the model outputs a prediction result, further includes: When the sleep apnea risk prediction model makes predictions on the input image, the GradCAM technology is used to generate a heat map of the face by analyzing the gradient information of the target category in the deep network.

8. A sleep apnea risk assessment system based on face recognition, characterized in that: include: An image acquisition module, used to acquire a user image, wherein the user image includes the user's face and neck; A face alignment module, used to detect key points in the user image and perform face alignment processing on the user image based on the key points; An image cropping module, used to identify and crop the face and neck area in the aligned user image to generate an input image; The model prediction module is used to input the input image into a preset sleep apnea risk prediction model, and the model outputs a prediction result.

9. An electronic device, characterized in that: The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a sleep apnea risk assessment method based on face recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and when the program is executed by the processor, it is used to implement the sleep apnea risk assessment method based on face recognition as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Lightweight unconstrained facial expression recognition method and system embedded with high-order information

    CN112801040A

  • Obstructive sleep apnea detection method and device based on deep learning

    CN113409944A

  • Audio and video feature extraction method and device, electronic equipment and computer readable storage medium

    CN115481283A

  • OSA judgment method and device based on face model, electronic equipment and medium

    CN116503935A

Cited By

  • OSA discrimination method based on face and neck contour image features

    CN119296161A