A method and system for facial segmentation in TCM diagnosis based on deep learning

Abstract

The present invention relates to a method and system for face segmentation in traditional Chinese medicine based on deep learning, and belongs to the field of image processing technology. The method comprises: extracting frames from a video image, and using a target detection algorithm based on deep learning to detect faces; performing facial analysis and segmentation to obtain a first segmentation mask; performing key point detection and face region segmentation to obtain a second segmentation mask; and correcting the mask regions of the first segmentation mask and the second segmentation mask to obtain a key region image of the face. The method uses a network model based on deep learning to recognize and segment faces, and realizes accurate segmentation of different parts of the face.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method and system for facial segmentation in traditional Chinese medicine inspection based on deep learning. Background Art

[0002] In the field of object detection, detection is based on improving detection accuracy, speed and efficiency. Common algorithms include region-based methods (such as R-CNN series, Fast R-CNN, Faster R-CNN), single-stage methods (such as YOLO series, SSD, RetinaNet) and some emerging methods (such as CenterNet, EfficientDet, etc.). In the field of face detection, there are methods such as Viola-Jones detector, deep learning-based methods (such as MTCNN, SSD, YOLO, etc.) and more advanced methods such as RetinaFace for detecting faces.

[0003] In face segmentation, the face part can be assigned to one category and the background to another category. Common semantic segmentation algorithms include FCN, U-Net, DeepLab, etc. Instance segmentation algorithms can not only perform pixel-level semantic segmentation on images, but also assign each semantic region to different instances. In face segmentation, different face instances can be assigned to different regions. Common instance segmentation algorithms include Mask R-CNN, YOLOv3, Panoptic FPN, etc.

[0004] In face location technology, there are algorithms that use features such as skin color, edges, eyes, and mouth, including Haar feature cascade classifiers, HOG features combined with support vector machines (SVM), etc. The normalized cross correlation (NCC) and square difference matching (SSD) algorithms use a pre-defined face template to slide and then calculate the similarity between the template and the local area of ​​the image to locate the face. At the same time, there are neural network algorithms based on face key point detection, such as PIPNet, PFLD's face 106 key point detection, etc.

[0005] At present, there are a lot of researches on the combination of western medicine and computers (artificial intelligence), such as the related technologies of combining CT images, magnetic resonance imaging and CNN (convolutional neural network), and using SHAP value to interpret the neural network of one-dimensional statistical data. However, compared with western medicine, the research on the combination of traditional Chinese medicine and artificial intelligence and related interpretability is still in a blank stage, and further research and in-depth exploration are still needed. Summary of the invention

[0006] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a method and system for TCM facial segmentation based on deep learning, which uses FasterRCNN as the backbone architecture to complete face detection, and adopts BiSeNet and Dlib library to complete face key point detection, thereby achieving accurate segmentation of different parts of the face.

[0007] According to one aspect of the present invention, the present invention provides a method for facial segmentation in TCM inspection based on deep learning, the method comprising the following steps:

[0008] Step 1: Extract frames from the video image and use a deep learning-based object detection algorithm to perform face detection;

[0009] Step 2: Perform facial analysis and segmentation to obtain the first segmentation mask;

[0010] Step 3: Perform key point detection and face area segmentation to obtain the second segmentation mask;

[0011] Step 4: Correct the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face.

[0012] Preferably, extracting frames from the video and performing face detection using a deep learning-based target detection algorithm includes:

[0013] S1: Extract key frames from video images and select N key frame images as input of S2;

[0014] S2: Use FasterRCNN to perform face detection on the key frame image and segment the target area according to the detection box.

[0015] Preferably, performing facial analysis and segmentation to obtain a first segmentation mask comprises:

[0016] S3: Use BiSeNet to perform facial analysis and part segmentation on the face image segmented by S2 to obtain the first segmentation mask, denoted as M1.

[0017] Preferably, the performing key point detection and face region segmentation to obtain a second segmentation mask comprises:

[0018] S4: Use the Dlib library to detect 68 key points of the face image segmented by S2. The set of detected key points is recorded as S1, and the i-th key point in S1 is recorded as P i , S1={P0,P2,…,P 67};

[0019] S5: Use CV2.Polylines to segment the line area according to the distribution of the predetermined area to be identified on the face of the human face, and use CV2.FillPoly to generate a mask to obtain a second segmentation mask, which is recorded as M2.

[0020] Preferably, the step of correcting the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face includes:

[0021] S6: Based on (M1, M2), CV2.bitwise_and is used to perform correction to obtain the final corrected mask M3;

[0022] S7: Apply the mask result M3 to the image segmented according to the detection frame to obtain a key area image that is finally segmented according to the predetermined area to be identified on the face of the person.

[0023] According to another aspect of the present invention, the present invention also provides a TCM facial segmentation system based on deep learning, the system comprising:

[0024] The detection module is used to extract frames from video images and perform face detection using a deep learning-based object detection algorithm;

[0025] A parsing module, used for performing facial parsing and segmentation to obtain a first segmentation mask;

[0026] A segmentation module, used for performing key point detection and face area segmentation to obtain a second segmentation mask;

[0027] The correction module is used to correct the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face.

[0028] Preferably, the detection module extracts frames from the video and uses a deep learning-based target detection algorithm to perform face detection, including:

[0029] Extract key frames from video images and select N key frame images as input data;

[0030] Use FasterRCNN to perform face detection on the key frame image and segment the target area according to the detection frame.

[0031] Preferably, the parsing module performs facial parsing and segmentation to obtain a first segmentation mask including:

[0032] The segmented face image is subjected to facial analysis and part segmentation using BiSeNet to obtain the first segmentation mask, denoted as M1.

[0033] Preferably, the segmentation module performs key point detection and face area segmentation to obtain a second segmentation mask including:

[0034] The segmented face image is used to detect 68 key points of the face using the Dlib library. The set of detected key points is denoted as S1, and the i-th key point in S1 is denoted as P i , S1={P0,P2,…,P 67};

[0035] According to the distribution of the predetermined area to be identified on the face, CV2.Polylines is used to segment the line area, and CV2.FilIPoly is used to generate a mask to obtain a second segmentation mask, which is recorded as M2.

[0036] Preferably, the correction module corrects the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face, including:

[0037] Based on (M1, M2), CV2.bitwise_and is used to correct the mask M3.

[0038] The mask result M3 is applied to the image segmented according to the detection frame to obtain a key area image that is finally segmented according to the predetermined area to be identified on the face of the person.

[0039] Beneficial effects: The TCM facial segmentation method and system based on deep learning provided by the present invention uses FasterRCNN as the backbone architecture to complete face detection, and adopts BiSeNet and Dlib library to complete face key point detection, thereby realizing the accurate segmentation of different parts of the face, including Mingtang, Que, Ting, Fan, Bi and other parts.

[0040] The features and advantages of the present invention will become clear through reference to the following drawings and detailed description of specific embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the facial segmentation method for TCM diagnosis based on deep learning;

[0042] Figure 2 It is a schematic diagram of the corresponding relationship between the "Mingtang Fanbi Tu" and the model division;

[0043] Figure 3 It is a schematic diagram of the segmentation polygonal boxes of the 68 key point distribution maps and the courtyard-related areas;

[0044] Figure 4 It is a schematic diagram of mask M2 of the relevant area of ​​the court;

[0045] Figure 5Schematic diagram of mask M1 for facial part segmentation;

[0046] Figure 6 Schematic diagram of the segmentation results of the courtyard-related area;

[0047] Figure 7 This is a schematic diagram of the TCM facial segmentation system based on deep learning. DETAILED DESCRIPTION

[0048] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Example 1

[0050] Figure 1 This is a flow chart of the facial segmentation method for TCM diagnosis based on deep learning. Figure 1 As shown, the present invention provides a method for facial segmentation in TCM inspection based on deep learning, the method comprising the following steps:

[0051] Step 1: Extract frames from the video image and use a deep learning-based object detection algorithm to perform face detection;

[0052] Step 2: Perform facial analysis and segmentation to obtain the first segmentation mask;

[0053] Step 3: Perform key point detection and face area segmentation to obtain the second segmentation mask;

[0054] Step 4: Correct the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face.

[0055] In this method, by detecting and segmenting the face, a first segmentation mask and a second segmentation mask are obtained respectively, and then the first segmentation mask and the second segmentation mask are corrected to obtain a key area image of the face, which can achieve accurate segmentation of different parts of the face.

[0056] Preferably, extracting frames from the video and performing face detection using a deep learning-based target detection algorithm includes:

[0057] S1: Extract key frames from video images and select N key frame images as input of S2;

[0058] S2: Use FasterRCNN to perform face detection on the key frame image and segment the target area according to the detection box.

[0059] Specifically, the image extraction process in S1 uses OpenCV to read the video image and uses the function CV_CAP_PROP_POS_FRAMES to jump to the specified frame of the video. Let the total number of video frames be F. At this time, to select N key frame images, the video needs to be divided into N+1 parts according to the total number of frames. Then the position of the i-th extracted image in the entire video frame is Considering that the behavior of the actor at the end and beginning of the video will be affected by environmental factors and become unstable, the last N+1 picture is discarded when extracting frames.

[0060] FasterRCNN in S2 uses the open source dataset CASIA-FaceV5 as training data. The CASIA-FaceV5 dataset contains 2,500 facial images of Asians, carefully organized into 500 separate folders, each with labels numbered sequentially from 000 to 499. Each folder represents a unique individual and contains five facial images taken under controlled conditions. The size of these images is standardized to 640x480 pixels to ensure the consistency of the dataset. This carefully curated dataset provides a comprehensive resource for research in the field of facial recognition and analysis, providing diverse images of 500 individuals with various facial expressions, poses, and lighting conditions. The model FasterRCNN uses ResNet50 as the backbone network for face detection.

[0061] The structure of FasterRCNN is mainly composed of four core modules: first, the convolution layer, which is responsible for extracting feature maps from the input image; then the region proposal network (RPN), which generates candidate regions that may contain target objects based on these feature maps; then the Rolpooling layer, which converts these candidate regions into fixed-size feature maps for subsequent processing; and finally the classification and regression layer, which classifies and adjusts the positions of candidate regions based on these feature maps to obtain the final object detection and recognition results. These four modules work together to enable FasterRCNN to achieve efficient and accurate object detection tasks.

[0062] Preferably, performing facial analysis and segmentation to obtain a first segmentation mask comprises:

[0063] S3: Use BiSeNet to perform facial analysis and part segmentation on the face image segmented by S2 to obtain the first segmentation mask, denoted as M1.

[0064] Specifically, BiSeNet in S3 is a bilateral segmentation network for real-time semantic segmentation, which consists of two parts: spatial path and context path. The spatial path consists of three convolutional layers to retain rich spatial information; the context path adopts a lightweight Xception model and adds a global average pooling layer to obtain the maximum receptive field and global context information. The features of the two paths are fused and refined through the Feature Fusion Module and the Attention Refinement Module. The entire structure achieves a balance between preserving spatial information and providing receptive field through parallel calculation of the spatial path and the context path, thereby achieving a balance between speed and segmentation performance.

[0065] Preferably, the performing key point detection and face region segmentation to obtain a second segmentation mask comprises:

[0066] S4: Use the Dlib library to detect 68 key points of the face image segmented by S2. The set of detected key points is recorded as S1, and the i-th key point in S1 is recorded as P i , S1={P0,P2,…,P 67};

[0067] S5: Use CV2.Polylines to segment the line area according to the distribution of the predetermined area to be identified on the face of the human face, and use CV2.FillPoly to generate a mask to obtain a second segmentation mask, which is recorded as M2.

[0068] Specifically, in step S5, the predetermined area to be identified on the face includes but is not limited to the areas of Mingtang, Que, Ting, Fan, Bi, Di, etc., which are divided according to the Mingtang Fanbi Diagram. The method in the Dlib library in S4 uses a cascaded residual regression tree (GBDT) to achieve face shape alignment. This method gradually regresses the current face shape to the real face shape. A residual regressor is stored on each leaf node of GBDT. When the input sample reaches a node, the residual will be added to the input sample to achieve the regression effect. Finally, all the residuals are superimposed together to complete the face shape alignment task.

[0069] Taking the face area to be identified as Mingtang, Que, Ting, Fan, Bi and other parts in the Mingtang Fanbi Diagram as an example, the relationship between the 68 key points in S5 and the areas divided by traditional Chinese medicine is shown in Table 2. The order of connection is as follows: Figure 4The 68 key point distribution diagram shown in the figure uses CV2.Polylines to connect the lines according to the line list corresponding to each mingtang fanbi map component in the table. For example, the line sequence list of Que is [21, 39, 27, 42, 22]. Then, a closed polygonal area can be obtained in the order of 21->39->27->42->22. Multiple lists mean that there are more than one segmented areas and the areas do not intersect with each other.

[0070] Table 2 The relationship between 68 key points and Mingtang Fanbitu

[0071]

[0072] *The Mingtang is difficult to generate a mask using key point technology due to its complex area, so it is left blank and no processing is done.

[0073] The order in the table is the order of connection. The polygons formed by the ready-made point sets are the corresponding mask areas. Among them, the effect of the 68 point set in the Mingtang is far inferior to that of the Mingtang (nose) area in facial analysis due to its complex area, so no mask is set.

[0074] For example, see Figure 2 The corresponding relationship diagram between the Mingtang Fanbi map and the model division is shown in Figure 1. After using the Dlib library to detect 68 key points of the face in S4, the following is obtained: Figure 3 The distribution of 68 key points is shown in Figure 3 It can be seen that the relevant points are [0, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 16]. According to the order of the above points 0->17->18->19->20->21->22->23->24->25->26->16, using CV2.Polylines to connect the lines can get the red polygon box shown in the figure, and using CV2.FillPoly can get the following Figure 4 The mask M2 associated with the courtyard area is shown. Figure 5 The mask M1 for facial segmentation is obtained from the S3 region. The blue region in the mask of M1 is subjected to CV2.bitwise_and operation with the M2 region to obtain the mask M3. The final result is obtained by applying it back to the original image of S2. Figure 6 shown.

[0075] Preferably, the step of correcting the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face includes:

[0076] S6: Based on (M1, M2), CV2.bitwise_and is used to perform correction to obtain the final corrected mask M3;

[0077] S7: Apply the mask result M3 to the image segmented according to the detection frame to obtain a key area image that is finally segmented according to the predetermined area to be identified on the face of the person.

[0078] Specifically, the correction process of S6 is as follows: if the area is a bright spot, the nose area of ​​M1 is selected as the result of M3. For other areas, the face area of ​​M1 and the corresponding area of ​​M2 are used to perform a bitwise AND operation as the mask result M3.

[0079] The deep learning-based TCM facial segmentation method and system provided in this embodiment uses FasterRCNN as the backbone architecture to complete face detection, and adopts BiSeNet and Dlib libraries to complete face key point detection, thereby realizing accurate segmentation of different parts of the face, including Mingtang, Que, Ting, Fan, Bi and other parts.

[0080] Example 2

[0081] Figure 7 This is a schematic diagram of the TCM facial segmentation system based on deep learning. Figure 7 As shown, this embodiment provides a TCM facial segmentation system based on deep learning, the system comprising:

[0082] The detection module 701 is used to extract frames from the video image and perform face detection using a deep learning-based target detection algorithm;

[0083] A parsing module 702, configured to perform facial parsing and segmentation to obtain a first segmentation mask;

[0084] A segmentation module 703 is used to perform key point detection and face region segmentation to obtain a second segmentation mask;

[0085] The correction module 704 is used to correct the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face.

[0086] Preferably, the detection module 701 extracts frames from the video and uses a deep learning-based target detection algorithm to perform face detection, including:

[0087] Extract key frames from video images and select N key frame images as input data;

[0088] Use FasterRCNN to perform face detection on the key frame image and segment the target area according to the detection frame.

[0089] Preferably, the parsing module 702 performs facial parsing and segmentation to obtain a first segmentation mask including:

[0090] The segmented face image is subjected to facial analysis and part segmentation using BiSeNet to obtain the first segmentation mask, denoted as M1.

[0091] Preferably, the segmentation module 703 performs key point detection and face region segmentation to obtain a second segmentation mask including:

[0092] The segmented face image is used to detect 68 key points of the face using the Dlib library. The set of detected key points is denoted as S1, and the i-th key point in S1 is denoted as P i , S1={P0,P2,…,P 67};

[0093] According to the distribution of the predetermined area to be identified on the face, CV2.Polylines is used to segment the line area, and CV2.FillPoly is used to generate a mask to obtain a second segmentation mask, which is recorded as M2.

[0094] Preferably, the correction module 704 corrects the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face, including:

[0095] Based on (M1, M2), CV2.bitwise_and is used to correct the mask M3.

[0096] The mask result M3 is applied to the image segmented according to the detection frame to obtain a key area image that is finally segmented according to the predetermined area to be identified on the face of the person.

[0097] The specific implementation process of the functions realized by each module in this embodiment 2 is the same as the implementation process in embodiment 1, and will not be repeated here.

[0098] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for facial segmentation in TCM based on deep learning, characterized in that: The method comprises the following steps: Step 1: Extract frames from the video image and use a deep learning-based object detection algorithm to perform face detection; Step 2: Perform facial analysis and segmentation to obtain the first segmentation mask; Step 3: Perform key point detection and face area segmentation to obtain the second segmentation mask; Step 4: Correcting the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face; extracting frames from the video and performing face detection using a deep learning-based target detection algorithm includes: S1: Extract key frames from video images and select N key frame images as input of S2; S2: Use FasterRCNN to perform face detection on the key frame image and segment the target area according to the detection frame; The performing facial analysis and segmentation to obtain a first segmentation mask includes: S3: Use BiSeNet to perform facial analysis and part segmentation on the face image segmented in S2 to obtain the first segmentation mask, denoted as M1; The performing of key point detection and face region segmentation to obtain a second segmentation mask comprises: S4: Use the Dlib library to detect 68 key points of the face image segmented by S2. The set of detected key points is recorded as S1, and the i-th key point in S1 is recorded as P i ,S1={P0,P2,…,P 67 }; S5: Use CV2.Polylines to segment the line area according to the distribution of the predetermined area to be identified on the face of the human, and use CV2.FillPoly to generate a mask to obtain a second segmentation mask, recorded as M2; The step of correcting the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face includes: S6: Use CV2.bitwise_and to correct (M1, M2) to obtain the final corrected mask M3; S7: applying the mask result M3 to the image segmented according to the detection frame to obtain a key area image segmented according to the predetermined area to be identified on the face of the person; The correction process of S6 is as follows: if the area is a bright spot, the nose area of ​​M1 is selected as the result of M3. For other areas, the face area of ​​M1 and the corresponding area of ​​M2 are used to perform a bitwise AND operation as the mask result M3.

2. A TCM facial segmentation system based on deep learning, characterized in that: The system comprises: The detection module is used to extract frames from video images and perform face detection using a deep learning-based object detection algorithm; A parsing module, used for performing facial parsing and segmentation to obtain a first segmentation mask; A segmentation module, used for performing key point detection and face area segmentation to obtain a second segmentation mask; A correction module, used for correcting the mask areas of the first segmentation mask and the second segmentation mask to obtain a key area image of the face; The detection module extracts frames from the video and uses a deep learning-based target detection algorithm to perform face detection, including: Extract key frames from video images and select N key frame images as input data; The key frame image is subjected to face detection using FasterRCNN to segment the target area according to the detection frame; the parsing module performs facial parsing and segmentation to obtain a first segmentation mask including: The segmented face image is subjected to facial analysis and part segmentation using BiSeNet to obtain a first segmentation mask, denoted as; the segmentation module performs key point detection and face area segmentation to obtain a second segmentation mask including: The segmented face image is used to detect 68 key points of the face using the Dlib library. The set of detected key points is denoted as S1, and the i-th key point in S1 is denoted as P i ,S1={P0,P2,…,P 67 }; According to the distribution of the predetermined area to be identified on the face of the human, CV2.Polylines is used to segment the line area, and CV2.FillPoly is used to generate a mask to obtain a second segmentation mask, recorded as M2; the correction module corrects the mask area of ​​the first segmentation mask and the second segmentation mask to obtain the key area image of the face, including: Use CV2.bitwise_and to correct (M1, M2) to get the final corrected mask M3; Applying the mask result M3 to the image segmented according to the detection frame to obtain a key area image segmented according to the predetermined area to be identified on the face of the person; The correction process of S6 is as follows: if the area is a bright spot, the nose area of ​​M1 is selected as the result of M3. For other areas, the face area of ​​M1 and the corresponding area of ​​M2 are used to perform a bitwise AND operation as the mask result M3.

Images