A method for calculating the correlation between a human face with acne and a macaque face

Abstract

The application discloses a kind of based on the relevance calculation method of human face and macaque face with acne, belong to digital image processing field and depth learning field.The application designs image processing, neural network, multi-modal fusion, contour similarity calculation, data set production and acquisition.The application eliminates the individual uniqueness by enlarging data set, superimposed fusion to the face segmentation result of many people, contour fitting, explores commonality problem.The conclusion whether there is relevance between human face acne occurrence area and macaque face hair area is obtained by the similarity calculation of fitting contour.

Description

Technical Field

[0001] This invention belongs to the fields of digital image processing and deep learning, specifically involving automatic segmentation technology for human facial acne and macaque bare face region contours based on multimodal fusion deep learning, as well as a method for verifying the correlation between acne distribution and macaque facial hairy areas. Background Technology

[0002] The "Chinese Guidelines for the Treatment of Acne (2019 Revised Edition)" points out that acne is a chronic inflammatory skin disease that commonly occurs during adolescence. More than 95% of the Chinese population will experience acne to varying degrees. Some patients will be left with scars, which can easily cause negative psychological effects such as anxiety, stress, and depression. Therefore, it is necessary to pay attention to and standardize treatment methods.

[0003] However, the pathogenesis of acne is still not fully understood. Its inducing factors include, but are not limited to, genetic factors, hormone secretion, excessive sebum secretion, and abnormal proliferation of Propionibacterium acnes. Current clinical dermatology generally believes that acne occurs when the sebaceous glands secrete excessive lipids, leading to abnormal proliferation of microorganisms in the hair follicles, causing inflammation and an immune response.

[0004] The T-zone of the face (the area formed by the forehead and nose) has many sweat glands and pores, resulting in excessive oil production daily; the U-zone (the areas of the left and right cheeks) has smoother skin and less oil production. Based on the aforementioned pathogenesis, the T-zone should be a common area for acne, while the U-zone should be a less common area. However, clinical observation of acne patients reveals that patients with severe acne have a very high incidence of acne in the U-zone, and acne scars such as pits often appear in the U-zone. Furthermore, observation of acne-prone areas in multiple individuals revealed a striking similarity between the contours of these acne-prone areas and the contours of fur-covered areas in macaques. Therefore, it is hypothesized that due to human genetic evolution, the contours of commonly acne-prone facial areas resemble those of fur-covered areas in macaques.

[0005] Therefore, to address the above issues, a verification method is needed to effectively verify the above conjectures, supplement the pathogenesis of acne, establish a connection with macaques, and enrich subsequent experimental and diagnostic methods. Summary of the Invention

[0006] This invention addresses the current clinical observation that the distribution area of ​​acne on the face of acne patients is similar to the hairy area on the face of macaques. It designs a verification method that combines image color space and multimodal deep learning to objectively describe the correlation between the acne-causing area on the face and the hairy area on the face of macaques.

[0007] The technical solution of this invention is a method for calculating the correlation between human facial features and macaque facial features based on acne. The method includes the following steps:

[0008] Step 1: Facial data acquisition; Use a high-resolution camera to acquire parallel polarization spectral images and near-infrared spectral images of the faces of the crowd;

[0009] Step 2: Filter the sample images collected in Step 1, remove images of patients with obvious skin diseases (such as vitiligo, lupus erythematosus, etc.), and divide the training set and test set according to a 1:9 ratio;

[0010] Step 3: Preprocess the parallel polarization spectrum image after Step 2;

[0011] Step 3-1: Perform grayscale processing on the parallel polarization spectrum image obtained in Step 2;

[0012] Step 3-2: Use OpenCV Dlib to perform facial landmark mapping on the image processed in Step 3-1;

[0013] Step 3-3: Using the facial key points obtained after processing in Step 3-2 and the OpenCV fillPoly function, determine the contours of the eyes, nose, and mouth of the face, and obtain the mask of the above facial organs;

[0014] Step 4: Annotate facial acne on the near-infrared spectral image obtained in Step 2;

[0015] Step 4-1: Overlay the facial organ mask obtained in Step 3-3 onto the near-infrared spectral image obtained in Step 1;

[0016] Step 4-2: Convert the overlay mask image obtained in Step 4-1 into an image represented in HSV color space, and set a threshold range based on the difference between the acne inflammation area and normal skin;

[0017] Step 4-3: Perform threshold segmentation on the image obtained in Step 4-2 to obtain a segmented image, where the corresponding segmented region is set to 1 and the background region is set to 0;

[0018] Step 4-4: Calculate the connected components of the image obtained in Step 4-3, and construct the bounding rectangle for each connected component.

[0019] Step 4-5: Convert the bounding rectangle data obtained in Step 4-4 into a JSON format file;

[0020] Steps 4-6: Combine the JSON file obtained in Step 4-5 with the parallel polarization spectrum image obtained in Step 2 and use the labeling software Labelme to correct it, ensuring that there are no omissions or errors in labeling;

[0021] Step 5: Process the labeled JSON file obtained in Step 4 and convert it into a VOC format file;

[0022] Step 6: Fuse the parallel polarization spectral image and the near-infrared spectral image obtained in Step 1 in the channel direction, and change the image size from (2000,3000,3) to (2000,3000,6);

[0023] Step 7: Construct a multimodal convolutional neural network structure based on YOLOv5;

[0024] Step 8: Input the VOC format annotation file obtained in Step 5 and the fused image obtained in Step 6 into the multimodal neural network built in Step 7 for training until training is successful;

[0025] Step 9: Use the multimodal convolutional neural network model trained in Step 8 and adjusted based on YOLOv5 to detect the test set obtained in Step 2. Save the detected acne regions as binary images, where the acne regions are 1 and the background regions are 0.

[0026] Step 10: Merge the binary images obtained in Step 9;

[0027] Step 10-1: Determine a facial template M, and use it as a basis to adjust other images to ensure that the facial features of all people are basically aligned;

[0028] Step 10-2: Perform grayscale processing on the parallel polarization spectrum image of the test set obtained in Step 2;

[0029] Step 10-3: Use OpenCV Dlib to perform facial landmark mapping on the image processed in Step 10-2;

[0030] Step 10-4: Calculate the center O of the bounding rectangle of all feature points in the image processed in Step 10-3, and calculate the difference between the center position of the bounding rectangle of the feature points and the center position of the bounding rectangle of the template M.

[0031] Step 10-5: Connect the two feature points at the top and bottom of the nose's central axis in the image processed in Step 10-3, and calculate the angle between the connecting line and the vertical line;

[0032] Step 10-6: Calculate the length and width of the bounding rectangle of all feature points in the image processed in Step 10-3, and calculate the ratio of the length to the width of the bounding rectangle of the feature points of template M.

[0033] Step 10-7: Translate the binary image obtained in step 9 according to the position difference obtained in step 10-4, rotate the image according to the included angle obtained in step 10-5, and scale it according to the aspect ratio obtained in step 10-6.

[0034] Step 10-8: Overlay all the binary images processed in Step 10-7 to obtain a map of the location of acne in multiple people, thus reducing specificity;

[0035] Step 11: Perform contour fitting on the images obtained in Steps 10-8 to obtain the internal contours of the acne distribution;

[0036] Step 12: Obtaining data on macaque faces;

[0037] Step 13: Annotate the monkey face data obtained in Step 12, and mark the boundary between the fur area and the bare area of ​​the monkey face;

[0038] Step 14: Using the file obtained in Step 13, train the U-net-based neural network model until training is successful;

[0039] Step 15: Use the U-net-based neural network model trained in Step 14 to segment the subsequent test set to obtain the monkey face bare face region;

[0040] Step 16: Extract the boundaries of all monkey-faced bare-faced regions obtained in Step 15, merge them into a single boundary and fill it to reduce specificity;

[0041] Step 16-1: Construct an outer rectangle for all the monkey-faced bare face regions obtained in Step 15;

[0042] Step 16-2: Define a rectangular template and determine the size of the rectangle and the angle between its central axis and the coordinate axis;

[0043] Step 16-3: Rotate and scale the bounding rectangle and its internal image obtained in Step 16-1 to make the bounding rectangle consistent with the template defined in Step 16-2;

[0044] Step 16-4: Calculate the connected domains of all monkey-faced bare surfaces obtained in Step 16-3 to obtain their center point O. Following the polar coordinate system, with O as the origin and 1° as the angle difference, emit 360 rays in all directions. The point where the ray intersects the edge of the connected domain is the sampling point.

[0045] Step 16-5: Take the sampling points with the same angle obtained in Step 16-4 as a group of data, calculate the average value of each group of data, and obtain 360 mean points.

[0046] Step 16-6: Smoothly connect and fill the mean points obtained in Step 16-5 to obtain a fusion map of the monkey face bare face region, reducing specificity;

[0047] Step 17: Calculate the similarity between the human face acne distribution contours obtained in Step 11 and Step 16 and the monkey face bare face region. Use three calculation methods, namely Euclidean distance, cosine distance and Hausdorff distance, to calculate the similarity between the two contours. The results are used to objectively evaluate the similarity between the two contours.

[0048] This invention employs a multimodal neural network to extract acne features from fused parallel polarization spectral images and near-infrared spectral images, resulting in higher recognition accuracy. It utilizes image rotation, translation, and scaling to distribute facial features and cheeks of different individuals in the same area as much as possible. It also uses image fusion and increased sample size to reduce differences between individuals and employs multiple evaluation parameters for calculation, making the similarity assessment more objective and accurate. Attached Figure Description

[0049] Figure 1 This describes the overall workflow of the present invention.

[0050] Figure 2 This invention presents a convolutional neural network process for intelligent monitoring of facial acne.

[0051] Figure 3 This describes the process for creating the facial acne dataset for this invention.

[0052] Figure 4 This is the monkey face contour fitting process of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and innovations of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this invention and do not limit this invention.

[0054] Please see Figures 1 to 4 This invention is a method for exploring the correlation between facial acne-causing regions and macaque fur regions based on image color and convolutional neural networks. It includes an automatic human facial acne segmentation and extraction module for automatically and intelligently segmenting and extracting acne from multiple facial images; a macaque bare-face region segmentation module for segmenting and extracting the bare-face region from multiple macaque facial images; an image merging and contour fitting module for contour fitting of the segmented facial acne images and the segmented macaque bare-face images; and a similarity calculation module for calculating the similarity between the fitted contour based on acne and the fitted contour based on the macaque bare face.

[0055] A method for calculating the correlation between facial acne-causing regions and macaque fur regions based on image color and convolutional neural networks includes the following steps:

[0056] Step 1-1: Face data acquisition. Using a high-resolution camera that can fix faces, acquire parallel polarization spectral images and near-infrared spectral images of multiple faces using two different light sources.

[0057] Step 1-2: Filter the sample images collected in Step 1-1 and divide them into training and test sets in a 1:9 ratio;

[0058] Steps 1-3: Perform grayscale processing on the parallel polarization spectrum image obtained in Step 1-2, and perform facial key point calibration. Based on the key points, obtain the masks for the eyes, nose, and mouth of the face.

[0059] Steps 1-4: After the near-infrared spectral image is overlaid with a mask, the overall image is converted from RGB representation to HSV color space representation, and a threshold range is set to initially extract acne.

[0060] Steps 1-5: Manually correct the rough labels to ensure there are no missing or incorrect labels, and create a VOC format file.

[0061] Steps 1-6: Train the input data using a multimodal convolutional neural network structure based on YOLOv5 until the loss decreases to a steady level.

[0062] Steps 1-7: Use the network model trained in Steps 1-6 to detect the test set obtained in Steps 1-2, and save the detected acne regions as binary images.

[0063] Steps 1-8: Merge the binary images obtained in Steps 1-7, and perform translation, rotation, and scaling operations on other images according to the template to make all faces basically aligned. Then, overlay the binary images obtained from model segmentation to obtain a map of the location of acne in multiple people, reducing specificity.

[0064] Step 2-1: Obtain and label the bare face areas of macaques;

[0065] Step 2-2: Train the input macaque face data using a U-net-based neural network model until training is successful;

[0066] Step 2-3: Use the network model trained in Step 2-2 to segment a large number of macaque face images to obtain the bare face regions of the monkeys;

[0067] Step 2-4: Calculate the connected components of all monkey face bare face regions obtained in Step 2-3, construct the bounding rectangle, and rotate and scale it according to the defined rectangle template until the bounding rectangles of all monkey face connected components are consistent.

[0068] Step 2-5: Calculate the connected domain of the monkey face processed in Step 2-4 to obtain the center point O. Similar to the polar coordinate system, with O as the origin and 1° as the angle interval, draw rays in all directions. The point where the ray intersects the edge of the connected domain is the sampling point of this angle.

[0069] Steps 2-6: Calculate the average value of the sampling points corresponding to the same angle obtained in Step 2-5 to obtain 360 mean points. Smoothly link and fill them in sequence to obtain the fitted monkey face bare face region and reduce specificity.

[0070] Step 3: Calculate the similarity between the human face acne distribution contours obtained in Steps 1-8 and Steps 2-6 and the monkey face bare face region. Use Euclidean distance, cosine distance and Hausdorff distance to calculate the similarity between the two contours. The results are used to objectively evaluate the similarity between the two contours.

[0071] This invention uses deep learning networks to automatically identify human acne features and the facial features of macaques that separate their bare and hairy faces, enabling batch data processing and reducing the subjectivity of manual segmentation. It also reduces the differences between individuals by increasing the sample size and using appropriate methods to perform image fusion on human and monkey faces separately. Furthermore, it uses multiple evaluation parameters for calculation, making the similarity assessment more objective and accurate.

Claims

1. A method for calculating the correlation between human and macaque faces based on acne, the method comprising: Step 1: Acquiring facial data; High-resolution cameras were used to acquire parallel polarization spectral images and near-infrared spectral images of the faces of people. Step 2: Filter the sample images collected in Step 1, remove images of patients with obvious skin diseases, and divide the training set and test set in a 1:9 ratio; Step 3: Preprocess the parallel polarization spectrum image after Step 2; Step 3-1: Perform grayscale processing on the parallel polarization spectrum image obtained in Step 2; Step 3-2: Use OpenCV Dlib to perform facial landmark mapping on the image processed in Step 3-1; Step 3-3: Using the facial key points obtained after processing in Step 3-2 and the OpenCV fillPoly function, determine the contours of the eyes, nose, and mouth of the face, and obtain the mask of the above facial organs; Step 4: Annotate facial acne on the near-infrared spectral image obtained in Step 2; Step 4-1: Overlay the facial organ mask obtained in Step 3-3 onto the near-infrared spectral image obtained in Step 1; Step 4-2: Convert the overlay mask image obtained in Step 4-1 into an image represented in HSV color space, and set the threshold range according to the difference between the acne inflammation area and normal skin; Step 4-3: Perform threshold segmentation on the image obtained in Step 4-2 to obtain a segmented image, where the corresponding segmented region is set to 1 and the background region is set to 0; Step 4-4: Calculate the connected components of the image obtained in Step 4-3, and construct the bounding rectangle for each connected component. Step 4-5: Convert the bounding rectangle data obtained in Step 4-4 into a JSON format file; Steps 4-6: Correct the JSON file obtained in Step 4-5 and the parallel polarization spectrum image obtained in Step 2 using the labeling software labelme to ensure that there are no omissions or errors in labeling; Step 5: Process the labeled JSON file obtained in Step 4 and convert it into a VOC format file; Step 6: Fuse the parallel polarization spectral image obtained in Step 1 with the near-infrared spectral image in the channel direction, and change the image size from (2000, 3000, 3) to (2000, 3000, 6). Step 7: Construct a multimodal convolutional neural network structure based on YOLOv5; Step 8: Input the VOC format annotation file obtained in Step 5 and the fused image obtained in Step 6 into the multimodal neural network built in Step 7 for training until training is successful; Step 9: Use the YOLOv5-based multimodal convolutional neural network model trained in Step 8 to detect the test set obtained in Step 2. Save the detected acne regions as binary images, where the acne regions are 1 and the background regions are 0. Step 10: Merge the binary images obtained in Step 9; Step 10-1: Determine a facial template M, and adjust other images based on this facial template M to ensure that the facial features of all people are basically aligned. Step 10-2: Perform grayscale processing on the parallel polarization spectrum images in the test set obtained in Step 2; Step 10-3: Use OpenCV Dlib to perform facial landmark mapping on the image processed in Step 10-2; Step 10-4: Calculate the center O of the bounding rectangle of all feature points in the image processed in Step 10-3, and calculate the difference between the center position of the bounding rectangle of the feature points and the center position of the bounding rectangle of the template M. Step 10-5: Connect the two feature points at the top and bottom of the nose's central axis in the image processed in Step 10-3, and calculate the angle between the connecting line and the vertical line; Step 10-6: Calculate the length and width of the bounding rectangle of all feature points in the image processed in Step 10-3, and calculate the ratio of the length to the width of the bounding rectangle of the feature points to that of template M. Step 10-7: Translate the binary image obtained in step 9 according to the position difference obtained in step 10-4, rotate the image according to the included angle obtained in step 10-5, and scale it according to the aspect ratio obtained in step 10-6. Step 10-8: Overlay all the binary images processed in Step 10-7 to obtain a map of the location of acne in multiple people, thus reducing specificity; Step 11: Perform contour fitting on the images obtained in Steps 10-8 to obtain the internal contours of the acne distribution; Step 12: Obtaining data on macaque faces; Step 13: Annotate the monkey face data obtained in Step 12, and mark the boundary between the fur area and the bare area of ​​the monkey face; Step 14: Use the labeled monkey face data obtained in Step 13 to train the U-net-based neural network model until training is successful; Step 15: Use the U-net-based neural network model trained in Step 14 to segment the test set and obtain the monkey face bare face region; Step 16: Extract the boundaries of all monkey-faced bare-faced regions obtained in Step 15, merge them into a single boundary and fill it to reduce specificity; Step 17: Calculate the similarity between the human face acne distribution contours obtained in Step 11 and Step 16 and the monkey face bare face region. Use three calculation methods, namely Euclidean distance, cosine distance and Hausdorff distance, to calculate the similarity between the two contours. The results are used to objectively evaluate the similarity between the two contours.

2. The method for calculating the correlation between human and macaque faces based on acne, as described in claim 1, is characterized in that... The specific method of step 16 is as follows: Step 16-1: Construct an outer rectangle for all monkey-faced bare face regions obtained in step 15; Step 16-2: Define a rectangular template and determine the size of the rectangle and the angle between its central axis and the coordinate axis; Step 16-3: Rotate and scale the bounding rectangle and its internal image obtained in Step 16-1 to make the bounding rectangle consistent with the template defined in Step 16-2; Step 16-4: Calculate the connected domains of all monkey-faced bare surfaces obtained in Step 16-3 to obtain their center point O. Following the polar coordinate system, with O as the origin and 1° as the angle difference, emit 360 rays in all directions. The point where the ray intersects the edge of the connected domain is the sampling point. Step 16-5: Take the sampling points with the same angle obtained in Step 16-4 as a group of data, calculate the average value for each group of data, and obtain 360 mean points; Step 16-6: Smoothly connect and fill the mean points obtained in step 16-5 to obtain a fusion map of the monkey face bare face region, reducing specificity.

Images