Pore detection method and system based on deep learning
By using deep learning methods of Swin-Transformer and multi-layer fusion network in pore detection, the problem of the existing technology identifying errors and loss of detailed information in complex environments is solved, and pore detection with high accuracy is achieved.
Patent Information
- Application Number
- CN202411267409.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2025-02-07
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing pore detection technologies are prone to misidentification when facing complex environments, and convolutional neural networks have limitations on the receptive field, resulting in loss of image details and making it difficult to accurately detect densely distributed pores.
Using deep learning-based pore detection method, a semantic segmentation network is constructed using Swin-Transformer and multi-layer fusion network. Through cross entropy and Dice coefficients as loss functions, the pore detection model is trained to enhance the long-distance dependence of data and retain image details. .
It realizes rapid and accurate detection of pore density and size in complex environments, improves the accuracy of pore recognition, and is suitable for dermatology and cosmetics applications.
Smart Images
Figure CN119399102A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of skin detection, and in particular to a pore detection method and system. Background Art
[0002] With the improvement of living standards, more and more people begin to pay attention to their skin quality, especially facial skin quality, such as the number, size and distribution of pores. For professionals in the beauty and medical industry, the state of pores is an important reference for facial skin cleaning and maintenance. There are two main pore detection and segmentation technology solutions: one is the traditional image processing method, and the other is the image detection and segmentation algorithm based on big data and deep learning to achieve the analysis of the target image and the capture of facial pores.
[0003] Although traditional image processing methods have fast processing speeds and high recognition accuracy under ideal conditions, in actual applications, they are prone to misrecognition when faced with complex environments (such as uneven lighting, facial accessories, changes in expression or posture, etc.). In contrast, algorithms based on deep learning are more robust and can not only cope with a variety of complex scenarios in real life, but also achieve higher recognition accuracy. With the development of hardware and the improvement of computing power, some deep learning models have been able to meet the real-time requirements in daily applications, so the processing speed advantage of traditional algorithms is gradually no longer obvious.
[0004] By combining a large amount of pore image data with image detection and segmentation algorithms in deep learning technology, a pore detection model that is more suitable for real-life applications can be obtained. At present, most image detection and segmentation algorithms in deep learning are constructed using convolutional neural networks (such as FCN, U-Net, DeepLab, etc.). These networks expand the receptive field and learn deeper features of images through deep structural design. However, convolutional neural networks naturally have limitations on the receptive field, and continuous downsampling will cause some image detail information to be lost, which may have a negative impact on the detection and segmentation of pores. The network built based on transformer can enhance the long-distance dependency of data and better obtain global context information. However, transformers are not good at capturing local information of images, and pores are densely distributed and closely spaced on facial skin. Short-distance information accounts for a small proportion of a token divided by them, which makes transformers perform poorly when dealing with these problems.
[0005] In addition, in human facial images, the foreground pixels of pores are far less than other background pixels. Therefore, when training the model, the model tends to not segment the pores to ensure the overall accuracy. Although the model has a high accuracy in detection results, it is unacceptable in practical applications if it can only capture a small number of pores and ignore a large number of pores.
[0006] Finally, the original image data has a high resolution and cannot be directly input into the network for learning. If the image is simply reduced without other processing, a lot of important information may be lost, which is very important for pore detection and segmentation, thus affecting the final segmentation effect. Summary of the invention
[0007] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a pore detection method and system based on deep learning.
[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0009] In a first aspect of the present invention, a pore detection method and system based on deep learning is provided, comprising the following steps:
[0010] Step 1, collecting pore images of different areas on the face of a person, collecting pore images of different areas on the face of a person through a hardware medium, and constructing training sets and test sets of pore images of different areas respectively;
[0011] Step 2: Configure the semantic segmentation network, using Swin-Transformer and the multi-layer fusion network as the encoder and decoder of the semantic segmentation network respectively;
[0012] Step 3: Input the training set into the configured semantic segmentation network for training, and then evaluate it with the test set to obtain the pore detection model. The pore detection model uses cross entropy and Dice coefficient as the loss function, and the formula is:
[0013] Loss=α*CE+(1-α)*Dice Loss;
[0014] CE=-Σp(x)log(q(x));
[0015]
[0016] Among them, α is the scale parameter, X and Y are the numbers of two sample elements; the formula for calculating model accuracy is:
[0017]
[0018] Among them, Y and Y are the number of two sample elements;
[0019] Step 4: segment the facial pores using the pore detection model.
[0020] As a preferred technical solution of the present invention, in the step 1, the pore images of different regions are detected by facial key points, the collected facial image is divided into a forehead region image and a cheek region image, and the horizontal and vertical pixels of the forehead region image and the cheek region image are reduced respectively;
[0021] The forehead area image and the cheek area image are respectively flipped at multiple angles, rotated at multiple angles, and cropped in multiple directions; the pore images existing in the forehead area image and the cheek area image are respectively marked, and the distinguishing colors between the pore area and the non-pore area are set.
[0022] As a preferred technical solution of the present invention, in step 2, the overall framework of the semantic segmentation network adopts the SegNet neural network, the encoder of the SegNet neural network performs feature extraction through Swin-Transformer, and the multi-layer fusion network adopts the MLP neural network.
[0023] In a second aspect of the present invention, a system for a pore detection method based on deep learning comprises:
[0024] A collection module, used to collect pore images of different areas on the face;
[0025] A training set and a test set construction module, used to construct training sets and test sets of pore images in different regions;
[0026] Configuration module, used to configure the semantic segmentation network, using Swin-Transformer and multi-layer fusion network as the encoder and decoder of the semantic segmentation network respectively;
[0027] The training module is used to input the training set into the configured semantic segmentation network for training, and then evaluate it with the test set to obtain the pore detection model;
[0028] The facial pore detection module is used to detect facial pores using the pore detection model.
[0029] As a preferred technical solution of the present invention, the acquisition module includes:
[0030] The face region segmentation module is used to segment the collected face image into a forehead region image and a cheek region image by detecting face key points;
[0031] The image data enhancement module is used to perform multi-angle flipping, multi-angle rotation and multi-directional cropping on the forehead area image and the cheek area image respectively.
[0032] As a preferred technical solution of the present invention, the image data enhancement module includes a color setting module, which is used to mark the pore images existing in the forehead area image and the cheek area image respectively, and set the distinguishing color between the pore area and the non-pore area.
[0033] In a third aspect of the present invention, a computer device is provided, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 3 or the functions of the system according to claims 4 to 6 when executing the computer program.
[0034] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 or the functions of the system according to claims 4 to 6 are implemented.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] In the pre-inspection of pore treatment equipment on the skin, the present invention can quickly and accurately provide pore density and pore size and the number of pores per unit area through the designed Swin-SegNet pore detection network, which can be widely used in dermatology and cosmetology. Compared with the existing technology, its accuracy greatly improves the degree of pore recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0038] Figure 1 is a flow chart of a facial pore detection method according to an embodiment of the present invention;
[0039] Figure 2 This is a flowchart of image data training in an embodiment of the present invention;
[0040] Figure 3 It is a schematic diagram of the structure of the SegNet semantic segmentation network in an embodiment of the present invention;
[0041] Figure 4 Schematic diagram of a facial pore detection system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0043] Example 1
[0044] like Figure 1 and Figure 2 As shown, in a first aspect, an embodiment of the present invention provides a facial pore detection method, the method comprising the following steps:
[0045] Step 1: collect pore images of different facial areas, and construct training sets and test sets of pore images of different facial areas respectively. The equipment used to collect pore images of different facial areas can adopt hardware media, such as digital cameras, SLR cameras and scanners, etc. The facial pore areas are manually marked by using drawing software.
[0046] In the present application, a full-face facial image may be collected, and the collected facial image may be segmented into a forehead area image and a cheek area image through facial key point detection, and the horizontal and vertical pixels of these area images may be reduced, and then multi-angle flipping, multi-angle rotation and multi-directional cropping may be performed to achieve the purpose of enhancing the image.
[0047] Then, the different colors between the pore area and the non-pore area are set for the forehead area image and the cheek area image, and the pores in these images are marked. In the marked file, the RGB value of the pixel in the non-pore area is (0, 0, 0), and the RGB value of the pore area is (255, 255, 255).
[0048] Step 2: Configure the semantic segmentation network, use Swin-Transformer and multi-layer fusion network as the encoder and decoder of the semantic segmentation network respectively. The overall framework of the semantic segmentation network adopts SegNet neural network. The encoder of SegNet neural network extracts features through Swin-Transformer, and the multi-layer fusion network adopts MLP neural network.
[0049] like Figure 3 As shown in the figure, the encoder network structure of Swin-Transformer, which downsamples the image four times, reducing the image resolution to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original. The Transformer architecture network usually has a large amount of computation, so fewer hidden layers are used to make the network lighter. Finally, four feature maps of different sizes are obtained as the output of the encoder network.
[0050] The decoder of the MLP neural network will fuse the four feature maps output by the encoder and finally output the segmentation result. The four feature maps are respectively expanded in resolution using bilinear interpolation, and then the channels are reduced using the MLP neural network. Finally, the four feature maps of different sizes are uniformly changed to a resolution of 1 / 4 of the original image size, superimposed in the channel dimension, and finally the MLP neural network is used to fuse the feature map information.
[0051] Step 3: Input the training sets of forehead area images and cheek area images into the configured semantic segmentation network for training, and then evaluate them through the test set to obtain the pore detection model.
[0052] The cross entropy and Dice coefficient are used as the loss function, and the formula is:
[0053] Loss=α*CE+(1-α)**Dice Loss.
[0054] CE = -∑p(x)log(q(x));
[0055]
[0056] Where α is the proportional parameter, CE = -∑p(x)log(q(x)), X and Y are the numbers of two sample elements; the formula for calculating model accuracy is:
[0057]
[0058] Among them, X and Y are the numbers of two sample elements.
[0059] α uses 0.5 as the ratio parameter to adjust the proportion of the two. The CE formula calculates the distance between two probability distributions. The smaller the loss value, the closer the distribution. The Dice Score formula is a set similarity measurement function, usually used to calculate the similarity of two samples, with a value range of 0 to 1; |X∩Y| is the intersection of two samples, in order to reconcile repeated elements. The Dice Loss formula and the Dice Score formula are complementary, that is, the higher the similarity, the smaller the loss.
[0060] The Swin-Transformer-based SegNet semantic segmentation network model is referred to as the Swin-SegNet network. The Swin-SegNet network is used to take the face image as the input of the model. After downsampling and encoding, the feature map of the deep semantic information of the image is obtained. Finally, after decoding, the prediction map of the final face pore area is obtained. The data set is input into the network training. After a certain number of iterations, the trained network gradually converges. The optimal weight file on the validation set is selected as the final model internal parameters to obtain the final pore detection model.
[0061] In another embodiment, if Figure 4 As shown, a facial pore detection system is also provided, including an acquisition module, a training set and test set construction module, a configuration module, a training module and a facial pore detection module. The acquisition module is used to acquire pore images of different areas on the face; the training set and test set construction module is used to construct training sets and test sets of pore images of different areas; the configuration module is used to configure the semantic segmentation network, and the Swin-Transformer and the multi-layer fusion network are used as the encoder and decoder of the semantic segmentation network respectively; the training module is used to input the training set into the configured semantic segmentation network for training, and then obtain the pore detection model after evaluation through the test set; the facial pore detection module is used to detect facial pores through the pore detection model.
[0062] The acquisition module includes a facial pore detection module and an image data enhancement module. The facial region segmentation module is used to segment the acquired facial image into a forehead region image and a cheek region image by detecting facial key points; the image data enhancement module is used to perform multi-angle flipping, multi-angle rotation, and multi-directional cropping on the forehead region image and the cheek region image. The image data enhancement module includes a color setting module, which is used to mark the pore images in the forehead region image and the cheek region image, and set the distinguishing colors between the pore region and the non-pore region.
[0063] In another embodiment, the present invention further provides a facial pore detection device, comprising:
[0064] At least one processor; a memory coupled to the at least one processor, the memory storing executable instructions, wherein the executable instructions, when executed by the at least one processor, implement the method steps of the first aspect of the present invention.
[0065] In a facial pore device provided by an embodiment of the present invention, a processor and a memory may be provided separately or integrated together.
[0066] For example, the memory may include a random access memory, a flash memory, a read-only memory, a programmable read-only memory, a non-volatile memory or a register, etc. The processor may be a central processing unit (CPU), etc. or a graphic processing unit (GPU). The memory may store executable instructions. The processor may execute the executable instructions stored in the memory to implement the various processes described herein.
[0067] It can be understood that the memory in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a ROM (Read-Only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), or a flash memory. The volatile memory can be a RAM (Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDRSDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memory 42 described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0068] In some embodiments, the memory stores the following elements, upgrade packages, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and an application program.
[0069] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application program.
[0070] In an embodiment of the present invention, the processor calls a program or instruction stored in a memory, specifically, a program or instruction stored in an application program, and the processor is used to execute the method steps provided in the first aspect.
[0071] In another embodiment, the present invention further provides a chip for executing the method in the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the method in the other embodiments.
[0072] In another embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method of other embodiments of the present invention are implemented.
[0073] For example, machine-readable storage media may include, but are not limited to, various known and unknown types of non-volatile memory.
[0074] In another embodiment, the present invention further provides a computer program product, comprising computer program instructions, which enable a computer to execute the above method.
[0075] In summary, a facial pore method and system according to an embodiment of the present invention adopts Swin-Transformer as a feature extraction network and designs a pore detection network called Swin-SegNet, which has a significant improvement in facial pore detection compared with other detection networks and can be applied to medical treatment, beauty and the lives of ordinary users.
[0076] It will be appreciated by those skilled in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0077] In the embodiments of the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, the division of units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system. In addition, the coupling between the various units can be direct coupling or indirect coupling. In addition, the various functional units in the embodiments of the present application can be integrated into a processing unit, or can be separate physical existences, etc.
[0078] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0079] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a machine-readable storage medium. Therefore, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a machine-readable storage medium, which can include several instructions to enable an electronic device to perform all or part of the technical solution described in the embodiments of the present application. The above-mentioned storage medium may include various media that can store program codes, such as ROM, RAM, removable disk, hard disk, magnetic disk or optical disk.
Claims
1. A pore detection method based on deep learning, characterized in that: The steps include: Step 1, collecting pore images of different areas on the face of a person, collecting pore images of different areas on the face of a person through a hardware medium, and constructing training sets and test sets of pore images of different areas respectively; Step 2: Configure the semantic segmentation network, using Swin-Transformer and the multi-layer fusion network as the encoder and decoder of the semantic segmentation network respectively; Step 3: Input the training set into the configured semantic segmentation network for training, and then evaluate it with the test set to obtain the pore detection model. The pore detection model uses cross entropy and Dice coefficient as the loss function, and the formula is: Loss=α*CE+(1-α)*Dice Loss; CE = -∑p(x)log(q(x)); Among them, α is the scale parameter, X and Y are the numbers of two sample elements; the formula for calculating model accuracy is: Among them, X and Y are the number of two sample elements; Step 4: segment the facial pores using the pore detection model.
2. A facial pore method according to claim 1, characterized in that: In the step 1, the pore images of different regions are detected by facial key points, the collected facial images are segmented into forehead region images and cheek region images, and the horizontal and vertical pixels of the forehead region images and cheek region images are reduced respectively; The forehead area image and the cheek area image are respectively flipped at multiple angles, rotated at multiple angles, and cropped in multiple directions; the pore images existing in the forehead area image and the cheek area image are respectively marked, and the distinguishing colors between the pore area and the non-pore area are set.
3. A facial pore method according to claim 2, characterized in that: In step 2, the overall framework of the semantic segmentation network adopts the SegNet neural network, the encoder of the SegNet neural network performs feature extraction through Swin-Transformer, and the multi-layer fusion network adopts the MLP neural network.
4. The system for pore detection based on deep learning according to claim 1 is used to implement the method according to any one of claims 1 to 3, characterized in that: include: A collection module, used to collect pore images of different areas on the face; A training set and a test set construction module, used to construct training sets and test sets of pore images in different regions; Configuration module, used to configure the semantic segmentation network, using Swin-Transformer and multi-layer fusion network as the encoder and decoder of the semantic segmentation network respectively; The training module is used to input the training set into the configured semantic segmentation network for training, and then evaluate it with the test set to obtain the pore detection model; The facial pore detection module is used to detect facial pores using the pore detection model.
5. The system according to claim 4, characterized in that The acquisition module comprises: The face region segmentation module is used to segment the collected face image into a forehead region image and a cheek region image by detecting face key points; The image data enhancement module is used to perform multi-angle flipping, multi-angle rotation and multi-directional cropping on the forehead area image and the cheek area image respectively.
6. A facial pore detection system according to claim 5, characterized in that: The image data enhancement module includes a color setting module, which is used to mark the pore images existing in the forehead area image and the cheek area image respectively, and set the distinguishing color between the pore area and the non-pore area.
7. A computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 3 or the functions of the system according to claims 4 to 6 when executing the computer program.
8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 or the functions of the system according to claims 4 to 6 are implemented.