Dectron2-based wrinkle identification method and device

Abstract

The invention discloses a detectron2-based wrinkle identification method and device. The method comprises the following steps: acquiring a common wrinkle image and marking the common wrinkle image; designing a neural network framework f (.), dividing an image data set into a training set and a test set according to a preset proportion, converting each label in the image data set into a one-dimensional vector format, using a feature pyramid network FPN and ResNet-50 as models of a backbone network, performing instance segmentation, and performing initialization by using a weight pre-trained on ImageNet; optimizing a network architecture training data set to prevent underfitting and overfitting; and designing an interface based on a flash framework and Ali cloud service. The system is good in use flexibility, efficient in operation, easy to operate, good in expansibility and wide in application.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning algorithms and deep learning-related classification and recognition in artificial intelligence, and specifically relates to a wrinkle recognition method and device based on detectron2, which is written into a backend interface form and deployed on a server. Background Art

[0002] Deep learning aims to build a form of advanced pattern classification by using artificial neural networks, which learn to distinguish samples of different categories and identify new samples as the correct category. With the release of large-scale training datasets (e.g., ImageNet-100, YFCC100M, etc.) and the advancement of processing power (e.g., GPU acceleration), deep learning models have achieved "deeper" data transformation layers and surpassed the level of human experts in many fields, including semantic segmentation, image recognition, natural language processing, speech recognition, autonomous driving, etc. At the same time, people have put forward higher and higher requirements for semantic segmentation and recognition, especially in face recognition, which is increasingly frequently used, among which wrinkle recognition is an important application in face recognition. Wrinkles are one of the significant signs of skin aging, which not only affects the appearance and psychological state of an individual, but may also reflect the individual's health status. Wrinkle recognition has important applications in beauty, medicine, health monitoring and other fields. For example, by analyzing the distribution and depth of wrinkles, personalized beauty and skin care recommendations can be provided, or certain health problems can be detected early. However, wrinkle recognition has the following difficulties:

[0003] 1. Data acquisition and annotation

[0004] Scarcity of high-quality datasets: High-quality, facial wrinkle image datasets for training wrinkle recognition models are relatively scarce. It is very difficult to obtain a diverse dataset that includes different ages, genders, ethnicities, and skin types.

[0005] Challenges of accurate labeling: Wrinkles are complex and subtle in shape, requiring accurate labeling. However, manually labeling wrinkle areas is time-consuming and labor-intensive, and the accuracy and consistency of the labeling results are difficult to guarantee.

[0006] 2. Complexity of wrinkle characteristics

[0007] Diversity and variability: The shape, depth, and distribution of wrinkles vary due to individual differences and external factors (e.g., lighting, expression), which increases the difficulty of model learning and generalization.

[0008] Fine-grained feature extraction: Wrinkles are usually small and complex texture features, mixed with skin color and other facial features. Extracting and identifying these fine-grained features is difficult.

[0009] 3. Model selection and optimization

[0010] Model complexity: Deep learning models (such as convolutional neural networks) usually have a large number of parameters and require a large amount of data for training, otherwise they are prone to overfitting or underfitting problems.

[0011] Computing resource requirements: Training deep learning models, especially large-scale models, requires high-performance computing resources, which places high demands on computing power and storage resources.

[0012] 4. Real-time and efficiency

[0013] Real-time processing: In practical applications, especially those that require real-time feedback, such as beauty applications, the inference speed and efficiency of the model are very important. It is necessary to optimize the inference speed of the model while ensuring accuracy.

[0014] Resource optimization: It is necessary to consider running the model efficiently under limited computing resources (such as mobile devices or embedded systems), which involves technologies such as model compression and quantization.

[0015] Although wrinkle recognition based on deep learning faces many challenges, these difficulties can be gradually overcome through the construction of high-quality data sets, the application of model optimization technology, and the enhancement of environmental adaptability, and high-precision and high-efficiency wrinkle recognition applications can be achieved. This requires multi-faceted technological progress and comprehensive innovation to fully realize the potential of deep learning in wrinkle recognition. Summary of the invention

[0016] In order to overcome the current problems in the field of wrinkle recognition, such as the difficulty in data acquisition and annotation and the high complexity of wrinkle features, the present invention provides a wrinkle recognition method and device based on detectron2, and writes it into a backend interface form and deploys it on a server.

[0017] This paper proposes an improved wrinkle recognition method based on Detectron2, which is a PyTorch-based computer vision library developed by the Facebook AI Research (FAIR) team. It is the successor of Detectron, focusing on object detection, segmentation, and other visual tasks. Detectron2 is highly modular and flexible, supports a variety of model architectures and datasets, and is widely used in academic research and industry. In the wrinkle recognition task, the framework based on Detectron2 can effectively utilize its powerful object detection and segmentation capabilities to identify wrinkle areas in the face. Through pre-trained models and transfer learning, we can quickly build a high-performance wrinkle recognition system and optimize and expand it using the flexibility of Detectron2. In order to enable the wrinkle recognition system to run efficiently in practical applications, we need to deploy it as a backend interface. By packaging the model into a RESTful API or gRPC service, the client application can easily call the model for real-time wrinkle detection. This architecture not only improves the scalability and maintainability of the system, but also enables the model to run in a distributed environment to adapt to different application scenarios and requirements. The wrinkle recognition framework based on Detectron2 combined with the deployment of the backend interface can provide efficient and accurate wrinkle detection services, which are widely used in the fields of beauty, medical treatment and health monitoring. This not only reflects the application potential of computer vision technology, but also demonstrates the important role of deep learning in solving practical problems.

[0018] A wrinkle recognition method based on detectron2 of the present invention comprises the following steps:

[0019] 1) Obtain common wrinkle images and annotate them, including:

[0020] 1.1) By cooperating with many local beauty institutions, we collected 1,500 commonly used wrinkle images while ensuring security and privacy. These images come from customers of various institutions and are closer to actual application scenarios.

[0021] 1.2) Label the images using labelme. Before that, the labelers received a series of professional training to enable them to distinguish and evaluate the types of wrinkles. Different wrinkles were labeled in different ways, such as line labeling for crow's feet and forehead wrinkles, and box labeling for nasolabial folds and Indian wrinkles. Each labeled image generates a json file, which contains information such as the labeling category, labeling method, label coordinates, image size, etc. Finally, a Python script was used to merge 1500 json files and convert them into the format of the coco dataset.

[0022] 1.3) Register the training set and validation set, where DatasetCatalog is used to register the dataset, and MetadataCatalog is used to store the metadata of the dataset, such as category name, color, key points, etc. Each dataset instance in the json file in DatasetCatalog is a dictionary containing the following key-value pairs:

[0023] file_name: The path to the image file.

[0024] image_id: Unique identifier of the image.

[0025] height: The height of the image.

[0026] width: The width of the image.

[0027] annotations: A list of annotations in the image. Each annotation dictionary contains the following key-value pairs:

[0028] o bbox: bounding box, in the format of [x,y,width,height].

[0029] o bbox_mode: Bounding box mode, using the BoxMode enumeration class.

[0030] o category_id: The identifier of the category.

[0031] 2) Design a neural network framework f(·), divide the image dataset into training set and test set according to a preset ratio, convert each label in the image dataset into a one-dimensional vector format, use the feature pyramid network FPN and ResNet-50 as the backbone network model, perform instance segmentation, and use the pre-trained weights on ImageNet for initialization, including:

[0032] 2.1) The image dataset is divided into a training set and a test set according to a preset ratio. Each label in the image dataset is converted into a one-dimensional vector format through one-hot encoding. For example, the label of the sample represented by [0, 0, 1, 0, 0, 0, 0, 0] is the third of the 9 categories. The cross entropy expression is as follows:

[0033]

[0034] where y i is the label of sample i, y′ i is the predicted value of sample i, H i is the cross entropy of sample i.

[0035] 2.2) Using FPN (Feature Pyramid Network) and ResNet-50 as the backbone network model, it will perform instance segmentation tasks and use the weights pre-trained on ImageNet for initialization. The training process will be repeated for 270,000 iterations, and the learning rate will be adjusted at 210,000 and 250,000 iterations. The segmentation accuracy reaches more than 90%, and the accuracy no longer increases significantly after 1,000 iterations. The final training iteration number is 1,000. Among them, the cross entropy loss function of the training is:

[0036]

[0037] 3) Optimize the network architecture training data set to prevent under-fitting, including:

[0038] The prediction of deep learning is greatly affected by the training data. During the model training process, the hidden layer will fit the feature distribution of the training data. For this reason, the project framework uses Faster R-CNN, which mainly consists of two parts: the loss of the region proposal network (RPN) and the target detection and classification loss based on the proposed region.

[0039] 3.1) Loss of Region Proposal Network (RPN)

[0040] The task of RPN is to generate a series of proposal boxes that may contain objects.

[0041] The loss function of RPN consists of two parts:

[0042] Classification loss (objectness): used to check whether the box contains an object.

[0043] Regression loss (bbox regression): used to adjust the position and size of the proposal box.

[0044] The total loss of RPN can be expressed as

[0045]

[0046] ·L cls is the classification loss, usually using cross entropy loss.

[0047] ·L reg is the bounding box regression loss, usually using a smooth L1 loss.

[0048] ·p i is the predicted class probability, is the actual category [1 for foreground, 0 for background].

[0049] ·t iare the predicted box regression parameters, are the regression parameters of the actual box.

[0050] ·λ is the balance coefficient, usually set to 1.

[0051] 3.2) Object detection and classification loss based on proposed regions;

[0052] After RPN generates the proposed boxes, Faster R-CNN uses these boxes for object detection and classification. The loss function of this part also includes two parts.

[0053] Classification loss (object classification): used to identify the specific category of the object in the box.

[0054] Regression loss (bbox regression): used to further adjust the position and size of the proposal box.

[0055] The total loss based on the proposed region can be expressed as

[0056]

[0057] The symbolic meaning here is similar to the loss function of RPN, but p i and t i It is the classification and regression results based on the proposed regions.

[0058] 4) Design interfaces based on the flask framework and Alibaba Cloud OSS services, including:

[0059] 4.1) Write an interface based on the flask framework and use it in the code

[0060] @app.route(' / get_data', methods = ['POST']) start url is IP +

[0061] The POST interface of get_datad, the incoming parameters are as follows:

[0062] img_url: The path of the image file in Alibaba Cloud OSS.

[0063] img_type: The type of input image.

[0064] task_name: The name of the task requested.

[0065] id: The id of the image file.

[0066] The operations after passing the parameters are as follows:

[0067] Verify the parameters, of which img_url and id are required parameters.

[0068] Use bucket.get_object(img_url).read() in the oss2 library to read the image at the corresponding location in Alibaba Cloud OSS. The reading method is byte type.

[0069] Convert the bytes to an array using np.asarray(bytearray(local_file)).

[0070] Decode the image using cv2.imdecode(img_array,cv2.IMREAD_COLOR).

[0071] Convert the image to RGB format using cv2.cvtColor(img_np,cv2.COLOR_BGR2RGB).

[0072] Pass the RGB function into the trained network to obtain the recognition type, confidence, recognition box coordinates, etc.

[0073] Use bucket.put_object() to store the recognized image into Alibaba Cloud OSS.

[0074] 4.2) Alibaba Cloud Object Storage Service (OSS) is a massive, secure, low-cost, and highly reliable cloud storage service provided by Alibaba Cloud. It can help users store and manage large amounts of unstructured data, such as pictures, videos, documents, and backup data. Use the following code to complete the configuration of Alibaba Cloud:

[0075] #Read Alibaba Cloud configuration

[0076] config=configparser.ConfigParser()

[0077] config.read('. / config / config.ini')

[0078] access_id=config.get("oss-config",'access_id')

[0079] access_key=config.get("oss-config",'access_key')

[0080] bucket_name=config.get("oss-config",'bucket_name')

[0081] endpoint=config.get("oss-config",'endpoint')

[0082] #Connect to Alibaba Cloud

[0083] auth=oss2.Auth(access_id,access_key)

[0084] bucket=oss2.Bucket(auth,endpoint,bucket_name)

[0085] The second aspect of the present invention relates to a wrinkle recognition device based on Detectron2, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a wrinkle recognition method based on Detectron2 of the present invention.

[0086] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a wrinkle recognition method based on detectron2 of the present invention.

[0087] The working principle of the present invention is:

[0088] Based on deep learning models, especially convolutional neural networks (CNNs), to identify and segment specific objects or patterns in images. For wrinkle recognition, the model may need to go through specific preprocessing steps, such as image enhancement and normalization, to improve recognition accuracy. Then, the trained model is used to detect wrinkle features in the image and distinguish different wrinkle instances through instance segmentation techniques.

[0089] The advantages of the present invention are:

[0090] 1) Modular design: The modular design of Detectron2 allows developers to easily replace or add custom components, which provides great flexibility for customizing models for specific tasks (such as wrinkle recognition).

[0091] 2) High performance: The optimized algorithm ensures efficient operation under complex tasks, which is crucial for wrinkle recognition tasks that require processing large amounts of image data.

[0092] 3) Easy to use: Detailed documentation and examples allow even beginners to quickly start a project and speed up the development process.

[0093] 4) Full support for custom datasets: Detectron2 provides complete process guidance from direct application of pre-trained models to training in specific scenarios, which means that developers can train models based on their own datasets to identify wrinkles.

[0094] 5) Containerized deployment options: Through Docker, Detectron2 makes environment configuration simple and also facilitates cross-platform work, which facilitates the deployment of wrinkle recognition models in different environments.

[0095] 6) High scalability: Detectron2’s architecture supports synchronous BatchNorm and support for new datasets, which enables the model to adapt to different data and requirements.

[0096] 7) Rich pre-trained models: Detectron2 provides rich pre-trained models that can be directly used for target detection tasks, which provides a basis for rapid wrinkle recognition.

[0097] 8) Wide range of practical applications: Detectron2 has a wide range of uses in practical applications. For example, in medical image analysis, it can be used to identify and analyze wrinkles, provide support for skin aging research or skin condition assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 This is an example of a labelme-annotated image in an embodiment of the present invention.

[0099] Figure 2 It is the detectron2 framework structure in the embodiment of the present invention.

[0100] Figure 3 is a structural diagram of a feature pyramid network in an embodiment of the present invention.

[0101] Figure 4 It is the json data returned after the APIfox request interface is completed in the embodiment. DETAILED DESCRIPTION

[0102] The present invention will be further described below in conjunction with the accompanying drawings.

[0103] Example 1

[0104] The embodiment of the present invention provides a wrinkle recognition framework based on detectron2 and its backend interface design, including the following steps:

[0105] Get an image dataset and use labelme to annotate wrinkle information on 1,500 commonly used wrinkle images. Use different annotation methods for different wrinkles. Each annotated image will generate a json file. Finally, use a Python script to merge the 1,500 json files and convert them into the format of the coco dataset.

[0106] Use the detectron2 framework to register the training set and validation set, where DatasetCatalog is used to register the dataset. Each instance in the json file is a dictionary, and MetadataCatalog is used to store the metadata of the dataset, such as category name, color, key points, etc.

[0107] The image data set is divided into a training set and a test set according to a preset ratio. The image data is converted into a one-dimensional vector format by encoding. The cross entropy expression is as follows, where y i is the label of sample i, y′ i is the predicted value of sample i, H i is the cross entropy of sample i:

[0108]

[0109] A model using FPN (Feature Pyramid Network) and ResNet-50 as the backbone network, which will perform instance segmentation tasks and is initialized using weights pre-trained on ImageNet. The cross entropy loss function for training is:

[0110]

[0111] The Faster R-CNN framework is used, which mainly consists of two parts: the loss of the region proposal network (RPN) and the target detection and classification loss based on the proposed region. The region proposal network includes classification loss and regression loss. The total loss of RPN can be expressed as

[0112]

[0113] The target detection and classification loss based on the proposed region includes two parts: classification loss for identifying the specific category of the object in the frame, and regression loss for further adjusting the position and size of the proposed frame. The total loss based on the proposed region can be expressed as

[0114]

[0115] ·Use the flask framework and Alibaba Cloud OSS service to design an interface, verify the parameters after entering them, convert the image into RGB array format, input it into the trained model for recognition, return the recognized information in JSON format, and save the image on Alibaba Cloud OSS.

[0116] Example 2

[0117] The present embodiment relates to a wrinkle recognition device based on Detectron2, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a wrinkle recognition method based on Detectron2 of Embodiment 1.

[0118] Example 3

[0119] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a wrinkle recognition method based on detectron2 of the present invention.

[0120] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A wrinkle recognition method based on detectron2, comprising the following steps: 1) Obtain common wrinkle images and annotate them; 2) Design a neural network framework f(·), divide the image dataset into training set and test set according to a preset ratio, convert each label in the image dataset into a one-dimensional vector format, use the feature pyramid network FPN and ResNet-50 as the backbone network model, perform instance segmentation, and use the weights pre-trained on ImageNet for initialization; 3) Optimize the network architecture training data set to prevent under-fitting; 4) Design interfaces based on the flask framework and Alibaba Cloud OSS services.

2. A wrinkle recognition method based on detectron2 as claimed in claim 1, characterized in that: Step 1) specifically includes: 1.1) Cooperate with beauty institutions to collect customers’ wrinkle images while ensuring security and privacy; 1.2) Labelme is used to annotate wrinkle images, distinguish and evaluate wrinkle categories. Different wrinkles are annotated in different ways. Lines are used to annotate crow's feet and forehead wrinkles, and boxes are used to annotate nasolabial folds and Indian wrinkles. A json file is generated for each annotated image, which contains information such as the annotation category, annotation method, label coordinates, image size, etc. Finally, a Python script is used to merge the json files and convert them into the format of the coco dataset. 1.3) Register the training set and validation set, where DatasetCatalog is used to register the dataset, and MetadataCatalog is used to store the metadata of the dataset, including category name, color, and key points; each dataset instance in the json file in DatasetCatalog is a dictionary containing the following key-value pairs: file_name: path to the image file; image_id: unique identifier of the image; height: the height of the image; width: the width of the image; annotations: A list of annotations in the image. Each annotation dictionary contains the following key-value pairs: οbbox: bounding box, format is [x, y, width, height]; οbbox_mode: bounding box mode, using the BoxMode enumeration class; οcategory_id: identifier of the category.

3. The wrinkle recognition method based on detectron2 as claimed in claim 1, characterized in that: Step 2) specifically includes: 2.1) Divide the image dataset into training set and test set according to the preset ratio, and convert each label in the image dataset into a one-dimensional vector format through one-hot encoding. The cross entropy expression is as follows: where y i is the label of sample i, y i ' is the predicted value of sample i, H i is the cross entropy of sample i; 2.2) Use the feature pyramid network FPN and ResNet-50 as the backbone network model for instance segmentation tasks and initialize them with the weights pre-trained on ImageNet. The cross entropy loss function of the training is:

4. The wrinkle recognition method based on detectron2 as claimed in claim 1, characterized in that: Step 3) specifically includes: The project framework uses Faster R-CNN, including the loss of the regional proposal network RPN and the target detection and classification loss based on the proposed region; 3.1) Loss of the Region Proposal Network (RPN); The loss function of RPN consists of two parts: Classification loss objectness: used to check whether the box contains an object; Regression loss bbox regression: used to adjust the position and size of the suggestion box; The total loss of RPN can be expressed as ·L cls is the classification loss, usually using cross entropy loss; ·L reg is the bounding box regression loss, usually using smooth L1 loss; ·p i is the predicted class probability, is the actual category [1 for foreground, 0 for background]; ·t i are the predicted box regression parameters, is the regression parameter of the actual box; ·λ is the balance coefficient, usually set to 1; 3.2) Object detection and classification loss based on proposed regions; After RPN generates the proposed boxes, Faster R-CNN uses these boxes for target detection and classification. The loss function of this part also includes two parts Classification loss object classification: used to identify the specific category of the object in the box; Regression loss bbox regression: used to further adjust the position and size of the suggestion box; The total loss based on the proposed regions is expressed as: Among them, p i and t i It is the classification and regression results based on the proposed regions.

5. The wrinkle recognition method based on detectron2 as claimed in claim 1, characterized in that: Step 4) specifically includes: 4.1) Write an interface based on the flask framework. Use @app.route(' / get_data', methods = ['POST']) in the code to start the POST interface with the url IP + get_datad. The incoming parameters are as follows: img_url: the path of the image file in Alibaba Cloud OSS; img_type: the type of input image; task_name: the name of the task being requested; id: the id of the image file; The operations after passing the parameters are as follows: Verify the parameters, where img_url and id are mandatory parameters; Use bucket.get_object(img_url).read() in the oss2 library to read the image at the corresponding location in Alibaba Cloud OSS. The reading method is byte type. Convert the bytes to an array using np.asarray(bytearray(local_file)) Decode the image using cv2.imdecode(img_array,cv2.IMREAD_COLOR); Convert the image to RGB format using cv2.cvtColor(img_np,cv2.COLOR_BGR2RGB); Pass the RGB function into the trained network to obtain the recognition type, confidence, recognition box coordinates, etc. Use bucket.put_object() to store the recognized image into Alibaba Cloud OSS; 4.2) Use the following code to complete the configuration of Alibaba Cloud: #Read Alibaba Cloud configuration config=configparser.ConfigParser() config.read('. / config / config.ini') access_id=config.get("oss-config",'access_id') access_key=config.get("oss-config",'access_key') bucket_name=config.get("oss-config",'bucket_name') endpoint=config.get("oss-config",'endpoint') #Connect to Alibaba Cloud auth=oss2.Auth(access_id,access_key) bucket=oss2.Bucket(auth,endpoint,bucket_name).

6. A wrinkle recognition device based on detectron2, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement a wrinkle recognition method based on detectron2 according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a wrinkle recognition method based on detectron2 as described in any one of claims 1 to 5 is implemented.

Images