Methods and devices for marking the background of labeled images, and for identifying skin problems.

Abstract

This invention relates to the field of image recognition technology, and particularly to a method for labeling the background of an labeled image, a method for training an artificial neural network for skin problem recognition, a skin problem recognition method / device, and a computer-readable storage medium. The method for labeling the background of an labeled image of this invention further processes a skin image with labeled regions and background regions. Regions in the background region adjacent to the labeled regions are labeled as neighboring regions. Weights are assigned to these neighboring regions and the remaining non-neighboring regions in the background region. The labeled skin image is then used as a training and validation set to train a multi-task convolutional neural network model. Finally, the skin image to be recognized is input into the trained skin problem recognition network for recognition, obtaining the labeled regions and the skin problem types in their labeling information, thus achieving skin problem recognition with a low probability of background misidentification.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method for labeling the background of an labeled image, a method for training an artificial neural network for skin problem recognition based on the labeling method, a skin problem recognition method / apparatus implemented by the skin problem recognition network trained by the training method, and a computer-readable storage medium that, when executed by a processor, can implement the above-mentioned labeling method, and / or training method, and / or skin problem recognition method. Background Technology

[0002] Current technology utilizes pre-trained artificial neural networks to automatically identify skin problems in facial images, such as freckles, age spots, acne, and moles. This requires manually labeling a large number of images to identify skin problems, specifically marking areas with skin issues. These labeled images are then fed into the artificial neural network for training. During recognition, each image identifies only one labeled problem at a time, such as freckles, while unlabeled areas are treated as background. However, in real-life facial images, multiple skin problems often coexist, such as age spots (A), acne (B), and moles (C). When identifying skin problems in a facial image, if only labeled age spots (A) are identified, the artificial neural network may mistakenly identify acne (B) and moles (C) as normal skin during training, thus affecting its ability to recognize acne (B) and moles (C) in subsequent training sessions. The current solution is to reduce the weight of the background during training. For example, the weight of age spot A, which is currently being identified, is 99%, while the weight of background areas containing skin problems such as sun spots B and pigmentation C is only 1%. This can reduce the interference of other skin problems such as sun spots B and pigmentation C that are not labeled to the background recognition. However, it will also lead to insufficient recognition weight of background areas, reduce the background recognition ability and cause background misidentification. Summary of the Invention

[0003] To address this, the inventors have provided a method for marking the background of an labeled image, which can reduce the probability of misidentification of the background when the image marked by this method is used for skin problem recognition.

[0004] The present invention provides a method for labeling the background of an labeled image, wherein the image is a skin image and the labeled image includes a labeled region and a background region, and includes the following labeling steps: labeling the regions in the background region of the image that are adjacent to the labeled region as neighboring regions, and labeling the weights of the neighboring regions and the remaining non-neighboring regions in the background region respectively.

[0005] Preferably, in the marked weights, the weight of the neighboring region exceeds the weight of the non-neighboring region by a preset degree.

[0006] Preferably, the weight of the neighboring region exceeding the weight of the non-neighboring region by a preset degree means that the weight of the neighboring region is ten times or more the weight of the non-neighboring region.

[0007] Preferably, the region adjacent to the labeled area refers to: the region extending outward from the outer edge of the labeled area by a predetermined distance, or, generating a closed line surrounding the labeled area, and recording the region between the closed line and the outer edge of the labeled area as the region adjacent to the labeled area.

[0008] Preferably, the labeled area of ​​the labeled image includes labeling information, which includes the type of skin problem in the labeled area.

[0009] Preferably, the skin problem type includes one or more of freckles, age spots, acne, and moles.

[0010] The present invention also provides a method for training an artificial neural network for skin problem identification, comprising the following steps:

[0011] (1) Obtain a skin image dataset, and use the skin images in the dataset that have been labeled in the above labeling steps as the training set and the validation set. The artificial neural network includes a multi-task convolutional neural network model, and the skin images in the training set are used as training samples as the training input of the multi-task convolutional neural network model.

[0012] (2) Identify each skin image in the training set respectively, wherein the identification weight of the labeled region is greater than the identification weight of the background region, and the identification weight of the neighboring region in the background region is greater than the identification weight of the non-neighboring region in the background region.

[0013] (3) The training set is matched according to the annotation information of the annotation region. The annotation information refers to the skin problem type of the annotation region. The matching difference value between the annotation region identified by the neural network model and the corresponding annotation region on the labeled image is obtained, and then the loss value is obtained by multi-task loss calculation.

[0014] (4) Update the weight file based on the calculated loss value. After all training sets are updated, perform validation set verification and accuracy calculation.

[0015] (5) Repeat steps (1) to (4) until the preset number of repetitions is reached or the accuracy of the validation set reaches the preset level, then terminate the iteration to obtain the trained multi-task convolutional neural network model as the skin problem recognition network.

[0016] The present invention also provides a method for identifying skin problems, comprising the following steps: inputting the skin image to be identified into a skin problem identification network trained by the above-described training method for artificial neural networks used for skin problem identification, and obtaining the labeled area and its labeled information.

[0017] The present invention also provides a skin problem identification device, comprising:

[0018] The image acquisition unit is used to acquire skin images;

[0019] A skin problem recognition unit is used to input the acquired skin image into a skin problem recognition network, and the skin problem recognition network outputs the labeled area and its labeling information. The skin problem recognition network is obtained by training an artificial neural network for skin problem recognition.

[0020] An output unit outputs a labeled area and a skin problem type for that labeled area obtained based on the labeled information. It includes an interconnected computer-readable storage medium and a processor, the computer-readable storage medium as described above.

[0021] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method for marking the background of an annotated image, and / or the above-described method for training an artificial neural network for skin problem identification, and / or the steps in the above-described skin problem identification method.

[0022] Beneficial effects

[0023] The method for marking the background of an labeled image of the present invention further processes the skin image with the labeled area and the background area, marks the area in the background area that is adjacent to the labeled area as the neighboring area, and marks the weight of the neighboring area and the remaining non-neighboring area in the background area respectively.

[0024] The present invention provides a training method for an artificial neural network used for skin problem identification. This method uses skin images labeled using the aforementioned labeling method as both a training and validation set to train a multi-task convolutional neural network model. The skin images from the training set are used as training samples and input to the multi-task convolutional neural network model. Each training iteration identifies one type of skin problem. During training, the labeled regions and background regions in the skin images are identified. After general training, validation, and accuracy calculations for the multi-task convolutional neural network model, a well-trained model capable of identifying the labeled regions of that specific skin problem type is obtained.

[0025] The recognition weights for labeled areas, neighboring areas within the background area, and non-neighboring areas decrease sequentially, with neighboring areas within the background area receiving a significantly higher recognition weight than non-neighboring areas. Labeled areas represent regions with skin problems. Since multiple skin problems typically don't overlap in the same location on human skin, labeled areas adjacent to areas with skin problems are more likely to be normal skin and can be considered part of the real background, thus reducing the chance of misidentification. Conversely, skin far from these labeled areas is more likely to have other skin problems; therefore, assigning these non-neighboring areas a lower background weight than neighboring areas reduces interference from other skin problems in background recognition.

[0026] The skin problem recognition method of the present invention specifically involves inputting the skin image to be recognized into the above-trained skin problem recognition network for recognition, obtaining the labeled area and the skin problem type in the labeled information, thereby achieving skin problem recognition with a low probability of background misidentification. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the method for labeling the background of an already labeled image.

[0029] Figure 2 This is a flowchart of the training method for an artificial neural network used for skin problem identification.

[0030] Figure 3 This is a flowchart of a skin problem identification method.

[0031] Figure 4 This is an example skin image from a skin image dataset.

[0032] Figure 5 Yes Figure 4 A diagram illustrating the annotation of a skin image.

[0033] Figure 6 Yes Figure 5 A schematic diagram of an already labeled image.

[0034] Figure 7 This is an example of a skin image used for skin problem identification.

[0035] Figure 8 Yes Figure 7A diagram illustrating the identification of skin problem types in skin images.

[0036] Figure 9 This is a schematic diagram of the structure of a skin problem training and recognition device. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In this invention, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0039] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0040] The method for marking the background of an annotated image according to the present invention is described in [reference needed]. Figure 1 The skin image with labeled and background regions is further processed by marking regions adjacent to the labeled regions in the background region as neighboring regions, and assigning weights to these neighboring regions and the remaining non-neighboring regions in the background region. The training method for the artificial neural network used for skin problem recognition according to this invention is described in [link to invention]. Figure 2 The skin images labeled using the above-mentioned labeling method are used as the training and validation sets to train a multi-task convolutional neural network model. Specifically, the skin images in the training set are used as training samples as the training input for the multi-task convolutional neural network model. Each training iteration identifies one type of skin problem. During training, the labeled regions and background regions in the skin images are identified. After general training, validation, and accuracy calculation of the multi-task convolutional neural network model, a trained multi-task convolutional neural network model capable of identifying the labeled regions of that skin problem type is obtained. The skin problem identification method of this invention is described in [link to invention]. Figure 3Specifically, the skin image to be identified is input into the pre-trained skin problem recognition network for identification, resulting in labeled regions and the types of skin problems identified in the labeled information. This invention achieves skin problem recognition with a low probability of background misidentification through the aforementioned labeling method, training method, and skin problem recognition method.

[0041] This embodiment takes the skin problem recognition process of facial skin images as an example to explain in detail the labeling method of the background of the labeled image, the training method of the artificial neural network used for skin problem recognition, and the specific steps of the skin problem recognition method.

[0042] The method for marking the background of an annotated image in this embodiment includes steps 101 to 104, which are described in detail below.

[0043] Step 101: Batch acquire facial skin images to obtain a skin image dataset. Figure 4 This is an example of a skin image from a skin image dataset.

[0044] Step 102: Label various skin problem types on each skin image in the skin image dataset, so as to... Figure 4 For example, different color blocks are used to mark the skin image (see...). Figure 5 Skin problem types such as age spots (A), acne (B), moles (C), and uncertain types (D) are identified, and labeled regions are formed. The labeling information corresponding to these labeled regions is recorded to obtain labeled images. The labeling information includes the skin problem type of the labeled region.

[0045] In step 102 of this embodiment, different people label different skin problem types on the same skin image, and during recognition, only one labeling type is selected for recognition each time, thereby making full use of the skin image. This step 102 can be changed to label only one skin problem type on a skin image at a time. In this way, the artificial neural network trained will be more accurate than the artificial neural network trained by recognizing all labeled skin problem types at once.

[0046] Before annotation, the facial contour line 1 is automatically identified. The annotation in step 102 is performed within the facial skin area enclosed by the facial contour line 1. Furthermore, non-skin areas such as eyebrows, eyes, nose, and mouth can be excluded using existing methods, which will not be elaborated upon here. Within the facial skin area, except for the annotated area, the rest is the background area of ​​the skin image.

[0047] For the labeled image obtained in step 102 (or other methods), perform the following labeling steps (steps 103 and 104).

[0048] Step 103: Mark the regions in the background area of ​​the image that are adjacent to the labeled regions as neighboring regions. For example, consider age spots (A) in the skin problem category. Figure 6 A closed line 3 is generated around the outer edge 2 of the labeled area of ​​senile spot A. The area between the closed line 3 and the outer edge 2 of the labeled area is recorded as the neighboring area of ​​the labeled area and labeled as the neighboring region G. The closed line 3 is generated by the automatic bounding box method in existing image recognition technology. The line type can be a rectangle, a circle, etc., which will not be elaborated here. When the outer edge of the labeled area is relatively flat or rounded, the neighboring region G can be changed to an area that automatically extends outward from the outer edge 2 of the labeled area by a preset distance (such as 5 mm, or 20% of the maximum diameter of the labeled area), without the need for automatic bounding box selection by image recognition.

[0049] Step 104: Label the weights of the neighboring region G and the remaining non-neighboring regions in the background region. Among the labeled weights, the weight of the neighboring region exceeds the weight of the non-neighboring region by a preset level. Specifically, the weight of the neighboring region is ten times or more than the weight of the non-neighboring region. For example, if the weight of the background region is 1, then the weight of the neighboring region is 0.092, and the weight of the non-neighboring region is 0.008. Figure 6 In the diagram, the non-adjacent region is the facial skin area enclosed by the facial contour line 1 outside the closed line 3. This region contains various other skin problems such as acne B, moles C, and uncertain type D. The weight of this region is a very low 0.008, which can reduce the interference of other skin problems on the background region. At the same time, the adjacent region G is normal skin without other skin problems. The weight of the adjacent region G is much greater than the weight of the non-adjacent region 0.092. This allows normal skin to be identified as the background region when the skin image labeled by this labeling method is used for skin problem recognition, thereby reducing the probability of false identification of the background.

[0050] All skin images in the skin image dataset are labeled using steps 101 to 104 above, and the background of the labeled images is marked. Then, the labeled skin images are fed into the artificial neural network for training.

[0051] The training method for the artificial neural network used for skin problem recognition in this embodiment includes the following steps. The training process of the artificial neural network is implemented using general neural network model training logic, which will not be elaborated in detail below.

[0052] (1) Obtain a skin image dataset. Use the skin images in the dataset that have undergone the labeling steps described above as the training set and validation set, and use the other labeled images in the dataset as the test set. The artificial neural network includes a multi-task convolutional neural network model, and use the skin images in the training set as training samples as the training input of the multi-task convolutional neural network model.

[0053] (2) Identify each skin image in the training set, where the recognition weight for the labeled region is greater than that for the background region, and the recognition weight for the neighboring regions in the background region is greater than that for the non-neighboring regions in the background region. Specifically, the recognition weights for the neighboring and non-neighboring regions in the background region are the weights assigned in step 104 of the labeling method. The recognition weights for the labeled region, the neighboring regions in the background region, and the non-neighboring regions decrease sequentially, with the recognition weight for the neighboring regions in the background region being much higher than that for the non-neighboring regions. The labeled region is the area with skin problems. Multiple skin problems typically do not overlap at the same location on the human skin surface. Therefore, the labeled region adjacent to the area with skin problems is more likely to be normal skin and can be considered as the real background, thus reducing the probability of misidentification of the background. Conversely, skin far from this labeled region has a higher proportion of other skin problems. Therefore, assigning these non-neighboring regions a lower background weight than the neighboring regions can reduce the interference of other skin problems on background recognition.

[0054] (3) The training set is categorized and matched according to the annotation information of the labeled regions. The annotation information refers to the skin problem type of the labeled region, including age spots (A), acne (B), moles (C), uncertain types (D), and freckles (E). The matching difference value between the labeled region identified by the neural network model and the corresponding labeled region on the labeled image is obtained, and the loss value is calculated by multi-task loss calculation. In this process, one type of skin problem is identified in each training session. After multiple training layers, the multi-task convolutional neural network model can identify multiple types of skin problems together.

[0055] (4) Update the weight file based on the calculated loss value. After all training sets are updated, perform validation set verification and accuracy calculation.

[0056] (5) Repeat steps (1) to (4) until the preset number of repetitions is reached or the accuracy of the validation set reaches the preset level (such as the accuracy tends to be stable and no longer increases), terminate the iteration, perform a final test on the test set, and use the error on the test set as an approximation of the generalization error, thereby obtaining the trained multi-task convolutional neural network model as a skin problem recognition network.

[0057] The above steps are used to train a skin problem recognition network that can identify various skin problems in skin images. This skin problem recognition network is then used in skin problem recognition methods.

[0058] The skin problem recognition method of this embodiment includes the following steps: inputting the skin image to be recognized into the skin problem recognition network trained by the above-mentioned training method for artificial neural networks used for skin problem recognition, and obtaining the labeled area and its labeled information (i.e. the identified labeled area and its corresponding skin problem type).

[0059] by Figure 7 Taking a skin image as an example, this image is input into a skin problem recognition network for skin problem identification. See [link to network details] for details on how the skin problem recognition network identifies this skin image. Figure 8 The skin problems in this skin image include: Figure 8 The skin problem identification method in this embodiment can quickly obtain the labeled areas and the types of skin problems in the labeled information, achieving skin problem identification with a low probability of background misidentification. (E) Large freckles on both upper cheeks, (C) Moles on the lower left side of the nose, (B) Two acne lesions on the forehead and lower left cheek, and (D) Scattered, uncertain skin problems.

[0060] Figure 9 This is a schematic diagram of the skin problem training and recognition device according to this embodiment. The skin problem training and recognition device is used to implement the above-described method embodiments of the present invention. Specifically, the skin problem training and recognition device includes a skin problem training device and a skin problem recognition device.

[0061] The skin problem training device includes:

[0062] The dataset acquisition unit is used to acquire skin image datasets and divide the acquired skin image datasets into training set, validation set and test set;

[0063] The multi-task convolutional neural network model unit is used to train, validate, and test the multi-task convolutional neural network model. Specifically, it uses skin images from the training set as training samples for the training input of the multi-task convolutional neural network model in this embodiment; it identifies each skin image in the training set, wherein the recognition weight for labeled regions is greater than the recognition weight for background regions, and the recognition weight for neighboring regions in the background region is greater than the recognition weight for non-neighboring regions in the background region; it classifies and matches the training set according to the annotation information of the labeled regions, and calculates the loss value based on the matching difference value of the labeled regions, thereby updating the weight file; after all training sets are updated, validation set verification and accuracy calculation are performed; it repeats the above steps until a preset number of repetitions is reached or the accuracy of the validation set reaches a preset level, then the iteration terminates, and a trained multi-task convolutional neural network model is obtained, which serves as the skin problem recognition network.

[0064] Skin problem recognition devices include:

[0065] The image acquisition unit is used to acquire skin images;

[0066] The skin problem recognition unit is used to input the acquired skin image into the skin problem recognition network, and output the labeled area and its labeling information through the skin problem recognition network. The skin problem recognition network is obtained by training an artificial neural network for skin problem recognition. The skin problem recognition unit includes a processor and a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor, it can implement the skin problem recognition method described in this embodiment.

[0067] The output unit outputs the labeled area and the skin problem type of the labeled area obtained from the labeled information.

[0068] The skin problem training and recognition device embodiments described above are merely illustrative. The system modules may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0069] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0070] Finally, it should be noted that the system and method for quickly launching mobile terminal payment methods disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for marking the background of an already labeled image, characterized by: The image is a skin image, and the labeled image includes a labeled area and a background area. The labeling steps include: labeling the area in the background area of ​​the image that is adjacent to the labeled area as a neighboring area; labeling the weights of the neighboring area and the remaining non-neighboring areas in the background area respectively; among the labeled weights, the weight of the neighboring area exceeds the weight of the non-neighboring area by a preset degree, so that the neighboring area is identified as the background area; the weight of the neighboring area exceeding the weight of the non-neighboring area by a preset degree means that the weight of the neighboring area is ten times or more than the weight of the non-neighboring area; the area adjacent to the labeled area refers to: the area extending outward from the outer edge of the labeled area by a preset distance, or, generating a closed line surrounding the labeled area, and recording the area between the closed line and the outer edge of the labeled area as the area adjacent to the labeled area.

2. The method for marking the background of an labeled image according to claim 1, characterized in that, The labeled area of ​​the labeled image includes labeling information, which includes the type of skin problem in the labeled area.

3. The method for marking the background of an annotated image according to claim 2, characterized in that, The types of skin problems include one or more of the following: freckles, age spots, acne, and moles.

4. A training method for an artificial neural network used for skin problem identification, characterized by: Includes the following steps: (1) Obtain a skin image dataset, and use the skin images in the dataset that have been labeled by the labeling step described in any one of claims 1 to 3 as the training set and the validation set. The artificial neural network includes a multi-task convolutional neural network model, and the skin images in the training set are used as training samples as the training input of the multi-task convolutional neural network model. (2) Identify each skin image in the training set respectively, wherein the recognition weight of the labeled region is greater than the recognition weight of the background region, and the recognition weight of the neighboring region in the background region is greater than the recognition weight of the non-neighboring region in the background region; (3) The training set is matched according to the annotation information of the annotation area. The annotation information refers to the skin problem type of the annotation area. The matching difference value between the annotation area identified by the neural network model and the corresponding annotation area on the labeled image is obtained, and then the loss value is obtained by multi-task loss calculation. (4) Update the weight file based on the calculated loss value. After all training sets are updated, perform validation set verification and accuracy calculation. (5) Repeat steps (1) to (4) until the preset number of repetitions is reached or the accuracy of the validation set reaches the preset level, then terminate the iteration to obtain the trained multi-task convolutional neural network model as the skin problem recognition network.

5. Skin problem identification methods, characterized by: The process includes the following steps: inputting the skin image to be identified into the skin problem recognition network trained according to claim 4 for recognition, and obtaining the labeled area and its labeling information.

6. A skin problem recognition device, characterized in that: include: The image acquisition unit is used to acquire skin images; A skin problem recognition unit is used to input the acquired skin image into a skin problem recognition network, and output the labeled area and its labeling information through the skin problem recognition network. The skin problem recognition network is obtained by training an artificial neural network for skin problem recognition. The skin problem recognition unit includes a processor and a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor, it can implement the skin problem recognition method as described in claim 5. The output unit outputs the labeled area and the skin problem type of the labeled area obtained from the labeled information.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement: a method for labeling the background of an labeled image as described in any one of claims 1 to 3, and / or a method for training an artificial neural network for skin problem identification as described in claim 4, and / or a method for identifying skin problems as described in claim 5.

Images