Eye bag detection method and device

Abstract

The present application provides an eye bag detection method and device, which relates to the field of face recognition technology, wherein the method comprises obtaining an image to be detected, the image to be detected comprises an eye bag ROI region of interest, detecting the eye bag ROI region through a preset convolutional neural network model, obtaining an eye bag detection score and eye bag position detection information, and when the eye bag detection score is within a preset score range, marking the image to be detected based on the eye bag detection score and the eye bag position detection information, and obtaining eye bag marking information. The technical solution provided by the present application can accurately identify the position and score of eye bags, significantly improving the accuracy of identifying eye bags.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, and in particular to an eye bag detection method and device. Background Art

[0002] Face recognition technology has been widely used in many fields such as photography, security, education and finance. As the application of face recognition technology continues to deepen, the precision of recognition is also attracting more and more attention. Among them, eye bags are an important recognition content. The survey shows that about 65% of users want to identify eye bags.

[0003] In the prior art, face key points are usually detected on the image to be detected to obtain eye key points, and then a preset area is determined as the eye bag area according to the eye key points.

[0004] However, since the existing technology does not actually identify eye bags, the shape and size of the preset area (i.e., the eye bag area) determined based on the key points of the eye is closely related to the shape and size of the eyes, while the shape and size of real eye bags have no specific correlation with the shape and size of the eyes. Therefore, the eye bag area determined by the existing technology is very different from the actual eye bag area and has a low accuracy rate. Summary of the invention

[0005] In view of this, the present application provides an eye bag detection method and device, which can improve the accuracy of identifying eye bags.

[0006] In order to achieve the above objectives, in a first aspect, an embodiment of the present application provides an eye bag detection method, comprising:

[0007] Acquire an image to be detected, wherein the image to be detected includes a region of interest (ROI) of an eye bag;

[0008] The eye bag ROI area is detected by a preset convolutional neural network (CNN) model to obtain an eye bag detection score and eye bag position detection information;

[0009] When the eye bag detection score is within a preset score range, the image to be detected is marked based on the eye bag detection score and the eye bag position detection information to obtain eye bag marking information.

[0010] In an embodiment of the present application, an image to be detected including an eye bag ROI region can be obtained, and then the eye bag ROI region can be directly detected through a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information. When the eye bag detection score is within a preset score range, that is, when it is determined that eye bags exist, the image to be detected can be marked by the eye bag detection score and the eye bag position detection information, thereby obtaining eye bag marking information for detecting eye bags. Since the eye bag detection score and the eye bag position detection information here are directly identified from the eye bag ROI region, rather than being set according to the size and shape of the eyes, the accuracy of detecting eye bags can be significantly improved.

[0011] Optionally, before detecting the eye bag ROI area by using a preset convolutional neural network model, the method further includes:

[0012] Performing facial key point detection on the image to be detected to obtain eye key points;

[0013] Based on the eye key points, the eye bag ROI area is determined from the image to be detected.

[0014] Optionally, the determining the eye bag ROI area from the image to be detected based on the eye key points includes:

[0015] Based on the eye key points, determining the eye center point;

[0016] Taking the center point of the eye as a reference point, an area of ​​a preset size and a preset shape is obtained from the image to be detected as the eye bag ROI area.

[0017] Optionally, the eye center point is located in the upper half of the eye bag ROI region, and is located at 1 / 2 of the width and 1 / 4 of the height of the eye bag ROI region.

[0018] Optionally, it also includes:

[0019] The eye bag ROI area is detected by using the preset convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0020] When the under-eye bags detection classification result is yes, based on the under-eye bags detection classification result and the under-eye bags position detection information, the under-eye bags in the image to be detected are marked to obtain under-eye bags marking information.

[0021] Optionally, the eye bag position detection information includes eye bag key points, and marking the eye bags in the image to be detected based on the eye bag detection score and the eye bag position detection information includes:

[0022] Perform interpolation fitting based on the eye bag key points to obtain the eye bag closed area;

[0023] The eye bags in the image to be detected are marked based on the eye bag detection score and the eye bag closed area.

[0024] Optionally, the eye bag position detection information includes an eye bag segmentation mask, and marking the eye bags in the image to be detected based on the eye bag detection score and the eye bag position detection information includes:

[0025] The eye bags in the image to be detected are marked based on the eye bag detection score and the eye bag segmentation mask.

[0026] Optionally, the under-eye bag position detection information includes under-eye bag key points, and marking the under-eye bag in the image to be detected based on the under-eye bag detection classification result and the under-eye bag position detection information includes:

[0027] Perform interpolation fitting based on the key points of the under-eye bags to obtain a closed area of ​​the under-eye bags;

[0028] Based on the under-eye bags detection and classification result and the under-eye bags closed area, the under-eye bags in the image to be detected are marked.

[0029] Optionally, the under-eye bag position detection information includes an under-eye bag segmentation mask, and marking the under-eye bag in the image to be detected based on the under-eye bag detection classification result and the under-eye bag position detection information includes:

[0030] Based on the under-eye bags detection and classification result and the under-eye bags segmentation mask, the under-eye bags in the image to be detected are marked.

[0031] Optionally, the preset convolutional neural network model includes multiple convolutional layers, wherein, except the first convolutional layer, the other convolutional layers include at least one depth-separable convolutional layer.

[0032] Optionally, the preset convolutional neural network model is trained on a plurality of sample images, and the sample images carry eye bag annotation scores and eye bag position annotation information.

[0033] Optionally, the sample image also carries under-eye bag annotation score and under-eye bag position annotation information.

[0034] Optionally, the eye bag ROI area includes a left eye bag ROI area and a right eye bag ROI area, and before detecting the eye bag ROI area by using a preset convolutional neural network model, the method further includes:

[0035] Based on the left eye bag ROI area and the right eye bag ROI area, the image to be detected is segmented to obtain a left eye bag ROI area image and a right eye bag ROI area image;

[0036] Mirroring the right eye bag ROI area image in the left and right directions;

[0037] The left eye bag ROI area image and the mirrored right eye bag ROI area image are input into the preset convolutional neural network model.

[0038] Optionally, marking eye bags in the image to be detected includes:

[0039] Marking the left eye bag ROI area image and the mirrored right eye bag ROI area image;

[0040] The marked right eye bag ROI area image is mirrored again along the left-right direction.

[0041] In a second aspect, an embodiment of the present application provides a method for training a convolutional neural network model, comprising:

[0042] Acquire multiple sample images, wherein the sample images include an eye bag ROI area, and the sample images carry an eye bag annotation score and an eye bag position annotation information;

[0043] The eye bag ROI area is detected by a convolutional neural network model to obtain an eye bag detection score and eye bag position detection information;

[0044] Based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score and the eye bag position annotation information, the model parameters of the convolutional neural network model are determined.

[0045] Optionally, the sample image also carries the classification result of the under-eye bags annotation and the under-eye bags position annotation information, and further includes:

[0046] The eye bag ROI area is detected by the convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0047] The step of determining the model parameters of the convolutional neural network model based on the eye bag detection score, the eye bag position detection information, the eye bag annotation score, and the eye bag position annotation information of the sample image includes:

[0048] Based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score, the eye bag position annotation information, the eye bag annotation classification result, the eye bag position annotation information, the eye bag detection classification result and the eye bag position detection information, the model parameters of the convolutional neural network model are determined.

[0049] In a third aspect, an embodiment of the present application provides an eye bag detection device, comprising:

[0050] An acquisition module, used for acquiring an image to be detected, wherein the image to be detected includes an eye bag ROI area;

[0051] A detection module, used to detect the eye bag ROI area through a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information;

[0052] A marking module is used to mark the eye bags in progress in the image to be detected based on the eye bag detection score and the eye bag position detection information when the eye bag detection score is within a preset score range, so as to obtain eye bag marking information.

[0053] Optionally, a determination module is also included;

[0054] The detection module is also used to perform facial key point detection on the image to be detected to obtain eye key points;

[0055] The determination module is used to determine the eye bag ROI area from the image to be detected based on the eye key points.

[0056] Optionally, the determining module is further used for:

[0057] Based on the eye key points, determining the eye center point;

[0058] Taking the center point of the eye as a reference point, an area of ​​a preset size and a preset shape is obtained from the image to be detected as the eye bag ROI area.

[0059] Optionally, the eye center point is located in the upper half of the eye bag ROI region, and is located at 1 / 2 of the width and 1 / 4 of the height of the eye bag ROI region.

[0060] Optionally, the detection module is further used to detect the eye bag ROI area through the preset convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0061] The marking module is also used to mark the under-eye bags in progress in the image to be detected based on the under-eye bags detection classification result and the under-eye bags position detection information to obtain under-eye bags marking information when the under-eye bags detection classification result is yes.

[0062] Optionally, the eye bag position detection information includes eye bag key points, and the marking module is further used for:

[0063] Perform interpolation fitting based on the eye bag key points to obtain the eye bag closed area;

[0064] The eye bags in the image to be detected are marked based on the eye bag detection score and the eye bag closed area.

[0065] Optionally, the eye bag position detection information includes an eye bag segmentation mask, and the marking module is further used to:

[0066] The eye bags in the image to be detected are marked based on the eye bag detection score and the eye bag segmentation mask.

[0067] Optionally, the under-eye bag position detection information includes under-eye bag key points, and the marking module is further used for:

[0068] Perform interpolation fitting based on the key points of the under-eye bags to obtain a closed area of ​​the under-eye bags;

[0069] Based on the under-eye bags detection and classification result and the under-eye bags closed area, the under-eye bags in the image to be detected are marked.

[0070] Optionally, the under-eye bag position detection information includes an under-eye bag segmentation mask, and the marking module is further used to:

[0071] Based on the under-eye bags detection and classification result and the under-eye bags segmentation mask, the under-eye bags in the image to be detected are marked.

[0072] Optionally, the preset convolutional neural network model includes multiple convolutional layers, wherein, except the first convolutional layer, the other convolutional layers include at least one depth-separable convolutional layer.

[0073] Optionally, the preset convolutional neural network model is trained on a plurality of sample images, and the sample images carry eye bag annotation scores and eye bag position annotation information.

[0074] Optionally, the sample image also carries under-eye bag annotation score and under-eye bag position annotation information.

[0075] Optionally, the eye bag ROI area includes a left eye bag ROI area and a right eye bag ROI area, and further includes:

[0076] A segmentation module, used for segmenting the image to be detected based on the left eye bag ROI area and the right eye bag ROI area to obtain a left eye bag ROI area image and a right eye bag ROI area image;

[0077] A mirroring module, used for mirroring the right eye bag ROI area image in the left and right directions;

[0078] An input module is used to input the left eye bag ROI area image and the right eye bag ROI area image after mirroring into the preset convolutional neural network model.

[0079] Optionally, the marking module is further used for:

[0080] Marking the left eye bag ROI area image and the mirrored right eye bag ROI area image;

[0081] The marked right eye bag ROI area image is mirrored again along the left-right direction.

[0082] In a fourth aspect, an embodiment of the present application provides a training device for a convolutional neural network model, comprising:

[0083] An acquisition module, used for acquiring a plurality of sample images, wherein the sample images include an eye bag ROI area, and the sample images carry an eye bag annotation score and an eye bag position annotation information;

[0084] A detection module, used to detect the eye bag ROI area through a convolutional neural network model to obtain an eye bag detection score and eye bag position detection information;

[0085] A determination module is used to determine the model parameters of the convolutional neural network model based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score and the eye bag position annotation information.

[0086] Optionally, the sample image also carries the classification result of the under-eye bags annotation and the under-eye bags position annotation information;

[0087] The detection module is also used to detect the eye bag ROI area through the convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0088] The determination module is also used to determine the model parameters of the convolutional neural network model based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score, the eye bag position annotation information, the eye bag annotation classification result, the eye bag position annotation information, the eye bag detection classification result and the eye bag position detection information.

[0089] In a fifth aspect, the present application provides a method for detecting undereye bags, comprising:

[0090] Acquire an image to be detected, wherein the image to be detected includes an eye bag ROI area;

[0091] The eye bag ROI area is detected by using a preset convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0092] When the under-eye bags detection classification result is yes, based on the under-eye bags detection classification result and the under-eye bags position detection information, the under-eye bags in the image to be detected are marked to obtain under-eye bags marking information.

[0093] In an embodiment of the present application, an image to be detected including an eye bag ROI region can be obtained, and then the eye bag ROI region is directly detected by a preset convolutional neural network model to obtain an undereye bag detection classification result and undereye bag position detection information. When it is determined that there are undereye bags (i.e., the undereye bag detection classification result is yes), the image to be detected can be marked by the undereye bag detection classification result and the undereye bag position detection information, thereby obtaining undereye bag marking information for undereye bag detection. Since the undereye bag detection classification result and the undereye bag position detection information here are directly obtained from the undereye bag ROI region, rather than being set according to the size and shape of the eyes, the accuracy of undereye bag detection can be significantly improved.

[0094] In a sixth aspect, an embodiment of the present application provides a method for training a convolutional neural network model, comprising:

[0095] Acquire multiple sample images, wherein the sample images include an eye bag ROI area, and the sample images carry an eye bag labeling classification result and an eye bag position labeling information;

[0096] The eye bag ROI area is detected by a convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0097] Based on the under-eye bag annotation classification result, the under-eye bag position annotation information, the under-eye bag detection classification result and the under-eye bag position detection information of the sample image, the model parameters of the convolutional neural network model are determined.

[0098] In a seventh aspect, the present application also provides an undereye bag detection device, comprising:

[0099] An acquisition module, used for acquiring an image to be detected, wherein the image to be detected includes an eye bag ROI area;

[0100] A detection module is used to detect the eye bag ROI area through a preset convolutional neural network model to obtain an under-eye bag detection classification result and under-eye bag position detection information;

[0101] The marking module is used to mark the under-eye bags in the image to be detected based on the under-eye bags detection classification result and the under-eye bags position detection information to obtain under-eye bags marking information when the under-eye bags detection classification result is yes.

[0102] In an eighth aspect, an embodiment of the present application provides a training device for a convolutional neural network model, comprising:

[0103] An acquisition module is used to acquire a plurality of sample images, wherein the sample images include an eye bag ROI area, and the sample images carry an eye bag labeling classification result and an eye bag position labeling information;

[0104] A detection module is used to detect the eye bag ROI area through a convolutional neural network model to obtain an under-eye bag detection classification result and under-eye bag position detection information;

[0105] A determination module is used to determine the model parameters of the convolutional neural network model based on the under-eye bag annotation classification result, the under-eye bag position annotation information, the under-eye bag detection classification result and the under-eye bag position detection information of the sample image.

[0106] In the ninth aspect, an embodiment of the present application provides a terminal, comprising: a memory and a processor, the memory being used to store a computer program; the processor being used to execute the method described in the first aspect, the second aspect, the fifth aspect or the sixth aspect above when calling the computer program.

[0107] In a tenth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect above is implemented.

[0108] In the eleventh aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal, enables the terminal to execute the method described in the first aspect, the second aspect, the fifth aspect or the sixth aspect.

[0109] It can be understood that the beneficial effects of the second to eleventh aspects mentioned above can be found in the relevant descriptions in the first or fifth aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 A schematic diagram of face recognition provided by the prior art;

[0111] Figure 2 A schematic diagram of the structure of a convolutional neural network model provided in an embodiment of the present application;

[0112] Figure 3 A flow chart of an eye bag detection method provided in an embodiment of the present application;

[0113] Figure 4 A schematic diagram of an eye bag ROI region provided in an embodiment of the present application;

[0114] Figure 5 A schematic diagram of the positions of key points of eye bags provided in an embodiment of the present application;

[0115] Figure 6 A schematic diagram of the positions of key points of the under-eye bags provided in an embodiment of the present application;

[0116] Figure 7 A flowchart of another eye bag detection method provided in an embodiment of the present application;

[0117] Figure 8 A schematic diagram of an eye bag closure area provided in an embodiment of the present application;

[0118] Fig. 9 A schematic diagram of a closed area of ​​the undereye bag provided in an embodiment of the present application;

[0119] Fig.10 A schematic diagram of the structure of an eye bag detection device provided in an embodiment of the present application;

[0120] Fig.11 A schematic diagram of the structure of a convolutional neural network model training device provided in an embodiment of the present application;

[0121] Fig.12 A schematic diagram of the structure of a device for detecting under-eye bags provided in an embodiment of the present application;

[0122] Fig.13 A schematic diagram of the structure of another convolutional neural network model training device provided in an embodiment of the present application;

[0123] Fig.14 A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0124] Fig.15 A software structure diagram of a terminal provided in an embodiment of the present application;

[0125] Fig.16 A schematic diagram of the structure of another terminal provided in an embodiment of the present application;

[0126] Fig.17 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0127] In order to facilitate understanding of the technical solutions in the embodiments of the present application, the application scenarios of the embodiments of the present application are first introduced below.

[0128] Eye bags refer to sagging, bloated, and bag-like skin under the lower eyelids. Eye bags can be divided into two categories based on the cause: primary and secondary. Secondary eye bags are the result of excessive orbital fat accumulation and weakened eyelid support structures. They are usually caused by improper massage, staying up late, aging, and other factors. Their shape and size are not directly related to the shape and size of the eyes. Although it will not affect the user's physical health, it will affect the appearance on the one hand, and on the other hand, it will also reflect some health problems, such as sub-health problems such as fatigue. At the same time, eye bags are also an important facial feature.

[0129] In view of the above reasons, the importance of eye bag recognition in the field of face recognition technology is becoming increasingly important. For example, in the process of shooting images, eye bags can be used to assist in facial detection and positioning; in beauty applications, eye bags can be identified and repaired (adjusting color and filtering), or, eye bags and eye bags can be distinguished to achieve beauty effects; in skin detection applications, eye bags can be identified to determine the user's skin health and give corresponding maintenance suggestions; in age simulation or face simulation applications, eye bags can be identified and the parameters such as the degree of relaxation, color, and size of eye bags can be adjusted to simulate and generate facial images of users at different ages.

[0130] Please refer to Figure 1 , which is a schematic diagram of face recognition provided by the prior art. The prior art is to obtain face key points (such as Figure 1 midpoint 1-68), and then determine the key points of the eye (such as Figure 1 37-42 and 43-48), determine a region of a preset size as the eye bag region (such as Figure 1 The distribution of key points of the eye is closely related to the size and shape of the eye, but the actual shape and size of the eye bag is not directly related to the shape and size of the eye. Therefore, the eye bag area determined by the prior art is different from the actual eye bag area (such as Figure 1 The shaded area surrounded by the solid line in the middle) is very different, for example, Figure 1 In the present invention, the eye bag area determined by the prior art is much larger than the actual eye bag area, and the shape is also very different from the shape of the actual eye bag area, and the accuracy is low.

[0131] To solve this technical problem, the present application provides an eye bag detection method. In an embodiment of the present application, an image to be detected including an eye bag ROI area can be obtained, and then the eye bag ROI area can be directly detected through a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information. When the eye bag detection score is within a preset score range, that is, when it is determined that eye bags exist, the image to be detected can be marked by the eye bag detection score and the eye bag position detection information, thereby obtaining eye bag marking information for detecting eye bags. Since the eye bag detection score and the eye bag position detection information here are directly identified from the eye bag ROI area, rather than being set according to the size and shape of the eyes, the accuracy of detecting eye bags can be significantly improved.

[0132] Before explaining the embodiments of the present application in detail, a convolutional neural network is first introduced.

[0133] CNN is a type of feedforward neural network with deep structure and convolutional computing. It is one of the representative algorithms of deep learning. CNN has the ability of representation learning and feature combination, and can perform shift-invariant classification on input information according to its hierarchical structure. It is widely used in many fields such as computer vision and natural language processing.

[0134] A convolutional neural network may include an input layer, a convolution layer, an excitation layer, a pooling layer, and a fully connected layer.

[0135] The input layer can be used to receive an input image to be detected.

[0136] Before inputting the image to be detected into the input layer, the image to be detected may be preprocessed, including resizing and normalizing pixels to the same value range (eg, [0, 1]).

[0137] The convolution layer can be used to extract features from the data in the input layer. The convolution layer can include filters, and each feature map can include multiple weights. These weights are also model parameters that need to be trained in the convolutional neural network model. When extracting features from the input image through the convolution layer, the image can be convolved with the filter to obtain a feature map, which can describe the characteristics of the image. Through multiple convolution layers connected in sequence, deeper feature maps can be extracted.

[0138] The excitation layer can be used to perform nonlinear mapping on the output results of the convolutional layer.

[0139] The pooling layer can be set after the convolution layer to compress the feature map, simplifying the complexity of network calculations on the one hand and extracting the main features on the other hand. The pooling layer can include an average pooling layer (Average Pooling) or a maximum pooling layer (Max Pooling).

[0140] The fully connected layer can be set at the end of the convolutional neural network to connect the features finally extracted by the previous layers and obtain classification or detection results.

[0141] Please refer to Figure 2 , is a structural diagram of a convolutional neural network model provided by an embodiment of the present application. The convolutional neural network model may include an input layer 100, a feature extraction subnetwork 200, and a feature combination detection subnetwork 300 connected in sequence. The feature extraction subnetwork 200 may include a plurality of convolutional layers 210 connected in sequence and a pooling layer 220, wherein, in order to reduce the amount of parameters and the amount of calculation, thereby reducing the model size and facilitating embedding in mobile terminal applications, the convolutional layers 210 after the second layer of the plurality of convolutional layers 210 may be depth-separable convolutional layers. The pooling layer 220 may be an average pooling layer. The feature combination detection subnetwork 300 may include at least one group of depth-separable convolutional layers and fully connected layers 310, wherein the depth-separable convolutional layers in the feature combination detection subnetwork 300 may be used to further perform feature learning for a specific task; each group of depth-separable convolutional layers and fully connected layers 310 is sequentially connected after the pooling layer 220 to determine the classification result of a task, such as the eye bag detection score or the eye bag position detection information mentioned above.

[0142] Among them, the size of the image input to the input layer 100 (i.e., the image to be detected or the sample image) and the size of each filter in the convolution layer 210 can be determined in advance. For example, the image input to the input layer 100 can be 112*112*3, and the size of the filter can be 3*3.

[0143] It should be noted that when the feature combination detection subnetwork 300 includes only one set of depth-separable convolutional layers and fully connected layers 310, the convolutional neural network model is a single-task learning network; when the feature combination detection subnetwork 300 includes multiple sets of depth-separable convolutional layers and fully connected layers 310, the convolutional neural network model is a multi-task learning network. Among them, the multi-task learning network can share the basic feature extraction subnetwork 200, thereby reducing the amount of calculation; and the single-task learning network can facilitate feature learning for a specific task. In general, the amount of extracted parameters increases, thereby significantly improving the accuracy of the detection results.

[0144] In the embodiment of the present application, the eye bag detection score (eye bag detection classification result) and the eye bag position detection information (eye bag position detection information) can be obtained respectively through two single-task learning networks, or the eye bag detection score (eye bag detection classification result) and the eye bag position detection information (eye bag position detection information) can be obtained simultaneously through a single-target multi-task learning network, thereby realizing the recognition of eye bags or eye bags alone. Alternatively, in other embodiments, the eye bag detection score, the eye bag position detection information, the eye bag detection classification result and the eye bag detection position information can be obtained simultaneously through a multi-target multi-task learning network.

[0145] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0146] Please refer to Figure 3 , is a flow chart of an eye bag detection method provided in an embodiment of the present application. It should be noted that the method is not based on Figure 3 The specific order described below is a limitation, and it should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps therein can be omitted or deleted.

[0147] S301, construct a data set.

[0148] In order to train the convolutional neural network model in the embodiment of the present application, thereby obtaining a preset convolutional neural network model capable of detecting eye bags, a data set can be constructed first.

[0149] When constructing a data set, multiple sample images can be obtained, and each sample image can include an eye bag ROI area, and then the eye bags in the sample images are annotated, including the eye bag annotation score and the eye bag position annotation information. Of course, since the eye bag ROI area is the same as the eye bags, the eye bag ROI area can also be used to detect the eye bags. When the eye bags are also detected, the eye bags in the sample images can also be annotated, including the eye bags annotation classification results and the eye bags position annotation information. Alternatively, the eye bags in the sample images can also be annotated separately, so that only the eye bags are detected. When the annotation is completed, the data set can be divided into two parts, one of which is used as a training set and the other as a test set.

[0150] It should be noted that the sample image here can be an image that only includes the eye bag ROI area, or it can also include other information. For example, the sample image can include the entire face, so the recognition result can be obtained by first detecting the key points of the face. Figure 1As shown, the sample image is then segmented according to the eye key points to obtain the eye bag ROI area.

[0151] The eye bag ROI area is the area of ​​interest for the machine when identifying eye bags or eye bags in an image. When the sample image is segmented to obtain the eye bag ROI area, the eye center point can be determined based on the eye key points, and then the eye center point is used as the reference point to obtain an area of ​​preset size and preset shape from the sample image as the eye bag ROI area.

[0152] It should be noted that the preset size and the preset shape can be obtained by determining in advance. Optionally, the eye bag ROI area can be a rectangular area, wherein the center point of the eye can be located at 1 / 2 of the width and 1 / 4 of the height of the area. Figure 1 Taking the face shown in FIG. 1 as an example, the left eye center point is determined according to the left eye key points 37-42, and the right eye center point is determined according to the right eye key points 43-48. Then, the face image is segmented according to the left eye center point and the right eye center point to obtain a rectangular left eye bag ROI area and a right eye bag ROI area, as shown in FIG. Figure 4 shown.

[0153] Since the details of the image will be affected by the terminal type and the light source, these details will affect the accuracy of eye bag detection. The convolutional network neural model trained using images taken by a certain terminal type as sample images may have difficulty accurately detecting eye bags in images taken by another terminal type; the convolutional network neural model trained using images taken under a certain light source as sample images may have difficulty accurately detecting eye bags in images taken under another light source. Therefore, in order to improve the robustness of the convolutional network neural model and ensure that the eye bag detection method provided in the embodiment of the present application can stably and reliably detect eye bags in different environments, when acquiring multiple sample images, images taken by multiple terminal types under multiple light source environments can be acquired as sample images.

[0154] For example, images taken by mobile phones from multiple manufacturers in environments such as 4000K (color temperature) 100Lux (brightness), 4000K 300Lux, white light, and yellow light may be obtained as sample images.

[0155] The eye bag annotation score, the eye bag position annotation information, the eye bag annotation classification result and the eye bag position annotation information can be obtained through annotation. Among them, the eye bag annotation score can indicate the severity of the marked eye bag; the eye bag position annotation information can indicate the marked eye bag position, and the eye bag position annotation information can include the marked eye bag key points or the eye bag segmentation mask; the eye bag annotation classification result can include yes or no; the eye bag position annotation information can include the marked eye bag position, and the eye bag position annotation information can include the marked eye bag key points or the eye bag segmentation mask. Among them, when the eye bag or the eye bag position is annotated by the eye bag segmentation mask or the eye bag segmentation mask, the convolutional neural network can be an image semantic segmentation network.

[0156] Taking the eye bag annotation score and eye bag position annotation information as an example, when annotating the sample image, in order to accurately distinguish the severity of eye bags and reduce the influence of subjective factors on the difference in eye bag perception, relevant technical personnel (such as ophthalmologists) can first determine and establish eye bag evaluation standards and eye bag score maps (including eye bag score intervals and preset score ranges). The preset score range can be used to indicate the eye bag score when there are eye bags. For example, the eye bag equal division interval can be [65-95], and the preset score range can be less than the score threshold of 85. The smaller the score, the more serious the eye bag. When the eye bag score is less than 85, it can be considered that there are eye bags, and when the eye bag score is greater than or equal to 85, it can be considered that there are no eye bags. Afterwards, the eye bag score map, eye bag line depth, protrusion, area size, relaxation and other dimensions can be used to score, so as to obtain the eye bag annotation score and eye bag position annotation information. Among them, the left and right eyes can use the same annotation order, and in order to reduce the annotation noise, at least three people can annotate the key points, and then take the average value as the final annotation.

[0157] It should be noted that the eye bag score can be the average score of the eye bag score, and in the embodiment of the present application, a positive score is used for the eye bag score, that is, the higher the score, the better the skin health of the user's eye bag area, but it can be understood that in other embodiments, a negative score can also be used, that is, the lower the eye bag score, the better the skin health of the user's eye bag area.

[0158] In addition, if the eye bag position marking information includes eye bag key points or the eye bags position marking information includes eye bags key points, that is, the eye bags or eye bags positions are marked by key points, the positions and number of eye bag key points or eye bags key points markings can be determined in advance.

[0159] For example, see Figure 5 and 6 , respectively, are a schematic diagram of the position of the key points of eye bags and the key points of eye bags provided in the embodiments of the present application. Figure 5In the figure, the eye bag key points include 5 key points, key points 1 and 2 are at the left and right corners of the eye respectively, key points 4 and 5 are in the middle area of ​​the eye bag, and key point 3 is at the bottom of the eye bag. Figure 6 In the figure, the key points of the eye bags include 2 key points, which are located in the middle area of ​​the eye bags.

[0160] S302, constructing a convolutional network neural model based on the data set.

[0161] Multiple sample images can be obtained from a training set, wherein the sample images include an eye bag ROI area, and the sample images carry an eye bag annotation score and an eye bag position annotation information. The eye bag ROI area is detected by a convolutional neural network model to obtain an eye bag detection score and an eye bag position detection information. Then, the eye bag detection score and the eye bag position detection information of each sample image are compared with the eye bag annotation score and the eye bag position annotation information. The model parameters of the convolutional neural network model (such as the weights in each filter mentioned above) are updated according to the comparison results until the convolutional neural network model converges or reaches a preset number of training times, and the model parameters of the convolutional neural network model are determined to obtain.

[0162] Optionally, if the convolutional neural network is also used to detect eye bags, the eye bags ROI area can be detected through the convolutional neural network model to obtain the eye bags detection classification results and the eye bags position detection information, and in a similar manner, based on the eye bags detection scores, eye bags position detection information, eye bags annotation scores, eye bags position annotation information, eye bags annotation classification results, eye bags position annotation information, eye bags detection classification results and eye bags position detection information of each sample image, the model parameters of the convolutional neural network model can be determined.

[0163] Among them, the eye bag detection score can explain the severity of the detected eye bags; the eye bag position detection information can explain the detected eye bag position; the under-eye bag detection classification result can include yes or no; the under-eye bag position detection information can explain the detected under-eye bag position.

[0164] Optionally, at the end of training, in order to test the accuracy of the convolutional neural network model, multiple sample images can be obtained from the test set, and the sample images can be recognized by the convolutional neural network model. Then, based on the accuracy of the recognition results (such as the difference between the eye bag detection score and the eye bag annotation score, and the difference between the eye bag position detection information and the eye bag position annotation information), it is determined whether to continue training the convolutional neural network model.

[0165] S303, performing eye bag detection based on a convolutional neural network model.

[0166] When the training of the convolutional neural network model is completed, the trained convolutional neural network model can be used to perform eye bag detection on the actual image to be detected.

[0167] In the above content, we have combined Figure 3 The eye bag detection method provided in the embodiment of the present application is briefly introduced, that is, it includes three steps: S301 constructing a data set, S302 constructing a convolutional network neural model based on the data set, and S303 detecting eye bags based on the convolutional neural network model. These three steps can be performed by one or more devices. For example, S301 can be executed by a terminal (camera or mobile phone) to collect a data set and transmit it to a server for storage; S302 can be executed by a server to train a convolutional neural network model based on the collected data set; the trained convolutional neural network model is obtained from the server through the terminal, and step 303 is executed to detect eye bags. In the following content, the eye bag detection method based on the convolutional neural network model in S303 will be explained in detail.

[0168] Please refer to Figure 7 , is a flow chart of an eye bag detection method provided in an embodiment of the present application. It should be noted that the method is not based on Figure 7 The specific order described below is a limitation, and it should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps therein can be omitted or deleted.

[0169] S701, obtaining an image to be detected.

[0170] Among them, the image to be detected can be obtained by calling a camera to shoot, or the camera can be called and an image can be obtained from a viewfinder as the image to be detected, such as in an augmented reality (AR) scene, or an image can be obtained from a memory as the image to be detected, or an image can be obtained from other devices as the image to be detected. Of course, in practical applications, the image to be detected can also be obtained by other means, and the embodiment of the present application does not specifically limit the way to obtain the image to be detected.

[0171] S702, performing facial key point detection on the image to be detected to obtain eye key points.

[0172] The purpose of detecting the key points of the face here is to obtain the key points of the eyes so as to facilitate the subsequent determination of the eye bag ROI area. Therefore, when performing human key point detection, all the key points of the face can be detected, or only the key points of the eyes can be detected, such as Figure 1 Or key points 37-42 and key points 43-48 in 4.

[0173] S703: Determine an eye bag ROI region from the image to be detected based on the eye key points.

[0174] Among them, the eye center point can be determined based on the eye key points in the image to be detected, and then the eye center point is used as a reference point to obtain a region of a preset size and a preset shape from the image to be detected as the eye bag ROI region.

[0175] It should be noted that the method of determining the eye center point according to the eye key points of the image to be detected and the method of obtaining the eye bag ROI area from the image to be detected using the eye center point can be the same as the method of determining the eye center point according to the eye key points of the sample image and the method of obtaining the eye bag ROI area from the sample image using the eye center point in the aforementioned S301, and they will not be repeated here.

[0176] It should also be noted that when determining the eye bag ROI area from the image to be detected, the eye bag ROI area can be cut out from the image to be detected to obtain the eye bag ROI area image, and then the eye bag ROI area image is input into the preset convolutional neural network model. Of course, the eye bag ROI area can be marked in the image to be detected, and the marked image to be detected can be input into the preset convolutional neural network model.

[0177] Optionally, since the human eye is symmetrical, the eye bag ROI area is also symmetrical, and the eye bag ROI area includes a left eye bag ROI area and a right eye bag ROI area. Therefore, in order to facilitate the preset convolutional neural network model to perform recognition, the image to be detected can be segmented based on the left eye bag ROI area and the right eye bag ROI area to obtain a left eye bag ROI area image and a right eye bag ROI area image. The right eye bag ROI area image is mirrored in the left and right directions, and the left eye bag ROI area image and the mirrored right eye bag ROI area image are input into the preset convolutional neural network model.

[0178] Of course, if the image to be detected is an image that only includes the eye bags ROI area, it is not necessary to execute the above S702-S703, that is, after S701, at least one of S704 and S708 can be directly executed to detect the eye bags or the eye bags separately, or to detect the eye bags and the eye bags at the same time.

[0179] S704, performing eye bag detection on the eye bag ROI area through a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information.

[0180] Among them, when the preset convolutional neural network model is a single-task learning network, the eye bag ROI area can be detected by two convolutional neural network models (which can be recorded as the first and second convolutional neural network models respectively), so as to obtain the eye bag detection score and the eye bag position detection information; when the preset convolutional neural network model is a single-target multi-task learning network, the eye bag ROI area can be detected by a convolutional neural network model (which can be recorded as the third convolutional neural network model), so as to obtain the eye bag detection score and the eye bag position detection information; when the preset convolutional neural network model is a multi-target multi-task learning network, the eye bag ROI area can be detected by a convolutional neural network model. Detection and eye bags are detected, so as to obtain the eye bags detection classification result and the eye bags position detection information. That is, S704 and S708 can be executed by the same convolutional neural network model, or by multiple convolutional neural network models, and there is no order restriction between the two steps.

[0181] S705, whether the eye bag detection score is within the preset score range, if so, execute S706, otherwise execute S707.

[0182] The eye bag detection score can be compared with a preset score range to determine whether the eye bag detection score is within the preset score range. Taking the preset score range as less than a certain score threshold as an example, if the eye bag detection score is greater than or equal to the score threshold, the skin health of the user's eye bag area is good and there are no eye bags. If the eye bag detection score is less than the score threshold, that is, the eye bag detection score is within the preset score range, it is possible that the skin health of the user's eye bag area is relatively poor and there are eye bags.

[0183] S706, marking the eye bags in the image to be detected based on the eye bag detection score and the eye bag position detection information to obtain eye bag marking information.

[0184] When it is determined that there are bags under the eyes, on the one hand, the bags under the eyes in the image to be detected can be marked based on the eye bag position detection information, so that the position information of the eye bags can be accurately and intuitively displayed to the user. On the other hand, the eye bags in the image to be detected can be marked based on the eye bag detection score, so as to accurately display the severity of the current eye bags to the user, so that the user can take timely care and adjust his or her work and rest habits.

[0185] The eye bag marking information is used to display the score and location of the eye bag detection to the user when eye bags exist.

[0186] As mentioned above, the eye bag position detection information may include eye bag key points or eye bag segmentation masks. Therefore, the way to mark the eye bags in the image to be detected can be divided into two cases:

[0187] In one marking method, when the eye bag position detection information includes eye bag key points, interpolation fitting can be performed based on the eye bag key points to obtain the eye bag closed area, and the eye bags in the image to be detected can be marked based on the eye bag detection score and the eye bag closed area.

[0188] Interpolation is to supplement the continuous function based on discrete data so that the obtained continuous curve can pass through all given discrete data points. Fitting is to connect the given points through a smooth curve. In the field of image processing, interpolation fitting can be equivalent to determining the closed area surrounded by multiple pixels based on given pixels. In this embodiment of the present application, the key points of the eye bags are the key points of the eye bag contour. By performing interpolation fitting processing on these key points of the eye bags, the closed area of ​​the eye bags can be obtained.

[0189] It should be noted that when performing interpolation fitting processing on the key points of eye bags, any difference fitting method can be selected, and the embodiment of the present application does not specifically limit this difference fitting method.

[0190] For example, Figure 5 The eye bag closed area can be obtained by interpolating and fitting the eye bag key points shown in FIG. Figure 8 shown.

[0191] It should be noted that in order to facilitate the understanding of the relationship between the eye bag key points and the eye bag closed area, Figure 8 In the present invention, the key points of eye bags are still retained. In practical applications, when the image to be detected is marked by the closed area of ​​eye bags, the key points of eye bags can be deleted.

[0192] In another marking method, since the eye bag segmentation mask has accurately covered the area where the eye bags are located, when the eye bag position detection information includes the eye bag segmentation mask, the eye bags in the image to be detected can be marked directly based on the eye bag detection score and the eye bag segmentation mask.

[0193] Optionally, as can be seen from the above, before the right eye bag ROI area image is detected, the right eye bag ROI area image is mirrored along the left and right directions. Then, when the detection is completed and the image to be detected is marked, in order to facilitate the user to view the detection results, the left eye bag ROI area image and the mirrored right eye bag ROI area image can be marked, and the marked right eye bag ROI area image can be mirrored again along the left and right directions to restore the right eye bag ROI area image.

[0194] It should be noted that the eye bag detection score and the eye bag closed area can be marked in the same image to be detected at the same time, or the image to be detected can be copied to obtain two identical images to be detected, and then the eye bag detection score and the eye bag closed area can be marked in one image to be detected.

[0195] It should also be noted that the operation of marking the eye bags in the image to be detected based on the eye bag detection score or the eye bag position detection information to obtain the eye bag marking information can be to directly add the eye bag detection score or the eye bag position detection information to the image to be detected, including adding it to the image to be detected in the form of pixels (that is, directly generating text information in the image to be detected) or adding it to the attribute information of the image to be detected in the form of attribute information; it can also be to separately store the eye bag detection score and the eye bag position detection information, and establish an association relationship between the eye bag detection score and the image to be detected, as well as an association relationship between the eye bag position detection information and the image to be detected.

[0196] The attribute information of the image to be detected may be used to describe the attribute information of the image to be detected, such as shooting parameters, etc. For example, it may include the Exchangeable image file format (EXif).

[0197] S707, the interface displays the eye bag detection result.

[0198] The interface may include a display screen of the terminal or an interface in a display. When the terminal performs eye bag detection according to the above method and obtains the detection result, the detection result may be displayed on the display screen of the terminal. Of course, the detection result may also be sent to other displays (such as smart TVs) for display. The embodiment of the present application does not specifically limit the way of displaying the eye bag detection result.

[0199] When the eye bag detection score is within the preset score range threshold (i.e., there are eye bags), the displayed eye bag detection result may include the image to be detected, and the user may view the eye bag detection score and the eye bag position detection information directly from the image to be detected; or, the eye bag detection score and the eye bag position detection information may be viewed from the attribute information of the image to be detected; or, the eye bag detection score and the eye bag position detection information may be obtained from the association relationship between the eye bag detection score and the image to be detected, and from the association relationship between the eye bag position detection information and the image to be detected.

[0200] It should be noted that, when displaying, the image to be detected, the eye bag detection score and the eye bag position detection information can be displayed in the same display area, or can be displayed in different display areas respectively. For example, the marked image to be detected (the image to be detected includes the eye bag position detection information marked in the form of pixels) can be displayed in one display area, and the eye bag detection score can be displayed separately in another display area. The embodiment of the present application does not specifically limit the setting method of this display area.

[0201] It should also be noted that when the eye bag marking information is displayed, personalized care suggestions can also be provided to the user, such as reminding the user to pay attention to rest, use care products to eliminate or lighten the eye bags, etc.

[0202] When the eye bag detection score is not within the preset score range, the eye bag detection result may include the eye bag detection score. Of course, the image to be detected may be marked by the eye bag detection score in a similar manner as when there are eye bags, and displayed in a similar manner as when there are eye bags.

[0203] S708, performing under-eye bag detection on the eye bag ROI area through a preset convolutional neural network model to obtain under-eye bag detection classification results and under-eye bag position detection information.

[0204] It should be noted that the method of detecting eye bags in the eye bag ROI area through a preset convolutional neural network model can be similar to the method of detecting eye bags in the eye bag ROI area through a preset convolutional neural network model, which will not be repeated here.

[0205] For example, two convolutional neural network models (which can be respectively recorded as the fourth and fifth convolutional neural network models) can be used to detect the eye bag ROI area respectively, so as to obtain the under-eye bag detection classification results and the under-eye bag position detection information; when the preset convolutional neural network model is a single-target multi-task learning network, a convolutional neural network model (which can be recorded as the sixth convolutional neural network model) can be used to detect the eye bag ROI area, so as to obtain the under-eye bag detection classification results and the under-eye bag position detection information.

[0206] Among them, the result of the eye bags detection can be represented by 1 or 0, 1 means there are eye bags, and 0 means there are no eye bags.

[0207] S709, whether there are bags under the eyes, if yes, execute S710, otherwise execute S711.

[0208] You can determine whether you have eye bags based on the eye bag test results.

[0209] S710, marking the under-eye bags in the image to be detected based on the under-eye bags detection classification result and the under-eye bags position detection information to obtain under-eye bags marking information.

[0210] Similar to the method of marking the image to be detected based on the eye bag detection score and the eye bag position detection information, the image to be detected can also be marked in the following two ways:

[0211] In one marking method, when the under-eye bag position detection information includes under-eye bag key points, interpolation fitting can be performed based on the under-eye bag key points to obtain the under-eye bag closed area, and the under-eye bag in the image to be detected can be marked based on the under-eye bag detection classification result and the under-eye bag closed area.

[0212] For example, Figure 6 The closed area of ​​the eyebag can be obtained by interpolating and fitting the key points of the eyebag shown in the figure. Fig. 9 Similarly, in order to understand the relationship between the key points of the eye bags and the closed area of ​​the eye bags, Fig. 9 In the 3D image, the key points of the eyebags are still retained. In practical applications, when the image to be detected is marked by the closed area of ​​the eyebags, the key points of the eyebags can be deleted.

[0213] In another marking method, when the under-eye bag position detection information includes an under-eye bag segmentation mask, the under-eye bag in the image to be detected can be marked based on the under-eye bag detection classification result and the under-eye bag segmentation mask.

[0214] It should be noted that the under-eye bag detection and classification results and the under-eye bag closed area can be marked in the same image to be detected at the same time, or the image to be detected can be copied to obtain two identical images to be detected, and then the under-eye bag detection and classification results and the eye bag closed area can be marked in one image to be detected.

[0215] It should also be noted that the operation of marking the under-eye bags in the image to be detected based on the under-eye bags detection classification results or the under-eye bags position detection information can be to directly add the under-eye bags detection classification results or the under-eye bags position detection information to the image to be detected; or it can be to separately store the under-eye bags detection classification results and the under-eye bags position detection information, and establish an association relationship between the under-eye bags detection classification results and the image to be detected, and an association relationship between the under-eye bags position detection information and the image to be detected.

[0216] S711, the interface displays the result of the eye bags detection.

[0217] It should be noted that the way the interface displays the results of the eye bags detection can be the same as the way the interface displays the results of the eye bags detection, which will not be repeated here.

[0218] When the classification result of the under-eye bag detection is Yes, the displayed under-eye bag detection result may include the image to be detected, and the user can directly view the under-eye bag detection classification result and the under-eye bag position detection information from the image to be detected; or, the under-eye bag detection classification result and the under-eye bag position detection information can be viewed from the attribute information of the image to be detected; or, the under-eye bag detection classification result and the under-eye bag position detection information can be obtained from the association relationship between the under-eye bag detection classification result and the image to be detected and the association relationship between the under-eye bag position detection information and the image to be detected.

[0219] When the under-eye bag detection classification result is none, the displayed under-eye bag detection result may include the under-eye bag detection classification result. Of course, the image to be detected may also be marked by the under-eye bag detection classification result in a manner similar to that when the under-eye bag detection classification result is sometimes, and displayed in a manner similar to that when the under-eye bag detection classification result is sometimes.

[0220] In addition, since eye bags and under-eye bags can be detected at the same time, in other embodiments, the eye bag detection score, eye bag closed area, under-eye bag detection classification result and under-eye bag closed area can be marked simultaneously in an image to be detected. The marked image to be detected can be used as both eye bag marking information and under-eye bag marking information. Accordingly, S707 and S711 can be combined into one step.

[0221] In an embodiment of the present application, an image to be detected including an eye bag ROI region can be obtained, and then the eye bag ROI region can be directly detected through a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information. When the eye bag detection score is within a preset score range, that is, when it is determined that eye bags exist, the image to be detected can be marked by the eye bag detection score and the eye bag position detection information, thereby obtaining eye bag marking information for detecting eye bags. Since the eye bag detection score and the eye bag position detection information here are directly identified from the eye bag ROI region, rather than being set according to the size and shape of the eyes, the accuracy of detecting eye bags can be significantly improved.

[0222] In actual tests, using different models, under 4 light source environments, including 4000K 100Lux, 4000K 300Lux, white light and yellow light, the intersection over union (IOU) of the detected closed area of ​​the eye bags and the actual (or marked) closed area of ​​the eye bags reached 72.54%, the IOU of the detected closed area of ​​the eye bags and the actual (or marked) closed area of ​​the eye bags reached 77%, the correlation coefficient of the eye bag score in a single environment reached 88%, and the correlation coefficient in multiple environments reached 87.6%. The standard deviation of the score in a single environment can be as low as 1.44, and the standard deviation in multiple environments is 1.66. Generally, if the standard deviation is less than 2 in a single environment and less than 3 in multiple environments, it is considered to meet the requirements. Therefore, the experimental results of the eye bag detection method provided in the embodiment of the present application far exceed the requirements.

[0223] Among them, IOU is a standard performance metric for object category segmentation problems. In the embodiment of the present application, the larger the value, the closer the detected closed area of ​​​​the under-eye bags (or eye bag detection area) is to the actual closed area of ​​​​the under-eye bags (or eye bag detection area), that is, the higher the detection accuracy; the correlation coefficient of the eye bag score is the correlation coefficient between the eye bag detection score and the eye bag annotation score. The higher the correlation coefficient, the higher the accuracy of eye bag detection.

[0224] Based on the same inventive concept, as an implementation of the above method, the embodiment of the present application provides an eye bag detection device, an undereye bag detection device and a training device for a convolutional neural network model. The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, the present device embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to and implement all the contents of the aforementioned method embodiment.

[0225] Fig.10 A schematic diagram of the structure of an eye bag detection device 1000 provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the eye bag detection device 1000 provided in this embodiment includes:

[0226] An acquisition module 1001 is used to acquire an image to be detected, where the image to be detected includes an eye bag ROI area;

[0227] The detection module 1002 is used to detect the eye bag ROI area through a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information;

[0228] The marking module 1003 is used to mark the eye bags in the image to be detected based on the eye bag detection score and the eye bag position detection information when the eye bag detection score is within a preset score range, so as to obtain eye bag marking information.

[0229] Optionally, a determination module is also included;

[0230] The detection module is also used to detect the key points of the face of the image to be detected to obtain the key points of the eyes;

[0231] The determination module is used to determine the eye bag ROI area from the image to be detected based on the eye key points.

[0232] Optionally, the determination module is further used to:

[0233] Based on the eye key point, determine the eye center point;

[0234] Taking the center point of the eye as a reference point, an area of ​​a preset size and shape is obtained from the image to be detected as the eye bag ROI area.

[0235] Optionally, the eye center point is located in the upper half of the eye bag ROI region, and is located at 1 / 2 of the width and 1 / 4 of the height of the eye bag ROI region.

[0236] Optionally, the detection module is further used to detect the eye bag ROI area through the preset convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0237] The marking module is also used to mark the under-eye bags in the image to be detected based on the under-eye bags detection classification result and the under-eye bags position detection information to obtain under-eye bags marking information when the under-eye bags detection classification result is yes.

[0238] Optionally, the eye bag position detection information includes eye bag key points, and the marking module is further used for:

[0239] Perform interpolation fitting based on the key points of the eye bag to obtain the closed area of ​​the eye bag;

[0240] Based on the eye bag detection score and the eye bag closed area, the eye bags in the image to be detected are marked.

[0241] Optionally, the eye bag position detection information includes an eye bag segmentation mask, and the marking module is further used for:

[0242] Based on the eye bag detection score and the eye bag segmentation mask, the eye bags in the image to be detected are marked.

[0243] Optionally, the under-eye bag position detection information includes under-eye bag key points, and the marking module is further used for:

[0244] Interpolation fitting is performed based on the key points of the under-eye bag to obtain the closed area of ​​the under-eye bag;

[0245] Based on the under-eye bag detection classification result and the under-eye bag closed area, the under-eye bag in the image to be detected is marked.

[0246] Optionally, the under-eye bag position detection information includes an under-eye bag segmentation mask, and the marking module is further used for:

[0247] Based on the under-eye bags detection and classification result and the under-eye bags segmentation mask, the under-eye bags in the image to be detected are marked.

[0248] Optionally, the preset convolutional neural network model includes multiple convolutional layers, wherein, except the first convolutional layer, the other convolutional layers include at least one depth-separable convolutional layer.

[0249] Optionally, the preset convolutional neural network model is trained on multiple sample images, and the sample images carry eye bag annotation scores and eye bag position annotation information.

[0250] Optionally, the sample image also carries the under-eye bag annotation score and under-eye bag position annotation information.

[0251] Optionally, the eye bag ROI area includes a left eye bag ROI area and a right eye bag ROI area, and further includes:

[0252] A segmentation module, used for segmenting the image to be detected based on the left eye bag ROI region and the right eye bag ROI region to obtain a left eye bag ROI region image and a right eye bag ROI region image;

[0253] A mirroring module, used for mirroring the right eye bag ROI area image in the left and right directions;

[0254] The input module is used to input the left eye bag ROI area image and the right eye bag ROI area image after mirroring into the preset convolutional neural network model.

[0255] Optionally, the tagging module is also used to:

[0256] Marking the left eye bag ROI area image and the mirror image of the right eye bag ROI area image;

[0257] The marked right eye bag ROI area image is mirrored again along the left-right direction.

[0258] The eye bag detection device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0259] Fig.11 A structural diagram of a convolutional neural network model training device 1100 provided in an embodiment of the present application, such as Fig.11 As shown, the training device 1100 of the convolutional neural network model provided in this embodiment includes:

[0260] An acquisition module 1101 is used to acquire a plurality of sample images, wherein the sample images include an eye bag ROI region and carry an eye bag annotation score and an eye bag position annotation information;

[0261] The detection module 1102 is used to detect the eye bag ROI area through a convolutional neural network model to obtain an eye bag detection score and eye bag position detection information;

[0262] The determination module 1103 is used to determine the model parameters of the convolutional neural network model based on the eye bag detection score, the eye bag position detection information, the eye bag annotation score and the eye bag position annotation information of the sample image.

[0263] Optionally, the sample image also carries the classification result of the under-eye bags annotation and the under-eye bags position annotation information;

[0264] The detection module is also used to detect the eye bag ROI area through the convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0265] The determination module is also used to determine the model parameters of the convolutional neural network model based on the eye bag detection score, the eye bag position detection information, the eye bag annotation score, the eye bag position annotation information, the eye bag annotation classification result, the eye bag position annotation information, the eye bag detection classification result and the eye bag position detection information of the sample image.

[0266] The training device for the convolutional neural network model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0267] Fig.12 A schematic diagram of the structure of an undereye bag detection device 1200 provided in an embodiment of the present application is shown as follows: Fig.12 As shown, the eyebag detection device 1200 provided in this embodiment includes:

[0268] An acquisition module 1201 is used to acquire an image to be detected, wherein the image to be detected includes an eye bag ROI area;

[0269] The detection module 1202 is used to detect the eye bag ROI area through a preset convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0270] The marking module 1203 is used to mark the under-eye bags in the image to be detected based on the under-eye bags detection classification result and the under-eye bags position detection information to obtain under-eye bags marking information when the under-eye bags detection classification result is yes.

[0271] The training device for the convolutional neural network model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0272] Fig.13 A structural diagram of a training device 1300 for a convolutional neural network model provided in an embodiment of the present application, such as Fig.13 As shown, the training device 1300 of the convolutional neural network model provided in this embodiment includes:

[0273] An acquisition module 1301 is used to acquire a plurality of sample images, wherein the sample images include an eye bag ROI region and carry an eye bag labeling classification result and eye bag position labeling information;

[0274] The detection module 1302 is used to detect the eye bag ROI area through a convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information;

[0275] The determination module 1303 is used to determine the model parameters of the convolutional neural network model based on the eyebag labeling classification result, the eyebag position labeling information, the eyebag detection classification result and the eyebag position detection information of the sample image.

[0276] The training device for the convolutional neural network model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0277] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0278] Please refer to Fig.14 , which is a structural diagram of a terminal 100 provided in the present application. The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0279] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0280] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0281] The controller may be the nerve center and command center of the terminal 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0282] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This reduces repeated accesses and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0283] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0284] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal 100.

[0285] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0286] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0287] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0288] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal 100.

[0289] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0290] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal 100, and can also be used to transmit data between the terminal 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminals, such as AR devices, etc.

[0291] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0292] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal 100. While the charging management module 140 is charging the battery 142, it may also power the terminal through the power management module 141.

[0293] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0294] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0295] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0296] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0297] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0298] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0299] In some embodiments, the antenna 1 of the terminal 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0300] The terminal 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0301] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0302] The terminal 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0303] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0304] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0305] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0306] Video codecs are used to compress or decompress digital videos. Terminal 100 may support one or more video codecs. Thus, terminal 100 may play or record videos in various coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0307] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0308] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0309] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0310] The terminal 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0311] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0312] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0313] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0314] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to microphone 170C to input the sound signal into microphone 170C. Terminal 100 may be provided with at least one microphone 170C. In other embodiments, terminal 100 may be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, terminal 100 may also be provided with three, four or more microphones 170C to realize collection of sound signals, noise reduction, identification of sound source, realization of directional recording function, etc.

[0315] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0316] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0317] The gyroscope sensor 180B can be used to determine the motion posture of the terminal 100. In some embodiments, the angular velocity of the terminal 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0318] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0319] The magnetic sensor 180D includes a Hall sensor. The terminal 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal 100 is a flip phone, the terminal 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0320] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal 100 in various directions (generally three axes). When the terminal 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the terminal posture and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0321] The distance sensor 180F is used to measure the distance. The terminal 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0322] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal 100 emits infrared light outward through the light emitting diode. The terminal 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal 100. When insufficient reflected light is detected, the terminal 100 can determine that there is no object near the terminal 100. The terminal 100 can use the proximity light sensor 180G to detect that the user holds the terminal 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0323] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal 100 is in a pocket to prevent accidental touches.

[0324] The fingerprint sensor 180H is used to collect fingerprints. The terminal 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0325] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal 100 heats the battery 142 to reduce abnormal shutdown of the terminal 100 caused by low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal 100 performs a boost on the output voltage of the battery 142 to reduce abnormal shutdown caused by low temperature.

[0326] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be arranged on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be arranged on the surface of the terminal 100, which is different from the position of the display screen 194.

[0327] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0328] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal 100 may receive key input and generate key signal input related to user settings and function control of the terminal 100.

[0329] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0330] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0331] The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the terminal 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal 100 interacts with the network through the SIM card to realize functions such as calls and data communications. In some embodiments, the terminal 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal 100 and cannot be separated from the terminal 100.

[0332] The software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal 100.

[0333] Fig.15 It is a software structure block diagram of the terminal 100 according to an embodiment of the present application.

[0334] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0335] The application layer can include a series of application packages.

[0336] like Fig.15 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0337] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0338] like Fig.15 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0339] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0340] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0341] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0342] The phone manager is used to provide communication functions of the terminal 100, such as management of call status (including connection, disconnection, etc.).

[0343] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0344] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal vibrates, the indicator light flashes, etc.

[0345] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0346] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0347] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0348] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0349] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0350] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0351] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0352] A 2D graphics engine is a drawing engine for 2D drawings.

[0353] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0354] The following is an illustrative description of the software and hardware workflow of the terminal 100 in conjunction with the capture and photo shooting scene.

[0355] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0356] Based on the same inventive concept, an embodiment of the present application also provides a terminal. Fig.16 A schematic diagram of the structure of the terminal 1600 provided in the embodiment of the present application is shown in FIG. Fig.16 As shown, the terminal 1600 provided in this embodiment includes: a memory 1610 and a processor 1620, the memory 1610 is used to store computer programs; the processor 1620 is used to execute the method described in the above method embodiment when calling the computer program.

[0357] The terminal provided in this embodiment can execute the above method embodiment to perform eye bag detection and / or eye bags detection. Its implementation principle and technical effect are similar and will not be repeated here.

[0358] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the above method embodiment is implemented to perform eye bag detection and / or under-eye bag detection, and a convolutional neural network model can also be trained.

[0359] An embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal, the terminal implements the method described in the above method embodiment to perform eye bag detection and / or under-eye bag detection, and can also perform training of a convolutional neural network model.

[0360] Based on the same inventive concept, an embodiment of the present application also provides a server. Fig.17 A schematic diagram of the structure of the server 1700 provided in the embodiment of the present application is shown in FIG. Fig.17 As shown, the server 1700 provided in this embodiment includes: a memory 1710 and a processor 1720, the memory 1710 is used to store computer programs; the processor 1720 is used to execute the method described in the above method embodiment when calling the computer program.

[0361] The server provided in this embodiment can execute the above method embodiment to train the convolutional neural network model. Its implementation principle and technical effect are similar and will not be repeated here.

[0362] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the above method embodiment is implemented to train a convolutional neural network model.

[0363] An embodiment of the present application also provides a computer program product. When the computer program product runs on a server, the server implements the method described in the above method embodiment to train a convolutional neural network model.

[0364] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0365] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0366] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0367] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0368] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0369] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0370] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0371] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0372] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0373] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting eye bags, characterized in that: include: Acquire the image to be detected; Determining the eye center point based on the eye key points in the image to be detected; Taking the center point of the eye as a reference point, obtaining an area of ​​a preset size and a preset shape from the image to be detected as an eye bag ROI area of ​​interest, wherein the eye bag ROI area is a rectangular area, and the center point of the eye is located in the upper half of the eye bag ROI area, and is located at 1 / 2 of the width and 1 / 4 of the height of the eye bag ROI area; The eye bag ROI area is detected by a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information, wherein the eye bag position detection information includes 5 eye bag key points, wherein key point 1 and key point 2 are located at the corners of the eyes, key point 4 and key point 5 are located in the middle area of ​​the eye bag, and key point 3 is located at the bottom of the eye bag; When the eye bag detection score is within a preset score range, the eye bags in the image to be detected are marked based on the eye bag detection score and the eye bag position detection information to obtain eye bag marking information.

2. The method according to claim 1, characterized in that Before determining the eye center point based on the eye key points in the image to be detected, the method further includes: Performing facial key point detection on the image to be detected to obtain the eye key points.

3. The method according to claim 1, characterized in that Also includes: The eye bag ROI area is detected by using the preset convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information; When the under-eye bags detection and classification result is yes, based on the under-eye bags detection and classification result and the under-eye bags position detection information, the under-eye bags in the image to be detected are marked to obtain under-eye bags marking information.

4. The method according to claim 3, characterized in that The eyebag mark information includes two eyebag key points located in the middle area of ​​the eyebag.

5. The method according to claim 1, characterized in that The step of marking the eye bags in the image to be detected based on the eye bag detection score and the eye bag position detection information includes: Perform interpolation fitting based on the eye bag key points to obtain the eye bag closed area; The eye bags in the image to be detected are marked based on the eye bag detection score and the eye bag closed area.

6. The method according to any one of claims 1 to 5, characterized in that: The preset convolutional neural network model is trained on a plurality of sample images, and the sample images carry eye bag annotation scores and eye bag position annotation information.

7. The method according to claim 6, characterized in that The preset convolutional neural network model includes multiple convolutional layers, wherein, except the first convolutional layer, the other convolutional layers include at least one depth-separable convolutional layer.

8. The method according to any one of claims 1 to 5, characterized in that: The preset convolutional neural network model includes multiple convolutional layers, wherein, except the first convolutional layer, the other convolutional layers include at least one depth-separable convolutional layer.

9. A method for training a convolutional neural network model, characterized in that: include: Acquire multiple sample images, wherein the sample images include an eye bag ROI region, and the sample images carry an eye bag annotation score and eye bag position annotation information, wherein the eye bag ROI region determines an eye center point based on eye key points in the sample images, and takes the eye center point as a reference point, and acquires a region of a preset size and a preset shape from the sample images, wherein the eye bag ROI region is a rectangular region, and the eye center point is located in the upper half of the eye bag ROI region, and is located at 1 / 2 of the width and 1 / 4 of the height of the eye bag ROI region; The eye bag ROI area is detected by a convolutional neural network model to obtain an eye bag detection score and eye bag position detection information, wherein the eye bag position detection information includes 5 eye bag key points, wherein key point 1 and key point 2 are located at the corners of the eyes, key point 4 and key point 5 are located in the middle area of ​​the eye bag, and key point 3 is located at the bottom of the eye bag; Based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score and the eye bag position annotation information, the model parameters of the convolutional neural network model are determined.

10. The method according to claim 9, characterized in that The sample image also carries the classification result of eye bags and the position information of eye bags, and also includes: The eye bag ROI area is detected by the convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information; The step of determining the model parameters of the convolutional neural network model based on the eye bag detection score, the eye bag position detection information, the eye bag annotation score, and the eye bag position annotation information of the sample image includes: Based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score, the eye bag position annotation information, the eye bag annotation classification result, the eye bag position annotation information, the eye bag detection classification result and the eye bag position detection information, the model parameters of the convolutional neural network model are determined.

11. An eye bag detection device, characterized in that: include: An acquisition module, used for acquiring an image to be detected, wherein the image to be detected includes an eye bag ROI area; A detection module, used to detect the eye bag ROI area through a preset convolutional neural network model to obtain an eye bag detection score and eye bag position detection information, wherein the eye bag position detection information includes 5 eye bag key points, wherein key point 1 and key point 2 are located at the corners of the eyes, key point 4 and key point 5 are located in the middle area of ​​the eye bag, and key point 3 is located at the bottom of the eye bag; a marking module, configured to mark the eye bags in the image to be detected based on the eye bag detection score and the eye bag position detection information when the eye bag detection score is within a preset score range, so as to obtain eye bag marking information; The device is also used to determine the center point of the eye based on the eye key points in the image to be detected; using the center point of the eye as a reference point, obtain an area of ​​a preset size and a preset shape from the image to be detected as an eye bag ROI area, wherein the eye bag ROI area is a rectangular area, and the eye center point is located in the upper half of the eye bag ROI area, and is located at 1 / 2 of the width and 1 / 4 of the height of the eye bag ROI area.

12. The device according to claim 11, characterized in that The detection module is also used to detect the eye bag ROI area through the preset convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information; The marking module is also used to mark the under-eye bags in the image to be detected based on the under-eye bags detection classification result and the under-eye bags position detection information to obtain under-eye bags marking information when the under-eye bags detection classification result is yes.

13. A training device for a convolutional neural network model, characterized in that: include: An acquisition module is used to acquire multiple sample images, wherein the sample images include an eye bag ROI area, and the sample images carry an eye bag annotation score and eye bag position annotation information, wherein the eye bag ROI area is determined based on the eye key points in the sample image, and the eye center point is used as a reference point to acquire an area of ​​a preset size and a preset shape from the sample image, wherein the eye bag ROI area is a rectangular area, and the eye center point is located in the upper half of the eye bag ROI area, and is located at 1 / 2 of the width and 1 / 4 of the height of the eye bag ROI area; A detection module, used to detect the eye bag ROI area through a convolutional neural network model to obtain an eye bag detection score and eye bag position detection information, wherein the eye bag position detection information includes 5 eye bag key points, wherein key point 1 and key point 2 are located at the corners of the eyes, key point 4 and key point 5 are located in the middle area of ​​the eye bag, and key point 3 is located at the bottom of the eye bag; A determination module is used to determine the model parameters of the convolutional neural network model based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score and the eye bag position annotation information.

14. The device according to claim 13, characterized in that The sample image also carries the classification result of under-eye bags annotation and under-eye bags position annotation information; The detection module is also used to detect the eye bag ROI area through the convolutional neural network model to obtain the under-eye bag detection classification result and under-eye bag position detection information; The determination module is also used to determine the model parameters of the convolutional neural network model based on the eye bag detection score of the sample image, the eye bag position detection information, the eye bag annotation score, the eye bag position annotation information, the eye bag annotation classification result, the eye bag position annotation information, the eye bag detection classification result and the eye bag position detection information.

15. A terminal, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the method according to any one of claims 1 to 8 or the method according to claim 9 or 10 when calling the computer program.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 or the method according to claim 9 or 10 is implemented.

Images