Systems and methods for counting, locating and visualizing acne

Abstract

Systems, methods, and techniques provide for the localization, counting, and visualization of acne. Images are processed using a model trained to identify objects. The model may be a deep learning (e.g., convolutional neural) network configured for object classification with a detection focus on small objects. The images may be frontal or profile facial images processed edge-to-edge. The model identifies and localizes different types of acne. Instances are counted and visualized, such as by annotating the original image. An exemplary annotation is an overlay identifying the type and location of each instance. Counting by acne type aids scoring. Products and / or services are recommended depending on the acne identification (e.g., type, location, count, and / or score).

Description

cross reference

[0001] This application claims national benefit to the United States and otherwise claims priority to U.S. Provisional Application No. 63 / 086,694, filed October 2, 2020, the entire contents of which are incorporated herein by reference where permitted. This application also claims priority to French Patent Application No. FR2013002, filed December 10, 2020, the entire contents of which are incorporated herein by reference where permitted. [Technical Field]

[0002] This application relates to the fields of computer image processing, convolutional neural networks and dermatology, and more particularly to systems and methods for counting, locating and visualizing acne, and e-commerce systems and methods utilizing the same. [Background technology]

[0003] Skin conditions such as acne often affect the face, but also other areas of the body. In acne, sebum or dead skin cells clog pores, causing various types of blemishes. It is common among teenagers, but can also be found in other age groups. The acne localization task aims to detect the presence of any acne in a given face image.

[0004] Acne localization is useful in downstream applications related to dermatology and image visualization. Summary of the Invention

[0005] According to embodiments, systems, methods, and techniques are provided for locating, counting, and visualizing acne. Images are processed with a model to identify (classify) objects. In one embodiment, the model is a convolutional neural network (CNN) configured for object classification with a small object detection focus. In one embodiment, the images are frontal or profile mode face images processed end-to-end by the CNN without cropping. In one embodiment, the model identifies and locates different types of acne. Instances of acne are counted (e.g., by type) and visualized, for example, by annotating the source image. An exemplary annotation is an overlay identifying the type and location of each instance. Counting by acne type aids scoring. In one embodiment, products and / or services can be recommended depending on the acne identification (e.g., type, location, count, and / or score). In one embodiment, purchasing is facilitated.

[0006] In one embodiment, a method is provided that includes analyzing an original image to determine a location of each instance of acne and visualizing the instances of acne on the original image for display, where the original image is analyzed using a model configured to detect at least one type of acne in the image and to perform focus detection of small objects in the image.

[0007] In one embodiment, the model is a deep learning neural network model configured for object classification and localization.

[0008] In one embodiment, the model is configured to operate end-to-end to process images at the pixel level and detect the location of pimples directly without cropping the original image.

[0009] In one embodiment, the model generates respective anchor boxes that provide location information for each detected instance of acne.

[0010] In one embodiment, the aspect ratio of the anchor box used to define one of the anchor boxes is calculated using k-means clustering from instances of acne identified in a dataset of images.

[0011] In one embodiment, the model is patch-based and the method includes providing patches of skin from the source image to the model for processing to detect instances of acne.

[0012] In one embodiment, the model is patch-based, and the method includes providing patches of skin from the original image to the model for processing to detect instances of acne, the patches being determined according to a skin mask.

[0013] In one embodiment, visualizing the acne instances is to indicate the respective location for each of the instances on the original image.

[0014] In one embodiment, visualizing the acne is showing each type of acne for each instance on the original image.

[0015] In one embodiment, the at least one type of acne comprises one or more of retentional acne, inflammatory acne, and pigmentary acne.

[0016] In one embodiment, the method includes determining a count of instances of acne.

[0017] In one embodiment, the method includes obtaining recommendations for specific products and / or services for treating instances of acne.

[0018] In one embodiment, the method includes communicating with an electronic commerce system to purchase products and / or services.

[0019] In one embodiment, the source images include face images in frontal or profile modes.

[0020] In one embodiment, the method includes obtaining a source image.

[0021] In one embodiment, a method is provided that includes analyzing an original image to determine a location of each instance of acne and generating and providing an acne score responsive to counting the instances, wherein the original image is analyzed using a model configured to perform focus detection of small objects in the image and to detect at least one type of acne in the image.

[0022] In one embodiment, the acne score is a function of one or more of acne location, count and type.

[0023] In one embodiment, the method includes generating a recommendation for one or more of specific products and services for treating the instance of acne.

[0024] In one embodiment, recommendations are generated according to factors selected from the group of acne type, count per type, score per type, acne location, purchaser location, delivery location, regulatory requirements, count display, gender, co-recommendations, likelihood of user following usage guidelines, and likelihood of user following usage guidelines.

[0025] In one embodiment, the method includes visualizing instances of acne on the source image.

[0026] In one embodiment, a computing device is provided that includes circuitry configured to perform a method according to any of the embodiments.

[0027] In one embodiment, a computing system is provided that includes circuitry configured to receive an original image and provide an interface for returning an annotated original image that visualizes instances of acne as determined by a model configured to process the original image, the model configured to perform focus detection of small objects in the image and detect at least one type of acne in the image.

[0028] In one embodiment, the computing system is configured to provide a recommendation component configured to recommend products and / or services for specifically treating at least some of the instances of acne, and an e-commerce component for facilitating the purchase of the products and / or services.

[0029] In one embodiment, the model comprises one of a patch-based model configured to receive patches of skin from a source image for processing to detect instances of acne, the patches determined according to a skin mask, and a single detection layer model configured to output multiple predictions including a three dimensional tensor encoding bounding box, objectness, and class predictions, where the multiple predictions are filtered to filter redundant detections on the same instance of acne. [Brief explanation of the drawings]

[0030] [Figure 1] A, B, and C are face images annotated with visualizations to show instances of acne by type and location, according to one embodiment. [Figure 2] FIG. 1 is a diagram of an example of a facial landmark-based skin mask used in processing a facial image, according to one embodiment. [Figure 3] FIG. 1 is a diagram of a face image with acne visualization and mask visualization, according to one embodiment. [Figure 4] 1 is an e-commerce network diagram illustrating a client computing device configured to detect instances of acne in an original facial image, obtain recommendations for product and / or service recommendations, and purchase the same, according to one embodiment. [Figure 5] FIG. 1 is a block diagram of a computing device according to one embodiment. [Figure 6] 1 is a flowchart of an operation according to one embodiment.

[0031]

[0032]

[0033]

[0034]

[0035] DETAILED DESCRIPTION OF THE INVENTION

[0036] In one embodiment, a model configured to process a source image to detect instances of acne is shown and described. The model is configured to determine the location of each of the acne instances and perform an acne localization task. In one embodiment, the model is configured to return bounding boxes or coordinates. In one embodiment, the model is also configured to classify the instances into a particular type of acne. In one embodiment, the source image is a full face image (an image with only the face).

[0037] In one embodiment, the model is a deep learning model. In one embodiment, the model focuses on small objects.

[0038] In one embodiment, the model is end-to-end and operates at the pixel level, which means that the model processes the entire image (e.g., of a face) and performs acne detection directly on the image without cropping.

[0039] The following description describes various embodiments of a model for processing source images, with a first group of embodiments ("first model embodiments" of the "first model type") based on a modified YOLO object detection deep learning network, and a second group of embodiments ("second model embodiments" of the "second model type") based on a patch-based deep learning network. (See JRedmon and AFiarhadi, YOLOv3: An Incremental Improvement, arXiv:1804.02767, Apr 8, 2018 URL: arxiv.org / abs / 1804.02767, which is incorporated by reference where permitted and hereinafter referred to as "YOLOv3").

[0040] It will be understood that various features described in connection with embodiments of one of the model types are applicable to embodiments of the second model type. By way of example, and not limitation, acne localization, acne type detection, acne counting (e.g., by type), and acne visualization features are applicable to both model types. In embodiments, the first and second model embodiments are useful for processing full facial images without cropping, whether frontal or profile images. In embodiments, the first and second model embodiments are useful for processing full facial images as guided by a skin mask generated from the facial image. Some of these features relate to the operation of embodiments of the model itself, such as acne detection features (e.g., localization and classification), while other features, such as counting and visualization, are operations related to processing output from the model. These and other features are common among the model embodiments. First Model Embodiment

[0041] 1A, 1B, and 1C show examples of acne visualizations after analysis of respective original images using a model according to one embodiment, in which three types of acne are detected and visualized. The types include retentional acne, inflammatory acne, and pigmentary acne. The types of acne that can be detected depend on the training data used in training the model. This approach is extensible to allow the model itself to learn and detect other types of acne if applicable training data is provided. It is understood that in one embodiment, fewer types of acne may be trained for detection.

[0042] As shown, according to one embodiment, the model also assists in acne detection in three input facial views (left profile / side, right profile / side and frontal modes). The model is viewpoint invariant in this embodiment and can process images from either viewpoint / mode.

[0043] 1A, 1B, and 1C are black-and-white (BW) simulated images 100, 102, and 104 showing annotated original images visualizing an instance of acne, according to one embodiment. Images 100, 102, and 104 are converted and edited from the original color images for patent application compliance purposes, although in practice, color images are used in embodiments to present a more accurate image of the subject. Image 100 is a left (side) view, image 102 is a right (side) view, and image 104 is a frontal view.

[0044] In this embodiment, the annotations for the three views intentionally focus on specific parts of the face in each view. For example, in the front view 104, pimples are prominent in the center of the image and around the mouth, so they are annotated. Similarly, the side views 100 and 102 are annotated with pimples around the "temples" that are not visible in the front view 104. In this embodiment, training data is defined (labeled) in such a way for each view, and the model is trained accordingly to identify data regarding such locations.

[0045] In this embodiment, instances are labeled with a type 106, a detection confidence measure (e.g., a numerical value) 108, and a bounding box 110. Acne may be closely adjacent to a human face, and therefore, labels may overlap when visualized at certain scales. In one embodiment (not shown in the black-and-white example), acne types are distinguished by using different colors for the labels or by other means. Other label types (e.g., color-coded bounding boxes without text) may also be used. In this example, acne types are indicated as "infr" (inflammatory), "rete" (retentive), and "pigum" (pigmented). The detection confidence measure is a scale between 0 and 1, where, for example, a value of 0.58 represents a 58% confidence that the detected instance is acne. In this embodiment, further filtering is used to select detected acne instances for visualization. For example, a 50% threshold is used to visualize instances above the threshold. The threshold may be variable. <<Structure of the first model type>> In one embodiment, the main architecture of the model is based on the YOLOv3 object detection algorithm. Bounding box prediction (position)

[0046] Briefly, YOLOv3 computes four coordinates (t x ,t y ,t h ,t w ) and predicts the bounding box by using the dimension clusters as anchor boxes. The box is calculated from the top left corner of the image (c x ,c y ) and the width and height of the box are offset by the center of gravity of the cluster (σ(t x ),σ(t y)). The box center coordinates are predicted relative to the position of the filtered image using a sigmoid function. Sum squared error loss is used during training.

[0047] YOLOv3 uses logistic regression to predict an objectness score for each bounding box. The score is set to 1 if a bounding box prior overlaps the ground truth object more than any other bounding box prior. If a bounding box prior is not the best but overlaps the ground truth object by more than some threshold, a threshold of 0.5 is used to ignore the prediction. YOLOv3 assigns one bounding box prior per ground truth object. If a bounding box prior is not assigned to a ground truth object, no loss is incurred for coordinate or class predictions, only objectness. Class prediction (e.g., object prediction by type or acne)

[0048] Multi-label classification is used to predict the class of each box via independent logistic classifiers, which are trained using binary cross-entropy loss.

[0049] YOLOv3 describes box predictions across three scales. From a base feature extractor, several convolutional layers are added. The final such prediction is a 3-D tensor encoding bounding box, an object likelihood, and a class prediction. Thus, for three boxes at each scale, the tensor is NxNx[3*(4+1+80)], with four bounding box offsets, one object likelihood prediction, and 80 class predictions.

[0050] The feature maps from the two previous layers are then upsampled by 2x, and the previous feature maps are merged with the upsampled features using concatenation. Several more convolutional layers are added to process the combined feature maps. A similar tensor is predicted at twice the size. A similar approach is used again to predict a third tensor at a third scale, combining precomputed and fine-grained features from previous processing. K-means clustering is used to determine the bounding box front, with nine clusters and three scales arbitrarily chosen, with the clusters evenly divided across the scales.

[0051] The backbone net for feature extraction described in YOLOv3 is a 53-layer convolutional network called DARKNET-53. Modification

[0052] According to one embodiment herein, the model disclosed in YOLOv3 is modified to focus on smaller objects as follows. - Miniaturizing the backbone network Keep only one YOLO detection layer and recalculate the anchor box size (aspect ratio), Fine-tune the model to get the best performance on the training dataset for the current task of acne localization.

[0053] If we keep only one YOLO detection layer, in this embodiment, we make predictions at one scale level. Therefore, in this embodiment, the YOLO detection layer is the layer that handles bounding box prediction. It receives features (e.g., the output of the backbone net layer), object likelihood, class prediction, and bbox location. The layer itself does not impose a limit on the number of predictions. In one embodiment, during inference, the layer outputs a large number of predicted boxes, which are filtered using non-maximum suppression (NMS) with a manually set confidence threshold.

[0054] In this embodiment, filtering using NMS aims, at least in part, to filter redundant detections of instances of the same pimple: if the model predicts more than one box on the same pimple, NMS filters out instances with lower confidence and keeps the box with the highest confidence on a particular instance.

[0055] In one embodiment, the model is trained to predict three classes of objects for each of the three acne types. There are no additional or background classes because YOLO does not perform background prediction (e.g., YOLOv3 does not draw a bounding box for the background to indicate it is background; YOLOv3 only predicts the target class). The number and types of classes can be easily adjusted to suit the format of the data. For example, if more classes of objects (e.g., acne or other objects) are identified in the data, the model can be adjusted and trained to predict more classes.

[0056] For tuning, in one embodiment, various operations were performed, including: 1) converting the image to HSV color space and adding random color jittering on the saturation and value channels; and adding a random affine transformation on the image; 2) using a multi-step learning rate scheduler; and 3) using an evolutionary algorithm to evolve hyper-parameters from the above augmentations by training multiple rounds and iteratively selecting the best setup.

[0057] In one embodiment, as described in YOLOv3, the aspect ratio of the anchor boxes is computed using k-means clustering of instances of acne identified in a (training) dataset of images (e.g., annotated by human reviewers (e.g., experts)). That is, the anchor boxes are clustered with respect to aspect ratio using k-means clustering operations. The anchor boxes are used to guide the bounding box prediction in YOLOv3. In other approaches, the operation finds patches of an image that potentially contain the object of interest, and then a bounding box around the object is predicted. However, this is a computationally very demanding task.

[0058] Understanding in the art has evolved to determine that there is no need to predict raw boxes because datasets should have some inherent characteristics. As an example, a car dataset should always have a rectangular bbox, and a face dataset should have an approximately square bbox. Using these principles, an image dataset is evaluated to cluster the bbox sizes and find some common sizes they share. These sizes are used as anchor boxes. The model then only needs to predict offsets from the anchor boxes, saving a lot of computation. In one embodiment, the algorithm used follows the same approach. The algorithm first obtains all bounding boxes from the training annotation, extracts and formats the boxes as (width, height) data pairs, and then uses k-means (a clustering technique) to divide the data into k individual clusters and find the cluster centroids. The number of clusters (k) is set manually and must match the YOLO layer, which captures a fixed number of anchor boxes. In one embodiment, k=5 for the model.

[0059] Table 1 shows the structure of the model per layer, according to one embodiment, following the modifications thus described. [Table 1]

[0060] The model itself does not directly predict the count of instances of an object (e.g., acne) by class or within an ensemble. Therefore, the counts by class or ensemble are determined by counting each instance of acne per class from the model's predictions (outputs) after filtering the predictions, e.g., using non-maximal suppression.

[0061] A comparison of the counts determined from the predictions with the counts generated from the ground truth is used to evaluate the model. Common object detection metrics rely on defining what counts as a hit and a miss. Often, a threshold of overlap ratio (Intersection over Union) is used, but this threshold can be set arbitrarily high or low, significantly affecting the accuracy figure.

[0062] In one embodiment, a metric is generated and used as an assessment of how well the model is performing.

[0063] In one embodiment, a count is generated from using the model's inference time for use in downstream tasks. Instead of (or in addition to) predicting the exact location of a pimple, the model's API can be configured to determine the count. Downstream applications can also use the count or score based on the predicted count for tasks such as estimating the severity of a skin condition, product and / or service recommendations, etc. Dataset

[0064] We collected a facial image dataset containing 348 facial images of 116 different subjects (3 views per subject). A group of dermatologists labeled the images and identified any visible acne belonging to one of three types: inflammatory acne, persistent acne, and pigmented acne. The identified acne was then saved in the form of center point coordinates to define ground truth for training, testing, etc.

[0065] In one embodiment, a group of three dermatologists (experts) reviewed and annotated the image set. Each expert annotated all images and identified all three types of acne. In one embodiment, the annotated data (three versions of annotations) were merged according to the following steps:

[0066] Merge all three versions. The total number of boxes after this step is simply the sum of the three original annotations.

[0067] Based on the same logic as NMS, it filters redundant boxes. That is, if the IoU (Intersection over Union) of two boxes is larger than a certain threshold, the larger box is used as the ground truth (GT). If multiple boxes overlap and all have large IoU, it uses the largest box as the GT. "evaluation"

[0068] The performance of acne localization was measured based on common object detection metrics, including precision ("P"), recall ("R"), "mAP" and "F1" scores. Note that mAP and F1 scores are direct formulations based on precision and recall. The performance evaluation for the localization task is shown in Table 2. [Table 2]

[0069] The following abbreviations for the types of acne assessed in the current model are used: pigm = pigmented, infl = inflammatory, rete = persistent, and P = precision, R = recall.

[0070] A similar model was also evaluated for the counting task, and the results are shown in Table 3. [Table 3]

[0071] The calculated error is taken as the absolute error (|#Predicted - #GroundTrue|). The metrics evaluated are the error by class (i.e., what proportion of test cases show an absolute error less than 5%), and the average error across all classes.

[0072] While the above embodiments are described in terms of processing the entire facial image, in one embodiment a skin mask is defined to indicate the portions of the image that depict skin, for example using facial landmarks, and the above first model embodiment can process the skin portions of the facial image as dictated by the skin mask.

[0073] In one embodiment, an operation may reject images that do not contain a face and not process the image for locating and counting pimples. In one embodiment, for example, an operation may process an image to locate facial landmarks and reject or further process the image accordingly. Second Model Embodiment—Patch-Based Acne Localization

[0074] A possible drawback of the YOLO-based approach of the first embodiment described above is that noisy annotations may result in poor results. Therefore, a second model embodiment using a patch-based approach to localize acne spots is provided herein. In one embodiment, the patch-based approach is applied to the entire face image. In one embodiment, the patch-based approach is applied using a skin mask to process one or more portions of the face image.

[0075] For example, in a mask-oriented embodiment, during inference, a source image of a face (e.g., a selfie) is preprocessed by a facial landmark detector to define a facial skin mask. An example landmark-based skin mask 200 is shown in FIG. 2, with masked portions in white indicating areas for acne treatment and unmasked portions in black indicating areas not to be treated. In this example, mask 200 shows an upper surface component 202 for the forehead, a T-line component 204 above the bridge of the nose, and lower surface components 206 for the checks, jawline, and chin, where portions of the eye region (e.g., 208), lower nose region 210, and lip region 212 are omitted. The background around the face is also omitted from the mask.

[0076] During inference, in a mask-related embodiment, resolution patches are created in the generated skin mask by scanning from top-left to bottom-right with a stride of one-third (1 / 3) patch width. Patch width is normalized by defining it as the face width divided by 15. Each patch is passed through a trained convolutional neural network (i.e., a second model embodiment) described below. In one embodiment, the model outputs a list of the following classes: inflammatory, persistent, and pigmented (three acne types or classes), and a probability of healthy skin. In one embodiment, non-maximum suppression (NMS) is applied to select the best box from the returned candidate detected acne.

[0077] In non-mask related embodiments, similar operations are performed using patches to process the entire face image, including non-skin areas such as background, hair, eyes, and lips. For example, during inference, in non-mask related embodiments, resolution patches are created across the entire image, scanning from top-left to bottom-right using a stride of one-third (⅓) patch width, as described below. Each patch is passed through a trained convolutional neural network (i.e., a second model embodiment) described below. In one embodiment, the model outputs a list of the following classes: inflammatory, persistent, and pigmented (three acne types or classes), and a probability of healthy skin. In one embodiment, non-maximum suppression (NMS) is applied to select the best box from the returned detected acne candidates. The healthy skin class provides the classifier with options other than one of the three acne classes.

[0078] In one embodiment, the selected boxes are useful for performing pimple counting and pimple visualization using the original image. In one embodiment, a filter is applied to select the best pimple box instances detected using a threshold confidence measure. For example, detected instances above a threshold (or only instances above a threshold) are counted and / or visualized.

[0079] According to one embodiment, FIG. 3 is a diagram of a screenshot 300 showing a source face image 302 that has been annotated to visualize acne (e.g., 304, 306, and 308). Screenshot 300 also shows the source image annotated to indicate the applicable skin mask 310 (here outlined with a dashed line) determined for the face. In FIG. 3, acne is visualized using a circle with the center point of the box for each detected instance of acne to be visualized. In one embodiment, the circles are actually represented with different colors or different grayscale values ​​to distinguish three acne classes. In this illustration, the circles are distinguished using a non-dashed (solid) line style (e.g., 304) or one of two dashed line styles (e.g., 306 and 308). In one embodiment, the skin mask is not visualized (not shown). A skin mask may not be visualized if it was not used or if it was used but is not desired to be shown. Structure and training of the second model type

[0080] In one embodiment, the patch-based model comprises a residual network backbone (e.g., ResNet-50 with 50 neural network layers) and three fully connected (FC) layers with rectified linear activation functions (e.g., Leaky Rectified Linear Unit (LeakyReLU)) sandwiched between adjacent fully connected layers. The final layers are denoted as FC1 → LeakyRelu1 → FC2 → LeakyRelu2 → FC3. (See He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015-12-10) "Deep Residual Learning for Image Recognition" arXiv:1512.03385, incorporated by reference where permitted.)

[0081] To create the dataset for the patch-based model, according to one embodiment, 2450 healthy patches and 3577 acne patches (patches containing instances of any of the three acne classes described hereinabove) were sampled from the full face images of the dataset described above with reference to the first model embodiment, respectively. In the training phase, the network is trained with a standard cross entropy loss function using data augmentation including random affine transformations, random horizontal flips, and channel shuffles.

[0082] In one embodiment, the frontal image was utilized for masking and annotation.

[0083] According to an embodiment in which a mask is used, to process the original image to detect acne instances by type, the original image is processed using landmark detection to define an applicable mask and define skin-bearing facial regions for analysis. According to the applicable mask, multiple overlapping patches (depending on patch size and stride) are extracted from the skin image and processed by a model. The model generates a list of the following classes: inflammatory, persistent, pigmented (three acne types or classes) and a probability of healthy skin. The mask instructs the operation to ignore eyes, lips, hair, and background in the original image. Multiple detected instances of the same acne are filtered (e.g., using NMS) to select the better / best instance. A second filter is applied to select detected instances with at least a minimum confidence level.

[0084] Similar to that described with reference to the first model embodiment, in an embodiment using the second model type, detected instances of acne by type are counted. In one embodiment, a score is calculated. In one embodiment, the detected acne is visualized relative to the original image. In one embodiment, a mask is visualized relative to the original image indicating which areas of the skin have been treated. Downstream Applications

[0085] The following description relates to either a first model embodiment or a second model embodiment. Thus, according to an embodiment, an analysis operation utilizes a model in which different types of acne (object classes) are distinguishable by the model and configured to output locations of acne instances detected in an original image (e.g., of skin, such as on a face in front or side mode). The analysis or other determination operation can, for example, count the instances to determine a total number and / or counts of each type.

[0086] The location, type, and / or count define acne data that is useful in downstream applications. In one embodiment, the acne instances are visualized on the original image or display using a visualization operation. In one embodiment, the original image is annotated (e.g., using an overlay or other technique) to indicate location. In one example, the location is the location of each of the instances in the original image. In one embodiment, the visualization operation shows the respective type of acne for each of the instances on the original image. In one embodiment, the visualization operation provides an indication of the count.

[0087] Acne data (e.g., count and type) are useful for predicting skin condition scores at surgery. For example, more persistent acne and less inflammatory acne indicates milder severity, and vice versa.

[0088] In one embodiment, based on the acne data and / or information derived therefrom (e.g., a score), an action recommends products and / or services to target the acne. An action (e.g., an e-commerce action), according to one embodiment, facilitates a purchase.

[0089] In one embodiment, the visualization uses the location to guide the application of the product, for example, to indicate the application location. In one example, the source image is used in the visualization to provide a user-specific tutorial via a graphical user interface (GUI).

[0090] Various paradigms and computing systems are envisioned for practical application of the model and its output. In one example, a computer network includes at least one client computing device (e.g., for a dermatologist or consumer) and a web-based e-commerce computing system. In this embodiment, the client computing device determines instances of acne in source images and obtains product and / or service recommendations that can be purchased.

[0091] The model for analyzing the original image and providing acne data as output is configured for run-time execution on the user device or on a server or other remote device. The application is configured to obtain the original image and generate acne data using the model. The acne data is used to provide one or more of an acne score, a visualization of the acne on the original image, and a recommendation for a product or service to specifically treat the acne.

[0092] 4 is a block diagram of an exemplary computer network 400 in which a personal computing device 402 operated by a user 404 communicates with remotely located server computing devices, namely, server 408 and server 410, via a communications network 406. In one embodiment, user 404 is a consumer. Also shown is a second user 412 and a second computing device 414 configured to communicate via communications network 406. In one embodiment, second user 410 is a dermatologist. In one embodiment, server 408 is configured to provide an instance of a model (418) for processing images to generate acne data and product and / or service recommendations for treating acne. In one embodiment, server 408 generates acne scores and recommendations from the received acne data (e.g., without processing the images themselves). In one embodiment, server 408 generates acne data, including location data for each detected instance of acne, such as for use in visualizing the acne. In one embodiment, server 408 generates the visualization, such as for providing to another device for display. In one embodiment, the server 408 generates acne data including counts by type (eg, without visualization) for purposes such as generating a score.

[0093] In one embodiment, the server 408 uses rules or models or other methods to generate recommendations (e.g., products, services (including practitioners / service providers, or both products and services)). In one embodiment, each product is associated with a respective treatment plan / usage guideline. In one embodiment, recommendations are generated depending on factors such as acne type, and a score or score is generated for each type. In one embodiment, recommendations are generated based on factors such as acne location (e.g., on the face), purchaser / delivery location, regulatory requirements (e.g., whether the product is available over the counter by prescription only), score indications (e.g., product should not be used during or before pregnancy, specific health conditions, etc.). The recommendations are generated depending on one or more factors including (e.g., not used if a condition is present, not used with other specific products, etc.), gender, co-recommendations (e.g., availability of co-recommendations), likelihood of the user following usage guidelines / treatment regimens, and other factors. In one embodiment, intended users of the product or service complete a user profile (e.g., a questionnaire or the like via their respective user device completed on behalf of other users (e.g., dermatologists / treatment practitioners / retail store sales staff, etc.)) for submission to server 408 to provide data for deriving recommendations.

[0094] In one embodiment, server 410 provides an e-commerce interface for purchasing products and / or services, such as those recommended by server 408 .

[0095] In one embodiment, computing device 402 is for personal use by user 404 and is not publicly available. However, services from the server are publicly available, where the public includes registered users and / or customers, etc. Publicly available computing device 416 (e.g., located in a brick-and-mortar store) is also coupled to network 406.

[0096] Computing device 402 is configured to perform pimple localization, i.e., assess pimple location, determine counts, etc., as described herein. In this embodiment, model (CNN) 418 is stored and utilized on computing device 402. In this embodiment, a second instance of model 418 is stored on server 408 and provided for use by other computing devices, such as via cloud services, web services, etc., for analysis of images received from computing devices (e.g., 416, etc.).

[0097] Computing device 402 is configured to communicate with server 408, for example, to provide acne data (which may include score data) and receive product / service recommendations based on the acne data and / or other information about the user, such as age, gender, etc. Computing device 402 (or alternatively server 408) is configured to communicate with server 410 to obtain e-commerce services for purchasing recommended products and / or services.

[0098] Computing device 402 is shown as a handheld mobile device (e.g., a smartphone or tablet). However, the physical device may be another computing device, such as a laptop, desktop, workstation, etc. Pimple locating and counting, etc., as described herein, may be implemented on other computing device types. Computing devices 402, 414, and 416 may be configured with, for example, one or more native or browser-based applications.

[0099] In this embodiment, the computing device 402 comprises, for example, a user device for acquiring one or more images of skin, particularly of a face, and processing the one or more images to generate respective acne data, etc. Such activity is referred to as performing a skin diagnosis. The skin diagnosis is performed in conjunction with a skin treatment plan, in which images are periodically acquired and analyzed to determine a skin score, such as for acne, as described. The scores are stored (locally, remotely, or both) and compared between sessions, for example, to indicate trends, improvement, etc. The skin scores and / or skin images are accessible to the user 404 of the computing device 402, made available (e.g., via the server 408) by another user of the computer system 400 (e.g., a second user 412), such as a dermatologist, or otherwise (electronically) communicated via the communications network 406. The second computing device 414 may also perform the skin diagnosis, as described. It can receive images from a remote source (e.g., computing device 402 or server 408, etc.) and / or capture images via an optical sensor (e.g., a camera) coupled to it or in any other manner. Model 418 can be stored and used on second computing device 414 or on server 408, as described.

[0100] An application can be provided to perform a skin diagnosis, recommend one or more products, and monitor changes in the skin over a period of time after one or more applications of the products (which can define treatment sessions in a treatment plan). The computer application can provide a workflow, such as a series of interactive graphical user interfaces (GUIs) and / or other user interfaces, which are typically interactive and receive user input, to perform any of the following activities:

[0101] - Skin diagnosis such as acne

[0102] Recommending products such as treatment plans

[0103] Purchase of products or other items

[0104] Product application reminders, instructions, and / or records (e.g., logging) for each treatment session

[0105] subsequent (e.g., one or more follow-up) skin examinations, and

[0106] Displaying results (e.g. comparison results)

[0107] For example, monitoring the progress of a skin treatment plan according to a treatment plan schedule. Any of these activities generate data that is stored remotely, for example, for review by user 412, for review by another individual, for aggregating data from other users (e.g., to collectively measure the effectiveness of the treatment plan), or otherwise.

[0108] Comparison results (e.g., before and after results), whether during and / or upon completion of the treatment plan, etc., are presented via computing device 402. As noted above, aspects of the skin diagnosis may be performed on computing device 402 or on a remotely coupled device (e.g., a server in the cloud or another configuration).

[0109] 5 is a block diagram of a computing device 402 in accordance with one or more aspects of the present disclosure. The computing device 402 includes one or more processors 502, one or more input devices 504, a gesture-based I / O device 506, one or more communication units 508, and one or more output devices 510. The computing device 402 also includes one or more memory devices 512 that store one or more modules and / or data. In one embodiment, the modules include a model 418, an application 516 having components for a graphical user interface (GUI 518) and / or workflow for monitoring a therapy (e.g., a therapy monitor 520), an image acquisition 522 (e.g., interface), and a therapy / product selector 530 (e.g., interface). The data may include one or more images for processing (e.g., image 524), diagnostic data 526 (e.g., acne data, respective scores, ethnicity, gender or other user data), logging data related to a particular treatment, treatment data 528 including a treatment plan with schedule of reminders, etc.

[0110] The application 516 provides the functionality to acquire one or more images, such as a video, and process the images to determine a skin diagnosis for the deep neural network provided by the model 418.

[0111] The storage device 512 may store an operating system 532 and other modules (not shown) including a communications module, a graphics processing module (e.g., for the GPU of the processor 502), a map module, a contacts module, a calendar module, and additional modules such as a photo / gallery module, a photo (image / media) editor, a media player and / or streaming module, social media applications, a browser module, etc. The storage device is referred to herein as a storage unit.

[0112] A communication channel 538 communicatively, physically, and / or operatively couples each of the components 502, 504, 506, 508, 510, 512 and any modules (e.g., 418 and 516) for inter-component communication. In some examples, the communication channel 338 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0113] One or more processors 502 can implement functions and / or execute instructions within computing device 402. For example, processor 502 is configured to receive instructions and / or data from storage device 512 to perform the functions of the modules shown in FIG. 5 (e.g., operating system, applications, etc.), among other things, and computing device 402 stores data / information in storage device 512. Some of the functions are further described herein below. It should be understood that operations may not fall exactly within modules 418 and 516 of FIG. 5, such that one module supports the functionality of another module.

[0114] Computer program code for carrying out operations may be written in one or more programming languages, for example, any combination of object-oriented programming languages ​​such as Java®, Smalltalk®, C++, or conventional procedural programming languages ​​such as C or a similar programming language.

[0115] The computing device 402 generates output for display on a screen of the gesture-based I / O device 506, or in some examples, for display by a projector, monitor, or other display device. It will be appreciated that the gesture-based I / O device 506 may be configured using a variety of technologies (e.g., resistive touchscreen, surface acoustic wave touchscreen, capacitive touchscreen, projective capacitance touchscreen, pressure sensitive screen, acoustic pulse recognition touchscreen, or other presence-sensitive screen technology for input capabilities, and liquid crystal display (LCD), light emitting diode (LED) display, organic light emitting diode (OLED) display, dot matrix display, e-ink, or similar monochrome or color display for output capabilities).

[0116] In the examples described herein, the gesture-based I / O device 506 includes a touchscreen device capable of receiving tactile interactions or gestures as input from a user interacting with the touchscreen. Such gestures include tap gestures, drag gestures or swipe gestures, flick gestures, and pause gestures in which the user touches or points at one or more locations on the gesture-based I / O device 506 (e.g., when the user touches the same location on the screen for at least a threshold period of time). The gesture-based I / O device 506 can also include gestures other than taps. The gesture-based I / O device 506 can output or display information to the user, such as a graphical user interface. The gesture-based I / O device 506 can expose various applications, functions, and capabilities of the computing device 402, including, for example, applications 516 for capturing images, viewing images, processing images, and displaying new images, messaging applications, telephony, contact and calendar applications, web browsing applications, gaming applications, e-book applications, and financial, payment, and other applications or functions, among others.

[0117] While this disclosure primarily illustrates and describes gesture-based I / O device 506 in the form of a display screen device (e.g., a touchscreen) with I / O capabilities, other examples of gesture-based I / O devices that can detect movement and do not themselves include a screen are available. In such cases, computing device 402 is connected to a display device that includes a display screen or that presents GUIs of new images and applications 516. Computing device 402 can receive gesture-based input from a trackpad / touchpad, one or more cameras, or another presence- or gesture-sensitive input device, where "presence" refers to aspects of a user's presence, including, for example, movement of all or part of the user.

[0118] One or more communication units 508 communicate with external devices (e.g., server 408, server 410, second computing device 412) for the purposes described and / or other purposes (e.g., printing), such as via communication network 404, by transmitting and / or receiving network signals over one or more networks. The communication units may include various antennas and / or network interface cards, chips (e.g., Global Positioning Satellite (GPS)), etc. for wireless and / or wired communication.

[0119] The input device(s) 504 and output device(s) 510 may include any of one or more buttons, switches, pointing devices, cameras, keyboards, microphones, one or more sensors (e.g., biometric, etc.), speakers, bells, one or more lights, haptic (vibration) devices, etc. One or more of the same may be coupled via a Universal Serial Bus (USB) or other communication channel (e.g., 338). The camera (input device 504) may be front-facing (i.e., on the same side) to allow a user to capture an image using the camera while looking at the gesture-based I / O device 506 and taking a "selfie."

[0120] The one or more storage devices 512 may take different forms and / or configurations, e.g., as short-term or long-term memory. The storage device 512 may be configured for short-term storage of information as a volatile memory that does not retain its stored contents when power is removed. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc. The storage device 512 may also include one or more computer-readable storage media, in some examples, to retain information when power is removed, e.g., for storing larger amounts of information than volatile memory and / or for storing such information for longer periods. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM).

[0121] In one embodiment, as described above, other user-oriented computing devices (e.g., 414 and 416) are configured similarly as computing device 402. In one embodiment, for a dermatologist user, application 516 is configured differently, for example, with functionality for working with multiple patients. In one embodiment, for devices located in a store, application 516 is configured differently, for example, with functionality for working with multiple customers, store branding, etc.

[0122] In another example, network model 418 is located remotely and a computing device (e.g., any of 402, 414, and 416) is enabled to process diagnostic data (e.g., acne data) and communicate images for return via an appropriately configured application 516. In such an example, application 516 is configured to perform these activities.

[0123] Although not shown, a computing device may be configured as a training environment for training neural network model 514 using, for example, a network such as that shown in FIG. 5 along with appropriate training and / or test data.

[0124] Model 418 is adapted to a light architecture for computing devices that are mobile devices (e.g., smartphones or tablets) that have fewer processing resources than "larger" devices such as laptops, desktops, workstations, servers, or other comparable production computing devices.

[0125] Although not shown in detail, a server device is a computing device similar in basic configuration (processor, storage device, communication device, input and output devices) to computing device 402, but is considered a server-side device. That is, a server device is often not configured with consumer-grade hardware, and has fewer input / output devices, user applications, a server-oriented OS, etc.

[0126] FIG. 6 is a flowchart of operations 600 according to one embodiment.

[0127] At 602, an original image (face image) is received. For example, the original image is a selfie image captured by a camera of a computing device.

[0128] Operation 604 is optional and is shown in dashed style. A facial skin mask is determined for use in guiding image processing.

[0129] In operation 606, the face image is processed with the model to generate localized instances of acne by type (e.g., one of three classes). In one embodiment, the model is configured to detect at least one type of acne in the image and to perform focus detection of small objects in the image. In one embodiment, the model is an embodiment of the first model. In an embodiment, the model is an embodiment of the second model. In one embodiment, processing is guided by the skin mask generated by operation 604.

[0130] At 608, the detected instances are filtered as described herein above to select better localizations and instances with higher confidence levels, and thus the final detected acne is determined by type.

[0131] At 610, a count is determined by type. At 610, a score is generated, for example, representing a scale of severity. At 612, the final detected acne is visualized in relation to the original image. For example, the image is annotated to show the final detected acne by type. In one embodiment, the image has annotations overlaid on it. At 612, the score is presented.

[0132] Recommendations are obtained at 614. Recommendations are presented at 614. In one embodiment, the scores are provided to an e-commerce service to generate recommendations for products and / or services.

[0133] In one embodiment, operations 600 are performed by a single computing device (eg, device 402) and, where applicable, in communication with other devices (eg, 408 and / or 410).

[0134] In one embodiment, operations are performed on two or more computing devices (e.g., 402, 408, and 410). For example, operation 606 includes communicating the original image to computing device 408, which provides the model as a service. That computing device 408 may also filter (step 608). That device 408 may also count the pimples (step 610). That device 408 may also visualize the pimples (step 612) and return the original image overlaid with the visualization and count and / or score (e.g., for presentation by device 402). A third computing device 410 may provide recommendations, for example, in response to a query including the score. In operation 600, steps therein may be divided into two or more steps or combined into fewer steps.

[0135] A practical implementation may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, and other ways of combining the features described herein. Several embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. Additionally, other steps may be provided or steps may be eliminated from the described processes, and other components may be added to or removed from the described systems. Accordingly, other aspects are within the scope of the following claims.

[0136] Throughout the description and claims of this specification, the terms "comprises," "includes," and variations thereof mean "including, but not limited to," and are not intended to (and do not) exclude other elements, integers, or steps. Throughout this specification, the singular encompasses the plural unless the context requires otherwise. In particular, where the indefinite article is used, it should be understood that the specification contemplates the plural as well as the singular, unless the context requires otherwise.

[0137] It should be understood that any feature, integer, property, or group described in connection with a particular aspect, embodiment, or example of the invention is applicable to any other aspect, embodiment, or example, except where incompatible therewith. All features disclosed herein (including any accompanying claims, abstract, and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any example or embodiment described above. The invention extends to any novel one or any novel combination of features disclosed herein (including any accompanying claims, abstract, and drawings), or any novel one or any novel combination of steps of any method or process disclosed. <Other> <Means> The method of technical idea 1 analyzes an original image to determine the location of each instance of acne, visualizes the instances of acne on the original image for display, and analyzes the original image with a model configured to detect at least one type of acne in the image and perform focus detection of small objects in the image. The method of Technical Idea 2 is characterized in that, in the method described in Technical Idea 1, the model is a deep learning neural network model configured for object classification and localization. The method of Technical Idea 3 is the method of Technical Idea 1 or 2, wherein the model is configured to operate end-to-end to process the image at the pixel level and directly detect the location of acne without cropping the original image. A method of technical idea 4 is a method according to any one of technical ideas 1 to 3, wherein the model generates respective anchor boxes that provide position information for each of the detected instances of acne. The method of Technical Idea 5 is used to define one of the anchor boxes in the method of Technical Idea 3, the aspect ratio of which is calculated using k-means clustering from instances of acne identified in a dataset of images. A method of Technical Idea 6 is a method according to any one of Technical Ideas 1 to 5, wherein the model comprises a single detection layer for outputting multiple predictions including 3D tensor-coded bounding boxes, object likelihoods, and class predictions, and the multiple predictions are filtered to filter redundant detections of the same acne instance. The method of Technical Idea 7 is a method according to any one of Technical Ideas 1 to 3, wherein the model is patch-based, and the method provides patches of skin from the source image to the model for processing to detect instances of acne. The method of technical idea 8 is a method according to technical idea 1 or 2, wherein the model is patch-based, and the method includes providing patches of skin from the original image to the model for processing to detect instances of acne, the patches being determined according to a skin mask. A method of technical idea 9 is a method according to any one of technical ideas 1 to 8, in which visualizing the instances of the acne comprises indicating a position for each of the instances on the original image. A method of Technical Idea 10 is a method according to any one of Technical Ideas 1 to 9, in which visualizing the acne is to show a respective type of acne for each of the instances on the original image. The method of Technical Idea 11 is the method of any one of Technical Ideas 1 to 10, wherein the at least one type of acne includes one or more of persistent acne, inflammatory acne, and pigmented acne. A method according to Technical Idea 12 is a method according to any one of Technical Ideas 1 to 11, in which the count of the instances of acne is determined. The method of technical idea 13 is a method according to any one of technical ideas 1 to 12, in which recommendations are obtained for one or more specific products and services for treating the instance of acne. The method of concept 14 is the method of concept 13, further comprising communicating with an electronic commerce system to effect the purchase of the product or service. The method of technical idea 15 is the method of technical idea 13 or 14, wherein the recommendations are generated according to a group of selected factors, including acne type, counts per type, scores per type, acne location, purchaser location, delivery location, regulatory requirements, count display, gender, common recommendations, likelihood of user following usage guidelines, and likelihood of user following usage guidelines. The method of technical idea 16 analyzes an original image to determine the location of each acne instance, generates and provides an acne score according to counting the instances, analyzes the original image using a model configured to perform focus detection of small objects in the image and detect at least one type of acne in the image, the model including: a single detection layer model configured to output multiple predictions including 3D tensor-coded bounding boxes, object likelihoods, and class predictions, the multiple predictions being filtered to filter redundant detections of the same acne instance; and a patch-based model configured to receive patches of skin from the original image for processing to detect acne instances, the patches being determined according to a skin mask. The method of Technical Idea 17 is the method of Technical Idea 16, wherein the acne score is based on one or more of acne location, count, and type. The method of technical idea 18 is the method of technical idea 16 or 17, which includes generating recommendations for one or more specific products and services for treating the instance of acne, wherein the recommendations are generated according to a group of selected factors including acne type, counts per type, scores per type, acne location, purchaser location, delivery location, regulatory requirements, count display, gender, common recommendations, likelihood of user following usage guidelines, and likelihood of user following usage guidelines. The method of Technical Idea 19 is the method of any one of Technical Ideas 16 to 18, in which the instance of a pimple on the original image is visualized. The computing device of concept 20 includes circuitry configured to perform the method of any of concepts 1-19. The computing system of technical idea 21 includes an interface for receiving an original image and returning an annotated original image visualizing instances of acne determined by a model configured to process the original image, the model performing focus detection of small objects in the image and detecting at least one type of acne in the image, and further includes a circuit configured to provide, for processing to detect instances of acne, either a patch-based model configured to receive patches of skin from the original image, the patches being determined according to a skin mask, or a single detection layer model configured to output multiple predictions including 3D tensor-coded bounding boxes, object likelihoods, and class predictions, the multiple predictions being filtered to filter redundant detections of the same instance of acne. The computing system of technical idea 22 is the computing system of technical idea 21, configured to provide a recommendation component configured to recommend products and / or services for specifically treating at least some of the instances of acne, and an e-commerce component for facilitating the purchase of the products and / or services.

Claims

1. 1. A method comprising: Analyze the source image to determine the location of each acne instance; visualizing the acne instance on the source image for display; analyzing the original image with a model configured to detect at least one type of acne in the original image and perform focus detection of small objects in the original image, the model comprising a deep learning neural network model configured for object classification and localization; The model further generates a respective anchor box providing location information for each of the detected instances of acne; In addition, the model a single detection layer YOLO-based model for outputting multiple predictions including a 3D tensor-coded bounding box, an object likelihood, and a class prediction, the single detection layer YOLO-based model having its performance fine-tuned for localizing the acne; wherein a plurality of the predictions are filtered to filter out redundant detections of the same acne instance.

2. the model is configured to operate end-to-end to process the source image at a pixel level and directly detect the location of the pimple without cropping the source image; 10. The method of claim 1, wherein the aspect ratio of one of the anchor boxes used to define the anchor box is calculated using k-means clustering from the acne instances identified in a dataset of images.

3. The fine adjustment is In YOLO, the backbone network for feature extraction, which is composed of multiple detection layers, is downsized to form a single detection layer; After recalculating the size of the anchor box, 3. The method of claim 1 or 2, characterized in that the method learns to obtain the best performance on a training dataset for the task of acne localization.

4. The fine adjustment is data augmentation by converting the original image to HSV color space, adding random color jittering on the saturation and lightness channels, and adding a random affine transformation on the original image; Using a multi-step learning rate scheduler; 4. The method of claim 1, wherein the method is performed using an evolutionary algorithm to evolve hyperparameters from the data augmentation by training multiple rounds and iteratively selecting the best setup.

5. The model is a patch-based model that obtains multiple overlapping skin patches from the source image for processing to detect the acne instances, the patches being determined according to a skin mask, a size and a stride of the patches; The patch-based model is resolution patches are created in the generated skin mask by scanning from top left to bottom right with a stride of one-third patch width; The patch width is normalized by defining it as the width of the face divided by 15; Each of the patches is passed through a convolutional neural network; outputting a list of the three acne types or classes, inflammatory, persistent, and pigmented, and the probability of healthy skin; 5. The method of claim 1, wherein non-maximal suppression is used to select the best anchor box from among the detected acne candidates returned.

6. visualizing the acne includes showing a respective type of acne for each of the instances on the source image; 4. The method of claim 1, wherein the at least one type of acne comprises one or more of persistent acne, inflammatory acne, and pigmented acne.

7. 4. The method according to claim 1, further comprising determining a count of the acne instances.

8. 4. The method of any of claims 1 to 3, further comprising obtaining recommendations for one or more specific products and services for treating the instance of acne.

9. 10. The method of claim 8, further comprising communicating with an electronic commerce system to effect the purchase of the product or service.

10. 10. The method of claim 8, wherein the recommendations are generated according to a group of selected factors including the acne type, counts per type, scores per type, location of the acne, location of the purchaser, delivery location, regulatory requirements, count display, gender, common recommendations, likelihood of user following usage guidelines, and likelihood of user following usage guidelines.

11. 1. A method comprising: Analyze the source image to determine the location of each acne instance; generating and providing a score for the acne according to the count of instances; analyzing the original image with a model configured to detect at least one type of acne in the original image and to perform focus detection of small objects in the original image; The model is a single detection layer YOLO-based model for outputting multiple predictions including a 3D tensor-coded bounding box, an object likelihood, and a class prediction, the single detection layer YOLO-based model having its performance fine-tuned for localizing the acne; wherein a plurality of the predictions are filtered to filter out redundant detections of the same acne instance.

12. 12. The method of claim 11, wherein the acne score is a function of one or more of the location, count, and type of the acne.

13. generating recommendations for one or more specific products and services for treating said instance of acne; 13. The method of claim 11 or 12, wherein the recommendations are generated according to a group of selected factors, including the type of acne, counts per type, scores per type, location of the acne, location of the purchaser, delivery location, regulatory requirements, count display, gender, common recommendations, likelihood of user following usage guidelines, and likelihood of user following usage guidelines.

14. A computing device comprising circuitry configured to perform the method of claim 1 or 11.

15. 1. A computing system comprising: an interface for receiving an original image and returning the original image with annotations visualizing instances of acne determined by a model configured to process the original image; the model detects at least one type of acne in the original image and performs focus detection of small objects in the original image; Furthermore, the model a single detection layer YOLO-based model for outputting multiple predictions including a 3D tensor-coded bounding box, an object likelihood, and a class prediction, the single detection layer YOLO-based model having its performance fine-tuned for localizing the acne; A computing system comprising circuitry configured to provide a plurality of the predictions to be filtered to filter redundant detections of the same acne instance.

16. a recommendation component configured to recommend products and / or services for specifically treating at least some of the instances of acne; and an electronic commerce component for facilitating the purchase of the products and / or services.

Images