Oral care based digital imaging system and method for determining perceptual appeal of facial image portions

Abstract

The invention provides a computer-implemented imaging system and method for determining the perceived attractiveness of a facial image portion of at least one person depicted in a digital image based on oral care. The method has the following steps: a) obtaining a digital image comprising at least one oral feature of at least one person, wherein the digital image comprises a facial image portion of the at least one person having both positive attributes as defined by pixel data of the digital image and negative attributes as defined thereby; b) analyzing the facial image portion; c) generating, based on the analyzed facial image portion in the obtained digital image, an attractiveness score indicative of the perceived attractiveness of the facial image portion; d) further generating, based on the attractiveness score, an image description identifying at least one area in the facial image portion; and e) presenting the image description to a user.

Description

TECHNICAL FIELD

[0001] The present invention relates to a digital imaging system and method based on oral care for processing information associated with image data, such as digital images, videos defined by a sequence of digital images (also referred to as frames). In particular, the present invention relates to a system and method for determining the perceived attractiveness of a facial image portion of at least one person depicted in a digital image. BACKGROUND

[0002] Body practices aimed at continuously improving the body and its impact on social relations are increasingly popular, as seen, attractiveness plays a central role in the focus of humans on their self-image. Visual cues strongly influence the attractiveness of a person when perceived by oneself or by a group of people. One visual cue is the face of a person, and concepts used to describe this face influence whether a person is perceived as attractive relative to another person or a group of people. However, attractiveness is highly subjective. Consumers also seek to improve their attractiveness by using various consumer products, including but not limited to oral care products, dental care products, or skin care products, etc. However, it is difficult to improve attractiveness without prior knowledge of the elements that influence attractiveness.

[0003] US patent 6,571,003 B1 describes an apparatus and method for displaying information associated with a plurality of skin defects, in particular for determining and displaying the location of one or more analysis regions and defect regions associated with a digital image of human skin, and for determining the severity of these defects, and displaying improvement and / or deterioration of the defect regions. US patent 8,073,212 describes a method and product for analyzing gingival tissue. US patent 10,405,754 describes standardized oral health assessment and scoring using digital images.

[0004] Therefore, there is a need for a method of determining the perceived attractiveness of a person's appearance, which method can then improve the person's ability to take measures or make informed decisions to improve the perceived attractiveness of their appearance. SUMMARY

[0005] The present invention relates to a computer-implemented method for determining the perceived attractiveness of a facial image portion of at least one person depicted in a digital image, the method comprising the steps of:

[0006] a) obtaining a digital image of at least one person, wherein the digital image comprises a facial image portion of the at least one person, wherein the facial image portion has both positive attributes and negative attributes;

[0007] b) analyzing the facial image portion;

[0008] c) generating, based on the analyzed facial image portion, an attractiveness score indicative of a perceived attractiveness of the facial image portion;

[0009] d) further generating, based on the attractiveness score, an image description identifying at least one region in the facial image portion; and

[0010] e) presenting the image description to a user. BRIEF DESCRIPTION OF DRAWINGS

[0011] It is to be understood that both the foregoing general description and the following detailed description describe various embodiments and are intended to provide an overview or framework for understanding the nature and character of the subject matter claimed. The accompanying drawings are included to provide a further understanding of the various embodiments, and are incorporated in and constitute a part of this specification. The drawings illustrate various embodiments described herein and, together with the description, serve to explain the principles and operations of the subject matter claimed.

[0012] Figure 1 is a diagram illustrating an exemplary system for determining a perceived attractiveness of a facial image portion over a network according to the present application;

[0013] Figure 2 is a diagram illustrating an exemplary functional block diagram associated with detecting a facial image portion according to the present application;

[0014] Figure 3 is a diagram illustrating an exemplary functional block diagram associated with generating an image description according to the present application;

[0015] Figure 4 is a diagram illustrating an exemplary functional block diagram associated with presenting an image description according to the present application;

[0016] Figure 5 is a flowchart illustrating a method for determining a perceived attractiveness of a facial image portion according to the present application;

[0017] Figure 6A is a screenshot illustrating an exemplary graphical user interface for presenting multiple image descriptions to a user for visualizing a perceived attractiveness of a smile in a digital image according to the present application;

[0018] Figure 6B is a digital image according to the present application Figure 6A comprising an image description presented in a heat map form on a facial image portion depicted in the digital image;

[0019] Figure 7is a schematic diagram showing the software architecture of a system according to the present application, which uses an exemplary convolutional neural network (CNN) for filtering detected facial image portions and generating image descriptors to determine the perceived attractiveness of facial image portions according to the present application;

[0020] Figure 8A and Figure 8B is Figure 10 a conceptual illustration of an exemplary filter visualization in the CNN showing the features of interest depicted in one or more filter feature maps according to the present application;

[0021] Figure 9 (a) through Figure 9 (c) are a series of process flow diagrams showing a method of acquiring digital images of facial image portions according to the present application;

[0022] Figure 10 is a flowchart showing a method of obtaining digital data images of detected facial image portions according to the present application;

[0023] Figure 11 (a), Figure 11 (b), and Figure 11 (c) are a series of process flow diagrams showing a method of generating image descriptions of facial image portions according to the present application;

[0024] Figure 12 (a) shows a digital image illustrating an exemplary presentation of image descriptions of facial image portions to a user according to the present application;

[0025] Figure 12 (b) is a detailed view of the facial image portion depicted in the screenshot of Figure 12 (a);

[0026] Figure 13 (a) shows a digital image illustrating a variation of an exemplary presentation of image descriptions of facial image portions to a user according to the present application;

[0027] Figure 13 (b) is a detailed view of the facial image portion depicted in the screenshot of Figure 13 (a);

[0028] Figure 14 is a flowchart showing a method of providing product recommendations to improve the perceived attractiveness of facial image portions according to the present application;

[0029] Figures 15A-15D are screenshots each showing an exemplary user interface for determining the perceived attractiveness of facial image portions according to the present application;

[0030] Figure 15E is a screenshot showing an exemplary user interface illustrating details for displaying a product recommendation for processing a facial feature defining a facial image portion of a person in order to improve the perceived attractiveness of the facial image portion;

[0031] Figure 16 is a flow chart illustrating a method according to the present application showing the efficacy of a customized oral care regimen in improving the perceived attractiveness of one or more oral features of at least one person depicted in a digital image;

[0032] Figure 17A and Figure 17B is a series of process flow charts illustrating a method according to the present application showing the efficacy of an oral care product; Figure 16

[0033] Figure 18 is a flow chart illustrating a method according to the present application showing the efficacy of an oral care product;

[0034] Figure 19A is a digital image showing at least a portion of a facial image portion of a person at the beginning of a predetermined period before processing with a product recommendation, wherein the perceived attractiveness of the facial image portion of the person is determined according to a method according to the present application;

[0035] Figure 19B is a digital image showing at least a portion of a facial image portion of a person after 1 week of use of a product recommendation;

[0036] Figure 19C is a digital image showing at least a portion of a facial image portion of a person after 2 weeks of use of a product recommendation;

[0037] Figure 20 is a flow chart illustrating a method according to the present application for tracking the improvement of the perceived attractiveness of a facial image portion of one or more oral features of at least one person depicted in a digital image; and

[0038] Figure 21 is a screenshot showing an exemplary presentation of an image description for a plurality of facial image portions of persons. DETAILED DESCRIPTION

[0039] The present application relates to a method, a device and a system for determining the perceived attractiveness of a facial image portion in a digital image, and a graphical user interface for visualizing the perceived attractiveness. The facial image portion is of a person and can include one or more facial features, facial expressions, or a combination thereof. The facial features can include nose, mouth, eyes, facial skin, teeth, gums. The facial expression can be a smile.

[0040] ​As described herein, the perceived attractiveness of the facial image portion provides a multi-faceted benefit, i.e., the perceived attractiveness provides both visual facial features that look healthy (hereinafter "well-hued facial features") and visual facial features that look problematic or less than well-hued facial features. In particular, the perceived attractiveness is influenced by positive attributes and negative attributes present in the facial image portion depicted in the digital image. Positive attributes can include tooth whiteness, gum pinkness, tooth surface smoothness, or positive appearance of teeth or gums. Negative attributes can include tooth stains, gum redness, or gum swelling, among others.

[0041] Before describing the present application in detail, the following terms are defined, and undefined terms shall have their ordinary meanings to one of ordinary skill in the relevant arts.

[0042] As used herein, "perceived attractiveness" means the attractiveness of a person whose facial image portion is depicted in a digital image as perceived by a group of people (hereinafter "the group of people"). The group of people can include professionals, industry experts, consumers, or a combination thereof. Perceived attractiveness can include, but is not limited to, the likeability or favorability of a person having the facial image portion depicted in the digital image; the attractiveness of the facial image portion in the context of the facial image portion of the person including something the person wants to do to improve the attractiveness of the facial image portion.

[0043] As used herein, "person" means a human being depicted in a digital image.

[0044] As used herein, "facial image portion" means any concept, digital image, or digital portion of an image based on the detection of one face or multiple faces of a person depicted, including but not limited to one or more facial features, one or more oral features, a facial expression, or a combination thereof, e.g., as determined or detected by pixel data or other pixels of one or more corresponding digital images.

[0045] As used herein, "facial feature" is a facial element and can include, but is not limited to, a tooth, a gum, a nose, a mouth, an eye, facial skin, including features such as determined or detected by pixel data or other pixels of one or more corresponding digital images.

[0046] As used herein, "facial expression" is one or more movements or positions of muscles under facial skin and can include, but is not limited to, a smile.

[0047] As used herein, "smile" consists of teeth and / or gums, but does not include lips, including, for example, as determined or detected by pixel data or other pixels of one or more corresponding digital images.

[0048] As used herein, "oral features" are elements of the mouth and can include, but are not limited to, oral soft tissue, gingiva, teeth, including, for example, as determined or detected by pixel data or other pixels of one or more corresponding digital images.

[0049] As used herein, "attractiveness score (attractiveness index)" means a probability value that indicates the degree of attractiveness of a facial image portion (e.g., teeth) depicted in a digital image to a group of people (hereinafter "the group of people") based on positive attributes and negative attributes of the facial image portion in the digital image. The probability value can be determined by a model constructed by a machine learning system trained by a training data set, wherein the training data set includes (i) a plurality of simulated images of facial image portions (e.g., teeth) including positive (white area) attributes and negative (colored area) attributes; and (ii) associated class definitions (e.g., facial coloring) based on the positive attributes and the negative attributes. The probability value can be a numerical value that indicates the perceived attractiveness of the facial image portion depicted in the digital image calculated by the system herein (hereinafter the attractiveness model is described as an example of the machine learning system) based on the positive attributes and the negative attributes of the facial image portion in the digital image.

[0050] The attractiveness model can be based on training data obtained from raw consumer choice data by way of hierarchical Bayesian (HB) estimation to estimate the main effects of the eight attributes and the local value utility of the limited interaction terms. The attractiveness score for any particular training image can then be calculated from the sum of the local value utilities in the selected attribute levels.

[0051] As used herein, "attribute" means a measurable characteristic of a facial image portion.

[0052] As used herein, "cosmetic dental attribute" means all cosmetic dental attributes that provide oral health effects or influence the appearance and / or feel of an oral area. Some non-limiting examples of cosmetic dental attributes can include gingiva inflammation / redness, gingiva firmness, gingiva bleeding, gingiva sensitivity, yellowness, lightness, anterior surface staining, interproximal (IP) staining between adjacent teeth, marginal staining (around gum line), opacity, shine.

[0053] A "convolutional neural network" is a type of feed-forward artificial neural network in which individual neurons are covered so that they respond to overlapping regions in the visual field.

[0054] As used herein, "oral care product" refers to a product that includes an oral care active and modulates and / or improves the condition of a cosmetic dental attribute. Oral care products can include, but are not limited to, toothpaste, mouthwash, dental floss, or whitening strips, among others.

[0055] As used herein, "digital image" refers to a digital image formed from pixels in an imaging system, including but not limited to standard RGB and the like, and obtained under different lighting conditions and / or modes. Non-limiting examples of digital images include color images (RGB), monochrome images, videos, multispectral images, hyperspectral images, and the like. Non-limiting light conditions include white light, blue light, UV light, IR light, light at specific wavelengths, such as light sources emitting 100 nm to 1000 nm, 300 nm to 700 nm, 400 nm to 700 nm, or any integer combination of the upper and lower limits described above or within the ranges listed above. The digital image can be a single photograph or a single frame of a series of frames defining a video.

[0056] As used herein, "image obtaining device" refers to a device configured for obtaining an image, including but not limited to a digital camera, a photo scanner, a computer readable storage medium capable of storing a digital image, and any electronic device including a photo taking capability.

[0057] As used herein, "user" refers to at least a person using the features provided herein, including, for example, a device user, a product user, and a system user, among others.

[0058] As used herein, "module" can be associated with software, hardware, or any combination thereof. In some implementations, one or more functions, tasks, and / or operations of a module can be carried out or performed by a software routine, a software process, hardware, and / or any combination thereof.

[0059] As used herein, "heat map" refers to a graphical representation of image data contained in a digital image, in which portions of a facial image portion depicted in the digital image are visually highlighted in order to identify an analysis target to be presented in the image description. For example, if the analysis target is a negative attribute of the facial image portion, the area of the facial image portion that includes the negative attribute is visualized.

[0060] As used herein, "processing" refers to providing product recommendations, customization instructions, using recommended products to improve the perceived attractiveness of a facial image portion of a subject depicted in a digital image. The subject is a person.

[0061] In the following description, the system is system 10 for determining the perceived attractiveness of a smile 521 of a person depicted in a digital image 51. Thus, the apparatus 14 is an apparatus 14 for determining the perceived attractiveness of a smile 521 of a person, and also a system for providing product recommendations to improve the perceived attractiveness of a smile 521 of a person depicted in a digital image is described. Thus, the positive and negative attributes of a smile 521 relate to cosmetic dental attributes as described above, i.e. all cosmetic dental attributes that provide an oral health effect to an oral region or influence its appearance and / or feel. However, it is conceivable that the apparatus and the method can be configured for various applications to determine the perceived attractiveness of other facial image portions, wherein the facial image portion is one or more facial features, including but not limited to nose, skin, lips, eyes, combinations thereof.

[0062] System

[0063] Figure 1 is a schematic diagram illustrating a system 10 for determining the perceived attractiveness of a facial image portion 52 of a person depicted in a digital image 51 according to the present application. In an exemplary embodiment, the system 10 is a cloud-based system configured for anywhere, such as for example by a portable electronic device 12 comprising an image obtaining means 18 and a display (not shown). The portable electronic device 12 can be connected to the apparatus 14 for generating a graphical user interface for display on the display for visualizing the perceived attractiveness of the facial image portion over the network 100. However, it is conceivable that the system 10 can be configured as a standalone system. It is also conceivable that the portable electronic device 12 can be a touch-sensitive display.

[0064] The system 10 can comprise a network 100, which can be embodied as a wide area network such as a mobile telephone network, a public switched telephone network, a satellite network, the Internet, and the like, a local area network such as a wireless fidelity, Wi-Max, ZigBee TM , Bluetooth TM , and / or other forms of networking functionality. Coupled to the network 100 are: a portable electronic device 12; and an apparatus 14 for generating a graphical user interface 30 (see Figure 6A ) for display on a display for visualizing the perceived attractiveness. The apparatus 14 is located remotely and connected to the portable electronic device 12 over the network 100. The network 100 can be used to obtain a digital image from the portable electronic device 12 and transmit the digital image to the apparatus 14 for determining the perceived attractiveness of the facial image portion 52 of the person depicted in the digital image 51 as described below with reference to Figure 7The input device 12a can be coupled to or integral with the portable electronic device 12 for receiving user input for initiating the processor 14b. The portable electronic device 12 can include an output device 12b for presenting the image description 53 of the facial image portion 52 depicted in the digital image 51. The input device 12a can include, but is not limited to, a mouse, a touch screen display, and the like. The output device 12b can include, but is not limited to, a touch screen display, a non-touch screen display, a printer, an audio output device such as a speaker.

[0065] The portable electronic device 12 can be a mobile phone, a tablet computer, a laptop computer, a personal digital assistant, and / or other computing device configured to capture, store, and / or communicate digital images such as digital photographs. Accordingly, the portable electronic device 12 can include an image obtaining device 18 for obtaining images such as a camera integral with the device 12 and an output device 12b for displaying images. The portable electronic device 12 can also be configured to communicate with other computing devices via the network 100. The apparatus 14 can include a non-transitory computer readable storage medium 14a (hereinafter "storage medium") that stores image obtaining logic 144a, image analysis logic 144b, and graphical user interface (hereinafter "GUI") logic 144c. The storage medium 14a can include random access memory such as SRAM, DRAM, and the like, read only memory (ROM), registers, and / or other forms of computing storage hardware. The image obtaining logic 144a, the image analysis logic 144b, and the GUI logic 144c define computer executable instructions. The processor 14b is coupled to the storage medium 14a, wherein the processor 14b is configured to implement the method 200 according to the present application for determining the perceived attractiveness of a facial image portion of one or more persons depicted in a digital image 51 based on the computer executable instructions, as described below with reference to the flowcharts of Figures 2-4 and described below with reference to the flowcharts of Figure 5 .

[0066] Figure 2 is a diagram illustrating an exemplary functional block diagram of a facial image portion pre-processing module 40 according to the present application that incorporates image obtaining logic 144a for obtaining a digital image 51 that includes a facial image portion 52. The pre-processing module 40 can include a first pre-processing submodule 40A for detecting the facial image portion 52 and a second pre-processing submodule 40B for detecting one or more features that define the facial image portion 52.

[0067] Figure 3is a diagram showing an exemplary functional block diagram of an attractiveness model module 42 according to the present application comprising image analysis logic 144b for analyzing a positive and negative attributes of a face image portion 52 of a person depicted in a digital image 51, generating an attractiveness score 57 and an image description 53. In particular, the attractiveness model module 42 can comprise a first attractiveness sub-module 42A for generating an attractiveness score 57 indicative of a perceived attractiveness of the face image portion 52 and a second attractiveness sub-module 42B for generating the image description 53.

[0068] Figure 4 is a diagram showing an exemplary functional block diagram of a visualization module 44 according to the present application comprising GUI logic 144c for presenting the image description 53. The visualization module 44 can comprise a first visualization sub-module 44A for presenting the attractiveness score 57, a second visualization sub-module 44B for presenting the image description 53 as a heat map and a third visualization sub-module 44C for presenting the image description 53 as a substitute text 531.

[0069] The face image portion pre-processing module 40, the attractiveness model module 42, or the visualization module 44 can be implemented partially or entirely as software, hardware, or any combination thereof. In some cases, the attractiveness model module 42 can be implemented partially or entirely as software running on one or more computing devices or computing systems, such as software running on a server computing system or a client computing system. For example, the attractiveness model module 42 or at least a portion thereof can be implemented as or within a mobile application (e.g. APP), program or applet, etc. running on a client computing system (such as the portable electronic device 12 of Figure 1 The computing system can be in communication with a content server configured to store the obtained digital image or the obtained plurality of digital images. The modules 40, 42, 44 can each be implemented using one or more computing devices or systems including one or more servers, such as web servers or cloud servers. In particular, the processor 14b is configured to implement the method 200 according to the present application for determining a perceived attractiveness of a face image portion of one or more persons depicted in a digital image 51 based on computer executable instructions, as described below with reference to the flowchart of Figure 5 The computing system can be in communication with a content server configured to store the obtained digital image or the obtained plurality of digital images. The modules 40, 42, 44 can each be implemented using one or more computing devices or systems including one or more servers, such as web servers or cloud servers. In particular, the processor 14b is configured to implement the method 200 according to the present application for determining a perceived attractiveness of a face image portion of one or more persons depicted in a digital image 51 based on computer executable instructions, as described below with reference to the flowchart of

[0070] Systems and methods

[0071] Accordingly, the following refers to Figure 5The steps 202, 204, 206, 208, 210, 212, 214 of the method 200 according to the present application will be described as individual processes for performing each step. Each process can also be described as a subroutine, i.e. a sequence of program instructions that performs the corresponding step according to the method 200 according to the present application.

[0072] When the processor 14b is started, the processor 14b causes a first digital image 51 of at least a portion of the face of the subject to be obtained, e.g. in step 202, via the image obtaining logic 144a. The first digital image 51 can be a dental image. The face image portion 52 is a smile 521 defined by a combination of teeth and gums as shown, and the smile comprises positive attributes and negative attributes. In step 204, to estimate the features of interest, the processor 14b uses a trained learning machine to analyze the face image portion 52. Figure 6B

[0073] In step 206, an attractiveness score 57 is generated for the face image portion 52.

[0074] The method 200 can further comprise generating, in step 208, an image description 53 comprising the face image portion 52 based on the attractiveness score 57, and presenting, in step 210, the image description 53 to a user to determine the perceived attractiveness of the face image portion 52. In particular, presenting the image description 53 can comprise one of displaying the image description 53 in the digital image 51 as a replacement text, displaying the image description 53 in the digital image 51 as a heat map, providing the image description 53 for presentation to the user in an audible manner, and combinations thereof.

[0075] By generating an attractiveness score 57 of a face image portion depicted in a digital image provided by a user (consumer), further generating an image description 53 based on the attractiveness score, and presenting the image description 53 to the consumer, the user and / or the consumer can obtain information about the face image portion 52 that affects the perceived attractiveness of the face image portion 52. It will be appreciated that the method 200 can also be applied to other image processing aspects of other face image portions, such as facial skin.

[0076] Human-machine user interface

[0077] ​The present application also relates to a human-machine user interface (hereinafter "user interface") for determining the perceived attractiveness of a facial image portion 52 in a digital image 51. The user interface can be a graphical user interface on a portable electronic device, including a touch screen display / display having an input device and an image obtaining device 18. The user interface can include a first area of the touch screen display that displays a first digital image of at least a portion of a subject's face, the first digital image including the facial image portion obtained from the image obtaining device 18 and a second digital image superimposed on the first digital image, the second digital image having at least a portion of the subject's face, the displayed facial image portion, and a displayed image description for the displayed facial image portion. The user interface can also include a second area of the touch screen display that is different from the first area, the second area displaying a selectable icon for receiving user input, wherein if the user activates the selectable icon, an image of at least one product recommendation item is displayed on the touch screen display to improve the perceived attractiveness of the facial image portion.

[0078] The method 200 for determining perceived attractiveness can be applied in a variety of different applications, including but not limited to providing product recommendations, providing personalized product usage instructions to a consumer, visualizing product efficacy, and monitoring progress in improving the perceived attractiveness of a facial image portion after using a recommended product. Although the following exemplary applications described below relate to oral features as specific examples of facial image portions, and such oral features include teeth, gums, and combinations thereof, it should be understood that the method is applicable to other facial features.

[0079] Figure 6A is a screen shot showing an exemplary graphical user interface 30 according to the present application that presents an image description 53 of a facial image portion 52 of a person in a digital image 51 to a user for determining the perceived attractiveness of the facial image portion 52.

[0080] The digital image 51 can include a facial image portion 52 that has been programmed by the processor 14b to determine the perceived attractiveness of the facial image portion, and the facial image portion 52 is detected by the processor 14b here by the pre-processing module 40 (hereinafter "detected facial image portion 52"). The facial image portion 52 can include one or more oral features, one or more facial expressions, or combinations thereof. The oral features can include a mouth, teeth, gums, or any feature in the oral cavity. The facial expressions can include a smile.

[0081] There is an image description 53 of the detected facial image portion 52, and a selectable input screen object 54 disposed in the graphical user interface 30.

[0082] Image description 53 may include: alternative text 531 displayed in graphical user interface 30; a heatmap 532 displayed on digital image 51, identifying at least one region in facial image portion 52 that includes negative attributes of facial image portion 52 (hereinafter referred to as "identified region"); or a combination of alternative text 531 and heatmap 532. Specifically, alternative text 531 includes a description indicating the influence of the identified region in facial image portion 52 on the perceived attractiveness of facial image portion 52. For example, heatmap 532 may show dental portions with different defects that require different corresponding oral care treatments. For example, heatmap 532 may include one or more regions of interest highlighted in a dental image associated with the person depicted in digital image 51.

[0083] The optional input screen object 54 may include a text label that describes the characteristics of the optional input screen object 54. The optional input screen object 54 may include a text label that describes the direction for processing a request for additional information about the facial image portion 52; for example, the text label may include a description related to proceeding to a different user interface, which relates to a method for providing product recommendations to enhance perceived appeal.

[0084] like Figure 6A As shown, the whiteness of the marked areas on the teeth can be improved. Therefore, this description may relate to understanding solutions for improving the whiteness of the marked areas on the teeth, thereby increasing the perceived attractiveness of the detected facial image portion 52.

[0085] Figure 6B It has a heatmap of 532. Figure 6A Digital image 51. Reference Figure 6B The detected facial image portion 52 is a smile 521 defined by a combination of oral features, teeth, and gums. The smile 521 includes positive and negative attributes as described below. Specifically, at least a portion of the smile 521 is defined by a first oral feature 521A and a second oral feature 521B.

[0086] The first oral cavity feature 521A may be a first tooth, and the second oral cavity feature 521B may be a second tooth located in a different portion of the area of ​​the facial image portion 52. The first oral cavity feature 521A includes a highlighted region of interest 533 of the heatmap 532 highlighted in the tooth image, thereby indicating a negative oral cavity attribute (yellowness). On the other hand, the second oral cavity feature 521B does not include the highlighted region of interest of the heatmap 532 highlighted in the tooth image, thereby indicating a positive oral cavity attribute (whiteness).

[0087] Figure 7is a schematic diagram showing an exemplary system architecture 80 configured for implementing the method 200 based on a convolutional neural network (CNN) model. Figure 9 (a) and Figure 9 (b) are Figure 7 conceptual diagram of an exemplary filter visualization in a CNN model showing a feature of interest depicted in one or more filter feature maps according to the present application.

[0088] In the following description, the CNN model is described as an example of a machine learning algorithm, in particular a deep learning algorithm, used to implement the methods and systems according to the present application. Deep learning algorithms involve building larger and more complex neural networks, and as described below, the present application involves analysis of models trained by labeling very large datasets of simulated data, such as digital images. Thus, other deep learning algorithms that can be used to implement the methods according to the present application can include, but are not limited to, recurrent neural networks (RNNs), long short-term memory networks (LSTMs), stacked autoencoders, deep Boltzmann machines (DBMs), deep belief networks (DBNs).

[0089] Figure 7 The system architecture 80 of the CNN model, the CNN components that make up the CNN model, and the exchanges between each of the CNN components for performing the method 200 according to the present application are shown. In general, the CNN model is able to extract a hierarchical structure of visual features through a stackable neural layer equipped with receptive fields that implement convolutional kernels that identify primary visual features as complex visual features of image components. In other words, each layer of the CNN model extracts rich information representing the original stimulus. With reference to Figure 7 The system architecture 80 of the CNN model includes CNN components operatively connected by CNN exchanges arranged to generate the attractiveness score 57, the details of the CNN components and their respective functions are described in Table 1 below.

[0090] Table 1

[0091]

[0092] The actions performed in each CNN exchange connecting each of the aforementioned CNN components are described in Table 2 below, and the order of the analysis step 204 and the generation step 206 is according to the CNN exchange direction as shown in Figure 7

[0093] Table 2

[0094]

[0095] ​

[0096] As Figure 7 illustrated, the analysis can comprise filtering the digital image 51 in a first exchange 90 to obtain one or more filtered feature maps comprising features of interest associated with the facial image portion 52, and analysing these features of interest. Figure 8A A first filtered feature map Y is illustrated having positive attributes Figure X , and Figure 8B A second filtered feature map Y is illustrated having negative attributes. The positive attributes can comprise tooth whiteness, pinkness of the gums, smoothness of the tooth surface, or positive appearance of the teeth or gums. The negative attributes can comprise tooth stains, gum redness, or gum swelling, among others.

[0097] As Figure 8A and Figure 8B illustrated, the first and second features of interest are different. In particular, with reference to Figure 7 and Figure 8A and Figure 8B , an attractiveness score 57 can be generated based on a first set of characteristics associated with the first feature of interest in the first filtered feature map (layer Y) and a second set of characteristics associated with the second feature of interest in the second filtered feature map (layer Y). With reference to Figure 8A , the first feature of interest can comprise a first plurality of oral features, including gums and teeth located in the upper part of the oral cavity. With reference to Figure 8B , the second feature of interest can comprise a second plurality of oral features, including gums and teeth located in the lower part of the oral cavity.

[0098] The method can further comprise generating an abnormality output 85 indicating the second feature of interest, the second feature of interest comprising negative attributes negatively affecting the condition of the first feature of interest.

[0099] Obtaining a digital image

[0100] With reference to a series of process flow diagrams illustrating how the digital image 51 is obtained (a), Figure 9 (b), and Figure 9 (c), the obtaining digital image step 202 of the method 200 according to the present application is described. Figure 9 is a flow diagram of a process 300 of obtaining the digital image 51 corresponding to step 202. Figure 10

[0101] Figure 9 ​(a) shows an input image 50a of a human face. Input image 50a may be captured by a user, for example, using a camera 18 of a portable electronic device 12. Input image 50a may also be further processed using machine learning and computer vision techniques to automatically detect human faces and / or facial image portions. For example, method 300 may include a face detection module that employs the Dlib face detection library to detect faces depicted in input image 50a and draws a first detector box 55 defining the detected faces in input image 50a. Examples of how the Dlib face detection library can be applied to find facial landmarks to detect faces in digital images can be found in the following published reference: DEKing. Dlib-ml: A machine learning toolkit. J.Mach. Learning Research, 10:1755–1758, 2009.

[0102] Figure 9 (b) shows the use Figure 9 (a) Step 302, which uses detector box 55 to crop input image 50a to obtain edited image 50b, as an example of facial image portion 52 according to the invention, the edited image includes at least a portion of a face containing a human smile. The second preprocessing submodule 40B may be a feature detection module configured to detect facial features defining facial image portion 52 (smile), such as teeth and / or gums, and to draw a second detector box defining facial image portion 52. Figure 9 As shown in (c), the second detector box can be used to further crop and edit image 50b to obtain digital image 51.

[0103] Generate image description

[0104] Reference Figure 11 (a) Figure 11 (b) and Figure 11 (c) describes the image generated according to the present invention, description 53.

[0105] Figure 11 (a) shows a digital image 51, which includes a facial image portion 52 depicted in the digital image 51 prior to analysis. The digital image 51 can be cropped according to the method 300 for obtaining the digital image. Figure 11 (b) shows a second digital image 51b, which includes an image description 53 that is visually presented as a heatmap 532 covering the digital image 50.

[0106] Displaying the image description 53 in the digital image 51 as a heat map 532 can comprise generating the heat map 532, wherein generating the heat map comprises overlaying the layer 120B onto at least a portion of the digital image 52 comprising the facial image portion, wherein the layer 120B is a pixel map identifying at least one region comprising at least one of the analyzed negative attributes.

[0107] In particular, the heat map 532 visualizes the positive attributes as the second layer 120A and the negative attributes as the layer 120B in at least one region in the depicted facial image portion 52 in the digital image 51. Figure 11 (c) shows a third digital image 51c comprising the facial image portion 52 with the heat map 532, wherein the layer 120B is only overlaid onto the facial image portion 52 to depict only the negative attributes present in the facial image portion 52. Although the above description only relates to a depiction of the negative attributes present in the facial image portion 52, it is to be understood that the heat map 532 can be configured to have the layer 120A only overlaid onto the facial image portion 52 to depict the positive attributes present in the facial image portion 52, as shown in the third digital image 51c.

[0108] Referring to Figure 12 (a), Figure 12 (b), the image description 53 can be presented as a color region 130 on the teeth to indicate the region for improving the perceived attractiveness of the person's smile (facial image portion 52). Alternatively, referring to Figure 13 (a), Figure 13 (b), the image description can be presented in the form of a color border line 131 framing a region on the teeth to indicate the region for improving the perceived attractiveness of the person's smile.

[0109] Product recommendation

[0110] Figure 14 is a flow chart illustrating a method 400 for providing a product recommendation to improve the perceived attractiveness of a depicted facial image portion 52 in a digital image 51. Figures 15A-15E are screen shots each showing an exemplary user interface cooperating with each other to provide a product recommendation according to the present application. Although the user interfaces of Figures 15A-15E are described as a series of user interfaces provided in sequence in response to the preceding user interfaces, it is to be understood that the user interfaces of Figures 15A-15E may be programmed in a variety of ways to define the entire user interface for providing a product recommendation according to the method according to the present application as described above. Preferably, all the user interfaces of Figures 15A-15E define exemplary user interfaces according to the present application for providing a product recommendation to improve the perceived attractiveness.

[0111] Referring toFigure 14 The method comprises, in step 402, transmitting a digital image of at least one person, wherein the digital image comprises a facial image portion of the at least one person, wherein the facial image portion has both positive attributes and negative attributes.

[0112] In step 404, an image description is received, wherein the image description identifies at least one region in the facial image portion that comprises at least one of the negative attributes analyzed using the method 200. In step 406, the image description is presented. In step 408, a product recommendation is presented to the user to increase the perceived attractiveness of at least one of the analyzed positive and / or negative attributes.

[0113] Figure 15A is a screenshot of a user interface 160A for transmitting an input image 50a of a person's face to a device 14 according to step 402 of the method 400. Figure 1 The user interface 160A can display a first text object 161, wherein the first text object 161 can comprise any terms and / or phrases that can be used to describe information about the method for determining the perceived attractiveness of a facial image portion of a person according to the present invention. In an exemplary embodiment, the first text object 161 can comprise text depicted in the user interface 160A that relates to a method of comparing the perceived attractiveness of a facial image portion 52 of a person to a group of people or a rhetorical question, i.e., "Do you have a captivating smile?" In particular, the first text object 161 can comprise an advertising device used in an advertisement, e.g., a question related to the facial image portion 52, to attract a consumer. The input image 50a can be captured by the user and transmitted to the device 14 as a selfie image using a selectable icon 162 displayed on the user interface 160, e.g., using a mobile phone.

[0114] Figure 15B is a screenshot of a user interface 160B displaying a second text object 163 to a user, the second text object indicating a status of the method 400. Referring to Figure 15C The user interface 160C displays an alternative text 531 indicating the attractiveness score 57 obtained in step 404. The image description 53 can also include an alternative text 531a displayed in the user interface 160D, wherein the alternative text 531a is associated with information about the attractiveness score. The alternative text 531a can be an attractiveness total score, such as, for example, a total score value calculated according to a mathematical formula based on the attractiveness score.

[0115] The user interface 160C also displays a selectable input icon 164 for transmitting an input image 50b to the device 14 according to step 404 of the method 400. Figure 15DAs shown, the following request is sent: in step 406, the image description 53 is presented in the form of a heat map 532. Reference is made to Figure 15D Figure 30, which is similar to the user interface 30 of Figure 6A Figure 29, and is shown to illustrate a sequence of user interfaces forming part of the above-described method.

[0116] The facial image portion 52 being determined is the smile of the person depicted in the digital image 51, and therefore Figure 15E the product recommendation shown in the user interface 170 of Figure 29 is an oral care product 172 for improving the perceived attractiveness of the smile.

[0117] The image description 53 can include alternative text 531 relating to the following oral care information:

[0118] 1) Brush more carefully and / or pay more attention to the areas of interest indicated in the heat map ("areas of interest")

[0119] 2) In the areas of interest, the whiteness is not optimized

[0120] 3) The areas of interest are not white enough

[0121] 4) The areas of interest are the corners of the mouth that are difficult to access with a toothbrush during brushing (hereinafter referred to as "brushing dead angles")

[0122] 5) The areas of interest are brushing dead angles that require more careful brushing.

[0123] Visualization of the efficacy of a customized oral care regimen

[0124] The present invention also relates to a method of demonstrating the efficacy of a customized oral care regimen to a user, and which can be used by a dental professional for remote oral care consultation for a user in need of treatment but unable to visit the dental clinic where the dental professional is located. Figure 16 is a flowchart illustrating a method 500 of demonstrating the efficacy of a customized oral care regimen in improving the perceived attractiveness of one or more oral features of at least one person depicted in a digital image, in accordance with the present invention. The customized oral care regimen can include: providing brushing guidance; and / or providing a recommendation of an oral care product for use in conjunction with the brushing guidance. The method 500 can be used to analyze weekly images of one or more oral features (e.g. teeth and / or gums) to visualize stain areas on the teeth.

[0125] Due to the professional specificity of clinical methods, it is often challenging to translate the clinical measured efficacy of an oral care regimen into consumer relevant benefits, and thus consumers find it difficult to compare / remember the "before and after" state. Therefore, it is important to visualize the progress of oral care regimen and / or oral care product efficacy by a method that provides image descriptions to explain the "before and after" state of oral features and makes the images "talking" and shareable.

[0126] According to the method 500, the user can receive personalized oral care advice with product usage instructions and pictures of their teeth analyzed according to the method when brushing is supervised. The use of the method 500 can include several key benefits:

[0127] • The first digital image 60 shows the areas of attention - the colored areas are marked in bright pink (as shown). Figure 17A

[0128] • It is more convenient for dental professionals to provide oral care such as brushing guidance.

[0129] • It is easy for the user to use at home.

[0130] The method 500 can comprise the following steps:

[0131] a) determining 502 the perceived attractiveness of the facial image portion of the person in the first digital image 60 before treatment with the customized oral care regimen or oral care product (see Figure 17A );

[0132] b) obtaining 504 a second digital image 61 of the facial image portion of the person depicted in the first digital image, wherein the second digital image comprises the facial image portion of the person, wherein the facial image portion in the second digital image is treated with the customized oral care regimen or oral care product for a predetermined period of time;

[0133] c) determining 506 the perceived attractiveness of the facial image portion in the second digital image 61 (see Figure 17B );

[0134] d) comparing the perceived attractiveness of the facial image portion in the second digital image 61 with the perceived attractiveness of the facial image portion in the first digital image 60.

[0135] In particular, Figure 17A is the first digital image 60 of the oral features of the subject determined for the perceived attractiveness on day 0 (i.e. the beginning of the predetermined period of time). Figure 17B ​is a second digital image of the oral feature of the subject determined for perceived attractiveness at the end of a predetermined period. The predetermined period can be two weeks of use of a consumer product, such as an oral care product. The image description in the first and second digital images identifies at least one area in the oral feature depicted in the digital image that includes the negative attribute analyzed. Thus, to determine whether the negative attribute of the oral feature has been reduced, the pixel count of the image description can be analyzed to show whether the negative attribute has been reduced. As shown in Figure 17A the image description of the oral care feature in the first digital image 60 has 12714 pixels, while the image description of the oral care feature in the second digital image 61 has 7894 pixels. The reduction in the number of pixels corresponds to a reduction in the negative attribute of the oral care feature. The negative attribute can be tooth stains. The facial image portion in the first digital image 60 can be treated with a reference product for comparative analysis between products. Figure 17B

[0136] Figure 18 is a flowchart showing a method 700 of showing efficacy of a consumer product or oral care regimen over a product use period.

[0137] The method 700 can include the following steps:

[0138] i) determining a perceived attractiveness of a facial image portion of a person in a first digital image before treating the facial image portion with a customized oral care guidance;

[0139] ii) obtaining a second digital image of the person depicted in the first digital image, wherein the second digital image includes the facial image portion of the person, wherein the facial image portion in the second digital image is treated with a consumer product for a predetermined period;

[0140] iii) determining a perceived attractiveness of the facial image portion in the second digital image;

[0141] iv) comparing the perceived attractiveness of the facial image portion in the second digital image to the perceived attractiveness of the facial image portion in the first digital image.

[0142] Figure 19A ​is a first digital image 511 comprising at least a portion of a face image portion 52 of a person. At the beginning of a predetermined period before performing a customized oral care guidance, the digital image 511 is analyzed according to the method 200 according to the application. The digital image 511 further comprises an image description identifying a plurality of regions 533 in the face image portion 52, the plurality of regions comprising at least one analyzed negative attribute. In particular, the image description is presented as a heat map 533. In order to provide the user with information related to the perceived attractiveness of the face image portion 52, in the digital image 511, an attractiveness score 534 corresponding to a numerical value (e.g. 27), a first numerical value 535 corresponding to the number of identified regions (e.g. 11 identified regions), and a second numerical value 536 corresponding to the total number of pixels defining the identified regions (e.g. 9742 pixels) are presented.

[0143] Figure 19B is a second digital image 512 showing at least a portion of a face image portion of a person after one week of use of the product recommendation item 1. Figure 19C is a third digital image 513 showing at least a portion of a face image portion of a person after two weeks of use of the product recommendation item 1. Table 3 below describes Figure 19B , Figure 19C the attractiveness score in Figure 19A the increase of the attractiveness score in

[0144] Table 3

[0145]

[0146] In particular, the decrease in the number of identified regions corresponding to negative attributes of oral features demonstrates that the use of the product recommendation reduces the negative attributes, thereby increasing the attractiveness score and, consequently, the perceived attractiveness of the face image portion.

[0147] The method can comprise the step of repeating the determining step and the comparison in step (iv) over a predetermined period. The predetermined period can be one week, preferably two weeks, more preferably three weeks. The technical effect consists in that it enables to track the increase of the perceived attractiveness of the face image portion over the predetermined period, thereby allowing the user to monitor the progress and the product usage accordingly. The perceived attractiveness of the face image portion can comprise one or more oral features of at least one person depicted in the digital image. The one or more oral features can comprise, but are not limited to, teeth, and the perceived attractiveness is teeth whiteness.

[0148] Method of tracking the increase of the perceived attractiveness

[0149] Figure 20is a flowchart illustrating a method 800 of tracking the improvement in perceived attractiveness of one or more oral features of at least one person depicted in a digital image of a face image portion according to the present application. For example, the oral feature can be teeth, and the method 800 can be used to keep a record and track of the digital image and analysis results of a person's teeth, such that the method can repeat the analysis based on a time schedule to show the improvement / progress in the person's teeth attributes (e.g. teeth whitening) and / or attractiveness over a period of time. This functionality can be used to support product advertising phrases, including but not limited to "21 days teeth whitening challenge", "whiten your teeth in a predetermined number of days", "long lasting whitening (lock-in whitening)", "long lasting (24 hours) whitening", or any teeth whitening related attributes.

[0150] The product can be an oral care product, including but not limited to toothpaste, whitening strips, mouthwash, or any form suitable for performing an oral care treatment. Although in the method 800, teeth attractiveness is described as a desired attribute related to perceived attractiveness, it is understood that the method 800 can be applied to other attributes, including but not limited to gum health, teeth shine, or any other consumer related description applicable to the image description of the oral feature attributes as described in Table 5 below.

[0151] The method 800 can comprise the following steps:

[0152] - obtaining 802 a digital image of at least a portion of a face of a subject to be obtained, e.g. via the image obtaining logic 144a. The digital image 51 can be a teeth image.

[0153] - analyzing 804 the face image portion 52 using a trained learning machine to estimate the feature of interest.

[0154] - generating 806 an analysis result for the face image portion 52.

[0155] - storing 808 the digital image and analysis result with a time code identifying the digital image with the analysis result. The time code can include but is not limited to date, time, or user information, etc.

[0156] - selecting 810 a time period, or presetting the time period, based on a product advertising phrase.

[0157] - generating 812 an image description based on the analysis result in step 806. Optionally, the image description can be presented in step 812.

[0158] - optionally, further presenting 814 a product recommendation.

[0159] The analysis result in step 806 can include an attractiveness score, at least one area of one or more oral features that the consumer still needs to improve, or other data generated by the attractiveness model described above.

[0160] In particular, generating an image description in step 812 can include generating an image summary and an analysis result data summary from the analysis result from the database. The database can be stored on a server coupled to the system. Optionally, the method 800 can further include presenting product recommendations in step 814, including but not limited to continuing to use a product (a product currently used by the consumer) for a predetermined number of days, adding a new product to the consumer’s oral care regimen for better results, or any suitable treatment for improving dental attractiveness.

[0161] Figure 21 is a screenshot of a plurality of digital images 51 of people, where for each detected facial image portion 52 for each person in the plurality of people, an image description 53 is presented. As shown, the detected facial image portion 52 is a smile 521, and an attractiveness score 57 is visually presented in the digital image 51. The attractiveness score 57 can include, but is not limited to, a numerical value indicating the perceived attractiveness of the smile 521 relative to a group of people. The attractiveness score 57 can be a smile score for the smile 521. Figure 3

[0162] Training data set

[0163] For example, the CNN model described above can be trained and evaluated on a data set of simulated dental images.

[0164] A training data set of simulated dental images (“simulated image data set”) for defining attractiveness scores can be constructed as described below. The training data set design criteria can be based on eight different dental attributes as described in Table 4 below, and different score levels ranging from 0% to 100% are assigned to each group of images belonging to the same dental attribute.

[0165] Table 4

[0166] Oral Feature Attributes Level 1 Level 2 Level 3 Level 4 Level 5 1 Brightness 20％ 40％ 60％ 80％ 100％ 2 Yellowness 20％ 30％ 40％ 50％ 60％ 3 Opacity 20％ 40％ 60％ 80％ 100％ 4 Facial Staining 20％ 40％ 60％ 80％ 100％ 5 Glossiness 20％ 40％ 60％ 80％ 100％ 6 Interproximal (IP) Staining 20％ 40％ 60％ 80％ 100％ 7 Marginal Staining 20％ 40％ 60％ 80％ 100％ 8 Gingival Redness 20％ 40％ 60％ 80％ 100％ .

[0167] ​There can be a set of simulated images for facial staining, each corresponding to a different level of score. The preparation of the simulated images is based on the assumption that a simulated image corresponding to a lower score has a predetermined area of front teeth surface with facial staining (a negative attribute) and a larger area of white front teeth surface (a positive attribute) and will be perceived as more attractive than another image with the same predetermined area of front teeth surface but corresponding to a higher score. The predetermined area of facial staining is the same from low score to high score, but the intensity of the color of the facial staining is increased from low score to high score in different images.

[0168] A set of three different images can be shown side by side to the consumer, representing a combination of all eight attributes. For each image, the specific levels of the eight attributes are determined by balanced, designed, discrete selection (joint) randomization. Thus, within each selection set, up to all eight attributes are different among the three images according to the randomization. This is to determine what they really perceive as most attractive.

[0169] For example, when a given set of three images can be shown to a consumer, the three images can be composed of any combination of the following set of attributes, including facial staining, where a given level of each attribute is represented in each set of teeth.

[0170] The attractiveness model based on training data can be obtained from the raw consumer selection data by estimating the main effects of the eight attributes and the local value utility of the limited interaction terms via hierarchical Bayesian (HB) estimation. Then, the attractiveness score for any particular training image can be calculated from the sum of the local value utility in the selected attribute levels.

[0171] The simulated image data set can be modified in the same way and built into the attractiveness model and analyzed accordingly based on knowing which skin attributes, such as pigmentation or other skin attributes, are to be defined. For example, if the facial image portion is skin, a simulated image data set can be generated by modifying skin images based on the design criteria for the data set described above for teeth, which is then applied to the attractiveness model to determine skin attractiveness.

[0172] The advantage of the simulated image data set is that it is easy to define the levels of the attributes that are relevant to the consumer, resulting in a better and controllable measure of the attributes that drive their perception of attractiveness. Using simulated images provides the advantage of using consumer-relevant data to generate a score that is consumer-relevant and not a random result generated by a random inventory of facial images.

[0173] Since each consumer-relevant image can be classified and labeled, using a simulated image data set to train a machine model will enable the machine model to generate consumer-relevant results.

[0174] Alternatively, real person images of a predetermined group size can be collected to build a training dataset based on real persons of the predetermined group, and a discrete choice model can be used to estimate attractiveness of facial image portions.

[0175] In one exemplary example, a process for building a training dataset can include the following steps:

[0176] (1) Create attribute images

[0177] (2) Randomize them in the design

[0178] (3) Collect consumer discrete choice data

[0179] (4) Estimate attribute image utilities into training data

[0180] (5) Build a machine learning algorithm based on training data utility scores

[0181] A training dataset can be created for any system that can be decomposed into body attributes and their levels. Discrete choice models can be used to describe these attributes. Preferably, the discrete choice model is a joint statistic that can be used to describe combinations of fixed (controlled) attribute images. Alternatively, the discrete choice model can be a MaxDiff analysis that can be used to describe a set of non-fixed (uncontrolled) attribute images (e.g., a large set of clinical images) that have known scores (e.g., clinical grading for color, yellowness, or any desired oral feature attribute) for identified attribute levels.

[0182] Further, a consumer can interpret attractiveness of one or more oral features, and thus, in accordance with the present invention, the term “attractiveness” can have multiple words for image descriptions shown in the following steps: step (e), presenting 210 image descriptions 53 to a user. Table 5 below is a non-exhaustive list of consumer-related descriptions that can be used for image descriptions described below with respect to relevant facial image portions (specifically, oral feature attributes).

[0183] Table 5

[0184]

[0185] Representative embodiments of the disclosure described above can be described as set forth in the following paragraphs:

[0186] A. A computer-implemented method for determining a perceived attractiveness of a facial image portion (52) of at least one person depicted in a digital image (51) based on oral care, the method comprising the steps of:

[0187] a) obtaining (202) a digital image (51) comprising at least one oral feature of at least one person, wherein the digital image (51) comprises a facial image portion (52) of the at least one person, wherein the facial image portion (52) has both positive attributes as defined by pixel data of the digital image (51) and negative attributes as defined thereby;

[0188] b) analyzing (204) the facial image portion (52);

[0189] c) generating (206) an attractiveness score (57) indicative of a perceived attractiveness of the facial image portion (52) based on the analyzed facial image portion (52) in the obtained digital image (51);

[0190] d) further generating (208) an image description (53) identifying at least one region in the facial image portion (52) indicative of the attractiveness score (57); and

[0191] e) presenting (210) the image description (53) to a user.

[0192] B. The method according to paragraph A, wherein the attractiveness score is generated as a probability value indicative of a degree of attractiveness of the facial image portion depicted in the digital image to a group of people based on the positive attributes and the negative attributes, preferably determined by a model structured by a machine learning system trained by a training data set, wherein the training data set comprises i) a plurality of simulated images comprising facial image portions of positive attributes and negative attributes; and (ii) associated class definitions based on the positive attributes and the negative attributes.

[0193] C. The method according to paragraph A or B, further comprising, after step (c), presenting the attractiveness score (57) to a user.

[0194] D. The method according to any one of paragraphs A-C, wherein the image description (53) further indicates an influence of the identified region in the facial image portion (52) on the attractiveness score (57).

[0195] E. The method according to any one of paragraphs A-D, wherein the facial image portion (52) is selected from the group consisting of: facial skin, one or more oral features, one or more facial expressions, and combinations thereof.

[0196] F. The method according to any one of paragraphs A-E, wherein the facial image portion (52) comprises one or more oral features selected from the group consisting of: oral soft tissue, gum, teeth, and combinations thereof.

[0197] G. The method of any one of paragraphs A-E, wherein the facial image portion (52) is a facial expression of the person, wherein the facial expression is a smile (521).

[0198] H. The method of any one of paragraphs A-E, wherein the facial image portion is defined by a first oral feature and a second oral feature associated with the facial image portion, the first oral feature and the second oral feature each selected from the group consisting of: oral soft tissue, gum, tooth, and combinations thereof.

[0199] I. The method of paragraph H, wherein the first oral feature comprises a first set of characteristics indicative of positive cosmetic dental attributes of the facial image portion (52), each positive cosmetic dental attribute assigned a positive value indicative that the first oral feature is healthy; wherein the second oral feature comprises a second set of characteristics indicative of negative cosmetic dental attributes of the facial image portion (52), wherein the first oral feature and the second oral feature are located in different portions of at least one region in the facial image portion (52).

[0200] J. The method of any one of paragraphs A-I, further comprising, prior to step (b), detecting the facial image portion (52) in the obtained digital image.

[0201] K. The method of any one of paragraphs A-J, wherein the analyzing in step (b) comprises filtering the facial image portion (52) to obtain one or more filtered feature maps comprising a first feature of interest and a second feature of interest, the first feature of interest and the second feature of interest each associated with the facial image portion (52); wherein the first feature of interest comprises a first set of characteristics indicative of positive attributes of the facial image portion (52), and the second feature of interest comprises a second set of characteristics indicative of negative attributes of the facial image portion (52), wherein the first feature of interest and the second feature of interest are located in different portions of at least one region in the facial image portion.

[0202] L. The method of any one of paragraphs A-K, wherein presenting the image description (53) comprises one of: displaying the image description (53) in the digital image (51) as a substitute text (531), displaying the image description (53) in the digital image (51) as a heat map (532), providing the image description (53) for presentation to the user in an audible manner, and combinations thereof.

[0203] M. The method according to paragraph L, wherein displaying the image description (53) in the digital image (51) as a heat map (532) comprises generating the heat map (532), wherein generating the heat map comprises overlaying a layer onto at least a portion of the digital image comprising the facial image portion, wherein the layer is a pixel map that identifies the at least one region comprising at least one of the analyzed positive and / or negative attributes.

[0204] N. The method according to any of paragraphs A-M, further comprising: receiving a request for additional information about the facial image portion (52); preferably, the additional information comprises providing information related to an increase in the attractiveness score.

[0205] O. The method according to any of paragraphs A-N, further comprising: receiving a request to share the image description (53) with a second user.

[0206] P. The method according to any of paragraphs A-O, wherein the image description (53) comprises a single face of a person depicted in the digital image (51).

[0207] Q. The method according to any of paragraphs A-P, wherein the image description (53) comprises a plurality of faces of a plurality of persons depicted in the digital image (51), and a separate image description (53) is presented for each face of the plurality of faces of the plurality of persons.

[0208] R. A method (400) for presenting a product recommendation to increase perceived attractiveness of a facial image portion, the method comprising:

[0209] transmitting a digital image (51) of at least one person, wherein the digital image comprises a facial image portion of the at least one person, wherein the facial image portion has both positive attributes and negative attributes;

[0210] receiving an image presentation that identifies at least one region in the facial image portion, the at least one region comprising at least one of the negative attributes analyzed using the method according to any of paragraphs A-Q;

[0211] presenting a product recommendation to increase perceived attractiveness of at least one of the analyzed positive and / or negative attributes.

[0212] S. A method of demonstrating efficacy of a customized oral care regimen in increasing perceived attractiveness of one or more oral features of at least one person depicted in a digital image, the method comprising:

[0213] obtaining (202) a digital image (51) of at least one person, wherein the digital image (51) comprises one or more oral features of the at least one person, wherein the one or more oral features have both positive attributes and negative attributes; wherein the one or more oral features are treated with a custom oral care regimen;

[0214] determining the perceived attractiveness of the one or more oral features using the method according to any of paragraphs A-Q.

[0215] T. A method for demonstrating the efficacy of a consumer product in increasing the perceived attractiveness of a facial image portion of at least one person depicted in a digital image, the method comprising:

[0216] i) determining the perceived attractiveness of a facial image portion (52) of a person in a first digital image (60) using the method according to any of paragraphs A-Q; wherein the facial image portion (52) in the first digital image (60) is untreated;

[0217] ii) obtaining a second digital image (61) of the person depicted in the first digital image (51), wherein the second digital image (61) comprises the facial image portion (52) of the person, wherein the facial image portion (52) in the second digital image (61) is treated with the consumer product for a treatment period;

[0218] iii) further determining the perceived attractiveness of the facial image portion (52) in the second digital image (61) using the method according to any of paragraphs A-Q;

[0219] iv) comparing the perceived attractiveness of the facial image portion (52) in the second digital image (61) with the perceived attractiveness of the facial image portion (52) in the first digital image (60).

[0220] U. The method according to paragraph T, further comprising: treating the facial image portion (52) in the first digital image (60) with a contrasting consumer product after step (i) and before step (ii) based on the treatment period of step (ii).

[0221] V. The method according to paragraph T or V, wherein the treatment period is two minutes to ten minutes, preferably two minutes to five minutes, more preferably three minutes.

[0222] W. The method according to any of paragraphs T-V, further comprising: repeating steps (iii) and (iv) over a period of time to track the increase in perceived attractiveness of the facial image portion; wherein the period of time is one day to three days, preferably three days to seven days, more preferably seven days to fourteen days.

[0223] X. A system (10) for determining a perceived attractiveness of a facial image portion of at least one person depicted in a digital image, the system (10) comprising:

[0224] a mobile application, the mobile application being compilable to run on a client computing system for obtaining a digital image comprising at least one oral feature of at least one person, wherein the digital image comprises a facial image portion of the at least one person, wherein the computing system is in communication with a content server configured to store the obtained digital image;

[0225] an image processing device (14) in communication with the mobile application over a network (100); wherein the image processing device (14) comprises a processor (14b) configured to generate, based on computer executable instructions stored in a memory (14a) to analyze the facial image portion, an attractiveness score indicative of a perceived attractiveness of the facial image portion based on the analyzed facial image portion in the obtained digital image; and further generate an image description identifying at least one region in the facial image portion indicative of the attractiveness score;

[0226] a display generation unit for generating a display item to display the image description indicative of the attractiveness score.

[0227] Y. A digital imaging method based on oral care for providing information to a graphical user interface to improve a perceived attractiveness of a facial image portion of at least one person depicted in a digital image, the digital imaging method based on oral care comprising:

[0228] implementing a graphical user interface (30) on a portable electronic device, the portable electronic device comprising a touch screen display or a display having an input device and an image obtaining device for obtaining a digital image comprising at least one oral feature of at least one person, wherein the digital image comprises a facial image portion of the at least one person;

[0229] on a first area of the display, displaying an image description (53) identifying at least one region in the facial image portion indicative of the attractiveness score;

[0230] on a second area of the display different from the first area, displaying a selectable icon (54) to receive a user input; and upon selection of the selectable icon (54) and by digitally coupling the device to a network interface of an image processing device, sending a request for additional information about the facial image portion (52), wherein the additional information is related to an improvement of the attractiveness score.

[0231] Z. The method according to paragraph R, further comprising:

[0232] Receive the selection corresponding to the product recommendations; and

[0233] Based on this choice, perform at least one of the following: (1) prepare products corresponding to the product recommendation for shipment, or (2) ship the products to the actual address.

[0234] Unless expressly excluded or otherwise limited, every reference cited herein, including any cross-references or related patents or patent applications, and any patent application or patent claiming priority to or benefiting from it, is incorporated herein by reference in its entirety. Reference to any reference is not an endorsement of it as prior art to any disclosed or protected art herein, nor is it an endorsement of any such invention, either on its own or in combination with any one or more references. Furthermore, where any meaning or definition of a term in this invention conflicts with any meaning or definition of the same term in referenced documents, the meaning or definition given to that term in this invention shall prevail.

[0235] While specific embodiments of the invention have been illustrated and described, it will be apparent to those skilled in the art that various other changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, it is intended that all such changes and modifications falling within the scope of the invention be covered in the appended claims.

Claims

1. A computer-implemented method for determining a perceived attractiveness of a facial image portion (52) of at least one person depicted in a digital image (51) based on oral care, the method comprising the steps of: a) obtaining (202) a digital image (51) comprising at least one oral feature of at least one person, wherein the digital image (51) comprises a facial image portion (52) of the at least one person, wherein the facial image portion (52) has both positive attributes as defined by pixel data of the digital image (51) and negative attributes as defined by pixel data of the digital image; b) analyzing (204) the facial image portion (52); c) generating (206), based on the analyzed facial image portion (52) in the obtained digital image (51), an attractiveness score (57) indicative of a perceived attractiveness of the facial image portion (52), wherein the attractiveness score is generated as a probability value indicative of a degree of attractiveness of the facial image portion of the at least one person depicted in the digital image to a group of people based on the positive attributes and the negative attributes of the facial image portion, wherein the probability value is determined by a model constructed by a machine learning system trained by a training data set, and wherein the training data set comprises: (i) a plurality of simulated images comprising facial image portions of simulated positive attributes and negative attributes; and (ii) associated class definitions based on the simulated positive attributes and negative attributes; d) further generating (208), based on the attractiveness score (57), an image description (53) identifying at least one region in the facial image portion (52); and e) presenting (210) the image description (53) to a user. After step (c), the attractiveness score (57) is presented to the user.

3. The method according to claim 1 or 2, wherein the image description (53) is further indicative of an influence of the identified region in the facial image portion (52) on the attractiveness score (57), wherein the image description (53) comprises a consumer-related description of a perceived attractiveness of the facial image portion.

4. The method according to claim 1 or 2, wherein the facial image portion (52) is selected from the group consisting of: facial skin, one or more oral features, one or more facial expressions, and combinations thereof, wherein the one or more oral features are selected from the group consisting of: oral soft tissue, gum, tooth, and combinations thereof, and wherein the one or more facial expressions comprise a smile (521).

5. The method according to claim 1 or 2, wherein the facial image portion is defined by a first oral feature and a second oral feature associated with the facial image portion, each of the first oral feature and the second oral feature being selected from the group consisting of: oral soft tissue, gum, tooth, and combinations thereof. ​ 2. The method of claim 1, further comprising: ​ ​ ​ ​ 6. The method of claim 5, wherein the first oral features comprise a first set of characteristics indicative of positive cosmetic dental attributes of the face image portion (52), each positive cosmetic dental attribute being assigned a positive value indicative that the first oral feature is healthy; wherein the second oral features comprise a second set of characteristics indicative of negative cosmetic dental attributes of the face image portion (52), wherein the first oral features and the second oral features are located in different portions of the at least one region in the face image portion (52).

7. The method of claim 1 or 2, further comprising: receiving a request for additional information about the face image portion (52), wherein the additional information comprises providing information related to an increase of the attractiveness score.

8. The method of claim 1 or 2, wherein the analyzing in step (b) comprises filtering the face image portion (52) to obtain one or more filtered feature maps comprising a first feature of interest and a second feature of interest, each associated with the face image portion (52), wherein the first feature of interest comprises a first set of characteristics indicative of positive attributes of the face image portion (52) and the second feature of interest comprises a second set of characteristics indicative of negative attributes of the face image portion (52), and wherein the first feature of interest and the second feature of interest are located in different portions of the at least one region in the face image portion.

9. The method of claim 1 or 2, wherein presenting the image description (53) comprises one of displaying the image description (53) in the digital image (51) as a replacement text (531), displaying the image description (53) in the digital image (51) as a heat map (532), providing the image description (53) for being presented audibly to the user, and combinations thereof.

10. The method of claim 9, wherein displaying the image description (53) in the digital image (51) as a heat map (532) comprises generating the heat map (532), wherein generating the heat map comprises overlaying a layer onto at least a portion of the digital image comprising the face image portion, wherein the layer is a pixel map identifying the at least one region comprising at least one of the positive and / or negative attributes of the analysis.

11. A method (400) for presenting product recommendations to increase perceived attractiveness of a face image portion, the method comprising: transmitting a digital image (51) of claim 1; receiving an image presentation identifying at least one region in the face image portion, the at least one region comprising at least one of the negative attributes analyzed using the method of any of the preceding claims; displaying alternative text indicative of an attractiveness score obtained in a step of receiving an image rendering identifying at least one region in the facial image portion, wherein the attractiveness score is generated as a probability value indicative of an attractiveness of the facial image portion of at least one person depicted in the digital image to a group of people based on positive attributes and negative attributes of the facial image portion, wherein the probability value is determined by a model constructed by a machine learning system trained by a training dataset, and wherein the training dataset comprises: (i) a plurality of simulated images comprising facial image portions of simulated positive attributes and negative attributes; and (ii) associated class definitions based on simulated positive attributes and negative attributes; and presenting a product recommendation to increase perceived attractiveness of at least one of the analyzed positive and / or negative attributes.

12. A method of demonstrating efficacy of a customized oral care regimen in increasing perceived attractiveness of one or more oral features of at least one person depicted in a digital image, the method comprising: obtaining (202) a digital image (51) of the at least one person, wherein the digital image (51) comprises one or more oral features of the at least one person, wherein the one or more oral features have both positive attributes and negative attributes, and wherein one or more oral features are treated with a customized oral care regimen; and determining perceived attractiveness of the one or more oral features using the method according to any one of the preceding claims.

13. A method for demonstrating efficacy of a consumer product in increasing perceived attractiveness of a facial image portion of at least one person depicted in a digital image, the method comprising: i) determining perceived attractiveness of a facial image portion (52) of a person in a first digital image (60) using the method according to any one of claims 1 to 12; wherein the facial image portion (52) in the first digital image (60) is untreated; ii) obtaining a second digital image (61) of the person depicted in the first digital image (60), wherein the second digital image (61) comprises the facial image portion (52) of the person, wherein the facial image portion (52) in the second digital image (61) is treated with a consumer product for a treatment period; iii) further determining perceived attractiveness of the facial image portion (52) in the second digital image (61) using the method according to any one of claims 1 to 12; and iv) comparing perceived attractiveness of the facial image portion (52) in the second digital image (61) to perceived attractiveness of the facial image portion (52) in the first digital image (60).

14. The method of claim 13, further comprising: based on the treatment period of step (ii), treating the facial image portion (52) in the first digital image (60) with a contrasting consumer product after step (i) and before step (ii), wherein the treatment period is from two minutes to ten minutes.

15. The method of claim 13 or 14, further comprising: repeating steps (iii) and (iv) over a period of time, wherein the period of time is from one day to fourteen days, to track the increase in the perceived appeal of the facial image portion.

16. The method of claim 13 or 14, wherein the perceived appeal of the facial image portion is teeth whiteness.

17. A system (10) for determining a perceived appeal of a facial image portion of at least one person depicted in a digital image, the system (10) comprising: a mobile application that is compilable to run on a client computing system for obtaining a digital image comprising at least one oral feature of at least one person, wherein the digital image includes a facial image portion of the at least one person, wherein the client computing system is in communication with a content server configured to store the obtained digital image; an image processing device (14) in communication with the mobile application over a network (100); wherein the image processing device (14) includes a processor (14b) configured to: generate, based on computer executable instructions stored in a memory (14a) to analyze the facial image portion, an appeal score indicative of a perceived appeal of the facial image portion based on the analyzed facial image portion in the obtained digital image; and further generate an image description that identifies at least one region in the facial image portion that is indicative of the appeal score; and a display generation unit for generating a display item to display the image description indicative of the appeal score; wherein the appeal score is generated as a probability value that is indicative of a degree of appeal of the facial image portion of the at least one person depicted in the digital image to a group of people based on positive attributes and negative attributes of the facial image portion, wherein the probability value is determined by a model that is constructed by a machine learning system trained by a training dataset, and wherein the training dataset includes: (i) a plurality of simulated images of facial image portions that include simulated positive attributes and negative attributes; and (ii) associated class definitions based on the simulated positive attributes and negative attributes.

18. A digital imaging method based on oral care for providing information to a graphical user interface to increase a perceived appeal of a facial image portion of at least one person depicted in a digital image, the digital imaging method based on oral care comprising: implementing a graphical user interface (30) on a portable electronic device, the portable electronic device including a touchscreen display or display having an input device and an image obtaining device for obtaining a digital image comprising at least one oral feature of at least one person, wherein the digital image includes a facial image portion of the at least one person; on a first region of the display, display an image depiction (53) that identifies at least one region in a facial image portion that is indicative of an appeal score, wherein the appeal score is generated as a probability value that indicates a degree of appeal of the facial image portion of at least one person depicted in the digital image to a group of people based on positive attributes and negative attributes of the facial image portion, wherein the probability value is determined by a model that is structured by a machine learning system trained by a training data set, and wherein the training data set includes: (i) a plurality of simulated images that include facial image portions of simulated positive attributes and negative attributes; and (ii) associated class definitions based on the simulated positive attributes and negative attributes; on a second region of the display that is different from the first region, display a selectable icon (54) to receive user input; and after selection of the selectable icon (54), and by digitally coupling the device to a network interface of an image processing apparatus, send a request for additional information about the facial image portion (52), wherein the additional information is related to an increase in the appeal score.

19. The method of claim 18, further comprising: receiving a selection corresponding to a product recommendation; and based on the selection, performing at least one of (1) preparing a product corresponding to the product recommendation for shipment, or (2) shipping the product to an actual address.

Images