Loop diagnostic system, apparatus and method

Abstract

SYSTEM, APPARATUS, AND METHOD FOR LOOP DIAGNOSTIC TECHNOLOGY The present invention relates to a system comprising: at least one server device; and a mobile user device that includes processing circuitry configured to run an application that receives user information input concerning a plurality of features of at least one user's hair, including at least one image of the user's hair; and the processing circuitry of the at least one server device receives the user information input and determines a loop pattern of the user based on the information received. Figure for abstract: none

Description

Title of the invention: SYSTEM, APPARATUS AND METHOD FOR loop diagnostics CONTEXT

[0001] Domain

[0002] This disclosure generally relates to a system, apparatus, and method for diagnosing a user's curly hair type, providing a recommendation on a product or treatment for the user, and using user and social feedback to reinforce both the curl diagnosis and the product recommendation. DETAILED DESCRIPTION OF THE IMPLEMENTATION METHODS

[0003] In the field of hair care and maintenance, there are many types of curly hair, and consumers with curly hair want the best and most appropriate care for their specific curl type, conformation, and shape. Many different curl pattern charts exist, and it can sometimes be difficult to self-diagnose the condition, type, and necessary care of one's own hair. Choosing the wrong chart can lead to selecting the wrong product, which can result in poor or suboptimal results. Therefore, there is a need for an application that can offer a method for diagnosing a customer's curly hair and providing a recommendation on a product or treatment to deliver the desired result.

[0004] The embodiments below describe a system, device, or application (“App”) that can receive as input at least one curly hair pattern from a user based on an image of the user's hair, and diagnose the curl pattern, the degree of shine of the hair, the color, and the texture. Based on this diagnosis, the application will recommend a specific product or regimen of products from a designated brand and the ability to purchase it on-site.

[0005] There is also an opportunity for the ability to adapt the diagnosis and recommendations based on changes in hair and consumer needs. One month, a consumer might receive a diagnosis based on their hair being dry, and perhaps their hair is changing and now it is oily. With this new information, they can rely on the application to redirect the diagnosis and recommendation according to their hair's changing condition.

[0006] The present invention relates to a system comprising:

[0007] at least one server device; and

[0008] a mobile user device that includes processing circuitry configured to run an application that receives user information input concerning a plurality of features of at least one of a user's hairs, including at least one image of the user's hair; and

[0009] The processing circuitry of at least one server device receives user information input and determines a user loop pattern based on the information received.

[0010] According to embodiments, the system according to the invention comprises one or more of the following features, in all technically possible combinations: - the user information input further includes a textual description concerning the plurality of characteristics of the user's hair, - the textual description concerns at least one current process for treating the user's hair and a current state of the user's hair, - the textual description concerns a first portion of the user's hair having one type of curl and another portion of the user's hair having a second type of curl, - The processing circuitry of at least one server device determines the user's loop pattern by inputting the user's hair image into a neural network that is trained to identify loop patterns in images, - The processing circuitry of at least one server device determines the user's loop pattern by identifying pixels in the user's hair image that correspond to the user's hair and performing linear regression analysis to determine the hair shape appearing in the image,

[0011] The present invention also relates to a system comprising:

[0012] a loop identification component including circuitry configured to generate a prediction score per pixel for the presence or absence of hair loops and to predict the presence or absence of a specific loop pattern in an image using one or more convolutional neural network image classifiers.

[0013] According to some embodiments, the system comprises one or more of the following features, depending on all of the following features: - the generation of the predicted score for the presence or absence of the specific loop pattern in the image includes the generation of an indicative score of the presence or absence of a combination of one or more of a twisted hair pattern, a curly hair pattern, a frizzy hair pattern, a wavy hair pattern or a straight pattern. - The predicted score for the presence or absence of the specific curl pattern in the image includes a score indicating the presence or absence of a twisted hair pattern, a curly hair pattern, a frizzy hair pattern, a wavy hair pattern, or a straight hair pattern. - the system also includes:

[0014] a loop appreciation component including circuitry configured to generate a user-selectable menu that allows a user to choose from a plurality of hair features and loop pattern images and to generate user hair appreciation information in response to one or more inputs associated with at least one prediction score for the presence or absence of hair loops and to predict a score for the presence or absence of a specific loop pattern.

[0015] [SYSTEM]

[0016] [Fig. 1] Fig. 1 shows a complete system 100 according to one embodiment. The basic components required are a user device 101 (such as a smartphone) and one or more server devices 102 (such as a cloud platform). A hardware description of these components will be provided later. The smartphone is shown as including a smartphone application (“app”).

[0017] The use of the smartphone application itself implies that the user effectively provides input by making selections which lead to the loop diagnostic functionality and product recommendations.

[0018] The smartphone application also ensures interactive communication with the cloud platform. For example, the smartphone application can receive the selection of relevant appearances described above, and can also provide direct user feedback on appearances previously sent by the cloud platform. It can also notify the cloud platform about the colors and recipes actually selected by the user and distributed by the distributor. Such feedback can be used in the form of machine learning by the cloud platform to improve the algorithms used by the cloud platform.

[0019] [Fig. 2] [Fig. 2] is a more detailed functional diagram illustrating an example of a user device 20 according to certain embodiments of this disclosure. In some embodiments, the user device 20 may be a smartphone. However, those skilled in the art will appreciate that the features described herein can be adapted for implementation on other devices (e.g., a laptop, a tablet, a server, an e-reader, a camera, a navigation device, etc.). The example user device 20 in [Fig. 9] includes a controller 110 and a wireless communication processor 102 connected to an antenna 101. A speaker 104 and a microphone 105 are connected to a voice processor 103.

[0020] The controller 110 is an example of the control unit 21 shown in [Fig. 1] and may include one or more central processing units (CPUs), and may control each element of the user device 20 to perform functions related to communication control, audio signal processing, audio signal processing control, still and moving image processing and control, and other types of signal processing. The controller 110 may perform these functions by executing instructions stored in a memory 150. Alternatively, or in addition to the local storage in the memory 150, the functions may be performed using instructions stored on an external device accessed over a network or on a non-transient computer-readable medium. As described above in relation to [Fig. 1], the controller 110 may perform these functions by executing instructions stored in a memory 150.l], the controller 110 can execute instructions enabling the controller 110 to function as the display control unit 211, the operations management unit 212 and the game management unit 213 schematically represented in [Fig. l]. .

[0021] Memory 150 is an example of the storage unit 22 shown in [Fig. 1] and includes, but is not limited to, read-only memory (ROM), random-access memory (RAM), or a memory array comprising a combination of volatile and non-volatile memory units. Memory 150 can be used as working memory by the controller 110 while executing the processes and algorithms of this disclosure. In addition, memory 150 can be used for long-term storage of, for example, image data and related information.

[0022] The user device 20 includes a command line CL and a data line DL as internal communication bus lines. Command data to / from the controller 110 can be transmitted via the command line CL. The data line DL can be used for transmitting voice data, display data, etc.

[0023] The antenna 101 transmits / receives electromagnetic wave signals between base stations to perform radio communications, such as various forms of cellular phone communication. The wireless communication processor 102 controls the communication between the user device 20 and other external devices via the antenna 101. For example, the wireless communication processor 102 can control the communication between base stations for cellular phone communication.

[0024] The loudspeaker 104 emits an audio signal corresponding to the audio data provided by the voice processor 103. The microphone 105 detects ambient audio and converts the detected audio into an audio signal. The audio signal can then be transmitted to the voice processor 103 for further processing. The voice processor 103 demodulates and / or decodes the audio data read from memory 150 or the data audio received by the wireless communication processor 102 and / or a short-range wireless communication processor 107. In addition, the voice processor 103 can decode the audio signals obtained by the microphone 105.

[0025] The example user device 20 may also include a display 120, a touch panel 130, an operation key 140, and a short-range communication processor 107 connected to an antenna 106. The display 120 may be a liquid crystal display (LCD), an organic electroluminescent display panel, or another display technology. In addition to displaying still and moving image data, the display 120 may display operational inputs, such as numbers or icons, that can be used to control the user device 20. The display 120 may also display a graphical user interface allowing a user to control aspects of the user device 20 and / or other devices.In addition, the display 120 can display characters and images received by the user device 20 and / or stored in memory 150, or accessed from an external device on a network. For example, the user device 20 can access a network such as the Internet and display text and / or images transmitted from a web server.

[0026] The touch panel 130 may include a physical touch panel display and a touch panel driver. The touch panel 130 may include one or more touch sensors for detecting an input operation on an operating surface of the touch panel display. The touch panel 130 also detects a touch shape and a touch area; here, the term "touch operation" refers to an input operation performed by touching an operating surface of the touch panel display with an instruction object, such as a finger, thumb, or stylus-type instrument.In the event that a stylus or similar is used in a touch operation, the stylus may include a conductive material at least at the tip of the stylus so that the sensors included in the touch panel 130 can detect when the stylus approaches / touches the operating surface of the touch panel display (as in the event that a finger is used for the touch operation).

[0027] One or more of the display 120 and the touch panel 130 are examples of the touch panel display 25 shown schematically in [Fig.1] and described above.

[0028] In certain aspects of this disclosure, the touch panel 130 may be arranged alongside the display 120 (for example, laminated) or may be integrated into the display 120. For simplicity, this disclosure assumes that the touch panel 130 is formed integrally with the display 120 and, consequently, the examples discussed herein may describe touch operations performed on the surface of the display 120 rather than on the touch panel 130. However, those skilled in the art will appreciate that this is not limiting.

[0029] For simplicity, this disclosure assumes that the touch panel 130 is a capacitive touch panel technology. However, it should be noted that aspects of this disclosure can readily be applied to other types of touch panels (e.g., resistance touch panels) with alternative structures. In certain aspects of this disclosure, the touch panel 130 may include transparent electrode touch sensors arranged in the XY direction on the glass surface of the transparent sensor.

[0030] The touch panel driver can be included in the touch panel 130 for control processing related to the touch panel 130, such as scanning control. For example, the touch panel driver can scan each sensor in a transparent electrostatic capacitance electrode pattern in the X and Y directions and detect the electrostatic capacitance value of each sensor to determine when a touch operation is performed. The touch panel driver can output a coordinate and a corresponding electrostatic capacitance value for each sensor. The touch panel driver can also output a sensor identifier that can be mapped to a coordinate on the touch panel screen.Furthermore, the touch panel driver and touch panel sensors can detect when an instruction object, such as a finger, is within a predetermined distance of an operating surface of the touch panel display. In other words, the instruction object does not necessarily need to be in direct contact with the operating surface of the touch panel display for touch sensors to detect the instruction object and perform the processing described here. For example, in some embodiments, the touch panel 130 can detect the position of a user's finger around an edge of the display panel 120 (for example, by grasping a protective case surrounding the display / touch panel). Signals can be transmitted by the touch panel driver, e.g.in response to the detection of a touch operation, in response to a request from another element based on a timed data exchange, etc.

[0031] The touch panel 130 and the display 120 can be surrounded by a protective housing, which can also enclose the other elements included in the user device 20. In some embodiments, the position of the user's fingers on the protective housing (but not directly on the surface of the display 120) can be detected by the sensors of the touch panel 130. Consequently, the controller 110 can perform the display control processing described herein based on the detected position of the user's fingers gripping the housing. For example, an element in an interface can be moved to a new location within the interface (e.g., closer to one or more fingers) depending on the detected finger position.

[0032] Furthermore, in certain embodiments, the controller 110 can be configured to detect which hand is holding the user device 20, based on the detected finger position. For example, the sensors on the touch panel 130 can detect a plurality of fingers on the left side of the user device 20 (for example, on an edge of the display 120 or on the protective housing), and detect a single finger on the right side of the user device 20. In this example scenario, the controller 110 can determine that the user is holding the user device 20 with their right hand, because the detected grip pattern corresponds to an expected pattern when the user device 20 is held only with the right hand.

[0033] The operation key 140 may include one or more external buttons or similar control elements, which can generate an operating signal based on user-detected input. In addition to the outputs of the touch panel 130, these operation signals can be provided to the controller 110 to perform the associated processing and control. In certain aspects of this disclosure, the processing and / or functions associated with external buttons and the like may be performed by the controller 110 in response to an input operation on the display screen of the touch panel 130 rather than on the external button, key, etc. In this way, external buttons on the user device 20 can be eliminated instead of performing inputs via touch operations, thereby improving water resistance.

[0034] The antenna 106 can transmit / receive electromagnetic wave signals to / from other external devices, and the short-range wireless communication processor 107 can control the wireless communication between the other external devices. Bluetooth, IEEE 802.11, and Near Field Communication (NFC) are non-limiting examples of wireless communication protocols that can be used for inter-device communication via the short-range wireless communication processor 107.

[0035] The user device 20 may include a motion detector 108. The motion sensor 108 may detect motion features (i.e., one or more movements) of the user device 20. For example, the motion sensor 108 may include an accelerometer to detect acceleration, a gyroscope to detect angular velocity, a geomagnetic sensor to detect direction, a geolocation sensor to detect location, etc., or a combination thereof to detect the movement of the user device 20. In some embodiments, the motion sensor 108 may generate a detection signal that includes data representing the detected movement. For example, the motion sensor 108 may determine a number of distinct movements within a single movement (e.g., from the beginning of the series of movements to The detection signal can be triggered by: a stoppage within a predetermined time interval, etc.), a number of physical shocks to the user device 20 (e.g., a jolt, a bump, etc., of the electronic device), a speed and / or acceleration of movement (instantaneous and / or temporal), or other motion features. The detected motion features can be included in the generated detection signal. The detection signal can be transmitted, for example, to the controller 110, whereby further processing can be performed based on the data included in the detection signal. The motion sensor 108 can operate in conjunction with a Global Positioning System (GPS) section 160. The GPS section 160 detects the current position of the terminal device 100. The current position information detected by the GPS section 160 is transmitted to the controller 110.An antenna 161 is connected to the GPS section 160 to receive and transmit signals to and from a GPS satellite.

[0036] The user device 20 may include a camera section 109, which comprises a lens and a shutter for capturing photographs of the surroundings of the user device 20. In one embodiment, the camera section 109 captures the environment on the opposite side of the user device 20 from the user. The images of the captured photographs may be displayed on the display panel 120. A memory section saves the captured photographs. The memory section may reside within the camera section 109 or be part of the memory 150. The camera section 109 may be a separate feature attached to the user device 20 or be an integrated camera feature.

[0037] [Fig. 3] Subsequently, a hardware description of one or more server devices 102 according to exemplary embodiments is given with reference to [Fig. 3]. In [Fig. 3], the device includes a CPU processor X00 that executes the processes described above / below. The process data and instructions can be stored in memory X02. These processes and instructions can also be stored on a storage medium X04 such as a hard disk drive (HDD) or portable storage medium, or can be stored remotely. Furthermore, the claimed advances are not limited by the form of the computer-readable media on which the instructions of the inventive process are stored. For example, the instructions can be stored on CD, DVD, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or any other information processing device with which the device communicates, such as a server or a computer.

[0038] Furthermore, the claimed advancements can be provided as a utility application, background daemon, or component of an operating system, or a combination thereof, running jointly with an X00 CPU and a system operating systems such as Microsoft Windows 7, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to a person skilled in the art.

[0039] The hardware components for implementing the [device] can be implemented using various circuitry components known to those skilled in the art. For example, CPU X00 could be a Xenon or Core processor from Intel of America or an Opteron processor from AMD of America, or other types of processors that would be recognized by those skilled in the art. Alternatively, CPU X00 could be implemented on an FPGA, ASIC, PLD, or using discrete logic circuits, as would be recognized by those skilled in the art. Furthermore, CPU X00 could be implemented as multiple processors working in parallel to execute the instructions of the inventive processes described above.

[0040] The [device] in Figure X also includes a network controller X06, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with the XX network. As can be seen, the XX network can be a public network, such as the Internet, or a private network such as a LAN or WAN, or any combination thereof, and can also include PSTN or ISDN subnets. The XX network can also be wired, such as an Ethernet network, or it can be wireless, such as a cellular network including EDGE, 3G, and 4G wireless cellular systems. The wireless network can also be Wi-Fi, Bluetooth, or any other known form of wireless communication.

[0041] The [device] further includes an X08 display controller, such as an NVIDIA GeForce GTX or Quadro graphics adapter from NVIDIA Corporation of America, for interfacing with the X10 display, such as a Hewlett Packard HPL2445w LCD monitor. A universal I / O interface X12 interfaces with an X14 keyboard and / or mouse, as well as an X16 touchscreen panel on or separate from the X10 display. The universal I / O interface also connects to various X18 peripherals, including printers and scanners, such as a Hewlett Packard OfficeJet or DeskJet.

[0042] An X20 sound controller is also supplied in the [device], such as Creative's Sound Blaster X-Fi Titanium, to interface with the X22 speakers / microphone and thus provide sounds and / or music.

[0043] The universal storage controller X24 connects the storage media disk X04 to the communication bus X26, which can be an ISA, EISA, VESA, PCI, or similar, to interconnect all the components of the [device]. A description of the general features and functionalities of the display X10, the keyboard and / or mouse XI4, as well as the display controller X08, the storage controller X24, the network controller X06, the sound controller X20, and the universal I / O interface X12, is omitted here for the sake of brevity, as these features are known.

[0044] The circuit element examples described in the context of this disclosure may be replaced by other elements and structured differently from the examples provided here. Furthermore, the circuitry configured to perform the functions described herein may be implemented in multi-circuit units (e.g., chips), or the features may be combined in circuitry on a single chip set.

[0045] [Fig. 3B] The functions and features described herein can also be performed by various distributed components of a system. For example, one or more processors can perform these system functions, with the processors distributed across multiple components communicating within a network. The distributed components can include one or more client and server machines, which can share processing, as shown in [Fig. 3B], in addition to various human interface and communication devices (e.g., display monitors, smartphones, tablets, personal digital assistants (PDAs)). The network can be a private network, such as a LAN or WAN, or it can be a public network, such as the Internet. Input into the system can be received via direct user input and received remotely in real time or as a discontinuous process.Furthermore, some implementations may be performed on modules or hardware that are not identical to those described. Therefore, other implementations fall within the scope that can be claimed.

[0046] The hardware description described above is a non-limiting example of a corresponding structure for the execution of the functionality described here.

[0047] [USER DATA COLLECTION]

[0048] L Collection of user information on the current state of the hair; Self-described information

[0049] [Fig.4][Fig.5] Figures 4-5 show the types of descriptive information that are Information received by the application from the user regarding their current hair condition. This information may include image and / or descriptive details.

[0050] For example, as shown in [Fig.4], as personal information, the application collects one or more pieces of information from among the user's name(s), age, ethnicity, location and contact.

[0051] As shown in [Fig. 5], the user can also provide information on their current hair care habits, which can be referred to as their "current hair care history". This information can be seen to include one or more of the following.

[0052] Whether or not the user uses heat to treat their hair

[0053] The frequency of use of shampoos and / or conditioning products

[0054] If the user uses a silk press treatment

[0055] If the user leaves their hair straightened

[0056] If the user's hair is treated with color

[0057] If the user's hair is chemically treated.

[0058] [Fig.6] As shown in [Fig.6], the user can also provide information on the current perceived condition of their hair. For example, the user can self-describe their hair as fine, medium, thick, oily, dry, brittle, porous, or curly.

[0059] The user can also provide information indicating whether their hair has different types of curls or characteristics in different areas. For example, while one type of curl may be present in the most visible parts of the hair, another type of curl may be present in a less visible area such as the nape of the neck. A conventional hair imaging system based on 1TA would not take this type of information into account and would likely have to be provided as user input.

[0060] [Fig. 7A] [Fig. 7A] shows that a user can take one or more "selfie" pictures of their hair. Specifically, the user can take a selfie, a 360° selfie, or a series of photos from different angles using the smartphone's camera.

[0061] [Fig. 7B] In a preferred embodiment, as shown in [Fig. 7B], the user must take a selfie of the portion of their hair that embodies the curl pattern, such as the side of their hair, or if possible, a view from the back of their hair. The image is preferably taken against a white background.

[0062] [APPEARANCE INFORMATION / DESIRED BENEFITS]

[0063] [Fig.8] Fig.8 shows the types of descriptive information that are received by the application from the user with regard to their desired (target) appearance.

[0064] The user can provide their target appearance using words and / or images. As shown in [Fig. 8], the user can enter descriptive terms regarding the desired benefits, such as achieving more definition, lengthening, hydration, frizz control, damage repair, shine, softness, or split-end reduction.

[0065] [Fig. 9] As shown in [Fig. 9], the user can enter one or more indicative images of their desired "final appearance." This can be done in conjunction with, or as an alternative to, descriptive terms provided by the user. For example, the user can select images taken from the Internet. The images can be provided by a plurality of images displayed to the user directly within the application itself. These images can be pre-associated text descriptors, such as representing an appearance with one or more

[0066] Alternatively, the images may not be pre-associated with textual descriptors, and an image analysis may be performed on the image to determine the characteristics of the model's hair in the image.

[0067] [LOOP DIAGNOSIS]

[0068] Once the descriptive and image information has been collected from the user, the application will initiate a loop diagnosis. Part of the loop diagnosis involves performing an image analysis to determine the user's loop type or pattern, as well as other attributes such as texture, damage level, gloss level, dryness, and color.

[0069] [Fig. 10] Figure 10 shows a graph of different types of loop patterns according to one embodiment. In this example, there are eight different types of loop patterns, which can represent wavy, curly, or twisted loop patterns. There are various image analysis methods.

[0070] One method involves detecting features in an image that indicate a specific loop pattern. For example, an angle in the hair pattern can be detected. Due to the contrast of the user's hair in the image against a white background, pixels of a certain color are detected, and the average loop is plotted on a 2D scatter plot. To facilitate this step, the white balance of the image can be optimized so that the white balance of the image makes it easier to detect the black / white contrast. The system can also automatically detect the percentage of black versus white within the white background.

[0071] [Fig. 11] A method corresponding to this type of detection is shown in [Fig. 11]. In step 1101, the system determines pixels of a predetermined color, which can be any color other than white, assuming that the image is taken against a white background. In step 1102, the mean angle and the loop are measured.

[0072] The mean angle and hair curl are measured using linear regression. Specifically, pixels in a "maximum expected" region are marked in red and fitted with polynomial regression (degree = 2). Using a pair of heuristics to determine which side of the parabola to choose, measurements are taken from one end of the parabola to the base. The heuristics are based on the side of the parabola with more data points and the concavity of the curve. This technique is described in more detail in US Patent No. 10,929,993.

[0073] Another method involves using deep learning or machine learning to train a model to determine a loop pattern in images. In this embodiment, the system implements one or more convolutional neural networks (CNNs), whose models can be trained using open datasets or crowdsourced datasets, as explained below. Other Machine learning techniques can be used in conjunction with the present invention, including, but not limited to, decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, aggregation, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparsity dictionary learning, genetic algorithms, rule-based machine learning, and learning classifiers. Additional techniques described in US Patent No. 8,442,321, US Patent No. 9,015,083, US Patent No. 9,536,293, US Patent No. 9,324,022, and US Publication PG No. 2014 / 0376819 A1L can be used with the present invention.In the descriptions that follow, it will be assumed that the machine learning logic implements a convolutional neural network, although the present invention is not limited to this. Those with expertise in artificial intelligence will recognize many techniques that can be used in conjunction with the present invention without departing from its intended spirit and scope.

[0074] [Fig. 12] [Fig. 12] is a flowchart illustrating an example of interaction with an embodiment of the invention. The interaction shown in [Fig. 12] is not intended to be limiting. The description in [Fig. 12] is intended to illustrate the functionality of the configuration shown in [Fig. 1]. Other features of the invention, beyond those described with reference to [Fig. 12], will be discussed below.

[0075] In operation 1210, the captured selfie image of the user is received

[0076] In operation 1220, image analysis and machine learning are performed to analyze the user's skin from the images. The system can perform analyses that determine, among other things, curl pattern type, hair texture, level of hair damage, level of hair shine, hair dryness, and hair color. Other hair conditions can be determined by the system. Further details of the analyses are provided below. Once the analyses are completed, as determined in operation 1225, process 1200 can proceed to operation 1230, whereby the analysis results and the prescribed regimen (products and routines) and / or regimen updates are sent to the user via an application interface.

[0077] In operation 1225, it is determined whether the analysis is complete and, in response to a positive determination, process 1200 can proceed to operation 1230, whereupon the application sends a recommended diet or diet updates to the user in operation 1230. The user can follow the diet as indicated in operation 1235 and, in operation 1240, it is determined whether a new interval has begun. If so, process 1200 repeats from operation 1210. The system can access calendars and timers (as well as GPS) on the client device and network-accessible calendars. Therefore, once a week, for example, the application can remind the user to take a picture of their hair—that is, remind them of the new interval. Over time, the system can determine from the images taken at each interval whether the recommended regimen is working and, if not, the system can revise the regimen—for example, change a product, recommend other lifestyle changes, schedule an appointment with a specialist, and so on.

[0078] [Fig. 13] [Fig. 13] is a data flowchart between an exemplary client interface 1340 (i.e., the application) and services of a service platform 1320. It should be noted that, in [Fig. 13], as illustrated in [Fig. 13], the machine learning logic 1354 may include a hair analyzer 1330, a facial appearance progression generator 335, and a diet recommendation generator 1350, and may be communicatively coupled to a user account database 310 and a product database 320. The machine learning logic 1354 may train and use machine learning models 1370 to recommend diets and track the user's progress under the diet.As the machine learning specialist will attest, training may involve selecting a set of features—for example, a curl pattern type, hair texture, hair damage level, hair shine level, hair dryness, and hair color—and assigning labels to image data that reflect the presence or prominence of these features. Labeling can be done by a subject matter expert or, as explained below, through crowdsourced data. Taking the assigned labels as real-world data, the machine learning logic can configure the models to predict the degree to which features are present in a test image, which may change over time.The present invention is not limited to a particular model representation, which may include binary models, multiclass classification models, regression models, etc.

[0079] The example user account database 1310 securely contains the data of all users of the system 100. This includes user profile data, current and past user photos 352 for each user, current and past hair analyses 358 for each user, current and past product recommendations 362 and current and past routine recommendations 364 for each user.

[0080] An example of a product database 1320 contains data on various products that can be used in a diet. The product database 1320 can contain records reflecting product names, active and inactive ingredients, label information, recommended uses, etc. In some embodiments, as illustrated as product entry 354, the user (and other users of the system) can provide feedback on different products and can enter products that are not already in the product database 1320. The present invention is not limited to particular products that can be entered into the product database 1320.

[0081] The hair analyzer 1330 is designed or otherwise configured to classify different hair analyzer conditions or artifacts from a user's hair image using machine learning techniques on the models 1370. In some embodiments, photographic images 352 of a user's hair are provided to the hair analyzer 1330 for analysis. The hair analyzer 330 can implement image preprocessing mechanisms that include cropping, rotating, registering, and filtering the input images before analysis. After such preprocessing, the hair analyzer 1330 can apply the models 1370 to the input image to locate, identify, and classify features of the user's hair.

[0082] A hair appearance progression generator 1335 can operate on the user's hair images to illustrate how the user's face would appear at some point in the future. Such progression can be in age, for which age progression techniques can be deployed, or can be in appearance following adherence to a regimen. A progression image 356 can be provided to the user via the client interface 1340.

[0083] The regimen recommendation generator 1350 can operate on the analysis results 358 obtained from the hair analyzer 1330 by prescribing a regimen to the user. The models 1370 can be trained to predict which products and routines (treatment, cosmetic, and lifestyle recommendations, etc.) would be effective in achieving the user's goal with respect to hair characteristics identified in the hair analysis. The regimen recommendation generator 1340 can format the analysis results 358 from the hair analyzer 1330 as a query in, for example, the product database 1320 based on the knowledge encoded on the models 1370. In response, the product database 1320 can return product data and metadata 1366, and product recommendations 362 and routine recommendations 364 can be provided to the client interface 1340.

[0084] As indicated above, training of 1370 models can be accomplished by labeling image data by an expert. However, instead of an expert, some embodiments of the invention exploit participatory data as training data.

[0085] [Fig. 14] [Fig. 14] is a diagram of one such embodiment of the invention. During training, users 410 are presented with a set of training images 420 and are asked to characterize curl patterns and / or hair features. In one embodiment, the curl pattern scale shown in [Fig. 10] is used, which allows users to classify the type of curl pattern. For example, each user 410 (over time) is presented with a large number of images and reviews a set of questions concerning the features of the person in the image. Using the scale, each user 410 is prompted to select one of the curl types. The answers to the questions can serve as labels used to train the machine language logic 1340.

[0086] [Fig. 15] With reference to [Fig. 15], an example 1500 is illustrated in which the application presents a photograph of a model 1510 and asks the user to determine which type of loop, among a plurality of loop types 1520, is shown in the image.

[0087] [Fig. 16] Fig. 16 illustrates an example of a trial operation in accordance with the participatory training mentioned above. A test image 710, i.e., a user's own image, can be presented to the machine learning logic 154, which analyzes the image according to the models trained on the participatory data 720. As illustrated in the figure, the machine learning logic 1340 estimates that 80% of respondents would classify the user's loop pattern as a type 7 as indicated in 722. Therefore, the machine learning logic 1340 can use loop type 7 as input data when making recommendations, as will be described below.

[0088] User text descriptors can be used to assist loop diagnosis as additional input to the model. For example, including a user's text describing their own hair in combination with visual features in the image of their hair will help to weight the probability of a loop pattern identified in the image.

[0089] The machine learning system described above may further include a loop identification component comprising circuitry configured to generate a per-pixel prediction score for the presence or absence of hair loops and to predict a score for the presence or absence of a specific loop pattern in an image using one or more neural network image classifiers Convolutional. In this process, generating the predicted score for the presence or absence of the specific loop pattern in the image includes generating an indicative score for the presence or absence of a combination of one or more of the following: a twisted hair pattern, a curly hair pattern, a kinky hair pattern, a wavy hair pattern, or a straight pattern. The predicted score for the presence or absence of the specific loop pattern in the image may include a score indicating the presence or absence of a twisted hair pattern, a curly hair pattern, a kinky hair pattern, a wavy hair pattern, or a straight pattern. The predicted score for the presence or absence of the specific loop pattern in the image includes an indicative score for the presence or absence of a loop pattern from a plurality of predetermined loop patterns on a loop scale.

[0090] A loop evaluation component may also be provided which includes circuitry configured to generate a user-selectable menu that allows the user to choose from a plurality of hair features and loop pattern images and to generate user hair appreciation information in response to one or more inputs associated with at least one of the prediction score for the presence or absence of hair loops and the prediction score for the presence or absence of a specific loop pattern.

[0091] A method can also be proposed for applying a convolutional neural network image classifier to a user image in order to obtain per-pixel prediction scores for the presence or absence of hair curls and hair curl types, and to generate a virtual representation of a portion of the user image and a predicted hair curl type based on the prediction scores for the presence or absence of hair curls and hair curl types.

[0092] A computer-implemented method can also be provided for training a neural network for hair curl detection. This method includes collecting a set of digital images of twisted hair patterns, curly hair patterns, frizzy hair patterns, or wavy hair patterns from a database;

[0093] applying one or more transformations to each digital hair image to create a modified set of digital hair images; creating a first training set comprising the collected set of digital hair images, the modified set of digital hair images, and a set of digital hair-free loop pattern images; training a neural network in a first stage using the first training set; creating a second training set for a second training stage comprising the first training set and digital hair-free loop images that are incorrectly detected as hair loop pattern images after the first training stage; and training the neural network in a second stage using the second training set.

[0094] [LOOP OR PRODUCT TREATMENT RECOMMENDATION]

[0095] While the above process describes how the system performs a loop diagnosis, the loop diagnosis is further combined with an input of the appearance or benefit desired by the user to obtain a product or treatment recommendation.

[0096] As shown in [Fig.8], the user enters the desired benefit he is looking for in the application, or as shown in [Fig.9], the user has made a selection of an "appearance" he is looking for.

[0097] [Fig. 17] As shown in [Fig. 17], a functional component called recommendation generator 1710 takes the loop diagnostic input 1720 and the benefit desired by the user 1730, and provides a recommendation 1740 as output.

[0098] Different methods for generating a recommendation are described below. Table consultation

[0099] As a first method, a consultation table can be used, which includes predetermined combinations of loop diagnosis and desired benefit in association with a predetermined output of a recommendation for a product or treatment.

[0100] [Fig. 18A] [Fig. 18A] shows an example of an entry in a lookup table as described above. A matrix-style table is shown where curl diagnosis possibilities populate one lookup axis and user-desired benefits populate the opposite lookup axis. Based on the inputs received by the recommendation generator 1710, an intersection of the inputs will lead to the recommendation to be generated. Although this example is simplified, the curl diagnosis could further include other features of the user's hair assessment as described above, and the user-desired benefits could also include other features. An increase in the number of feature types contained in the input data will obviously require an expanded lookup table. The lookup table type is not limited to the type shown in [Fig. 18B].The table itself can be generated manually from expert data.

[0101] [Fig. 18B][Fig. 18C] For example, Figures 18B and 18C show expert information that maps a user's curl type to other characteristics, pain points, and treatment recommendations. How this information can be used to supplement the lookup table in [Fig. 18A] is obvious. For reference, a "zigzag" hair type means that the strands form a zigzag, not a curl or wave. This is referred to as "Twisted" hair is characterized by tightly twisted strands. This type of hair is very versatile but can be fragile, especially if the strands are fine, as they may have a thin outer layer. "Curly" hair strands clump together and coil around themselves in spirals or looser waves. This texture needs plenty of moisture to maintain a defined pattern, but a little frizz can add character. Hair is "wavy" when the strands curve or form an "S" shape.

[0102] A second method for creating the consultation table involves generating a recommendation based on machine learning. In this case, a set of "before and after" images can be used in conjunction with a treatment label and / or a product that has been used to teach a machine which products or treatment processes lead to certain results. The images can be associated with text descriptors, such as those provided by the user to describe their current hair condition and desired benefits. This can be used to create a model that can be continuously updated with new training data as it is collected.

[0103] [Fig.19] Figures 18-19 show further details on how Deep learning is performed to enable the smartphone app (or cloud platform) to estimate a recommendation for a user. In Figure 18, training is performed for the deep learning model. Inputs are provided at level 1910, where snapshots (which can be 360 ​​video selfies or snapshot selfies) are entered with text descriptors and a label. Since this is the training phase, the users' actual before-and-after images can be used to show the originally diagnosed loop pattern and the resulting loop pattern as a pair of images. Additionally, the users' self-perceived words describing their loops and text descriptors of their desired benefits can be entered as a pair of texts. Finally, the "label" given to the input data can be the product and / or treatment that achieved the desired outcome.

[0104] Inputs are provided to a deep learning algorithm in step 1920. The deep learning algorithm used can be based on available software known in the art, such as TensorFlow, Keras, MXNet, Caffe, or PyTorch. The result of the labeled training will be a neural network in step 1940. The resulting neural network includes nodes from each layer that are aggregated; the aggregates overlap, and each aggregate transmits data to multiple nodes in the next layer.

[0105] [Fig.20] Fig.20 shows the use of the deep learning model once the training has reached an adequate level. This is what is called the "time "Inference" is used because the recommendation will be inferred from the unlabeled input data. As you can see, the input stage does not include a label. Furthermore, the image pair includes the selfie image entered by the user and the image selected by the user within the application. These inputs are passed to the neural network, which will provide an output of a product and processing recommendation.

[0106] [OUTPUT TYPES]

[0107] [Fig.21] Fig.21 shows the different types of product recommendations and treatments generated by the recommendation generator according to the process described above. As you can see, the recommendation can take the form of a specific product in multiple categories, such as hair care, styling, and scalp care. Hair care products can include shampoo, conditioner, leave-in conditioner, mask, and dry shampoo. Styling products can include hairspray, refresher, gels, and primer. Scalp care products can include oil or cream.

[0108] [Fig.22] In addition to the recommended products, the recommendation generated by the The recommendation generator can also include tutorials and guides to help educate the user. As shown in [Fig. 22], the recommendation can include guides related to hair science, which explain hair structure and growth using illustrations, publications, videos, and / or animations. These tutorials can be specific to the user's needs.

[0109] [Examples of scenarios]

[0110] Examples of scenarios of how the application works for different user needs are described below.

[0111] A situation is described in which a consumer "A" has very distinct hair types but doesn't know how to classify them. When searching for a specific / personalized hair care routine based on their hair type, the consumer may feel overwhelmed by the sheer number of products available. In this scenario, consumer A can perform the following actions using the application according to one embodiment.

[0112] The consumer downloads the app.

[0113] Between information on the habits and current actions of her hair: frequency of washing, styling habits, additional types of damage such as the use of colour or heat.

[0114] He takes a picture of her hair in standard lighting.

[0115] Selects the type of products he is looking for: styling products or hair care.

[0116] Selects a picture to simulate what he wants his hair to look like.

[0117] Selects the types of results he is looking for: frizz control, shine, smooth character...

[0118] The app diagnoses what type of curls they are, their shine (or lack thereof), their level of damage and their color, as well as the recommended products.

[0119] Finally, the app will offer the opportunity to make in-store purchases. It will also include tutorials on what curly hair is and why it behaves the way it does.

[0120] Another situation is described in which a consumer "B" has very curly hair and is tired of buying so many different products to find "the one" that meets their needs. In this scenario, consumer B can do the following using the application according to one embodiment.

[0121] The consumer downloads the app.

[0122] Between information on the habits and current actions of her hair: frequency of washing, styling habits, additional types of damage such as the use of colour or heat.

[0123] He takes a picture of her hair in standard lighting.

[0124] Selects the type of products he is looking for: styling products or hair care.

[0125] Selects the types of results it is looking for: frizz control, shine, smoothness...

[0126] The app diagnoses what type of curls they are, their shine (or lack thereof), their level of damage and their color, as well as the recommended products.

[0127] Finally, the app will provide the opportunity to make purchases on site.

[0128] Another situation is described in which a consumer "C" has very curly hair and has used the application and noticed that their hair has changed to the specific / customized hair type, but is unsure of their hair type. In this scenario, consumer C can perform the following using the application according to one embodiment.

[0129] Enter new information on how hair now behaves.

[0130] He takes a new shot of his hair under standard lighting.

[0131] Selects the type of products he is looking for: styling products or hair care.

[0132] Selects the types of results it is looking for: frizz control, shine, smoothness...

[0133] The app provides a beauty diagnosis adaptable to one's hair attributes.

[0134] Finally, the app will provide the opportunity for new hair products, on-site shopping, and tutorials on what curly hair is and why it behaves the way it does.

[0135] Finally, another situation is described in which consumer "D" is very confused, but does not understand why they behave the way they do. He seeks to learn and follow their hair journey. In this scenario, consumer D can do the following using the application according to one embodiment.

[0136] The consumer downloads the app.

[0137] Between information on the habits and current actions of her hair: frequency of washing, styling habits, additional types of damage such as the use of colour or heat.

[0138] He clicks on the learning tools and videos about hair

[0139] He can upload pictures of his hairstyles, present uses of products as well as results and reviews and share this data with friends / family via a universal link which will produce a shareable version of his journal.

[0140] The app will also recommend potential products he might want to try with an option to buy on the spot.

[0141] Therefore, the above example scenarios show how the application described in these embodiments can meet a variety of consumer needs.

[0142] [ENHANCED FEATURES]

[0143] Training in product / treatment process recommendation

[0144] Although the above embodiment illustrates a scenario from the point of view of an individual user, a global system can be realized which uses a community of users to enhance both loop diagnosis and product recommendation (similar to some applications we have filed for Perso).

[0145] [Fig.23] [Fig.23] shows a 2700 system in which a plurality of users are connected to the server / cloud platform 102. As noted above, each user can keep a "hair diary" to track their results with snapshots. While this can be used to adjust the treatment or recommendation for each individual user, as described above, the "hair diary" can also be used to allow the system to "learn" how different treatments have worked with different users.

[0146] In other words, for each user who has obtained satisfactory results using a product or treatment to achieve a desired appearance, all parameters associated with that user can be entered into a machine learning model. These parameters may include the following.

[0147] User profile

[0148] Initial user hair condition (curl diagnosis)

[0149] Desired user appearance

[0150] Percentage of satisfaction with the result

[0151] Therefore, with continuous inputs, the system will learn the parameters for an optimal probability of success when a new user with a certain profile, starting hair condition and desired appearance begins to use the application.

[0152] The use of additional data on satisfaction can be used to weight a particular treatment, when the visual results between two users are similar, but different treatments have been used.

[0153] Community training in loop diagnosis

[0154] Users provide their own descriptions of what they think their current hair looks like, along with a photograph. This can be treated as labeled data within the system itself. Furthermore, users can be asked to identify the hair type they see in other photographs, which may be based on photographs from other users or models. The same applies to the labeling process. This also provides a method for better associating the images attributed to curl patterns with the textual words used to describe those images.

[0155] In one example, as shown above in [Fig.15], in the application, the user may be asked to provide their own input on the loop pattern they think is present in an image.

[0156] There may also be several "game" features that are not only useful for entertaining the user, but also provide valuable data and feedback for optimizing system features and providing customized functionalities for the user.

[0157] [Fig.24] For example, [Fig.24] shows a graph where data are collected The user's selections on a loop pattern are shown for a displayed image. Additionally, the correct loop pattern is shown for comparison.

[0158] The type of game illustrated in [Fig. 24] can be used for at least two purposes. First, it can be used to label an unlabeled image with a loop pattern. In this situation, the 'correct answer' shown at the end of the game can be the answer with the highest current percentage provided by a community of users.

[0159] [Fig.25] Secondly, it can be used to adjust the entire graph of the loop pattern itself. In [Fig. 25], we can see that the aggregate of user selections does not converge to the correct loop pattern label. If a threshold quantity of user selections is shifted from a supposedly correct loop type on the loop scale, then a correction may be necessary for the types of hair images used to determine a loop type.

[0160] The first advantage of the game shown in [Fig.24] is that the user can develop an eye for curl patterns in hair.

[0161] A second benefit of the game shown in [Fig. 24] is that each user's selections can be collected by a central server. This data can be very useful. For example, it can be used to learn about the preferences of the general public. For instance, with additional user data, such as age, location, and other lifestyle habits, a cosmetics company can determine if there are trends in user preferences based on different user categories and locations. Local purchases

[0162] Although the product recommendation described above can be made using a large number of products available from a large number of retailers or on the Web, it would not be useful to the user if he were in a store when using the app and the recommended product was not available there.

[0163] [Fig.26A] Therefore, the application may include a feature that allows to perform a "store scan" based on the products actually on the shelves. First, [Fig. 26A] shows that a globally recommended product 3010 can be output by the recommended generator, as noted above. However, an additional option 3020 can be displayed for "Check products in store".

[0164] [Fig.26B] If the user selects option 3020, a new screen is displayed on [Fig. 26B]. The user can then choose between two options. The first option (3030) involves checking the current store's stock online. In this option, the application can use the smartphone's GPS to determine the user's current store location and display one or more recommended products currently in stock at that store. This option assumes that this information is actually available online.

[0165] [Fig.27A] The second option 3040 is to perform a shelf scan near from the user's actual physical location. If the user selects this option, the smartphone's camera function will be activated, and the user can capture an image of multiple products located on a shelf. For example, the user can capture an image of a shelf of hair care products in a retail store, as shown in [Fig. 27A].

[0166] A scan of the products can be performed using methods included in the art. For example, a technique can be used as described in US Patent No. 10,579,962.

[0167] [Fig.27B] Following the scan of the ray, a suitable product can be identified visually and / or textually as shown in [Fig. 27B]. This product may be similar to, or have similar ingredients to, the recommended product in Figure 26A.

[0168] Obviously, many modifications and variations to this disclosure are possible in light of the above teachings. It should therefore be understood that, within the scope of the appended claims, the invention may be implemented in a manner other than as specifically described herein.

Claims

Demands

1. A hair curl pattern determination and hair curl identification system, comprising: at least one server device; and a mobile user device that includes processing circuitry configured to run an application that receives user information input concerning a plurality of features of at least one user's hair, including at least one image of the user's hair; and processing circuitry of the at least one server device that receives the user information input and determines a curl pattern of the user based on the information received;and a loop identification component including circuitry configured to generate a per-pixel prediction score for the presence or absence of hair loops and to predict the presence or absence of a specific loop pattern in the image using one or more convolutional neural network image classifiers; and a loop appreciation component including circuitry configured to generate a user-selectable menu that allows a user to choose from a plurality of hair features and loop pattern images and to generate user hair appreciation information in response to one or more inputs associated with at least one prediction score for the presence or absence of hair loops and a score for the presence or absence of a specific loop pattern.

2. System according to claim 1, wherein the user information input further includes a textual description concerning the plurality of characteristics of the user's hair.

3. System according to claim 2, wherein the textual description relates to at least one actual method of treating the user's hair and an actual state of the user's hair.

4. System according to claim 2, wherein the textual description relates to a first portion of the user's hair having one type of curl and another portion of the user's hair having a second type of curl.

5. System according to claim 1, wherein the processing circuitry of at least one server device determines the user's loop pattern by inputting the user's hair image into a neural network which is trained to identify loop patterns in images.

6. System according to claim 1, wherein the processing circuitry of at least one server device determines the user's loop pattern by determining pixels in the user's hair image that correspond to the user's hair and performing a linear regression analysis to determine a hair shape appearing in the image.

7. System according to claim 1, wherein the generation of the predicted score for the presence or absence of the specific loop pattern in the image includes the generation of an indicative score of the presence or absence of a combination of one or more of a twisted hair pattern, a curly hair pattern, a frizzy hair pattern, a wavy hair pattern or a straight pattern.

8. System according to claim 1, wherein the predicted score for the presence or absence of the specific loop pattern in the image includes a score indicating the presence or absence of twisted hair pattern, curly hair pattern, frizzy hair pattern, wavy hair pattern or straight hair pattern.

Images