System, apparatus and method for assessing and personalizing curly hair

Abstract

SYSTEM, APPARATUS, AND METHOD FOR ASSESSING AND PERSONALIZING CURLY HAIR The disclosure in this document generally relates to a system, apparatus, and method for diagnosing a user's curly hair type, providing a product or treatment recommendation for the user, and utilizing user and social feedback to improve both curl diagnosis and product recommendation. Figure for abstract: none

Description

Description of the invention: System, apparatus and method for assessing and curly hair customization CONTEXT Domain The disclosure in this document generally relates to a system, device, and method for diagnosing a user's curly hair type, providing a recommendation on a product or treatment for the user, and using user and social feedback to improve both the curly diagnosis and the product recommendation. DETAILED DESCRIPTION OF THE IMPLEMENTATION METHODS When it comes to hair care and maintenance, there are many types of curly hair, and consumers with curly hair want the best and most appropriate care for their specific curl type and shape. Numerous curl charts exist, and it can sometimes be difficult to self-diagnose your hair's condition, type, and necessary care. Choosing the wrong chart can lead to selecting the wrong product, resulting in poor or suboptimal results. Therefore, what's needed is an application that can offer a process for diagnosing a client's curly hair and providing a product or treatment recommendation to help the consumer achieve their desired outcome. To this end, the invention relates to a system comprising: at least one server device; and a mobile user device that includes processing circuitry configured to run an application that receives user information input concerning a plurality of characteristics of at least one user's hair, including at least one of the user's curl patterns; and receives input from the user regarding a desired hair treatment outcome, in which the processing circuitry of at least one server device receives user information input and input of the desired processing result and provides output a product or treatment recommendation based at least partially on the user's determined loop pattern and the user's desired result. According to embodiments of the invention, the system comprises one or more of the following features, taken individually or according to all embodiments technically compatible: The user enters a desired result by providing text descriptors. of the user's desired hair condition, The user enters a desired result based on the selection of an image from a type of loop pattern. the processing circuitry of at least one server device determines the re- ordering a product or treatment by entering the information user and input of the desired processing result into a network neuronal which is trained to provide output recommendations based based on a combination of image and textual information, the processing circuitry of at least one server device determines the re- ordering a product or treatment by entering the information user and the desired processing result in a lookup table. The product or treatment recommendation provided at the output includes a product and / or treatment recommendation related to at least one of the following: Hair care, styling products and scalp care, The product or treatment recommendation provided at the output includes guides related to hair science, which explain the structure and growth hair using at least one illustration, publication, video and / or animation, and after providing the product or treatment recommendation as output the user, the processing circuitry of the mobile user device receives an image of a variety of hair products that are located in a place retail outlet where the user is located, and the processing circuitry of at least one server device is configured to generate a recommendation recommendation of at least one of the hair care products that are found in the retail location. The embodiments below describe a system, device, or application (“App”) that can receive as input at least one curly hair pattern from a user based on a photograph of the user's hair, and diagnose the curl pattern, hair shine, color, and texture. Based on this diagnosis, the application will recommend a specific product or product regimen from a designated brand and offer the option to make in-store purchases. It is also possible to adapt the diagnosis and recommendations based on changes in hair type and consumer needs. One month, a consumer might receive a diagnosis based on their hair being dry, but perhaps their hair has changed and is now oily. With this new information, they can rely on the application to adjust the diagnosis and recommendations. order based on the adaptation of her hair. [SYSTEM] [Fig. 1] Fig. 1 shows a complete system 100 according to one embodiment. The basic components required are a user device 101 (such as a smartphone) and one or more server devices 102 (such as a cloud platform). A hardware description of these components will be provided later. The smartphone is shown with an included smartphone application (“application”). Using the smartphone application requires the user to actually provide input by making selections that lead to loop diagnostic functionality and product recommendations. The smartphone app also communicates interactively with the cloud platform. For example, the app can receive the relevant selection of gaits described above and can also provide direct feedback from the user on gaits previously sent by the cloud platform. It can also notify the cloud platform about the colors and recipes actually selected by the user and distributed by the distributor. Such feedback can provide a form of machine learning for the cloud platform and improve the algorithms it uses. [Fig. 2] Figure 2 is a more detailed functional diagram illustrating an example of a user device 20 according to certain embodiments of this disclosure. In some embodiments, the user device 20 may be a smartphone. However, the skilled craftsman will appreciate that the features described in this document can be adapted for implementation on other devices (e.g., a laptop, tablet, server, e-reader, camera, navigation device, etc.). The example user device 20 in Figure 9 includes a control device 110 and a wireless communication processor 102 connected to an antenna 101. A speaker 104 and a microphone 105 are connected to a voice processor 103. The control device 110 is an example of the control unit 21 shown in [Fig. 1] and may include one or more central processing units (CPUs). It can control each element of the user device 20 to perform functions related to communication control, audio signal processing, audio signal processing control, still and moving image processing and control, and other types of signal processing. The control device 110 can perform these functions by executing instructions stored in a memory 150. Alternatively, or in addition to local storage in the memory 150, the functions can be executed using instructions stored on an external device accessed over a network or on a non-transient, computer-readable medium. As described above in connection with [Fig.1], the control device 110 can execute instructions enabling the control device 110 to function as the display control unit 211, the operations management unit 212 and the game management unit 213 schematically represented in [Fig.1]. Memory 150 is an example of the storage unit 22 shown in [Fig. 1] and includes, but is not limited to, read-only memory (ROM), random-access memory (RAM), or a memory array comprising a combination of volatile and non-volatile memory units. Memory 150 can be used as working memory by the control device 110 while executing the processes and algorithms of this disclosure. Furthermore, memory 150 can be used for long-term storage of, for example, image data and related information. User device 20 has a command line (CL) and a data line (DL) as internal communication bus lines. Command data to / from control device 110 can be transmitted via the command line (CL). The data line (DL) can be used for transmitting voice data, display data, etc. Antenna 101 transmits / receives electromagnetic wave signals between base stations to perform radio communications, such as various forms of cellular phone communication. Wireless communication processor 102 controls communication between user device 20 and other external devices via antenna 101. For example, wireless communication processor 102 can control communication between base stations for cellular phone communication. The loudspeaker 104 emits an audio signal corresponding to audio data provided by the voice processor 103. The microphone 105 detects ambient audio and converts it into an audio signal. This audio signal can then be output to the voice processor 103 for further processing. The voice processor 103 demodulates and / or decodes audio data read from memory 150 or audio data received by the wireless communication processor 102 and / or a short-range wireless communication processor 107. In addition, the voice processor 103 can decode audio signals obtained from the microphone 105. The example user device 20 may also include a display 120, a touch panel 130, an operating key 140, and a short-range communication processor 107 connected to an antenna 106. The display 120 may be a liquid crystal display (LCD), an organic electroluminescent display panel, or another display technology. In addition to displaying still and moving image data, the display 120 may display operational inputs, such as numbers or icons, which can be used for The display 120 can also display a graphical interface for a user to control aspects of the user device 20 and / or other devices. Furthermore, the display 120 can display characters and images received by the user device 20 and / or stored in memory 150, or accessed from an external device on a network. For example, the user device 20 can access a network such as the Internet and display text and / or images transmitted from a web server. The Touch Panel 130 may include a physical touch panel display screen and a touch panel driver. The Touch Panel 130 may include one or more touch sensors to detect an input operation on a touch panel display screen's operating surface. The Touch Panel 130 also detects a touch shape and a touch area. In this document, the term "touch operation" refers to an input operation performed by touching a touch panel display's operating surface with an instruction object, such as a finger, thumb, or stylus-like instrument.In the event that a stylus or similar is used for a touch operation, the stylus may have a conductive material at least at the tip of the stylus so that the sensors included in the touch panel 130 can detect when the stylus approaches / comes into contact with the operating surface of the touch panel display (as in the event that a finger is used for the touch operation). One or more of the display 120 and the touch panel 130 are examples of the touch panel display 25 schematically represented in [Fig.1] and described above. In certain aspects of this disclosure, the touch panel 130 may be positioned adjacent to the display 120 (e.g., laminated) or may be integrated into the display 120. For simplicity, this disclosure assumes that the touch panel 130 is formed as a single unit with the display 120 and, therefore, the examples discussed herein may describe touch operations performed on the surface of the display 120 rather than on the touch panel 130. However, the skilled craftsman will appreciate that this is not limiting. For simplicity, this disclosure assumes that the Touch Panel 130 is a capacitive touch panel technology. However, it should be noted that aspects of this disclosure can readily be applied to other types of touch panels (e.g., resistive touch panels) with alternative structures. In certain aspects of this disclosure, the Touch Panel 130 may include transparent electrode touch sensors arranged in the XY direction on the sensor's transparent glass surface. The touch panel driver can be included in the 130 touch panel for control processing related to the 130 touch panel, such as scan control. For example, the touch panel driver can scan each sensor in a transparent electrostatic capacitance electrode structure in the X and Y directions and detect the electrostatic capacitance value of each sensor to determine when a touch operation is performed. The touch panel driver can output a coordinate and a corresponding electrostatic capacitance value for each sensor. The touch panel driver can also display a sensor ID that can be mapped to a coordinate on the touch panel display. Furthermore, the touch panel driver and touch panel sensors can detect when an instruction object, such as a finger, is within a predetermined distance of an operating surface on the touch panel display.Specifically, the instruction object does not necessarily need to be in direct contact with the operating surface of the touch panel display for the touch sensors to detect the instruction object and perform the processing described herein. For example, in some embodiments, the touch panel 130 can detect the position of a user's finger around an edge of the display panel 120 (for example, grasping a protective case surrounding the screen / touch panel). Signals can be transmitted by the touch panel driver, for example, in response to the detection of a touch operation, in response to a query of another element based on a timed data exchange, etc. The touch panel 130 and the display 120 can be enclosed in a protective housing, which can also contain the other components included in the user device 20. In some embodiments, the position of the user's fingers on the protective housing (but not directly on the surface of the display 120) can be detected by the sensors of the touch panel 130. Consequently, the control device 110 can perform display control processing as described in this document based on the detected position of the user's fingers gripping the housing. For example, an element of an interface can be moved to a new location within the interface (e.g., closer to one or more fingers) based on the detected finger position. Furthermore, in certain embodiments, the control device 110 can be configured to detect which hand is holding the user device 20, based on the detected finger position. For example, the sensors on the touch panel 130 can detect a plurality of fingers on the left side of the user device 20 (e.g., on an edge of the display 120 or on the protective housing), and detect a single finger on the right side of the user device 20. In this example scenario, the control device 110 can determine that the user is holding the device 20 with the right hand, because the detected grip pattern corresponds to an expected pattern when device 20 is held only in the right hand. The operating key 140 may include one or more external buttons or similar control elements, which can generate an operating signal based on user-detected input. In addition to the outputs of the touch panel 130, these operating signals can be provided to the control device 110 to perform related processing and control. In certain aspects of this disclosure, the processing and / or functions associated with external buttons and the like may be performed by the control device 110 in response to an input operation on the display screen of the touch panel 130, rather than on the external button, key, etc. In this way, the external buttons of the user device 20 can be eliminated instead of performing inputs via touch operations, thus improving sealing. Antenna 106 can transmit / receive electromagnetic wave signals to / from other external devices, and the short-range wireless communication processor 107 can control the wireless communication between these other external devices. Bluetooth, IEEE 802.11, and Near Field Communication (NFC) are non-limiting examples of wireless communication protocols that can be used for communication between devices via the short-range wireless communication processor 107. The user device 20 may include a motion sensor 108. The motion sensor 108 may detect motion features (i.e., one or more displacements) of the user device 20. For example, the motion sensor 108 may include an accelerometer to detect acceleration, a gyroscope to detect angular velocity, a geomagnetic sensor to detect direction, a geolocation sensor to detect position, etc., or a combination thereof to detect the motion of the user device 20. In some embodiments, the motion sensor 108 may generate a detection signal that includes data representing the detected motion. For example, the motion sensor 108 may determine a number of distinct displacements within a movement (e.g., from the beginning of the series of displacements to a standstill, within a predetermined time interval, etc.).), a number of physical shocks to the user device 20 (e.g., a jolt, a bump, etc., of the electronic device), a speed and / or acceleration of movement (instantaneous and / or temporal), or other motion features. The detected motion features can be included in the generated detection signal. The detection signal can be transmitted, for example, to the control device 110, whereby further processing can be performed based on the data. included in the detection signal. The motion sensor 108 can operate in conjunction with a Global Positioning System (GPS) section 160. The GPS section 160 detects the current position of the terminal device 100. The current position information detected by the GPS section 160 is transmitted to the computer 110. An antenna 161 is connected to the GPS section 160 to receive and transmit signals to and from a GPS satellite. The user device 20 may include a camera section 109, which comprises a lens and a shutter for capturing photographs of the environment around the user device 20. In one embodiment, the camera section 109 captures the environment on the opposite side of the user's device 20. Images of the captured photographs may be displayed on the display panel 120. A memory section saves the captured photographs. The memory section may be located within the camera section 109 or it may be part of the memory 150. The camera section 109 may be a separate feature attached to the user device 20 or an integrated camera feature. Next, a hardware description of one or more server devices 102, according to exemplary embodiments, is given with reference to [Fig. 3]. [Fig. 3] comprises a CPU X00 that performs the processes described above / below. Process data and instructions can be stored in memory XO2. These processes and instructions can also be stored on a storage medium XO4, such as a hard disk drive (HDD) or portable storage medium, or can be stored remotely. Furthermore, the claimed advances are not limited by the form of the computer-readable media on which the instructions of the inventive process are stored. For example, the instructions can be stored on CD, DVD, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk drive, or any other information processing device with which the [device] communicates, such as a server or a computer. In addition, the claimed advancements can be provided as a utility application, background daemon or component of an operating system, or a combination thereof, running jointly with the X00 CPU and an operating system such as Microsoft Windows 7, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to the person skilled in the art. The hardware components needed to create the device can be implemented using various circuitry elements, known to those skilled in the art. For example, the X00 CPU could be an Intel Xenon or Core processor (in America), an AMD Opteron processor (in America), or other types of processors that would be recognized. by a person skilled in the art. Alternatively, the X00 CPU can be implemented on an FPGA, ASIC, PLD, or using discrete logic circuits, as a person skilled in the art would recognize. Furthermore, the X00 CPU can be implemented as multiple processors working together to execute the instructions of the inventive processes described above. The device in Figure X also includes a network control device X06, such as an Intel Ethernet PRO network interface card from Intel Corporation in America, to serve as an interface with network XX. As can be seen, network XX can be a public network, such as the Internet, or a private network, such as a LAN or WAN, or any combination thereof, and may also include PSTN or ISDN subnets. Network XX can also be wired, such as an Ethernet network, or wireless, such as a cellular network including EDGE, 3G, and 4G wireless cellular systems. The wireless network can also be Wi-Fi, Bluetooth, or any other known form of wireless communication. The device further includes an XO8 display control device, such as an NVIDIA GeForce GTX or Quadro graphics adapter from NVIDIA Corporation in America, for interfacing with the X10 display, such as a Hewlett Packard HPL2445w LCD monitor. A general-purpose X12 I / O interface serves as an interface with an X14 keyboard and / or mouse, as well as an X16 touchscreen panel on or separate from the X10 display. The universal F / O interface also connects to various X18 peripherals, including printers and scanning devices, such as a Hewlett Packard OfficeJet or DeskJet. An X20 sound control device is also supplied in the [device], such as Creative's Sound Blaster X-Fi Titanium, to interface with the X22 speakers / microphones and thus provide sounds and / or music. The X24 universal storage controller connects the X04 storage media disk to the X26 communication bus, which can be an ISA, EISA, VESA, PCI, or similar, to interconnect all the components of the [device]. A description of the general features and functionalities of the X10 display, the X14 keyboard and / or mouse, as well as the X08 display controller, the X24 storage controller, the X06 network controller, the X20 sound controller, and the X12 universal I / O interface is omitted here for brevity, as these features are known. The circuit element examples described in the context of this disclosure may be substituted with other elements and structured differently from the examples provided here. Furthermore, the circuitry configured to achieve the features described herein may be implemented in multi-circuit units (e.g., chips), or the features may be combined in a circuit. cooking on a single set of chips. The functions and features described herein can also be performed by various distributed components of a system. For example, one or more processors can execute these system functions, with the processors distributed across multiple components communicating within a network. The distributed components may include one or more client and server machines, which can share processing, as shown in [Fig. 3B], in addition to various human interface and communication devices (e.g., display monitors, smartphones, tablets, personal digital assistants (PDAs)). The network may be a private network, such as a LAN or WAN, or a public network, such as the Internet. Input to the system may be received directly from the user and received remotely in real time or as discontinuous processing.Furthermore, some implementations may be performed on modules or hardware that are not identical to those described. Consequently, other implementations fall within the scope that can be claimed. The hardware description described above is a non-limiting example of a corresponding structure to achieve the functionality described here. [USER DATA COLLECTION] Collection of information | hair 1: Information described by the user themselves [Fig. 4-5] Figures 4 and 5 show the types of descriptive information received by the application from the user regarding the current state of their hair. The information may include one or both image information and descriptive information. For example, as shown in [Fig.4], as personal information, the application collects one or more of the user's name, age, ethnicity, location and contact details. As shown in Fig. 5, the user can also provide information about their current hair care habits, which can be referred to as their "current hair history." This information can be seen to include one or more of the following elements. Whether or not the user uses heat to treat their hair The frequency of use of shampoos and / or conditioning products If the user uses a silky smoothing treatment If the user leaves their hair straightened If the user's hair is colored If the user's hair is chemically treated. [Fig. 6] As shown in [Fig. 6], the user can also provide in- training on the current perceived condition of their hair. For example, the user can describe their hair as fine, medium, coarse, oily, dry, fragile, porous or frizzy. The user can also indicate if their hair has different types of curls or characteristics in different areas. For example, while one type of curl might be present in the most visible parts of the hair, another type of curl might be present in a less visible area such as the back of the neck. A traditional AI-based hair imaging system would not take this type of information into account, and it would likely need to be provided by the user. [Fig. 7A] Figure 7A shows that a user can take one or more "self-portrait" images of their hair. More specifically, the user can take a single self-portrait, a 360° self-portrait, or a series of photographs from different angles using the smartphone's camera capabilities. In a preferred embodiment, as shown in [Fig. 7B][Fig. 7B], the user must take a self-portrait image of the portion of their hair that embodies the curl pattern, such as the side of their hair, or if possible, a view from the back of their hair. The image is preferably taken against a white background. [INFORMATION ON APPEARANCE / DESIRED BENEFITS] [Fig.8] Fig.8 shows the types of descriptive information received by the application from the user regarding the desired (target) gait. The user can provide their target look using words and / or images. As shown in [Fig.8], the user can enter descriptive terms regarding desired benefits, such as achieving more definition, lengthening, hydration, frizz control, damage repair, shine, softness, or split end reduction. [Fig. 9] As shown in [Fig. 9], the user can enter one or more indicative images or the desired "final look." This can be done in conjunction with, or as an alternative to, descriptive terms provided by the user. For example, the user can select images taken from the internet. The images can be provided by a plurality of images displayed to the user directly within the application itself. These images can be pre-associated text descriptors, such as representing a look with one or more Alternatively, images may not be pre-associated with textual descriptors, and image analysis may be performed on the image to determine the characteristics of the model's hair in the image. [LOOPS DIAGNOSIS] Once the descriptive and image information has been collected from The application will trigger a curl diagnosis for the user. Part of the curl diagnosis involves performing an image analysis to determine the user's curl type or pattern, as well as other attributes such as texture, damage level, gloss level, dryness, and color. [Fig. 10] Figure 10 shows a table of different types of loop patterns according to one embodiment. In this example, there are eight different types of loop patterns, which can represent wavy, curly, or frizzy loops. There are various methods for performing image analysis. One method involves detecting features in an image that indicate a specific curl pattern. For example, an angle in the hair pattern can be detected. Due to the contrast of the user's hair against a white background in the image, pixels of a certain color are detected, and the average curl is plotted on a 2D diffusion diagram. To facilitate this step, the image's white balance can be optimized so that the image's white balance makes it easier to detect the black / white contrast. The system can also automatically detect the percentage of black / white on the white background. A method using this type of detection is shown in [Fig. 11][Fig. 11]. In step 1101, the system determines the pixels of a predetermined color, which can be any color other than white, assuming the image is taken against a white background. In step 1102, the mean angle and the loop are measured. The mean angle and hair curl are measured using linear regression. Specifically, pixels in a "maximum expected" region are labeled in red and correspond to a polynomial regression (degree = 2). Using heuristics to determine which side of the parabola to choose, measurements are taken from one of the parabola's points to the base. The heuristics are based on which side of the parabola has more data points and the concavity of the curve. This technique is described in more detail in US Patent No. 10,929,993. Another method involves using deep learning or machine learning to train a model to determine a loop pattern in images. In this embodiment, the system implements one or more convolutional neural networks (CNNs), whose models can be trained using open-source or collaborative datasets, as explained below. Other machine learning techniques can be used in conjunction with the present invention, including, but not limited to, decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, and Bayesian networks. Reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, rule-based machine learning, and learning classifiers. Additional techniques described in US Patent No. 8,442,321, US Patent No. 9,015,083, US Patent No. 9,536,293, US Patent No. 9,324,022, and US Publication PG No. 2014 / 0376819 A1 may be used with the present invention. In the following descriptions, it will be assumed that the machine learning logic implements a convolutional neural network, although the present invention is not so limited. Those with expertise in artificial intelligence will recognize many techniques that can be used in conjunction with the present invention without departing from its intended spirit and scope. [Fig. 12] Fig. 12 is a flowchart illustrating an example of interaction with an embodiment of the invention. The interaction shown in Fig. 12 is not intended to be limiting. The description in Fig. 12 is intended to illustrate the functionality of the configuration shown in Fig. 1. Other features of the invention, beyond those described with reference to Fig. 12, will be discussed below. At operation 1210, the user's captured self-portrait image is received. At step 1220, image analysis and machine learning are used to analyze the user's skin from the images. The system can perform analyses that determine, among other things, curl pattern type, hair texture, hair damage level, hair shine level, hair dryness, and hair color. Other hair conditions can also be determined by the system. Further details about the analyses are provided below. Once the analyses are complete, as determined at step 1225, process 1200 can proceed to step 1230, at which point the analysis results and the prescribed regimen (products and routines) and / or regimen updates are sent to the user via an application interface. At operation 1225, it is determined whether the analysis is complete, and in response to a positive determination, process 1200 can proceed to operation 1230, whereupon the application sends a recommended diet or diet updates to the user at operation 1230. The user can follow the diet as indicated at operation 1235, and at operation 1240, it is determined whether a new interval has begun. If so, process 1200 repeats operation 1210. The system can access calendars and timers (as well as GPS) on the client device and network-accessible calendars. Consequently, say once a week, the application can remind the user to Taking a picture of her hair serves as a reminder of the new interval. Over time, the system can determine from the images taken at each interval whether the recommended regimen is working and, if not, the system can revise the regimen, for example, change a product, recommend other lifestyle changes, schedule an appointment with a specialist, etc. [Fig. 13] [Fig. 13] is a data flowchart between an example customer interface 1340 (i.e., the application) and the services of a service platform 1320. It should be noted that, in [Fig. 13], as illustrated in [Fig. 13], the machine learning logic 1354 may include a hair analyzer 1330, a facial appearance progression generator 335, and a diet recommendation generator 1350, and may be coupled in communication to a user account database 310 and a product database 320. The machine learning logic 1354 can train and use the machine learning models 1370 to recommend diets and track the user's progress under the diet.As anyone skilled in machine learning will attest, training may involve selecting a set of features—for example, a curl pattern type, hair texture, hair damage level, hair shine level, hair dryness, and hair color—and assigning labels to image data that reflect the presence or prominence of these features. Labeling can be done by a subject matter expert or, as explained below, through collaborative data. Taking the assigned labels as real-world data, machine learning logic can configure models to predict the degree to which features are present in a test image, which may change over time.The present invention is not limited to a particular model representation, which may include binary models, multi-category classification models, regression models, etc. The sample user account database 1310 securely contains the data of all users of system 100. This includes user profile data, current and past user photos 352 for each user, current and past hair analyses 358 for each user, current and past product recommendations 362 and current and past routine recommendations 364 for each user. The sample product database 1320 contains data for various products that can be used in a diet. The product database 1320 can contain records reflecting product names, active and inactive ingredients, label information, recommended uses, and so on. In some embodiments, such as illustrated as product entry 354, The user (and other users of the system) can provide feedback on different products and can enter products that are not already in the product database 1320. The present invention is not limited to particular products that can be entered into the product database 1320. The Hair Analyzer 1330 is built or otherwise configured to classify different states or artifacts of the hair analyzer from a user's hair image using machine learning techniques against the models 1370. In some embodiments, photographic images 352 of a user's hair are provided to the Hair Analyzer 1330 for analysis. The Hair Analyzer 330 can implement image preprocessing mechanisms that include cropping, rotating, registering, and filtering the input images before analysis. After such preprocessing, the Hair Analyzer 1330 can apply the models 1370 to the input image to locate, identify, and classify the user's hair features. The hair appearance progression generator 1335 can operate on the user's hair images to represent the user's future facial appearance. Such progression can be an age progression, for which age progression techniques can be deployed, or it can be an appearance progression following a diet. A progress image 356 can be provided to the user via the client interface 1340. The 1350 regimen recommendation generator can operate on the 358 analysis results obtained from the 1330 hair analyzer to prescribe a regimen to the user. The 1370 models can be trained to predict which products and routines (treatment, cosmetic and lifestyle recommendations, etc.) would be effective in achieving the user's goal with respect to the hair characteristics identified in the hair analysis. The regime recommendation generator 1340 can format the analysis results 358 from the hair analyzer 1330 as a query in, for example, the product database 1320 based on the knowledge encoded on the models 1370. In response, the product database 1320 can return product data and metadata 1366, and product recommendations 362 and routine recommendations 364 can be provided to the client interface 1340. As mentioned above, training 1370 models can be achieved by labeling image data with an expert's input. However, instead of an expert, some embodiments of the invention use collaborative data as training data. [Fig. 14] Figure 14 is a diagram of such an embodiment of the invention. During training, users 410 are presented with a set of images 420 training exercises on which they are asked to characterize curl patterns and / or hair features. In one embodiment, the curl pattern scale shown in [Fig. 10] is used so that users can evaluate the type of curl pattern. For example, each of the 410 users (over time) is presented with a large number of images and reviews a set of questions concerning the features of the person in the image. Using the scale, each user 410 is asked to select one of the curl types. The answers to the questions can serve as labels used for training automatic language logic 1340. [Fig.15] With reference to [Fig.15], an example 1500 is illustrated in which the application presents a photograph of a model 1510 and asks the user to determine which type of loop, among a plurality of loop types 1520, is shown in the image. [Fig. 16] Figure 16 illustrates an example of trial operation in accordance with the collaborative training discussed above. A test image 710, i.e., a user's own image, can be presented to the machine learning logic 154, which analyzes the image according to the models trained on the collaborative data 720. As illustrated in the figure, the machine learning logic 1340 estimates that 80% of respondents would rate the user's loop pattern as a type 7, as indicated in 722. Consequently, the machine learning logic 1340 can use loop type #7 as an input data element when making recommendations, as described below. User text descriptors can be used to aid in curl diagnosis as additional input to the model. For example, including a user's text describing their own hair, in combination with visual features in the image of their hair, will help weight the probability of a curl pattern being identified in the image. The machine learning system described above may further include a loop identification component, comprising circuitry configured to generate a per-pixel prediction score for the presence or absence of hair loops and to predict a score for the presence or absence of a specific loop pattern in an image using one or more convolutional neural network image classifiers. In this process, generating the predicted score for the presence or absence of the specific loop pattern in the image involves generating a score indicative of the presence or absence of a combination of one or more of the following: a curly hair pattern, a wavy hair pattern, a frizzy hair pattern, a wavy hair pattern, or a straight hair pattern. The predicted score for the presence or absence of the specific loop pattern in the image may include a An indicative score for the presence or absence of a curly, wavy, coily, or straight hair pattern. The predicted score for the presence or absence of a specific curl pattern in the image includes an indicative score for the presence or absence of a curl pattern from a plurality of predetermined curl patterns on a curl scale. A loop appreciation component may also be provided which includes circuitry configured to generate a selectable menu that allows a user to choose from a plurality of hair features and loop pattern images and to generate user hair appreciation information in response to one or more inputs associated with at least one of the prediction scores for the presence or absence of hair loops and the prediction of a score for the presence or absence of a specific loop pattern. A method can also be provided for applying a convolutional neural network image classifier to a user image to obtain per-pixel prediction scores for the presence or absence of loops and types of hair loops, and to generate a virtual representation of a portion of the user image and a predicted type of hair loop based on the prediction scores for the presence or absence of loops and types of hair loops. A computer-implemented method can also be provided for training a neural network for hair curl detection. This method involves collecting a set of digital images of curly hair patterns, images of wavy hair patterns, images of kinky hair strands, or images of wavy hair patterns from a data store; applying one or more transformations to each digital hair image to create a modified set of digital hair images; creating a first training set comprising the collected set of digital hair images, the modified set of digital hair images, and a set of digital hair images without curls; training a neural network in a first step using the first training set; creating a second training set for a second training step comprising the first training set and digital hair images without curls that are incorrectly detected as images of hair curl patterns after the first training step; and training the neural network in a second step using the second training set. [LOOP OR PRODUCT TREATMENT RECOMMENDATION] While the above process describes how the system performs loop diagnostics, loop diagnostics is further combined with an input of the desired look or benefit for the user to obtain a product or treatment recommendation. As shown in [Fig.8], the user enters the desired benefit they are looking for in the application, or as shown in [Fig.9], the user has selected a "look" they are looking for. As shown in [Fig.17][Fig.17], a functional component called the recommendation generator 1710 takes the loop diagnostic input 1720 and the benefit desired by the user 1730, and provides a recommendation 1740 as output. Different methods for generating a recommendation are described below. Consultation Table As a first method, a consultation table can be used, which includes predetermined combinations of loop diagnosis and desired benefit in association with a predetermined output of a product or treatment recommendation. [Fig. 18A] [Fig. 18A] shows an example of input into a lookup table as described above. A matrix table style is shown where curl diagnosis options fill one lookup axis and user-desired benefits fill the opposite lookup axis. Based on the inputs received by the Recommendation Generator 1710, an intersection of the inputs will lead to the generated recommendation. Although this example is simplified, curl diagnosis could also include other user hair assessment features as described above, and user-desired benefits could also include other features. An increase in the number of feature types contained in the input data will obviously require an expanded lookup table. The lookup table type is not limited to the type shown in [Fig. 18B][Fig. 18B].The table itself can be generated manually based on expert input. For example, Figures 18B and 18C show expert information that matches a user's curl type to other characteristics, tender points, and treatment recommendations. Clearly, this information can be used to supplement the lookup table in Fig. 18A. For reference, a zigzag hair type means that the strands form a zigzag, not a curl or wave. Hair is referred to as "curly" when each strand forms tight, coiled curls. It is very versatile but can be fragile, especially if the strands are fine, as it may have a thin outer layer. "Curly" hair strands clump together and wrap in spirals or looser curls. This texture needs plenty of moisture to promote a defined pattern, but a little frizz can add character. "Wavy" hair when the strands curve or form an "S" shape. A second method for creating the consultation table involves generating a recommendation based on machine learning. In this case, a set of "before and after" images can be used in conjunction with a treatment label and / or a product that has been used to teach a machine which products or treatment processes lead to certain results. The images can be associated with textual descriptors, such as those provided by the user to describe the current condition of their hair and the desired benefits. This can be used to create a model that can be continuously updated with new training data as it is collected. Figures 18 and 19 provide further details on how deep learning is performed to enable the smartphone app (or cloud platform) to estimate a recommendation for a user. In Figure 18, training is performed for the deep learning model. Inputs are provided in step 1910, where snapshots (which can be 360° video selfies or photo selfies) are entered along with text descriptors and a label. Since this is the training phase, actual before-and-after snapshots of the users can be used to show the originally diagnosed loop pattern and the resulting loop pattern as a pair of images. Additionally, the users' self-perceived words describing their loops and text descriptors of their desired benefits can be entered as a pair of texts.Finally, the "label" given to the entered data can be the product and / or treatment that achieved the desired result. The inputs are fed to a deep learning algorithm in step 1920. The deep learning algorithm used can be based on readily available software such as TensorFlow, Keras, MXNet, Caffe, or PyTorch. The result of the labeled training will be a neural network in step 1940. The resulting neural network consists of nodes in each layer that are grouped together, with overlapping groupings, and each grouping transmits data to multiple nodes in the next layer. [Fig. 20] Figure 20 shows the use of the deep learning model once training has reached an adequate level. This is called the "inference time," because the recommendation will be inferred from the unlabeled input data. We can see that the input stage is unlabeled. Furthermore, the image pair includes the self-portrait image entered by the user and the image selected by the user in the application. These inputs are fed into the neural network, which will provide an output product and a processing recommendation. [OUTPUT TYPES] [Fig. 21] Figure 21 shows the different types of products and treatment recommendations generated by the recommendation generator based on the process described above. As can be seen, the recommendation can take the form of a specific product in several categories, such as hair care, styling products, and scalp care. Hair care products can include shampoo, conditioner, leave-in conditioner, hair mask, and dry shampoo. Styling products can include hairspray, clarifying spray, gels, and base. Scalp care products can include oil or cream. In addition to recommended products, the recommendation generated by the recommendation generator may also include tutorials and guides to help educate the user. As shown in [Fig. 22][Fig. 22], the recommendation may include guides related to hair science, which explain hair structure and growth using illustrations, publications, videos, and / or animations. These tutorials can be specific to the user's needs. [Scenario examples] Examples of application operating scenarios for different user needs are described below. One situation is where consumer A has very distinct hair types but doesn't know how to classify them. When searching for a specific / personalized hair care routine for their hair type, the consumer may feel overwhelmed by the sheer number of products available. In this scenario, consumer A can perform the following actions using the application in one embodiment. The consumer downloads the app. Between information on habits and current actions on its Hair: washing frequency, styling habits, additional types damage such as the use of color or heat. I take a picture of her hair in standard lighting. Select the type of products he is looking for: styling products or hair care products pillars. Select an image to simulate how he wants his hair to look. seem. Select the types of results he is looking for: frizz control, shine, smooth texture… The app diagnoses your curl type, shine (or lack thereof) shine), its level of damage and its color, as well as the re- products ordered. Finally, the app will offer the option to make purchases on-site. With tutorials on what curly hair is and why it behaves as they do. Another situation is where consumer "B" has very curly hair and is tired of buying so many different products to find one that meets their needs. In this scenario, consumer B can do the following using the application according to one embodiment. The consumer downloads the app. Between information on habits and current actions on its Hair: washing frequency, styling habits, additional types damage such as the use of color or heat. I take a picture of her hair in standard lighting. Select the type of products he is looking for: styling products or hair care products pillars. Select the types of results he is looking for: frizz control, shine, smooth texture… The app diagnoses your curl type, shine (or lack thereof) shine), its level of damage and its color, as well as the re- products ordered. Finally, the app will offer the possibility to make purchases on site. Another situation is where a consumer, "C," has very curly hair and has used the application. They have noticed that their hair has changed in terms of a specific / personalized routine for their hair type, but are unsure of their hair type. In this scenario, consumer C can perform the following actions using the application according to one embodiment. Among new information on the current behavior of hair. I] takes a new photo of her hair in standard lighting. Select the type of products he is looking for: styling products or hair care products pillars. Select the types of results he is looking for: frizz control, shine, smooth texture… The app provides a beauty diagnosis that can be adapted to one's hair type. Finally, the app will offer the possibility to purchase new products. hair care services on-site and tutorials on what curly hair is and why they behave the way they do. Another situation is where consumer "D" has very curly hair but doesn't understand why it behaves the way it does. They are looking to learn and follow their hair journey. In this scenario, the Consumer D can perform the following using the application according to one embodiment. The consumer downloads the app. Between information on habits and current actions on its Hair: washing frequency, styling habits, additional types damage such as the use of color or heat. I click on the learning tools and videos about hair 1 can upload photos of their hairstyles, showcase uses of products as well as results and evaluations and share this data with his friends / family via a universal link which will output a version by- manageable from his journal. The app will also recommend any products he might want try with an option to buy on site. Therefore, the example scenarios above show how the application described in these embodiments can meet various consumer needs. [IMPROVED FEATURES] Entraï de | 4 dui de trai Although the above embodiment illustrates a scenario from the perspective of an individual user, a global system can be realized that uses a community of users to improve both loop diagnosis and product recommendation (similar to some of the requests we have made for Perso). [Fig. 23] Figure 23 shows a 2700 system in which multiple users are connected to the server / cloud platform 102. As noted above, each user can keep a "hair diary" to track their results with snapshots. While this can be used to adjust the treatment or recommendation for each individual user, as described above, the "hair diary" can also be used to allow the system to "learn" how different treatments have worked with different users. In other words, for each user who has achieved satisfactory results using a product or treatment to achieve a desired appearance, all parameters associated with that user can be entered into a machine learning model. These parameters may include the following. User Profile Hair condition at startup by user (curl diagnosis) Appearance desired by the user Percentage of satisfaction with the result Therefore, with continuous input, the system will learn the parameters for an optimal probability of success when a new user with a certain profile, starting hair condition, and desired look begins using the application. Additional satisfaction data can be used to weight a particular treatment when visual results between two users are similar, but different treatments have been used. Community training in loop diagnosis Users provide their own descriptions of what they think their current hair looks like, along with a photo. This can be treated as labeled data within the system itself. Additionally, users can be asked to identify the hair type they see in other photos, which may be based on photos from other users or models. This also provides a labeling process. Furthermore, it provides a method for better associating images attributed to curl patterns with the textual words used to describe those images. In one example, as shown above in [Fig.15], in the application, the user may be asked to provide their own input on the loop pattern they believe is present in an image. There may also be several "game" features that are not only useful for entertaining the user, but also provide valuable data and feedback to optimize system features and deliver personalized results for the user. For example, [Fig. 24] shows a graph where the data collected on the user's selections on a loop pattern are shown for a displayed image. In addition, the correct loop pattern is shown for comparison. [Fig. 24] The type of game shown in [Fig. 24] can be used for at least two purposes. First, it can be used to label an unlabeled image with a loop pattern. In this case, the "correct answer" shown at the end of the game can be the answer with the highest current percentage provided by a community of users. Secondly, it can be used to adjust the entire loop pattern array itself. In [Fig. 25][Fig. 25], we can see that the grouping of user selections does not converge to the correct loop pattern label. If a threshold number of user selections are off by an assumed correct loop type on the loop scale, then a correction may be necessary for the hair image types used to determine a loop type. The first advantage of the game shown in [Fig.24] is that the user can develop an eye for curl patterns in hair. A second advantage of the game shown in [Fig. 24] is that each user's selections can be collected by a central server. This data can be very useful. For example, it can be used to learn about the preferences of the general public. For instance, with additional user data, such as age, location, and other lifestyle habits, a cosmetics company can determine if there are trends in user preferences based on different user categories and locations. On-site purchases Although the product recommendation described above can be made using a large number of products available in a large number of stores or on the Web, it would not be useful to the user if they were in a store when using the app and the recommended product was not available there. Therefore, the application may include a feature that allows for a "store scan" based on products actually on the shelves. Firstly, [Fig. 26A][Fig. 26A] shows that a globally recommended product 3010 can be output by the recommendation generator, as noted above. However, an additional option 3020 can be displayed for "Check products in store". If the user selects option 3020, a new screen appears as shown in [Fig. 26B]. The user can then choose between two options. The first option, 3030, involves checking the current store's inventory online. In this option, the application can use the smartphone's GPS to determine the user's current store location and provide one or more recommended products currently in stock at that store. This option assumes that this information is actually available online. The second option, 3040, involves performing a shelf scan near the user's actual physical location. If the user selects this option, the smartphone's camera function is activated, and the user can capture an image of multiple products located on a shelf. For example, the user can capture an image of a shelf of hair care products in a retail store, as shown in [Fig. 27A][Fig. 27A]. Product scanning can be performed using methods included in the technique. For example, a technique can be used as described in US Patent No. 10,579,962. Following the ray scan, a suitable product can be identified visually and / or textually as shown in [Fig. 27B][Fig. 27B]. This product may be similar to, or have similar ingredients to, the recommended product in Figure 26A. Obviously, many modifications and variations to this disclosure are possible in light of the above teachings. It is therefore understood that, within the scope of the attached claims, the invention may be implemented in ways other than as specifically described herein.

Claims

Claims

1. Recommendation system, comprising: at least one server device; and a mobile user device that includes circuitry processing configured to run an application that receives user information input regarding a plurality of characteristics of at least one user's hair comprising at least less one user loop pattern; and receives an input of a desired hair treatment outcome from the user, wherein processing circuitry of the at least one device server receives user information input and result input desired treatment and provides an output recommendation product or treatment based at least partially on the motive of user-determined loop and the user's desired outcome.

2. The system of claim 1, wherein the user enters a desired result by providing textual descriptors of a state desired hair length of the user.

3. The system of claim 1, wherein the user enters a desired result based on selecting an image of a model type of loops.

4. The system of claim 1, wherein the processing circuitry of the at least one server device determines the recommendation of product or processing by entering user information input and inputting the desired processing result into a neural network that is trained to provide output recommendations based on a combination of image information and text information.

5. The system of claim 1, wherein the processing circuitry of the at least one server device determines the recommendation of product or processing by entering user information input and the desired processing result in a lookup table.

6. The system of claim 1, wherein the recommendation of product or treatment provided at the output includes a recommendation mandate of products and / or treatment linked to at least one of hair care, styling care and scalp care.

7. The system of claim 1, wherein the recommendation of product or processing provided as output includes guides related to the science of hair, which explains the structure and growth of hair using at least one illustration, publication, video and / or animation. [Claim A system according to claim 1, wherein, after providing in output the product or treatment recommendation to the user, the processing circuitry of the mobile user device receives an image of a plurality of hair products which are located in a place of retail where the user is located, and the processing circuitry of the at least one server device is configured to generate a recommendation mandate of at least one of the hair care products that found in the retail location.