System and method for determining human skin properties and processing

Abstract

The disclosure relates to methods and systems for automatically determining and generating human skin attributes and attribute maps by a skin diagnostic and aesthetic processing system. The disclosure proposes a process for automated determination of various skin attributes by using one or more trained models that use the determined skin attributes to identify processing parameters of an energy-based processing system.

Description

[0001] Related applications

[0002] This application is a continuation-of-patent application of U.S. Provisional Application No. 63 / 428,827, filed on November 30, 2022, entitled “System and Method for Skin Type Determination”; U.S. Provisional Application No. 63 / 428,832, filed on November 30, 2022, entitled “System and Method for Determining Human Skin Attributes”; U.S. Provisional Application No. 63 / 428,835, filed on November 30, 2022, entitled “System and Method for Masking Hair in a Skin Diagnostic System”; U.S. Provisional Application No. 63 / 428,849, filed on November 30, 2022, entitled “System and Method for Identifying Vascular Structure Deptin Skin”; and U.S. Provisional Application No. 63 / 428,850, filed on November 30, 2022, entitled “System and Method for Determining Pigment Intensity in a Diagnostic and U.S. Provisional Application No. 63 / 428,877, filed on November 30, 2022, entitled “System and Method for Identifying Pigment Structure Depth in Skin,” which are incorporated herein by reference in their entireties. Background Art

[0003] The target skin is treated and aesthetically treated using therapeutic and aesthetic energy-based treatments. Typically, medical personnel diagnose various skin conditions and set the parameters of the machine that delivers the energy-based treatment. Energy-based treatments can be treatments that are directed to the tissue of the target skin, absorbed by one or more chromophores, and cause a series of reactions, including photochemical, photothermal, thermal, photoacoustic, acoustic, healing, ablation, coagulation, biological, firming, or any other physiological effects. These reactions produce the desired treatment results, such as permanent hair removal, hair growth, treatment of pigment or vascular lesions of soft tissue, rejuvenation or firming, acne treatment, cellulite treatment, venous collapse, or tattoo removal (which may include mechanical destruction and scabbing of tattoo pigment).

[0004] Therapeutic and aesthetic treatments focus on changing the aesthetic appearance by treating conditions including scars, sagging skin, wrinkles, moles, liver spots, excess fat, cellulite, unwanted hair, skin discoloration, spider veins, etc. The target skin is treated using energy-based systems (such as lasers and / or light-based systems). In these treatments, light energy with predefined parameters can generally be projected onto the target skin area. Medical personnel may have to consider skin attributes such as skin type, presence of tanning, hair color, hair density, hair thickness, blood vessel diameter and depth, lesion type, pigment depth, pigment intensity, tattoo color and type in order to decide on the treatment parameters to be used. Summary of the Invention

[0005] In one aspect of the disclosure, a system for determining skin properties and treatment parameters for a target skin for an aesthetic skin diagnosis and treatment unit includes: a display; at least one illumination light source; an image capture device; a source for providing energy-based treatment; and a processor. Furthermore, a memory is communicatively coupled to the processor, wherein the memory stores processor-executable instructions that, when executed, cause the processor to: activate the at least one illumination light source to illuminate at a plurality of monochromatic wavelengths; obtain an image at the plurality of monochromatic wavelengths from the image capture device; receive target skin data including data for each pixel of the obtained image; analyze the target skin data using a plurality of trained skin property models; determine at least one skin property classification for the target skin using the trained skin property models; analyze the at least one skin property classification for the target skin using the trained skin treatment models; identify treatment parameters for the energy-based treatment source for the at least one determined skin property classification using the trained skin treatment models; and display the treatment parameters identified as treating the skin properties.

[0006] In one aspect of the disclosure, a system generates and displays a list of target skin attributes based on analysis using trained skin attribute models. An energy-based treatment source is activated to treat the target skin using the determined treatment parameters. Furthermore, multiple trained skin attribute models are trained by: (i) providing the skin attribute models with multiple labeled images of at least one skin attribute stored in a database; and (ii) configuring the skin attribute models to classify the multiple labeled images into the at least one skin attribute.

[0007] In another aspect of the disclosure, the plurality of different wavelengths includes 450 nm, 490 nm, 570 nm, 590 nm, 660 nm, 770 nm, and 850 nm. The processor is further configured to register and align the images of the plurality of monochromatic wavelengths after acquiring the images, and to generate and display a map of the target skin using any combination of the plurality of monochromatic wavelengths, or to generate and display a map of the target skin from wavelengths representing red, green, and blue.

[0008] In another aspect of the disclosure, one of the skin attributes is hair on the target skin, and the hair masking model is a skin attribute model among multiple skin attribute models, and the processor is further configured to: receive target skin data of one monochromatic wavelength among multiple monochromatic wavelengths; and determine one of two classifications of hair or background for each pixel of the image of one monochromatic wavelength using the hair masking model.

[0009] In another aspect of the disclosure, the processor is further configured to instruct the additional skin attribute model to remove hair pixels marked as hair by the hair masking model from further analysis of the target skin. One of the skin attributes is skin type, and the skin type model is one of the plurality of skin attribute models. The processor is further configured to receive skin type data comprising average calibrated reflectance values ​​for the total pixels of each monochrome image; and determine six classifications of skin type using the skin type model. The skin attribute is at least one of: melanin density, vascular density, or scattering.

[0010] In some aspects of the disclosure, the processor is further configured to: receive skin type data comprising a plurality of absolute reflectance values ​​for each pixel representing a plurality of wavelengths; analyze the plurality of absolute values ​​for each pixel by comparing them to a lookup table (LUT) value using at least one of a melanin model or a vascular model, wherein the LUT includes values ​​for the skin model representing the lighting effects of a known physical model on human skin and physical measurements representing concentrations of skin properties in the target skin; and, for each pixel, identify a LUT entry that is closest in distance to the plurality of measured absolute values ​​for each pixel for at least one of melanin density or vascular density, wherein the distance can be an approximation of a specific distance.

[0011] In another aspect of the disclosure, one of the skin attributes is vascular lesion depth, and the vascular depth model is one of the plurality of skin attribute models. The processor is further configured to: receive target skin data of a plurality of monochromatic wavelengths; determine, for each pixel, one of four classifications: deep vessel, medium vessel, superficial vessel, or background using the vascular lesion model; and generate and display a labeled graph illustrating the classification of the vascular lesion depth.

[0012] In another aspect of the disclosure, one of the skin attributes is pigmented lesion depth, and the pigment depth model is one of the plurality of skin attribute models. The processor is further configured to: receive target skin data for two monochromatic wavelengths of a plurality of monochromatic wavelengths, wherein one monochromatic wavelength represents a lowest wavelength value for the system and the second monochromatic wavelength represents a highest wavelength value for the system; receive classified pixels for vascular depth from the vascular depth model; analyze pixels not classified by the vascular depth model for outliers in the dark color for each of the two monochromatic wavelengths; determine a classification for each analyzed pixel using the pigmented lesion model, wherein outliers with the lowest wavelength value are lightly pigmented lesions and outliers with the highest wavelength value are darkly pigmented lesions; and generate and display a plot with labels to illustrate the classification of pigmented lesion depth.

[0013] In another aspect of the disclosure, one of the skin attributes is pigment lesion intensity, and the pigment intensity model is one of the plurality of skin attribute models. The processor is further configured to: receive target skin data of three features from each of a plurality of monochrome images, wherein the features are a threshold representing a 99th percentile of melanin concentration of the lesion, a median melanin level for the entire image calculated from the melanin density model, and a 99th percentile subtracted from the calculated median melanin level; and determine whether the pigment lesion intensity is a light lesion or a dark lesion based on the features.

[0014] In one aspect of the disclosure, the processor is further configured to: receive a value of at least one of a melanin density value from a melanin model or a vascular density value from a vascular model in a LUT; calculate a new value of the melanin density value or the vascular density value based on setting other skin attributes on the LUT to be closest to zero; and generate a graph of the melanin density or the vascular density using the calculated new value wavelength.

[0015] In additional aspects of the disclosure, a processor having a trained skin treatment model is further configured to receive the following information: treatment safety parameters, energy treatment source capability parameters, at least one skin area to be treated from a user, at least one skin problem indication from the user for treatment based on the skin area to be treated, and outputs of multiple skin attribute models related to the at least one skin problem indication. The processor and the trained skin treatment model then determine target skin treatment parameters for the energy-based treatment based on the received information, and display the target skin treatment parameters for the energy-based treatment.

[0016] The determination of skin treatment parameters is accomplished using a processing lookup table, and the processor is further configured to: determine a skin treatment lookup table from a plurality of skin treatment lookup tables, each of which is based on a specific skin problem indication; match the outputs of a plurality of skin attribute models with the processing parameters of the determined skin treatment lookup table; and display the matched skin treatment parameters of the energy-based processing.

[0017] Furthermore, the processor having the trained skin treatment model is further configured to: generate and display a red, green, and blue (RGB) image of the target skin; generate and save to memory at least one of a plurality of maps; and display the generated at least one map, wherein the at least one of the plurality of maps comprises: a melanin density map, a vascular density map, a pigmented lesion depth map, a vascular lesion depth map, pigment intensity, or any combination thereof. The at least one skin problem indicator is at least one of: a pigmented lesion, a vascular lesion, a combined pigmented and vascular lesion, hair removal, or any combination thereof.

[0018] In another aspect of the disclosure, a method for determining skin properties and treatment parameters of a target skin is provided, comprising: providing a display, at least one illumination light source, an image capture device, a source for providing energy-based processing, a memory, and a processor; activating, by the processor, at least one illumination light source to illuminate at multiple monochromatic wavelengths; obtaining, by the processor, an image at multiple monochromatic wavelengths from the image capture device; receiving, by the processor, target skin data including data for each pixel of the obtained image; analyzing, by the processor, the target skin data using multiple trained skin property models; determining, by the processor, at least one skin property classification of the target skin using the trained skin property models; analyzing, by the processor, at least one classification of the skin properties of the target skin using the trained skin processing models; identifying, by the processor, treatment parameters of the energy-based processing source for at least one determined skin property classification using the trained skin processing models; and displaying, by the processor, the treatment parameters identified as treating the skin properties.

[0019] The method may also include the skin attribute being at least one of: melanin density, vascular density, and scattering, and wherein the method further includes: receiving, by the processor, skin type data comprising a plurality of absolute reflectance values ​​for each pixel representing a plurality of wavelengths; analyzing, by the processor, the plurality of absolute values ​​for each pixel by comparison with a lookup table (LUT) value using at least one of a melanin model or a vascular model, wherein the LUT includes values ​​of the skin model representing the lighting effects of a known physical model on human skin and a physical measurement representing the concentration of the skin attribute in the target skin; and identifying, by the processor, for each pixel, a LUT entry for at least one of melanin density or vascular density that is closest in distance to the plurality of measured absolute values ​​for each pixel, wherein the distance may be an approximation of a specific distance.

[0020] A method for generating a map, wherein the method further includes: receiving, by a processor, a value of at least one of a melanin density value from a melanin model or a vascular density value from a vascular model in a LUT; calculating, by the processor, a new value of the melanin density value or the vascular density value based on setting other skin attributes on the LUT to be closest to zero; and generating, by the processor, a map of the melanin density or the vascular density using the calculated new value wavelength.

[0021] In another aspect of the disclosure, the method further includes: receiving, by a processor having a trained skin treatment model, the following information: a treatment safety parameter, an energy treatment source capability parameter, at least one skin area to be treated from a user, at least one skin problem indication from the user for treatment based on the skin area to be treated, and outputs of multiple skin attribute models related to the at least one skin problem indication. The processor having the trained skin treatment model then determines, by the processor having the trained skin treatment model, target skin treatment parameters for the energy-based treatment based on the received information; and displaying, by the processor having the trained skin treatment model, the target skin treatment parameters for the energy-based treatment.

[0022] In a final aspect of the disclosure, determination of skin treatment parameters is accomplished using a treatment lookup table, and the method further includes: determining, by a processor having a trained skin treatment model, one of a plurality of skin treatment lookup tables, wherein each of the skin treatment lookup tables is based on a specific skin problem indication; matching, by the processor, outputs of a plurality of skin attribute models with treatment parameters of the determined skin treatment lookup table; and displaying, by the processor, the matched skin treatment parameters of the energy-based treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various embodiments of the disclosure may be further explained with reference to the accompanying drawings, in which similar structures are referred to by like reference numerals throughout the several views. The drawings shown are not necessarily drawn to scale, but rather generally emphasize illustrating the principles of the disclosure. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching one or more illustrative embodiments to those skilled in the art.

[0024] Figure 1A and Figure 1B is a block diagram of the skin diagnosis system of the present invention.

[0025] Figures 2A to 2C A diagram depicting an apparatus as part of the skin diagnostic system of the present invention.

[0026] Figure 3 A series of monochromatic images obtained by the system of the present invention are shown.

[0027] Figure 4A and Figure 4B Depicted are the uneven illumination of an image and the correction used in the present invention.

[0028] Figure 4C Depicted is an enhanced view of a blood vessel obtained by combining two images acquired at different wavelengths as the output of the present invention.

[0029] Figure 5 is a flow chart depicting a method of the present invention for determining an attribute or characteristic of target skin.

[0030] Figure 6 An example of a machine learning model of the present invention is shown.

[0031] Figure 7 Shown is the image output of the hair masking machine learning model of the present invention.

[0032] Figure 8 A second example of a machine learning model of the present invention is shown.

[0033] Figure 9 is an example of a lookup table used in the present invention.

[0034] Figure 10 is a graph of the absorption coefficients of the major chromophores in the target skin used in the present invention.

[0035] Figure 11 A third example of a machine learning model of the present invention is shown.

[0036] Figure 12 is a second example of a lookup table used in the present invention.

[0037] Figure 13A and Figure 13B Graphs depicting melanin / pigment and vascular / erythema density as outputs of the present invention.

[0038] Figure 14A is a flow chart depicting a method of the present invention for determining properties or characteristics of a target skin using a lookup table (LUT).

[0039] Figure 14B is a flow chart depicting a method of the present invention for utilizing a LUT to generate an RGB map depicting attributes or characteristics of a target skin.

[0040] Figure 15 A vascular lesion depth map is depicted as an output of the present invention.

[0041] Figure 16A and Figure 16B A melanin lesion depth map is depicted as an output of the present invention.

[0042] Figure 17A is a flow chart depicting the method of the present invention for generating a combined pigment and vascular lesion map.

[0043] Figure 17B Depicted are vascular lesion and melanin lesion depth maps as output of the present invention.

[0044] Figure 18 is a flow chart depicting a method of the present invention for generating recommendations for treatment parameters. DETAILED DESCRIPTION

[0045] Various detailed embodiments of the disclosure are disclosed herein in conjunction with the accompanying drawings; however, it should be understood that the disclosed embodiments are merely illustrative. In addition, each example provided in conjunction with the various embodiments of the disclosure is intended to be illustrative and not restrictive.

[0046] Skin tissue is a highly complex biological organ. While the basic structure is common to all humans, there is considerable variation within and between different regions within a given individual. Variations include skin color (melanin content in the basal layer), hair color and thickness, collagen integrity, vascular structure, various types of vascular and pigmented lesions, and foreign bodies such as tattoos.

[0047] Various embodiments of the disclosure provide a technical solution that assists medical personnel in selecting optimal treatment presets and determining target skin properties associated with skin conditions, skin diseases, or skin responses to treatments using a target skin diagnostic system that can be included in a skin treatment system. In some embodiments, data for a skin area (target skin) is collected before and after treatment, and the data can be compared for immediate analysis of how to proceed with treatment of the target skin. In some embodiments, the target skin's response to treatment is further used to determine the efficacy of the treatment and to train a treatment module, such as, for example, determining the moisture present in the skin after treatment.

[0048] The disclosure relates to methods and systems for determining multiple properties, characteristics, and features of a target person's skin (hereinafter referred to as skin attributes) using a skin diagnostic system, which may be part of an aesthetic skin treatment system. The disclosure proposes automating the process of determining multiple skin attributes by type using one or more trained models.

[0049] One or more trained models are trained using a large number of parameters related to the classification of multiple skin attributes of the target skin to output specific skin attributes of the target skin of the person. Skin attributes may include, but are not limited to: skin type using the Fitzpatrick scale, pigment or melanin (hereinafter referred to as melanin), vascularity or erythema (hereinafter referred to as vascularity), pigmented lesion intensity, pigmented lesion depth, vascular lesion depth, masking hair data, and the scattering coefficient of the skin. The scattering coefficient is a measure of the ability of a particle to scatter photons from a light beam.

[0050] In some embodiments, skin properties can be determined for tattoo removal. In tattoo removal, the challenge is twofold. First, in order to selectively destroy tattoo ink, the optimal energy-based method (such as laser wavelength) should be selected to achieve selective absorption of a specific ink color or colors while minimizing nonspecific effects. However, commonly used tattoo inks are poorly regulated, and such ink compositions are highly variable. Therefore, ink colors that appear similar may have a wide range of peak absorption, and medical personnel cannot determine the exact type / properties of a specific ink, and thus the optimal treatment to use. Furthermore, in addition to the color properties of the ink, the skin type (amount of melanin), the depth and amount of the ink should also be considered to achieve the optimal energy-based settings and clinical outcomes. Second, in addition to the color properties of the ink, the skin type (amount of melanin), the depth and amount of the ink should also be considered to achieve the optimal parameters and clinical outcomes.

[0051] In addition, principal component analysis (PCA) can be used, which enables robust classification of valuable parameters while reducing the overall dimensionality of the acquired data. In other words, PCA distinguishes data features relative to their importance to the ultimate clinical outcome. The most relevant parameters can be used to develop physical energy-based treatment interaction models, including, for example, thermal relaxation and soft tissue coagulation. In addition, the large amount of highly correlated data allows the construction of empirical equations based on quantitative immediate biological responses (such as erythema in hair removal and frost formation in tattoo removal treatments). Currently, immediate responses are subjectively assessed by medical personnel in a non-quantitative manner without any dynamic quantification. Details regarding the use of PCA and methods / systems for tattoo removal are further described in U.S. application serial number 17 / 226,235 filed on April 9, 2021, which is hereby incorporated by reference in its entirety.

[0052] Values ​​and / or maps for skin properties such as, but not limited to, melanin density, vascular density, pigment depth maps, vascular depth maps, and optical property maps are generated by the skin diagnostic system, and these properties may or may not reveal the physical condition of the target skin.

[0053] Figure 1A An exemplary block diagram of a skin diagnostic system 100 that can be integrated into an energy-based processing system is shown. Energy-based processing can include, but is not limited to, laser, intense pulsed light, radio frequency, ultrasound, visible light, ultraviolet light, light-emitting diodes (LEDs), or any combination thereof. According to some embodiments of the disclosure, the skin analysis module 103 can include one or more modules 107 that can be located in the memory of the skin diagnostic system. It should be understood that such modules can be represented as a single module or a combination of different modules. It should be understood that these modules can be represented as a single processing module or a combination of different processing modules.

[0054] Figure 1B Also shown is an example of a block diagram in which the skin diagnostic system 100 includes a processor or controller 104 (hereinafter referred to as the processor). The skin diagnostic system 100 may also include a memory (not shown) and input / output interfaces and devices 105, such as, but not limited to, a display, a computer keyboard, and a mouse.

[0055] In some embodiments, as Figure 1A and Figure 1BAs seen in FIG, one or more modules 107 may include, but are not limited to, a target skin data receiving module 201, a target skin data analyzing module 202, an operating parameter determining module 203, a processing module 109, and one or more other modules (not shown) associated with the skin diagnostic system. In some embodiments, the target skin data receiving module 201 receives target skin data of the target skin being analyzed. In some embodiments, the target skin data analyzing module 202 is used to analyze, parse training data, and utilize the training data to train the skin diagnostic system. In some embodiments, one or more skin processing modules 109 are skin treatment models for analyzing, parsing, and outputting parameters to treat the target skin. In some embodiments, there are preset operating parameters for the skin treatment system, which include, but are not limited to: technical specification limitations of the aesthetic skin treatment unit, safety parameters as a function of the expected treatment and / or clinical effect for the patient's specific skin type, skin areas that should not receive treatment (such as "no treatment" areas), or any combination thereof.

[0056] In some embodiments, one or more modules 107 are configured so that the module collects and / or processes the data results and then is stored in the memory of the skin diagnostic system as part of data 108, such as training data, operational processing parameter data, or analyzed target skin data (not shown). In some embodiments, data 108 can be processed by one or more modules 107. In some embodiments, one or more modules 107 can be implemented as dedicated units, and when implemented in such a manner, these modules can be configured with the functionality defined in the disclosure to produce a novel hardware device. As used herein, the term module can refer to an application specific integrated circuit (ASIC), an electronic circuit, a field programmable gate array (FPGA), a programmable system on chip (PSoC), a combinational logic circuit, and / or other suitable components that provide the described functionality. In some embodiments, training data is used to train the module to successfully identify the target skin attributes. In some embodiments, unsuccessful identification of the target skin attributes is included for training the model.

[0057] In some embodiments, and as Figure 1BAs seen in , the skin diagnostic system 100A is a combined skin diagnostic and energy-based processing system having components for generating energy-based processing, and an additional component is a processor or controller component 102 (hereinafter referred to as a PC component). The combined system 100A may also include an input / output interface 105 and devices such as, but not limited to, a display, a computer keyboard, and a mouse. The PC component 102 may have two different processors (not shown) connected to each other, and the connection may be an Ethernet cable. The first of the two processors may be configured with modules for collecting images, analyzing the collected images using a plurality of trained models to generate skin properties, and indicating the flow to the user via the input / output module. The second of the two processors may be configured with modules for managing a graphical user interface (GUI) for the input / output module, controlling processing energy, and analyzing skin properties using a skin processing module to determine the processing to use.

[0058] In some embodiments, the combined system further includes a module 210 configured to control acquisition of image data using an image capture device, such as a multispectral camera, which may be part of the handpiece 1300. In some embodiments, the combined system further includes a processing component with the handpiece to deliver energy-based treatment 1350.

[0059] In some embodiments, the target skin data includes skin properties or at least one property of the target skin tissue to be analyzed. In some embodiments, the target skin data includes at least one pre-processed target skin property (pre-processed target skin data) and at least one real-time target skin property (real-time target skin data). The pre-processed target skin data can be a skin property associated with the target skin before the aesthetic treatment is performed on the target skin. The real-time target skin data can be a skin property obtained in response to the real-time aesthetic treatment. In some embodiments, the target skin data is obtained before the aesthetic treatment, during regular time intervals, and immediately after the aesthetic treatment, or any combination thereof. The target skin data at any time before and after the aesthetic treatment can be analyzed to develop different treatment parameters. The treatment can be completed in a short period of time, such as a laser shot, so the acquisition and decision-making of the image data will also be expected to be completed quickly, that is, the feedback signal can be delivered in less than a few milliseconds.

[0060] In some embodiments, upon receiving the target skin data, the target skin analysis module 202 can be configured to analyze the target skin data using multiple trained models to determine multiple skin properties of the target skin. In some embodiments, the multiple trained models can be multiple machine learning models, deep learning models, or any combination thereof. Each of the multiple trained models can be trained separately and independently. In some embodiments, the training data can be used to pre-train each of the multiple trained models. In some embodiments, the target skin data is associated with skin properties and includes, but is not limited to, melanin, the anatomical location, spatial and depth distribution (epidermis / dermis) of melanin, the spatial and depth distribution (epidermis / dermis) of blood, melanin morphology, vascular morphology, venous (capillary) network morphology diameter and depth, collagen spatial and depth distribution (epidermis / dermis), water content, melanin / blood spatial homogeneity, hair, temperature, or morphology.

[0061] Figure 2A A diagram depicts a device 1000 as part of a skin diagnostic system for sensing and analyzing skin conditions, according to some embodiments of the skin diagnostic system. Device 1000 can be a diagnostic standalone unit (i.e., without an energy-based treatment source). According to some embodiments, the skin diagnostic system can also include an energy-based treatment source.

[0062] The apparatus can include a frame 1023 configured to circumscribe the target skin 1030 to stretch or flatten the target tissue 1030 to capture diagnostic images. In some embodiments, the target skin data includes captured diagnostic images of the target skin 1030. The frame 1023 can include one or more fiducial markers 1004. The fiducial markers 1004 can be included in the image and used to digitally register multiple captured images of the same target tissue 1030.

[0063] The apparatus may include an electro-optical unit 1001 comprising an illuminator assembly 1040 , an optical assembly 1061 , and an image sensor assembly 1053 .

[0064] The illuminator assembly 1040 can be configured to illuminate the target tissue 1030 during image capture. The illuminator assembly 1040 can include multiple groups of one or more illumination elements (also referred to as illumination light sources (such as LEDs)), each group having a different optical output spectrum (e.g., peak wavelength). When capturing an image of the target skin tissue 1030, a combination of one or more spectrums can be used for illumination. Images under each spectrum can be captured separately and then combined. Alternatively or additionally, the illumination elements of the illuminator assembly having multiple spectrums can be illuminated simultaneously to capture an image. The optical assembly 1061 focuses the reflected / backscattered illumination light onto the image sensor of the image sensor assembly 1053.

[0065] In this example, the device may also include a processor 1050, or the processor 104 from the previous figures. The skin diagnostic system may have more than one processor. The processor 1050 may be responsible for controlling the imaging parameters of the illuminator assembly 1040 and the image sensor assembly 1053. The imaging parameters may include frame rate, image acquisition time, the number of frames added to the image, the illumination wavelength, and any combination thereof. The processor 1050 may also be configured to receive a start signal from a user of the device (e.g., pressing a trigger button) and may communicate with the skin diagnostic system.

[0066] Figure 2B and Figure 2C 1 is a skin imaging handpiece 1300 according to some embodiments of the present invention. In some embodiments, handpiece 1300 includes a trigger button 1301, a heat sink 1302, and a frame 1303 including fiducial markers 1304. In some embodiments, frame 1303 is removable from handpiece 1300, thereby enabling interchange between frames of various sizes or shapes according to treatment instructions. Figure 2C Frame 1303 is shown removed from handpiece 1300. Details of systems and methods including treatment components with a handpiece to deliver energy-based treatment are further described in U.S. application serial number 17 / 565,709, filed December 30, 2021, and U.S. application serial number 17 / 892,375, filed August 22, 2022, both of which are hereby incorporated by reference herein in their entirety.

[0067] In some embodiments, Figure 3 A skin diagnostic system is depicted having an image capture system configured to capture multiple monochrome images at different peak wavelengths (hereinafter referred to as wavelengths) via an image sensor. In some embodiments, there are seven monochrome image captures, each at a different wavelength, for example, 450 nm, 490 nm, 570 nm, 590 nm, 660 nm, 770 nm, and 850 nm. Figure 3 As seen in.

[0068] Preprocessing

[0069] In some embodiments of the skin diagnostic system, there are multiple pre-processing methods for the captured images. The captured images can be cropped or resized to accommodate the measurement of the energy-based treatment site. Additional pre-processing functions that can be utilized are quality inspection, lighting correction, registration, and reflectance calibration.

[0070] Figure 4A shows an example of an image depicting uneven illumination, Figure 4B An example of a corrected illumination image after pre-processing illumination correction is shown.Registration between all monochrome images aligns all monochrome images.

[0071] Reflectance calibration can be done in real time. Real-time calibration can be done according to the following formula: Calibrated image = (1 registered image / 2 calibration coefficient) x (marker calibration value / measured marker)

[0072] The registration images are multiple monochrome images aligned with each other. The calibration coefficients are obtained from the reflective material (which can be ) for each monochrome image. The average of the multiple reflectance values ​​can be used as a calibration coefficient. The calibration coefficient is typically determined when the skin diagnostic system is manufactured. The marker calibration value relates to the fiducial marker 1304. The same process is used for using the calibration coefficient, except that it is determined from a cropped image of only the fiducial marker, which is also determined at the time of manufacturing. The marker measurement is a real-time current value of the calibration of the fiducial marker cropped image. After pre-processing, the incoming image data can then be parsed for input into a module or model.

[0073] In some embodiments, the skin diagnostic system generates a color map or RGB image from a monochrome image. The color map can be a 24-bit RGB image in an uncompressed or compressed image format. The image is constructed using a 650nm wavelength, a 570nm wavelength, and a 450nm wavelength. In some embodiments, each wavelength used in the color map first has a global brightening step and a local contrast enhancement step performed before the wavelengths are combined. In some embodiments, any monochrome image can be combined. Other wavelength combinations can have the effect of enhancing specific skin structures / conditions, such as in Figure 4C As can be seen, two wavelengths are used to show the approximate blood vessel map.

[0074] Figure 5 is a general flow chart depicting a method for determining a property or characteristic of target skin.

[0075] At block 501 , a skin diagnostic system is configured to receive target skin data including a multispectral image.

[0076] At block 503 , the skin diagnostic system is configured to analyze target skin data using at least one trained model to determine properties of the target skin.

[0077] At block 505, the system is configured to output the analyzed skin attributes of the target skin. In some embodiments, these attributes are associated with skin conditions, skin diseases, skin response to treatment, or any combination thereof.

[0078] Hair masking

[0079] In some embodiments, the hair masking module in one or more modules 107 is used to automatically identify hair in the target skin data and remove (mask) the hair from further analysis. In some embodiments, the deep learning model used for hair masking is a U-Net deep learning semantic segmentation classifier model with three layers of depth, for example, see Figure 6 , hereinafter referred to as the hair masking model. In some embodiments, the hair masking model is trained to detect hair or background (everything except hair) for each pixel of the image. In some embodiments, the hair masking model is trained using a labeled target skin image by pixel-wise labeling of hair in the target skin image.

[0080] In some embodiments, the hair masking model receives a single monochrome image in the target skin image. This single image can be at a wavelength between about 590 nm and 720 nm. Figure 7 As seen in the image, and in some embodiments, the output of the hair masking model is the classification of hair or background in the image and the removal of hair from the target skin image and target skin data. In some embodiments, the removal of hair from the target skin image and target skin data removes hair data and pixels from any further analysis of the target skin by instructing other models and modules in the skin diagnostic system to ignore pixels marked as hair by the hair masking model. In some embodiments, the hair masking data can be collected and stored in memory for further development of the hair masking model.

[0081] Skin type

[0082] In some embodiments, a skin type module in one or more modules 107 automatically determines the skin type of a person's skin based on the Fitzpatrick scale. The Fitzpatrick scale is a measure of the skin's response to ultraviolet (UV) light and is a designation for a person's entire body. Typically, a trained medical professional makes such a determination. In some embodiments, the skin type module includes a machine learning multilayer perceptron-type neural network model (hereinafter referred to as the skin type model). In some embodiments, the skin type model is trained using images of the target skin numbered 1 to 6 with appropriate skin type labels. In some embodiments, the images labeled for training are labeled by a medical professional.

[0083] Figure 8 is a non-limiting example of a multilayer perceptron-type neural network with two hidden layers used in the skin type model, comprising: a first hidden layer 801 having twenty neurons, a second hidden layer 803 having ten neurons, and an output layer 805 having three neurons. The neural network utilized in the skin type model may have a sigmoid nonlinear activation function, where the output is a nonlinear function of a weighted sum of the inputs. Other typical features of a neural network are that W represents weights, which are parameters within the neural network that transform input data within the network's hidden layers. B represents bias, which is a constant value (or constant vector) added to the product of the neural network's inputs and weights.

[0084] In some embodiments, the skin type model receives skin type data comprising the average calibrated reflectance value of the total pixels of each monochrome image [the average spectrum of all monochrome images], and the output of the skin type model classifies the skin type into one of six skin types. The skin type data can be collected in a memory for further development of the skin type model.

[0085] In some embodiments, the output is the skin type of the target skin to be treated and is automatically determined by a skin type module in the one or more modules.

[0086] Lookup Table - Skin Properties

[0087] The reflectance image from skin tissue can be determined by two physical properties: chromophore absorption and the resulting reduced scattering of illumination. Integrating these parameters by tissue depth yields the reflectance image. Thus, reflectance imaging (different wavelengths, polarizations, and patterns) provides information about fundamental skin optical properties at depths up to several millimeters.

[0088] In some embodiments, skin properties related to the spectral analysis are automatically determined and generated. In some embodiments, a lookup table (LUT) is constructed using a known physical model of the effects of lighting on skin and generating multiple skin property values ​​for the skin model, such as Figure 9 .

[0089] Skin property values ​​may include, but are not limited to, melanin (pigment) density, vascular (erythema) density, and light scattering coefficient.In some embodiments, a LUT with skin properties per wavelength is accomplished using physical equations and spectral analysis. Figure 10 A graph shows the absorption coefficients of the primary chromophores in the target skin as a function of illumination wavelength. The primary chromophores are melanin, oxygenated and deoxygenated hemoglobin, and water. Together with a physical model of light scattering in human skin, this provides the basis for completing the LUT. Furthermore, the LUT values ​​represent a physical measure of the concentration of a skin property within the human skin volume. For example, if melanin is determined to be 0.06, then the concentration of melanin is 0.06%.

[0090] In some embodiments, a machine learning model receives image skin data and links the spectral wavelength response to the amount of skin chromophores. In some embodiments, this can be the amount of other skin chromophores (color-producing regions of molecules), such as, but not limited to, vascular regions, melanin regions, and collagen.

[0091] Each pixel of each of the multiple wavelength images is input into a machine learning model to search on the LUT. Each of the multiple skin attribute values ​​and images utilizes a different machine learning model (hereinafter referred to as a universal model) to determine each skin attribute value on the target skin. For example, when seven wavelength images are used, seven numbers for each pixel are input into the universal model to obtain an output of one number for each pixel. A brute force or naive search in a long LUT will typically analyze every row of the table and is very slow and time-consuming, especially for every pixel in multiple monochromatic wavelength images. Therefore, when using a LUT, a universal model is utilized to achieve faster and more efficient functions.

[0092] Optionally, an estimate of a specific LUT skin property in the target skin is output for each pixel using a general model of the LUT. In some embodiments, once the estimate for each pixel is determined, anomalies or outliers of the LUT skin property are identified. In some embodiments, the abnormal level of the LUT skin property is determined by the following equation: Abnormal Level ≥ Average (LUT Skin Property) + cx STD (LUT Skin Property), where c is an arbitrary coefficient, for example, 2. In some embodiments, the coefficient is experimentally determined by analyzing the distribution of the LUT skin property in a large number of images. The coefficient is different for each LUT skin property. Non-abnormal levels are then classified, and normal skin and abnormal levels are identified as specific LUT skin property densities.

[0093] Optionally, a basic map is generated showing areas with abnormal levels of the LUT skin attribute and corresponding color bars. In some embodiments, the scale of the map is adjusted so that a range of 0-15% is mapped to a 0-255 numerical level for displaying the map. In some embodiments, the abnormal level pixels are compared to the total number of pixels to determine the relative area of ​​the abnormal region. For example, LUT skin attribute value = total pixels in the map with abnormal values ​​ / total pixels in the image display unit: image area % (0-100%).

[0094] In some embodiments, a general model is trained using multiple pixels from the image skin data on the LUT data to determine the properties in the target skin to be identified. Machine learning models for specific skin properties are discussed further below.

[0095] Melanin density

[0096] In some embodiments, the melanin density and map are automatically determined by the melanin module in the one or more modules 107 using the LUT. In some embodiments, the machine learning model for melanin density and map is a machine learning regression tree model for identifying melanin (hereinafter referred to as a melanin tree model), an example of which is shown in FIG. Figure 11 As a specific example, the melanin tree model has a tree depth of 25 layers (e.g., see 1101) and 132 leaves (e.g., see 1103). The LUT discussed above is used by the melanin tree model to determine melanin density and generate a melanin map.

[0097] In some embodiments, the melanin tree model receives image skin data for each pixel, the image skin data having a plurality of absolute reflectance values ​​representing a plurality of wavelengths imaged. The melanin tree model then analyzes the plurality of absolute values ​​for each pixel for comparison with the LUT values ​​and identifies, for each pixel, a LUT entry whose value is closest in distance to the plurality of measured absolute values ​​for each pixel. The distance can be an approximation of a specific distance, such as a cosine, Euclidean distance, or any combination thereof.

[0098] In some embodiments, as Figure 13A The melanin density map 1310 shown is generated from multiple wavelengths (e.g., seven wavelengths) by determining the value that is closest in distance to the measured value using the approximation of the specific distance already described above. Next, in some embodiments, the processor calculates the value in the LUT based on a computer program that best represents the melanin density value that has been determined (see Figure 12 , 1203), while the other skin attributes on the LUT are closest to zero. Figure 12 In the example, the vascularity value is another skin attribute. The row of the LUT with the closest melanin value and where the other skin attributes are closest to zero is used to represent the RGB map of melanin. In this example, Figure 12 Line 1204.

[0099] Vessel density

[0100] In some embodiments, the vessel density and map are automatically determined by the vessel module in one or more modules 107 using the LUT. In some embodiments, the machine learning model used for the vessel density and map is a machine learning regression tree model for identifying vascular regions (hereinafter referred to as a vascular tree model). As a specific example, the vascular tree model has 41 layers (e.g., see Figure 11 1101) and 35,855 leaves (see, for example, Figure 11 The LUT is also used by the vascular tree model to determine vascular density and generate a vascular map.

[0101] In some embodiments, and similar to the melanin tree model above, the vascular tree model receives image skin data and links the spectral wavelength response to the amount of skin chromophores, in this case to the vascular density.

[0102] In some embodiments, the vascular tree model receives image skin data for each pixel, the image skin data having a plurality of absolute reflectance values ​​representing a plurality of wavelengths imaged. The vascular tree model then analyzes the plurality of absolute values ​​for each pixel for comparison with the LUT values ​​and identifies, for each pixel, a LUT entry whose value is closest in distance to the plurality of measured absolute values ​​for each pixel. The distance can be an approximation of a specific distance, such as a cosine distance, a Euclidean distance, or any combination thereof.

[0103] In some embodiments, as Figure 13B The illustrated vascular density map 1312 is generated from multiple wavelengths (e.g., seven wavelengths) by determining the value closest in distance to the measured value using the specific distance approximation described above. Next, in some embodiments, the processor, based on a computer program, calculates the value in the LUT that best represents the determined vascular density value while the other skin attributes on the LUT are closest to zero. The row of the LUT with the closest vascular density value and in which the other skin attributes are closest to zero is used to represent the RGB map of vascular density.

[0104] scattering

[0105] In some embodiments, the scattered light values ​​are automatically determined by the scattering module in one or more modules 107, also using a LUT. In some embodiments, the machine learning model used for the scattered light values ​​is a machine learning regression tree model (hereinafter referred to as a scattering tree model) for identifying the scattering properties of the target skin. As a specific example, the scattering tree model has a tree depth of 35 levels and 81,543 leaves. The LUT described above is used by the scattering tree model to generate the scattering values.

[0106] In some embodiments, the scattering tree model receives image skin data per pixel, the image skin data having a plurality of absolute reflectance values ​​representing a plurality of wavelengths imaged.

[0107] The scattering tree model then analyzes the multiple absolute values ​​for each pixel and identifies, for each pixel, a LUT entry whose value is closest in distance to the multiple measured absolute values ​​for each pixel, compared to the LUT values. The distance can be an approximation of a specific distance, such as cosine, Euclidean distance, or any combination thereof.

[0108] In some embodiments, skin chromophore estimation predicts treatment energy absorption to predict treatment outcomes (assuming known melanin / pigment and blood response to energy / temperature). In some embodiments, output values ​​and plots for melanin density, vascular density, and scattered light can be collected in memory for further development of machine learning models.

[0109] Examples of using LUTs

[0110] Figure 14A is a flow chart depicting a method for a machine learning model in a machine learning model that employs a LUT to determine, for each pixel, a specific skin attribute value on the LUT table.

[0111] At block 1401 , a skin diagnosis system is configured to receive image skin data comprising a plurality of monochrome images of target skin.

[0112] At block 1403 , the skin diagnostic system is configured to analyze each pixel of a plurality of monochrome images of the target skin.

[0113] At block 1405, the system is configured to measure absolute reflectance values ​​for each pixel of a plurality of monochrome images of target skin for a particular skin property value being sought.

[0114] At block 1407 , the system using the machine learning module is configured to map the absolute reflectance value for each pixel to a value for the same pixel represented in the LUT.

[0115] Figure 14B is a flow chart depicting a method for generating an RGB map of skin properties determined on a LUT.

[0116] At block 1421 , the skin diagnostic system is configured to receive the LUT entry whose value is closest in distance to the absolute value of the specific skin property sought for each pixel.

[0117] At block 1423 , the skin diagnostic system is configured to determine, for each pixel, a second LUT entry value representing one skin attribute to be displayed, and also to set all additional skin attributes listed in the LUT to a value closest to zero.

[0118] At block 1425 , the system is configured to generate a display of red, green, and blue wavelengths for each pixel based on the determined second LUT entry value.

[0119] Vascular lesion depth map

[0120] In some embodiments, a vascular lesion depth map is automatically determined and generated by a vascular depth module in one or more modules 107. In some embodiments, the deep learning model used for vascular depth determination is a U-Net deep learning semantic segmentation classifier model with four layers of depth (hereinafter referred to as the vascular depth model). In the disclosed text, vascular lesions are vascular structures visible to the human eye. In some embodiments, the vascular depth model is trained to detect four categories per pixel using all monochrome images of the image data. In some embodiments, the four categories are deep vascular lesions, medium vascular lesions, shallow vascular lesions, and background.

[0121] In some embodiments, the vascular model is trained using a labeled target skin image and each pixel in the target skin image is labeled with a classification (as a specific example, four classifications). In some embodiments, the target skin image is labeled for training using classifications by experienced medical personnel.

[0122] In some embodiments, the vascular depth model receives multiple monochrome images of target skin data. In some embodiments, the output of the vascular depth model is an array in which the image has scores for four classifications, with scores used to match each training class. In some embodiments, the vascular depth model further analyzes the four probability matrix (output for the four classifications) by processing the relevant three probability layers into a three probability matrix (i.e., three depths of vascular lesions). In some embodiments, the three probability matrix is ​​further analyzed by the vascular depth model, and the output is the model probability of the following three classes: shallow, medium (not shallow or deep), and deep vascular lesions. The classification with the maximum score can be selected as the predicted class for the pixel. In some embodiments, vascular structure lesion data can be collected in the memory of the skin diagnostic system for further development of the vascular depth model.

[0123] In some embodiments, a vascular lesion depth map is generated from the vascular depth model. The vascular lesion depth map includes a semi-transparent RGB or grayscale image overlaid with markers that segment vascular lesions into shallow, medium, and deep. In some embodiments, the vascular module determines which pixels are marked with any of three colors or markers. The vascular lesion depth map can use different colors or other markers to represent the depth of the vascular lesion, such as Figure 15 As seen in.

[0124] In some embodiments, the depth determination as one of shallow, medium, or deep is automatically determined by a mark lesion module among the one or more modules 107 for a single image of the vascular lesion map.

[0125] In some embodiments, a labeled vascular lesion model is trained using labeled target skin images that are labeled as images with classifications. In some embodiments, the target skin images are labeled with classifications by experienced medical personnel.

[0126] The per-pixel labels output by the vessel module are fed into a label vessel lesion module. A label vessel lesion module is a machine learning classifier model that outputs a single label for the image: light, medium, or dark.

[0127] Pigmented lesion depth map

[0128] In some embodiments, a pigment depth map is automatically determined and generated by a pigment depth module in one or more modules 107. Typically, a pigment lesion is an abnormal level of melanin based on a person's skin type. In some embodiments, the pigment depth module includes a machine learning 1D classifier model (hereinafter referred to as a pigment depth model). In some embodiments, the pigment depth model is trained using images labeled as "epidermal" (shallow) or "junctional" (dark) pigment lesions by trained medical personnel.

[0129] In some embodiments, the pigment depth model receives results from the vascular depth model and the hair masking model, thereby removing hair and vascular lesion information for each pixel from the pigment depth module analysis. Thus, removing hair and vascular lesions from images and data removes those pixels from any further analysis of the target skin by instructing other modules and / or models to ignore those pixels.

[0130] In some embodiments, the pigment depth model receives the measured brightness intensity of each pixel of the image at two wavelengths. Typically, a low wavelength value (such as 450 nm) captures lighter images of the target skin, and a high wavelength value (such as 850 nm) captures darker images of the target skin. Additionally, pigment / melanin typically absorbs light (reducing the amount of reflected light), resulting in darker image areas.

[0131] In some embodiments, the low wavelength value image is analyzed per pixel by the pigment depth model to determine pigmented lesions, and if a pigmented lesion is present in the pixel, it is labeled as a lightly pigmented lesion pixel. In some embodiments, the high wavelength value image is analyzed per pixel by the pigment depth model to determine pigmented lesions, and if a pigmented lesion is present in the pixel, it is labeled as a darkly pigmented lesion pixel.

[0132] In some embodiments, pigmented lesion pixels are identified by assigning a brightness value to each pixel at any wavelength, where a brightness value of 255 represents white and a value of zero represents dark. In some embodiments, a standard deviation calculation is used to identify dark pixel outliers. In some embodiments, the pigment depth model identifies outlier brightness intensity pixels by performing a statistical analysis of the distribution of intensity levels within the standard deviation. The pigment depth model can then identify a threshold to classify outliers as pigmented lesions present in the target skin. In some embodiments, pigmented lesions of more than two depths can be classified.

[0133] In some embodiments of the skin diagnostic system, a pigmented lesion depth map is generated using the outlier pixels in each image at the lowest and highest wavelengths. The pigmented lesion depth map may use different colors or other markers to indicate the depth of the pigmented lesions, such as Figure 16BOutlier pixels in the lowest wavelength image will be labeled as dark pigmented lesions, and outlier pixels identified in the highest wavelength image will be labeled as shallow depth pigmented lesions. In some embodiments, the pigmented lesion data is collected in memory for further development of the pigment depth model.

[0134] In some embodiments, the pigment depth model receives multiple monochrome images of target skin data and does not require input from the vascular depth model. In some embodiments, the output of the pigment depth model is an array in which the image has scores for four classifications, with scores used to match each training class. In some embodiments, the pigment depth model further analyzes the four probability matrices (output for the four classifications) by processing the relevant three probability layers into a three probability matrix and background (i.e., three depths of vascular lesions). The output is the model probability of the following three classes: epidermal (shallow), junctional (now medium), or dermal (deep) lesions. The classification with the maximum score can be selected as the predicted class for the pixel.

[0135] Depth maps of vascular and melanotic lesions

[0136] In some embodiments, the skin diagnosis system automatically combines the vascular depth map and the melanin depth map. The system combines the vascular lesion map generated by the vascular depth model and the pigment lesion map output by the pigment depth module for each pixel.

[0137] Figure 17A is used to generate Figure 17B Flowchart of the method for combined vascular lesion and skin lesion depth mapping as seen in FIG.

[0138] At block 1701 , a skin diagnosis system is configured to receive image skin data comprising a plurality of monochrome images of target skin.

[0139] At block 1703, the skin diagnosis system is configured to identify a label for each of the plurality of monochrome images and pixels using a vascular depth model, wherein the label is one of four categories: background, deep vascular lesion, medium vascular lesion, and shallow vascular lesion.

[0140] At block 1705 , the skin diagnostic system is configured to receive image skin data comprising two monochrome images of target skin via a pigment depth model.

[0141] At block 1707, the skin diagnostic system is configured to also receive the output of the vascular depth model for classification of vascular lesions regardless of depth via the pigment depth model. The pigment depth model does not analyze pixels that have been labeled as vascular lesions.

[0142] At block 1709 , the system is configured to determine outliers in the dark color at two wavelength values ​​via the pigment depth model.

[0143] At block 1711 , the system is configured to label low wavelength value outliers as per-pixel light pigmented lesions and high wavelength values ​​as per-pixel dark pigmented lesions via the pigment depth model.

[0144] At block 1713, the system is configured to generate a display using each pixel of an image that is labeled as one of the six determined categories: background, dark pigmented lesion, light pigmented lesion, deep vascular lesion, medium vascular lesion, and light vascular lesion.

[0145] In some embodiments, a vascular lesion value and a pigmented lesion value are calculated and displayed to medical personnel. For vascular, the vascular lesion area is relative to the total image pixels. For example, vascular value = vascular lesion area relative to the total image pixels. For example, vascular value = total pixels in the vascular lesion map / total pixels in the image display unit: image area % (0-100%). For pigment, the pigmented lesion area is relative to the total image pixels. For example, pigmented lesion value = total pixels in the pigmented lesion map / total pixels in the image display unit: image area % (0-100%).

[0146] In some embodiments, the skin diagnostic system calculates and displays a percentage of vascular lesions to pigmented lesions for medical professionals. This helps medical professionals determine which lesions to address first. For example, the vascular to pigmented lesion ratio = total pixels (or mm) in the vascular lesion image / total pixels (or mm) in the pigmented lesion image.

[0147] Pigment intensity

[0148] In some embodiments, the pigment intensity of the pigmented lesion is automatically determined by a pigment intensity module in one or more modules 107. Typically, pigment intensity is the contrast between the pigmented lesion of target skin tissue and the background skin. This contrast between the lesion and the surrounding target skin is typically determined by a medical professional and, therefore, by the human eye. Thus, the contrast is determined not only based on the empirical difference between the intensity of the pigmented lesion and the intensity of the surrounding target skin, but also based on the nonlinear human impression of the baseline (dark or light background) of the surrounding skin. The intensity of the pigmented lesion (the contrast between the pigmented lesion and the surrounding skin) can be used as a processing input for calculating the amount of energy required to treat the pigmented lesion.

[0149] In some embodiments, the pigment intensity module comprises a machine learning random forest classification model (hereinafter referred to as the pigment intensity model) with two outputs (light or dark lesions). Typically, the intensity or contrast of brightness is nonlinear and depends on the baseline intensity of the skin. In some embodiments, the pigment intensity model is trained using images of the target skin labeled with lesion intensity.

[0150] In some embodiments, the pigment intensity model receives data for three features from each of the multiple monochrome images. Feature 1 is a threshold representing the 99th percentile of the melanin concentration of the lesion. Feature 2 is the calculated median melanin level for the entire image, i.e., the output from the melanin density module using a lookup table (LUT). Finally, Feature 3 consists of subtracting Feature 1 from Feature 2. The output of the image intensity model is a light or dark lesion. In some embodiments, the pigment intensity data is stored in memory for further development of the pigment intensity model.

[0151] Hair properties

[0152] In some embodiments, the hair attributes in the target skin are automatically determined by a hair attribute module in one or more modules 107. In some embodiments, the hair attribute module receives the output of the hair masking model to identify hair in the target skin. In some embodiments, the hair attribute module includes a machine or deep learning classifier model trained to detect hair color and hair texture using labeled skin images of medical personnel (hereinafter referred to as a hair attribute model).

[0153] In some embodiments, a hair attribute model is trained using labeled target skin images to determine hair color by labeling hair pixels as colors. After subjective training using labeled skin images, the classifier generates a number of categories for hair color. In some embodiments, hair color is the following four categories: blonde / red, light brown, dark brown, and black.

[0154] In some embodiments, the hair attribute model is trained to determine hair texture. In some embodiments, the input data for determining hair texture is a single monochrome image from the target skin image. This single image can be at a wavelength between about 590 nm and about 720 nm. Each image is of known size, so counting the pixels per hair (specifically, pixels per hair width) can determine the hair diameter of each hair. Similarly, counting the pixels per hair compared to the total number of pixels can determine hair density. Information about hair density and hair diameter, along with subjective labeling training of a machine learning classifier, can generate a classification for hair texture. In an alternative approach, a threshold value for the diameter of each classification can be determined for classification. In some embodiments, hair texture is classified into the following three categories: fine, medium, and coarse. In some embodiments, the hair attribute model also determines hair thickness, hair melanin level, and hair count.

[0155] Model-based processing

[0156] In some embodiments, the skin diagnostic system generates the above-mentioned skin attributes and maps as input to the skin treatment modules of one or more processing modules 109 to generate parameters for treating the target skin. In some embodiments, the skin treatment modules include machine or deep learning models (hereinafter referred to as skin treatment models). Treatment parameters may include peak energy, energy fluence, pulse width, time distribution, site size, wavelength, pulse train, etc. In some embodiments, the skin diagnostic system skin attribute and map data can be collected and stored in memory for further development and training of diagnostic and skin treatment models.

[0157] Skin lesions or problems to be treated (hereinafter referred to as skin problem indications) include, but are not limited to, vascular lesions, pigmented lesions, melasma, telangiectasia, poikiloderma, age spots, facial acne, non-facial acne, and hair removal. Vascular lesions and pigmented lesions that can be treated may include, but are not limited to, port-wine stains, hemangiomas, leg veins, rosacea, rosacea erythema, pigmented spots, keratosis (keratin growth on the skin), café au lait spots, hemosiderin, Baker's nevus (non-cancerous, large, brown birthmark), nevus of Ota / Nevus of Ito (melanosis of the skin around the eyes), acne, melasma, and hyperpigmentation. Some skin conditions are a combination of pigmented and vascular lesions, such as, but not limited to, poikiloderma, age spots, and telangiectasia.

[0158] Figure 18 is a flow chart depicting a method for generating suggested processing parameters.

[0159] At box 1801, the skin treatment model of the skin diagnosis system is configured to receive predetermined target skin areas to be treated and skin problem indications to be treated from medical personnel and / or users of the system. In some embodiments, the skin treatment model also receives parameters for processing safety parameters and the capabilities of the energy treatment source. The user of the system can select multiple skin areas where the target skin is located, as well as multiple skin problem indications to be treated for each skin area. The user of the system can be instructed on a display to guide the user to where to aim the skin image handpiece 1300 to collect the image skin data required for the multiple skin attribute models.

[0160] At block 1803 , the skin treatment model of the skin diagnosis system is configured to receive outputs of a plurality of skin property models of the target skin that are associated with predetermined skin problem indications to be treated.

[0161] In some embodiments, when the treatment is for vascular lesions, the inputs to the skin treatment model are skin type and vascular lesion depth. In some embodiments, when the treatment is for pigmented lesions, the inputs to the skin treatment model are skin type, pigmented lesion depth, and pigment intensity, which uses melanin density to determine pigment intensity. In some embodiments, when the treatment is for combined pigmented and vascular lesions, the inputs to the skin treatment model are skin type, vascular lesion depth, pigmented lesion depth, and pigment intensity, which uses melanin density to determine pigment. In some embodiments, when the treatment is for hair removal, the inputs to the skin treatment model are skin type, hair color, and hair texture.

[0162] At block 1805, the skin treatment model of the skin diagnostic system is configured to analyze skin attributes for a predetermined skin treatment. In some embodiments, the skin treatment model utilizes multiple skin treatment lookup tables (one for each of the skin problem indications to be treated) to match skin attributes with appropriate skin treatment parameters. In some embodiments, the treatment lookup tables are specifically developed for IPL energy-based treatments. The multiple skin treatment lookup tables can be generated from input from medical personnel and from the vast amount of data collected through clinical trials.

[0163] At box 1807, the skin diagnostic system is configured to determine and generate a display of the treatment parameters of the suggestion. In some skin diagnostic systems, the system is configured to display an RGB image of the target skin with the treatment parameters of the suggestion. In some embodiments, multiple maps related to the treatment of the target skin are displayed, such as but not limited to a melanin density map, a vascular density map, a pigmented lesion depth map, a vascular lesion depth map, pigment intensity or any combination thereof. These maps can help medical personnel and / or users change the treatment parameters to be used. In some embodiments, a report of the recommended treatment, the completed treatment, and multiple maps of the target skin are all saved in a database for future training of the machine learning model, for future display to the user, and for future generation of reports for each patient.

[0164] In some embodiments, the skin diagnostic system 100 or the combined system 100A can include a diagnostic module having a deep learning or machine learning model to diagnose a skin problem indication to be treated using the image skin data without requiring input from a medical professional or user. In some embodiments, the combined system 100A can also include a treatment determination module having a deep learning or machine learning model to analyze and determine a treatment for the target skin based on the image skin data.

[0165] In some embodiments, the diagnostic module and / or treatment determination module are trained using images that may utilize additional skin attribute data not previously considered for treatment determination. In some embodiments, the system may capture an image of a target skin area and determine a treatment based on the image and depth and / or machine learning and output a simulated image of the target skin area after treatment.

[0166] In some embodiments, the treatment source is an intense pulsed light (IPL) treatment source. In some embodiments, the IPL treatment source uses different filters for treatment, and as a specific example, uses special filters to treat acne.

[0167] In some embodiments, the treatment source has an image capture device and treatment source in the same handpiece. In these cases, the handpiece can operate in two modes: a treatment mode for delivering energy-based treatment from, for example, an intense pulsed light (IPL) source to a patient's skin area; and a diagnostic mode for acquiring images of the skin area. In some embodiments, the device is a handpiece of a therapeutic IPL source system and is coupled to the system via a tethered connection.

[0168] The system can switch between these two modes in a relatively short time (in some embodiments, at most a few seconds), enabling in-process monitoring. In addition, in some embodiments, the device sends image data to a skin diagnostic system, which analyzes the image and calculates optimal treatment parameters (at least the optimal parameters for the next delivery) and sends the optimal treatment process to the controller or display of the device in real time. The device implements an iteration of imaging the skin after delivery of, for example, energy-based treatment and determining the parameters for the next delivery without excessive delay. "In-process" monitoring does not mean that monitoring must be performed simultaneously with treatment. The system can switch between treatment mode and diagnostic mode in a time period that is short enough for the user (i.e., a few seconds). During treatment, the system can switch between treatment mode and diagnostic mode multiple times. Details about skin treatment and real-time monitoring and the combined treatment and image capture handpiece are further described in PCT serial number PCT / IL2023 / 050785, filed on July 30, 2023, which is hereby incorporated by reference in its entirety.

[0169] Throughout this specification, the following terms take the meanings explicitly associated with this document unless the context clearly dictates otherwise. As used herein, the phrases "in one embodiment" and "in some embodiments" do not necessarily refer to the same embodiment, although they may refer to the same embodiment. Furthermore, as used herein, the phrases "in another embodiment" and "in some other embodiments" do not necessarily refer to different embodiments, although they may refer to different embodiments. Thus, as described below, various embodiments may be readily combined without departing from the scope or spirit of the disclosure.

[0170] The terms "a," "an," and "the" mean "one or more" unless expressly specified otherwise.

[0171] Throughout this specification, the word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or implementation of the present subject matter described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0172] The terms "comprises," "comprising," or any other variations thereof are intended to cover a non-exclusive inclusion, such that an arrangement, apparatus, or method that comprises a list of components or steps includes not only those components or steps but may also include other components or steps not expressly listed or inherent to such arrangement, apparatus, or method. In other words, the phrase "comprising..." preceding one or more elements of a system or device does not, without further constraints, preclude the presence of other or additional elements in the system or method.

[0173] The terms "comprising," "having," or any other variations thereof are intended to cover a non-exclusive inclusion, such that an arrangement, apparatus, or method having a list of components or steps includes not only those components or steps but may also include other components or steps not expressly listed or inherent to such arrangement, apparatus, or method. In other words, without further constraints, the phrase "having" preceding one or more elements of a system or device does not preclude the presence of other or additional elements in the system or method.

[0174] In addition, the term "based on" is not exclusive and allows for being based on additional factors that are not described unless the context clearly dictates otherwise. In addition, throughout the specification, the meanings of "a," "an," and "the" include plural references. The meaning of "in" includes "in" and "on."

[0175] Should be understood that at least one aspect / functionality of various embodiments described herein can be performed in real time and / or dynamically.As used herein, term " real time " or " near real time " relate to the event / action that can occur instantly or almost instantly when another event / action has occurred.For example, " real time processing ", " real time calculation " and " real time execution " all relate to and perform calculation during the actual time that relevant physical process (for example, user interacts with the application on mobile device) occurs, so that the result of calculation can be used to guide physical process.In some embodiments, can occur in real time, near real time and / or based on at least one predetermined periodicity of following according to the event and / or action of open text: nanosecond, several nanoseconds, millisecond, several milliseconds, second, several seconds, minute, several minutes, every hour, several hours, every day, several days, every week, every month etc.As used herein, term " dynamically " and term " automatically " and their logic and / or language related words and / or derivatives mean that particular event and / or action can trigger and / or occur when there is no human intervention.

[0176] Computer systems and systems, as used herein, may include any combination of hardware and software. Examples of software may include software components, programs, applications, operating system software, firmware, software modules, routines, subroutines, functions, methods, processes, software interfaces, application programming interfaces (APIs), computer code, data variables, or any combination thereof that can be processed by a computing device as computer-executable instructions.

[0177] In some embodiments, one or more computer-based systems of the disclosed text may include or be partially or fully incorporated into at least one personal computer (PC), laptop computer, tablet computer, portable computer, smart device (e.g., smart phone, smart tablet or smart TV), mobile Internet device (MID), instant messaging device, data communication device, server computer, etc.

[0178] Figure 7 The illustrated method illustrates specific events that occur in a specific order. In alternative embodiments, specific operations can be performed, modified, or removed in different orders. In addition, steps can be added to the above logic and still comply with the described embodiment. In addition, the operations described herein can occur sequentially or can process specific operations in parallel. In addition, operations can be performed by a single processing unit or by distributed processing units.

[0179] Finally, the language used in the specification is primarily selected for readability and instructional purposes and may not be selected to limit or delimit the subject matter of the present invention. Therefore, the scope of the present invention is not intended to be limited by this detailed description, but rather by any claims issued based on the application hereof. Therefore, the disclosure of the embodiments of the present invention is intended to illustrate and not limit the scope of the invention, which is set forth in the appended claims.

Claims

1. A system for determining skin properties and treatment parameters of a target skin for an aesthetic skin diagnosis and treatment unit, comprising: monitor; at least one lighting source; Image capture device; a source for providing energy-based processing; processor; a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions that, when executed, cause the processor to: activating the at least one illumination light source to illuminate at a plurality of monochromatic wavelengths; obtaining images at the plurality of monochromatic wavelengths from the image capture device; receiving target skin data including data for each pixel of the obtained image; analyzing the target skin data using a plurality of trained skin property models; determining at least one skin attribute classification of the target skin using the trained skin attribute model; analyzing the at least one classification of the skin attribute of the target skin using a trained skin treatment model; identifying treatment parameters of the energy-based treatment source for the determined at least one skin attribute classification using the trained skin treatment model; as well as The processing parameters identified as processing the skin attribute are displayed. 2 . The system of claim 1 , wherein the system generates and displays a list of properties of the target skin based on the analysis of the trained skin property model.

3. The system of claim 1, wherein the energy-based treatment source is activated to treat the target skin using the determined treatment parameters.

4. The system of claim 1 , wherein the plurality of trained skin property models are trained by: (i) providing a plurality of labeled images of at least one skin attribute stored in a database to the skin attribute model, and (ii) configuring the skin attribute model to classify the plurality of labeled images into at least one skin attribute.

5. The system of claim 1, wherein the plurality of different wavelengths comprises 450 nm, 490 nm, 570 nm, 590 nm, 660 nm, 770 nm, and 850 nm.

6. The system of claim 1, wherein the processor is further configured to register and align the images of the plurality of monochromatic wavelengths after obtaining the images.

7. The system of claim 1, wherein the processor is further configured to generate and display a map of the target skin using any combination of the plurality of monochromatic wavelengths.

8. The system of claim 1, wherein the processor is further configured to generate and display a map of the target skin from the wavelengths representing red, green, and blue.

9. The system of claim 1 , wherein one of the skin attributes is hair on the target skin, and a hair masking model is one of the plurality of skin attribute models, and the processor is further configured to: receiving the target skin data of one of the plurality of monochromatic wavelengths; and The hair masking model is used to determine one of two classifications of hair or background for each pixel of the one monochromatic wavelength image.

10. The system of claim 9, wherein the processor is further configured to: The additional skin property model is instructed to remove hair pixels marked as hair by the hair masking model from further analysis of the target skin.

11. The system of claim 1, wherein one of the skin attributes is skin type and a skin type model is one of the plurality of skin attribute models.

12. The system of claim 11, wherein the processor is further configured to: receiving skin type data comprising an average calibrated reflectance value for total pixels of each monochrome image; and Six categories of skin type are determined using the skin type model.

13. The system of claim 1, wherein the skin attribute is at least one of: Melanin density, vascular density, and scattering.

14. The system of claim 13, wherein the processor is further configured to: receiving skin type data comprising a plurality of absolute reflectance values ​​for each pixel representing the plurality of wavelengths; analyzing the plurality of absolute values ​​for each pixel by comparison with a lookup table (LUT) value using at least one of a melanin model or a vascular model, wherein the LUT comprises skin model values ​​representing the lighting effects of a known physical model on human skin and physical measurements representing the concentration of the skin property in the target skin; and For each pixel, for at least one of melanin density or vascular density, a LUT entry is identified whose value is closest in distance to the plurality of measured absolute values ​​for each pixel, wherein the distance can be an approximation of a specific distance.

15. The system of claim 1, wherein one of the skin attributes is vascular lesion depth, and a vascular depth model is one of the plurality of skin attribute models.

16. The system of claim 15, wherein the processor is further configured to: receiving the target skin data of the plurality of monochromatic wavelengths; and Determining one of four classifications of deep vessel, medium vessel, superficial vessel or background for each pixel using the vascular lesion model; and A map with labels is generated and displayed to illustrate the classification of vascular lesion depth.

17. The system of claim 1, wherein one of the skin attributes is pigmented lesion depth and a pigment depth model is one of the plurality of skin attribute models.

18. The system of claim 17, wherein the processor is further configured to: receiving the target skin data for two monochromatic wavelengths of the plurality of monochromatic wavelengths, wherein one monochromatic wavelength represents a lowest wavelength value of the system and the second monochromatic wavelength represents a highest wavelength value of the system; receiving classified pixels of vascular depth from the vascular depth model; analyzing the pixels not classified by the vessel depth model for outliers in the dark color for each of the two monochromatic wavelengths; Determining a classification for each pixel analyzed using the pigmented lesion model, wherein the outlier value of the lowest wavelength value is a lightly pigmented lesion and the outlier value of the highest wavelength value is a darkly pigmented lesion; as well as A map with labels is generated and displayed to illustrate the classification of pigmented lesion depth.

19. The system of claim 1, wherein one of the skin attributes is pigmented lesion intensity and a pigment intensity model is one of the plurality of skin attribute models.

20. The system of claim 19, wherein the processor is further configured to: receiving the target skin data of three features from each of the plurality of monochromatic images, wherein the features are a threshold representing a 99th percentile of melanin concentration of the lesion, a median melanin level for the entire image calculated from a melanin density model, and subtracting the 99th percentile from the calculated median melanin level; and The intensity of the pigmented lesion is determined to be a light lesion or a dark lesion based on the feature.

21. The system of claim 14, wherein the processor is further configured to: receiving at least one of the melanin density value from the melanin model or the vascular density value from the vascular model in the LUT; Calculating a new value of the melanin density value or the vascular density value based on setting other skin attributes on the LUT to be closest to zero; and A map of the melanin density or the vascular density is generated using the calculated new value wavelength.

22. The system of claim 1, wherein the processor having the trained skin treatment model is further configured to: Receive the following information: Processing security parameters, Energy handling source capability parameters, at least one skin area to be treated from a user, at least one skin problem indication from the user to be treated based on the skin area to be treated, outputs of the plurality of skin attribute models associated with the at least one skin problem indication; determining target skin treatment parameters for the energy-based treatment based on the received information; as well as The target skin treatment parameters of the energy-based treatment are displayed.

23. The system of claim 22, wherein said determining of said skin treatment parameters is accomplished using a treatment lookup table, and wherein said processor is further configured to: determining one of a plurality of skin treatment lookup tables, each of the skin treatment lookup tables being based on a specific skin problem indication; matching the outputs of the plurality of skin property models to processing parameters of the determined skin treatment lookup table; as well as The matched skin treatment parameters of the energy-based treatment are displayed.

24. The system of claim 22, wherein the processor having the trained skin treatment model is further configured to: generating and displaying a red, green, and blue (RGB) image of the target skin; generating at least one of the plurality of graphs and saving it to memory, displaying the generated at least one graph, wherein the at least one of the plurality of graphs comprises: Melanin density map, Vascular density map, Pigmented lesion depth map, Vascular lesion depth map, Pigment intensity, or any combination thereof.

25. The system of claim 22, wherein the at least one skin problem indicator is at least one of: Pigmented lesions, Vascular lesions, Combined pigmented and vascular lesions, Hair removal, or any combination thereof.

26. A method for determining skin properties and treatment parameters of a target skin, comprising: providing a display, at least one illumination source, an image capture device, a source for providing energy-based processing, a memory, and a processor; activating, by the processor, the at least one illumination light source to illuminate at a plurality of monochromatic wavelengths; obtaining, by the processor, images at the plurality of monochromatic wavelengths from the image capture device; receiving, by the processor, target skin data comprising data for each pixel of the obtained image; analyzing, by the processor, the target skin data using a plurality of trained skin property models; determining, by the processor, at least one skin attribute classification of the target skin using the trained skin attribute model; analyzing, by the processor, the at least one classification of the skin attribute of the target skin using a trained skin treatment model; identifying, by the processor, treatment parameters of the energy-based treatment source for the determined at least one skin attribute classification using the trained skin treatment model; as well as The processing parameters identified as processing the skin attribute are displayed by the processor.

27. The method of claim 26, wherein the skin attribute is at least one of: melanin density, vascular density, and scattering, and wherein the method further comprises: receiving, by the processor, skin type data comprising a plurality of absolute reflectance values ​​for each pixel representing the plurality of wavelengths; analyzing, by the processor, the plurality of absolute values ​​for each pixel by comparison with a lookup table (LUT) value using at least one of a melanin model or a vascular model, wherein the LUT comprises values ​​of a skin model, the values ​​representing the lighting effects of a known physical model on human skin and physical measurements representing the concentration of the skin property in the target skin; as well as The processor identifies, for each pixel, a LUT entry having a value closest in distance to the plurality of measured absolute values ​​for at least one of melanin density or vascular density, wherein the distance can be an approximation of a specific distance.

28. The method according to claim 27, wherein the method further comprises: receiving, by the processor, at least one of the melanin density value from the melanin model or the vascular density value from the vascular model in the LUT; calculating, by the processor, a new value for the melanin density value or the vascular density value based on setting other skin attributes on the LUT to be closest to zero; and A map of the melanin density or the vascular density is generated by the processor using the calculated new value wavelength.

29. The system of claim 26, wherein the method further comprises: The following information is received by the processor having the trained skin treatment model: Processing security parameters, Energy handling source capability parameters, at least one skin area to be treated from a user, at least one skin problem indication from the user to be treated based on the skin area to be treated, outputs of the plurality of skin attribute models associated with the at least one skin problem indication; determining, by the processor having the trained skin treatment model, target skin treatment parameters for the energy-based treatment based on the received information; as well as The target skin treatment parameters for the energy-based treatment are displayed by the processor having the trained skin treatment model.

30. The system of claim 29, wherein said determining of said skin treatment parameters is accomplished using a treatment lookup table, and said method further comprising: determining, by the processor having the trained skin treatment model, one of a plurality of skin treatment lookup tables, wherein each of the skin treatment lookup tables is based on a specific skin problem indication; matching, by the processor having the trained skin treatment model, the outputs of the plurality of skin attribute models with treatment parameters of the determined skin treatment lookup table; as well as The matched skin treatment parameters of the energy-based treatment are displayed by the processor.

Images