Method for providing information about demodex mites and device for providing information about demodex mites using same

Abstract

The present invention comprises a method for providing information about demodex mites and a device based on the method, the method comprising the steps of: extracting a first feature of the demodex mites by inputting a face image of an individual to a trained first prediction model; generating a saliency map from the first feature; acquiring regions of interest in the face as one or more patches; extracting a second feature of the demodex mites by inputting the one or more patches to a second prediction model; and predicting a final density probability of the demodex mites for the individual by inputting the first feature and the second feature to a third prediction model trained to predict the density probability of the demodex mites for the individual.

Description

Method for providing information on demodex mites and a device for providing information on demodex mites using the same

[0001] The present invention relates to a method for providing information on demodex mites and a device using the same.

[0002] Demodex species, or Demodex folliculorum, are microorganisms found primarily in human skin, particularly in the sebaceous glands. Their subspecies include Demodex folliculorum and Demodex brevis. Demodex folliculorum is found in hair follicles of all ages, except newborns, and is most commonly found in adults over 30.

[0003] Skin lesions caused by Demodex mites mainly appear as inflammatory lesions with papules, pustules, and follicular keratosis on the face, and Demodex species are known to act as a major aggravating factor in rosacea, an inflammatory skin disease.

[0004] However, the pathogenesis of Demodex mites is caused by changes in their numbers rather than their presence itself. More specifically, Demodex mites are found in normal adult skin when they grow to a size of 1 cm. 2 If more than 10 are found per day, the host's immune response may be altered by the Demodex mites' larvae or metabolites, causing proliferation and hyperkeratinization of the sebaceous gland infundibulum structure, which may lead to hair follicle occlusion.

[0005] Rosacea, one of the skin diseases caused by these Demodex mites, is a common skin disease that occurs in about 10% of the population. It is a chronic disease characterized by the appearance of vasodilation, papules, and pustules, mainly in the central areas of the face such as the nose and cheeks.

[0006] According to a big data analysis conducted by the Korean Dermatological Association in 2017, the number of patients with rosacea has been increasing by approximately 20% annually since the 2000s, and Demodex folliculitis is attracting attention as the main cause of rosacea.

[0007] More specifically, dysregulation of skin immune responses by Demodex has been reported in patients with rosacea, and previous studies have shown that TLR2-mediated inflammatory cytokines (IL-1, IL-6, IL-8, and LL-37) increase with increasing Demodex population, and expression of TRPV receptors, which are associated with the development of neurovascular symptoms of rosacea, also increased.

[0008] Furthermore, Demodex overgrowth has been observed in other conditions that present with facial erythema, including inflammatory folliculitis and seborrheic dermatitis, in addition to rosacea. Demodex overgrowth is more common in patients with both rosacea and contact dermatitis and has been linked to specific allergens (nickel metal).

[0009] As mentioned above, since Demodex mites are the cause of a wide range of skin diseases, identification of Demodex mites can be important in differential diagnosis and treatment selection for patients complaining of erythema. Therefore, there is a need to develop a method that can accurately diagnose and identify Demodex mites.

[0010] The background technology of the invention has been prepared to facilitate a better understanding of the present invention. It should not be construed as an admission that the matters described in the background technology of the invention constitute prior art.

[0011] Currently, methods for measuring Demodex mites include conventional punch biopsy, standardized skin surface biopsy (SSSB), and extrusion. However, all of these are invasive biopsy methods. These biopsy-based analysis methods require extensive training and, because they are dependent on the examiner's skill, may not produce consistent results across examiners or measurement periods.

[0012] That is, these conventional methods of measuring and evaluating folliculitis are based on the subjective intuition or experience of the examiner, so they may not be objective or consistent, and may increase the burden of invasive biopsies on patients.

[0013] Accordingly, the method of measuring demodex mites is not directly used in actual clinical settings due to difficulties in accessibility to the test itself, and measurement of demodex mites is performed only in a very small number of large hospitals.

[0014] Meanwhile, AI-based information systems, such as U-net, are being widely applied in skin diagnosis. However, for skin diagnosis related to Demodex, the number of medical institutions capable of diagnosing and measuring Demodex is limited, limiting the application of AI technology in this area. Furthermore, conventional skin diagnosis methods suffer from limitations such as poor reproducibility depending on the type of facial skin image.

[0015] Accordingly, the inventors of the present invention have observed that when measuring (confirming) and evaluating Demodex mites, more consistent results can be achieved by segmenting specific areas within the face and performing unit predictions. Specifically, the inventors of the present invention have recognized that when areas within the face are segmented into patterns and unit predictions are performed, artificial intelligence can learn more effectively to predict various skin lesions.

[0016] Furthermore, the inventors of the present invention discovered that a specific area within the face, i.e., an area of ​​interest where follicle mites are predicted to be distributed, can be segmented more accurately by artificial intelligence.

[0017] More specifically, the inventors of the present invention have recognized that when the lesion area is segmented into the entire area (global) and a specific subarea (local) and learned and predicted by an algorithm (model), the density of the folliculitis mite of an individual (patient) can be predicted with high accuracy.

[0018] Furthermore, the inventors found that when using weakly supervised learning, a region of interest (ROI) patch for segmenting a specific sub-region can be automatically extracted by the model.

[0019] Finally, the inventors of the present invention developed a density prediction system for Demodex mites with higher accuracy by including an ensemble model that combines a model (algorithm) that takes a facial image as input, a model (algorithm) that automatically searches for images that highlight only the main areas of interest within the facial image and extracts them as patches, and then takes these as input again, and a model (algorithm) that takes clinical data as input.

[0020] Furthermore, the inventors of the present invention have found that when the main region of interest within a facial image is not selected based on anatomical structure, but is automatically selected and presented by artificial intelligence, the density and distribution of the follicle mites can be predicted with higher accuracy.

[0021] Accordingly, the problem to be solved by the present invention is to provide a method for providing information on demodex mites and a device based thereon, which is configured to first predict the density probability of demodex mites using each prediction model that inputs a facial image, a patch, and clinical information, and then provide a final density probability with high accuracy using a final prediction model that is relearned by combining data derived from each prediction model.

[0022] The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0023] In order to solve the above-described problem, a method for providing information on Demodex mite according to an embodiment of the present invention is provided. The method for providing information comprises the steps of: receiving a face image of an individual; inputting the face image into a first prediction model trained to extract a first feature for Demodex mite by inputting the face image as an input, and extracting a first feature for Demodex mite; generating a saliency map from the first feature; acquiring regions of interest within the face as one or more patches from the saliency map; The method includes: extracting a second feature for a Demodex mite by inputting one or more patches into a second prediction model trained to extract a second feature for a Demodex mite by inputting one or more patches; and predicting a final density probability for a Demodex mite for an object by inputting the first feature and the second feature into a third prediction model trained to predict a density probability for a Demodex mite for an object by inputting the first feature and the second feature.

[0024] According to a feature of the present invention, after the receiving step, a step of de-identifying a facial landmark from the received facial image may be further included.

[0025] According to another feature of the present invention, the de-identifying step may include extracting coordinates for an anatomical structure within the face from the received face image, and de-identifying a landmark within the face based on the extracted coordinates.

[0026] According to another feature of the present invention, after the step of extracting the first feature, the method may further include a step of inputting the first feature into a first prediction model that is further trained to predict the density probability of the demodex mite for the object by inputting the first feature into the first prediction model.

[0027] According to another feature of the present invention, the first density probability may be a global prediction for the face image.

[0028] According to another feature of the present invention, after the step of extracting the second feature, the method may further include a step of predicting the second density probability of the demodex mite for the object by inputting the second feature into a second prediction model that is further trained to predict the density probability of the demodex mite for the object by inputting the second feature.

[0029] According to another feature of the present invention, the second density probability may be a local prediction for the patch image.

[0030] According to another feature of the present invention, the receiving step may further include a step of receiving clinical information of the subject, and after the receiving step, a step of inputting the clinical information into a fourth prediction model trained to extract features of the demodex mite by inputting the clinical information may further include a step of extracting a third feature for the demodex mite.

[0031] According to another feature of the present invention, the third prediction model is further trained to predict the density probability of demodex mites for an object by inputting the first feature, the second feature, and the third feature, and the step of predicting the final density probability may include the step of predicting the final density probability of demodex mites for an object by inputting the first feature, the second feature, and the third feature into the third prediction model.

[0032] According to another feature of the present invention, the step of outputting characteristic importance based on clinical information may be further included.

[0033] According to another feature of the present invention, the first feature of the step of predicting the final density probability may be, but is not limited to, a pooled vector value.

[0034] According to another feature of the present invention, the second feature in the step of predicting the final density probability may be, but is not limited to, an attention-weighted representation Z vector value to which an attention score is applied.

[0035] According to another feature of the present invention, the step of extracting the second feature may further include the step of applying an attention score to the extracted second feature.

[0036] According to another feature of the present invention, the step of applying an attention score may include the steps of inputting one or more patches as a batch to one or more FC layers; outputting an attention score based on a logit output from the FC layer; and multiplying the output attention score by a second feature.

[0037] According to another feature of the present invention, the first prediction model may be a model trained to predict the probabilities for class 0 and class 1 by inputting a facial image, but is not limited thereto.

[0038] According to another feature of the present invention, the second prediction model may be, but is not limited to, a model trained to predict probabilities for class 0 and class 1 using patches as input.

[0039] According to another feature of the present invention, the third prediction model may be a model trained to predict the probability for class 0 and class 1 using the first feature and the second feature as inputs, but is not limited thereto.

[0040] According to another feature of the present invention, classification 0 is 1 cm 2 The number of hair follicle mites per swarm is 0 or more and less than 5, and category 1 is 1 cm 2 The number of hair follicle mites per colony may be 5 or more, but is not limited thereto.

[0041] In order to solve other problems as described above, a device for providing information on hair follicle mites according to one embodiment of the present invention is provided. A device for providing information on Demodex of the present invention comprises a communication unit configured to receive a face image of an individual, and a processor connected to communicate with the communication unit, wherein the processor is configured to input a face image to a first prediction model trained to extract a first feature for Demodex by using the face image as an input, extract a first feature for Demodex, generate a saliency map from the first feature, acquire regions of interest in the face from the saliency map as one or more patches, input one or more patches to a second prediction model trained to extract a second feature for Demodex, extract a second feature for Demodex, and input the first feature and the second feature to a third prediction model trained to predict a density probability of Demodex for the individual by using the first feature and the second feature as inputs, and predict a final density probability of Demodex for the individual.

[0042] According to a feature of the present invention, the processor may be further configured to de-identify facial landmarks from the received facial image.

[0043] According to another feature of the present invention, the processor may be configured to extract coordinates for anatomical structures within the face from the received face image, and to de-identify landmarks within the face based on the extracted coordinates.

[0044] According to another feature of the present invention, the processor may be further configured to, after extracting the first feature, input the first feature into a first prediction model that is further trained to predict the density probability of the folliculitis mite for the object by using the first feature as an input, and to predict the first density probability of the folliculitis mite for the object.

[0045] According to another feature of the present invention, the first density probability may be a global prediction for the face image.

[0046] According to another feature of the present invention, the processor may be further configured to, after the step of extracting the second feature, input the second feature into a second prediction model that is further trained to predict the second density probability of the demodex mite for the object by using the second feature as an input.

[0047] According to another feature of the present invention, the second density probability may be a global prediction for the patch image.

[0048] According to another feature of the present invention, the communication unit may be further configured to receive clinical information of the subject, and the processor may be further configured to input the clinical information into a fourth prediction model trained to extract features of the demodex mite by inputting the clinical information, and to extract third features for the demodex mite.

[0049] According to another feature of the present invention, the third prediction model is further trained to predict the probability of the density of the demodex mite for the object by using the first feature, the second feature, and the third feature as inputs, and the processor can be configured to predict the final probability of the density of the demodex mite for the object by using the first feature, the second feature, and the third feature as inputs to the third prediction model.

[0050] According to another feature of the present invention, the processor may be further configured to output a feature importance based on clinical information.

[0051] According to another feature of the present invention, the first feature input to the third prediction model may be a pooled vector value, but is not limited thereto.

[0052] According to another feature of the present invention, the second feature input to the third prediction model may be, but is not limited to, an attention-weighted representation Z vector with an attention score applied.

[0053] According to another feature of the present invention, the processor may be further configured to, but is not limited to, apply an attention score to the extracted second feature.

[0054] According to another feature of the present invention, the processor may be further configured to input one or more patches as a batch to one or more FC layers, output an attention score based on a logit output from the FC layer, and multiply the output attention score by a second feature, but is not limited thereto.

[0055] According to another feature of the present invention, the first prediction model may be a model trained to predict the probabilities for class 0 and class 1 by inputting a facial image, but is not limited thereto.

[0056] According to another feature of the present invention, the second prediction model may be, but is not limited to, a model trained to predict probabilities for class 0 and class 1 using patches as input.

[0057] According to another feature of the present invention, the third prediction model may be a model trained to predict the probability for class 0 and class 1 using the first feature and the second feature as inputs, but is not limited thereto.

[0058] According to another feature of the present invention, classification 0 is 1 cm 2The number of hair follicle mites per swarm is 0 or more and less than 5, and category 1 is 1 cm 2 The number of hair follicle mites per colony may be 5 or more, but is not limited thereto.

[0059] Specific details of the embodiments are included in the detailed description and drawings.

[0060] The present invention can objectively and quickly diagnose and predict Demodex mites without any intrusion, thereby reducing the psychological and financial burden on patients associated with biopsies and improving the workflow of medical staff in actual clinical practice. Furthermore, medical staff can quickly select an appropriate treatment method based on the results, contributing to a positive prognosis for skin diseases caused by Demodex mites, including the injection of the present invention.

[0061] Furthermore, since the present invention includes a de-identification method, personal and personal information about an individual can be safely protected.

[0062] Moreover, the present invention provides anonymized facial images and patch images, thereby minimizing interference from areas other than the area of ​​interest, and enabling more accurate prediction of the density and distribution of hair follicle mites within the area of ​​interest.

[0063] Furthermore, the present invention can predict the density probability of Demodex mites for the entire face and each facial region, thereby making it possible to predict the density probability of Demodex mites in more detail, as well as confirm and predict the distribution of Demodex mites within the face.

[0064] Furthermore, the present invention can prevent data bias by performing cross-validation so that all data can be used for learning and evaluation, and can achieve high accuracy even with a small amount of data, thereby allowing prediction of the density probability of hair follicle mites with high reliability.

[0065] Furthermore, since the present invention includes a saliency map for a facial image, it is possible to more easily visually confirm the contribution of each region within the face to the density probability of the follicle mites, i.e., the distribution of lesions.

[0066] Furthermore, by including a step of outputting the feature importance based on clinical information, the contribution of the clinical information of an individual to the probability of the density of Demodex mites can be more easily and visually confirmed.

[0067] The effects according to the present invention are not limited to those exemplified above, and more diverse effects are included in this specification.

[0068] FIG. 1a illustrates an example of a system for providing information on demodex mites based on a device for providing information on demodex mites according to one embodiment of the present invention.

[0069] FIG. 1b is a block diagram showing the configuration of a device for providing information on hair follicle mites according to one embodiment of the present invention.

[0070] FIG. 1c is a block diagram of a configuration of a medical device according to one embodiment of the present invention.

[0071] FIG. 1d is a schematic diagram of a user device according to one embodiment of the present invention.

[0072] Figure 2 is a flowchart of a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0073] Figure 3a is a schematic diagram of a de-identification process according to a method for providing information on a hair follicle mite according to one embodiment of the present invention.

[0074] Figure 3b is a schematic diagram of coordinates in a de-identification process according to a method for providing information on a hair follicle mite according to one embodiment of the present invention.

[0075] FIG. 4 is an exemplary diagram of a learning and inference process for a first prediction model in a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0076] FIG. 5 is an exemplary diagram of a patch acquisition process in a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0077] FIG. 6a is an exemplary diagram of a learning and inference process for a second prediction model in a method for providing information on a hair follicle mite according to one embodiment of the present invention.

[0078] FIG. 6b is a block diagram showing the configuration of an attention score module in a method for providing information on a hair follicle mite according to one embodiment of the present invention.

[0079] Figure 7 is an exemplary diagram of the entire learning and inference process of a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0080] FIG. 8 is an example of a SHAP result in a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0081] FIG. 9 is an exemplary diagram of the final density probability of a demodex mite for a method for providing information on a demodex mite according to one embodiment of the present invention.

[0082] Figure 10 shows various image results derived from a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0083] Figures 11a and 11b are evaluation results for various prediction models used in a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0084] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0085] In this document, the expressions "has," "may have," "includes," or "may include" indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), but do not exclude the presence of additional features.

[0086] In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A or / and B" can include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" can all refer to cases where (1) at least one A is included, (2) at least one B is included, or (3) at least one A and at least one B are included.

[0087] The terms "first," "second," "first," or "second," as used herein, may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, without limiting the components. For example, a first user device and a second user device may represent different user devices, regardless of order or importance. For example, without departing from the scope of the rights set forth in this document, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0088] When it is said that a component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the component is directly coupled to the other component, or can be connected via another component (e.g., a third component). Conversely, when it is said that a component (e.g., a first component) is "directly coupled to" or "directly connected to" another component (e.g., a second component), it should be understood that no other component (e.g., a third component) exists between the first component and the other component.

[0089] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in hardware. Instead, in some contexts, a "device configured to" can mean that the device, together with other devices or components, is "capable of." For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0090] The terms used in this document are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this document. Terms defined in general dictionaries among the terms used in this document may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this document. In some cases, even if a term is defined in this document, it cannot be interpreted to exclude the embodiments of this document.

[0091] The individual features of the various embodiments of the present invention can be partially or wholly combined or combined with each other, and as can be fully understood by those skilled in the art, various technical connections and operations are possible, and each embodiment can be implemented independently of each other or can be implemented together in a related relationship.

[0092] For clarity in the interpretation of this specification, the terms used in this specification are defined below.

[0093] As used herein, the term "subject" may refer to any subject for which a diagnosis of Demodex mites is sought. For example, the subject may be a subject suspected of having a skin disease caused by Demodex mites. In this case, the subject disclosed herein may be any mammal other than humans, but is not limited thereto.

[0094] As used herein, the term "facial image" may refer to a facial image acquired from an imaging diagnostic device. In this case, the facial image may be a medical image including the facial region of an object.

[0095] Meanwhile, the facial image may be a video composed of multiple cuts. In this case, the face, i.e., the facial area of ​​the subject, the distribution of the folliculitis mites, and the density of the folliculitis mites can be determined and predicted for each frame of the video according to a method for providing information on folliculitis mites according to an embodiment of the present invention.

[0096]

[0097] Hereinafter, with reference to FIGS. 1a to 1d, a system for providing information on a demodex mite based on a device for providing information on a demodex mite according to one embodiment of the present invention will be described.

[0098]

[0099] FIG. 1a illustrates an example of a system for providing information on demodex mites based on a device for providing information on demodex mites according to one embodiment of the present invention.

[0100] First, referring to FIG. 1A, a system (1000) for providing information on Demodex may be a system configured to provide information related to Demodex based on a facial image of an individual. At this time, the system (1000) for providing information on Demodex may be configured to determine a density probability of Demodex, a distribution probability of Demodex, and a skin treatment method for an individual based on an image including the individual's face, i.e., the facial area, of the individual, a device (100) for providing information on Demodex, a medical staff device (200) for receiving information on Demodex, and a user device (300).

[0101] At this time, the device (100) for providing information on demodex mites, the medical device (200) for receiving information on demodex mites, and the user device (300) can transmit and receive various information through wired and wireless communication.

[0102] More specifically, the device (100), the medical staff device (200), and the user device (300) for providing information on hair follicle mites can communicate via a wired connection that is directly connected by a cable and transmitted and received, but preferably, the cable is omitted and the communication can be wirelessly transmitted and received.

[0103] Accordingly, the device (100) for providing information on hair follicle mites, the medical staff device (200), and the user device (300) may be connected to a network for wireless communication. The network may be a closed network such as a local area network (LAN) and a wide area network (WAN), or an open network such as the Internet, or may be a short-range wireless communication such as Bluetooth, NFC (Near Field Communication), RFID (Radio-Frequency Identification), Wi-Fi, and / or Zigbee, but is not limited thereto.

[0104] A device (100) for providing information on demodex mites according to one embodiment of the present invention may include a general-purpose computer, laptop, and / or data server that performs various operations to predict the density probability of demodex mites and / or the distribution probability of demodex mites based on a facial image of an individual provided from a user device (300) and to determine a skin treatment method for the individual.

[0105] More specifically, referring to FIG. 1B, a block diagram illustrating the configuration of a device for providing information on a demodex mite according to one embodiment of the present invention is illustrated. The device (100) for providing information on a demodex mite may include a communication interface (110), a memory (120), an I / O interface (130), and a processor (140), and each component may communicate with one another via one or more communication buses or signal lines.

[0106] The communication interface (110) may refer to a communication unit (110) and may be connected to a medical device (200) and a user device (300) via a wired / wireless communication network to exchange data. For example, the communication interface (110) may receive a facial image of an object in real time from the user device (300). As another example, the communication interface (110) may transmit data on a folliculitis mite from the medical device (200).

[0107] Meanwhile, the communication interface (110) that enables transmission and reception of such data includes a wired communication port (111) and a wireless circuit (112), wherein the wired communication port (111) may include one or more wired interfaces, for example, Ethernet, Universal Serial Bus (USB), FireWire, etc. In addition, the wireless circuit (112) may transmit and receive data with an external device via an RF signal or an optical signal. In addition, the wireless communication may use at least one of a plurality of communication standards, protocols, and technologies, for example, GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

[0108] The memory (120) can store various data used and derived from the device (100) for providing information on Demodex mites. For example, the memory (120) can store various data such as a facial image of an individual, clinical information on the individual, a density probability for Demodex mites derived from the facial image and clinical information, a distribution probability for Demodex mites, Grad-CAM data derived from the facial image, a visualization image based on Grad-CAM, feature importance, etc.

[0109] In addition, the memory (120) can store the first prediction model, the second prediction model, the third prediction model, and the fourth prediction model, and can also store various data extracted or generated (predicted) from the aforementioned prediction models. For example, the memory (120) can include, but is not limited to, the first feature extracted from the first prediction model, the second feature extracted from the second prediction model, the final density probability extracted from the third prediction model, and the third feature extracted from the fourth prediction model.

[0110] Furthermore, the memory (120) may further include, but is not limited to, data derived from the aforementioned prediction model, as well as a saliency map generated from the first feature and patches for regions of interest within the face obtained from a facial image or saliency map.

[0111] In various embodiments, the memory (120) may include a volatile or non-volatile storage medium capable of storing various data, commands, and information. For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0112] In various embodiments, the memory (120) may store configurations of at least one of an operating system (121), a communication module (122), a user interface module (123), and one or more applications (124).

[0113] An operating system (121) (e.g., embedded operating systems such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers to control and manage general system operations (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.

[0114] The communication module (123) can support communication with other devices through the communication interface (110). The communication module (120) can include various software components for processing data received by the wired communication port (111) or wireless circuit (112) of the communication interface (110).

[0115] The user interface module (123) can receive a user's request or input from a keyboard, touch screen, keyboard, mouse, microphone, etc. through an I / O interface (130) and provide a user interface on the display.

[0116] The application (124) may include a program or module configured to be executed by one or more processors (140). Here, the application for providing information about the folliculitis may be implemented on a server farm.

[0117] The I / O interface (130) can connect at least one of an input / output device (not shown) of a device (100) for providing information on hair follicle mites, such as a display, a keyboard, a touch screen, and a microphone, to the user interface module (123). The I / O interface (130) can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module (123) and process commands according to the received input.

[0118] The processor (140) is connected to a communication interface (110), a memory (120), and an I / O interface (130) to control the overall operation of the device (100) for providing information on the hair follicle mite, and can perform various commands to extract information on the hair follicle mite through an application or program stored in the memory (120).

[0119] The processor (140) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Furthermore, the processor (140) may be implemented in the form of an integrated chip (IC), such as a SoC (System on Chip) in which various computing devices are integrated. Alternatively, the processor (140) may include a module for calculating an artificial neural network model, such as an NPU (Neural Processing Unit).

[0120] Referring again to FIG. 1A, a device (100) for providing information on demodex mites can receive a facial image from a user device (300) and provide the density probability of demodex mites, the distribution probability of demodex mites, and the skin treatment method to a medical device (200) from the received facial image.

[0121] Data provided from the device (100) for providing information on hair follicle mites in this manner may be provided as a web page via a web browser installed on a medical device (200), or may be provided in the form of an application or program. In various embodiments, such data may be provided in a form included in a platform in a client-server environment.

[0122] The medical device (200) is an electronic device that requests information on the subject's hair follicle mites and provides a user interface for displaying the information data, and may include at least one of a smartphone, a tablet PC (personal computer), a laptop, and / or a PC.

[0123] More specifically, referring to FIG. 1C, a block diagram of a configuration of a medical device according to one embodiment of the present invention is illustrated. The medical device (200) may include a memory interface (210), one or more processors (220), and a peripheral interface (230). Various components within the medical device (200) may be connected by one or more communication buses or signal lines.

[0124] The memory interface (210) is connected to the memory (250) and can transmit various data to the processor (220). Here, the memory (250) can include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0125] In various embodiments, the memory (250) may store at least one of an operating system (251), a communication module (252), a graphical user interface module (GUI) (253), a sensor processing module (254), a telephone module (255), and an application module (256). Specifically, the operating system (251) may include instructions for processing basic system services and instructions for performing hardware operations. The communication module (252) may communicate with at least one of one or more other devices, computers, and servers. The graphical user interface module (GUI) (253) may process a graphical user interface. The sensor processing module (254) may process sensor-related functions (e.g., processing voice input received using one or more microphones (292). The telephone module (255) may process telephone-related functions. The application module (256) may perform various functions of a user application, such as electronic messaging, web browsing, media processing, navigation, imaging, and other processing functions.

[0126] Additionally, the medical device (200) can store one or more software applications (256-1, 256-2) associated with a certain type of service in the memory (250). At this time, the application (256-1) can provide information about hair follicle mites to the medical device (200).

[0127] In various embodiments, the memory (250) may store a digital assistant client module (257) (hereinafter, DA client module), and accordingly, may store commands for performing client-side functions of the digital assistant and various user data (258) (e.g., user-customized vocabulary data, preference data, user's electronic address book, to-do list, other lists, etc.).

[0128] Meanwhile, the DA client module (257) can obtain the user's voice input, text input, touch input, and / or gesture input through various user interfaces (e.g., I / O subsystem (240)) provided in the medical device (200).

[0129] Additionally, the DA client module (257) can output data in audiovisual and tactile forms. For example, the DA client module (257) can output data consisting of a combination of at least two or more of voice, sound, notification, text message, menu, graphic, video, animation, and vibration. In addition, the DA client module (257) can communicate with a digital assistant server (not shown) using a communication subsystem (280).

[0130] In various embodiments, the DA client module (257) may collect additional information about the surroundings of the medical device (200) from various sensors, subsystems, and peripheral devices to construct a context associated with the user input. For example, the DA client module (257) may provide context information along with the user input to a digital assistant server to infer the user's intent. Here, the context information that may accompany the user input may include sensor information, such as lighting, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, the context information may include the physical state of the medical device (200) (e.g., device orientation, device position, device temperature, power level, speed, acceleration, motion patterns, cellular signal strength, etc.). As yet another example, the context information may include information related to the software state of the medical device (200) (e.g., processes running on the medical device (200), installed programs, past and present network activity, background services, error logs, resource usage, etc.).

[0131] In various embodiments, the memory (250) may include added or deleted instructions, and further, the medical device (200) may include additional configurations other than those illustrated in FIG. 1C, or may exclude some configurations.

[0132] The processor (220) can control the overall operation of the medical device (200) and execute various commands to implement various information interfaces for the folliculitis by running applications or programs stored in the memory (250).

[0133] The processor (220) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). In addition, the processor (120) may be implemented in the form of an integrated chip (IC) such as a SoC (System on Chip) in which various computing devices such as an NPU (Neural Processing Unit) are integrated.

[0134] The peripheral interface (230) can be connected to various sensors, subsystems, and peripheral devices to provide data so that the medical device (200) can perform various functions. Here, the function performed by the medical device (200) can be understood as being performed by the processor (220).

[0135] The peripheral interface (230) can receive data from a motion sensor (260), a light sensor (light sensor) (261), and a proximity sensor (262), through which the medical device (200) can perform orientation, light, and proximity detection functions, etc. For another example, the peripheral interface (230) can receive data from other sensors (263) (positioning system - GPS receiver, temperature sensor, biometric sensor), through which the medical device (200) can perform functions related to the other sensors (263).

[0136] In various embodiments, the medical device (200) may include a camera subsystem (270) connected to a peripheral interface (230) and an optical sensor (271) connected thereto, which enables the medical device (200) to perform various photographing functions, such as taking pictures and recording video clips.

[0137] In various embodiments, the medical device (200) may include a communication subsystem (280) connected to a peripheral interface (230). The communication subsystem (280) may be comprised of one or more wired / wireless networks and may include various communication ports, radio frequency transceivers, and optical transceivers.

[0138] In various embodiments, the medical device (200) includes an audio subsystem (290) coupled to the peripheral interface (230), the audio subsystem (290) including one or more speakers (291) and one or more microphones (292), such that the medical device (200) can perform voice-activated functions, such as voice recognition, voice replication, digital recording, and telephony.

[0139] In various embodiments, the medical device (200) may include an I / O subsystem (240) connected to a peripheral interface (230). For example, the I / O subsystem (240) may control a touch screen (243) included in the medical device (200) via a touch screen controller (241). As an example, the touch screen controller (241) may detect a user's contact and movement or cessation of contact and movement using any one of a plurality of touch sensing technologies, such as capacitive, resistive, infrared, surface acoustic wave technology, or a proximity sensor array. As another example, the I / O subsystem (240) may control other input / control devices (244) included in the medical device (100) via other input controller(s) (242). As an example, the other input controller(s) (242) may control one or more buttons, rocker switches, thumb-wheels, infrared ports, USB ports, and pointer devices such as a stylus.

[0140] Referring back to FIG. 1a, the medical device (200) can receive various information about demodex mites, such as the density probability of demodex mites, the distribution probability of demodex mites, and the skin treatment method for the individual, from the device (100) for providing information about demodex mites, as it includes the configuration described above.

[0141] Meanwhile, the user device (300) may include the same configuration as the medical device (200) described above in FIG. 1C. Accordingly, the user device (300) may include a camera subsystem connected to a peripheral interface and an optical sensor connected thereto, thereby capturing a user's facial image (310) and providing it to the device (100) for providing information on hair follicle mites and / or the medical device (200).

[0142] More specifically, referring to FIG. 1D , a schematic diagram of a user device according to one embodiment of the present invention is illustrated. The user device (300) may be configured with an interface screen (350) to enable the user's face to be captured as an image (310).

[0143] Furthermore, the user device (300) may be configured with an interface screen (350) so that the user can input not only the user's facial image (300), but also the user's information and symptoms of a disease.

[0144] Accordingly, the device (100) for providing information on hair follicle mites can more easily obtain and confirm the user's status from the user device (300).

[0145] In addition, since the user device (300) includes the same configuration as the aforementioned medical device (200), it receives information on the density probability of the demodex mites, the distribution probability of the demodex mites, and the skin treatment method for the object from the device (100) for providing information on the demodex mites in the same manner as the medical device (200), and the received data can be displayed and provided to the user through the interface screen (350).

[0146] Referring back to FIG. 1A, the user device (300) may refer to any device capable of providing a facial image (310) to the device (100) for providing information on hair follicle mites. Accordingly, the user device (300) may include, but is not limited to, a smartphone, a tablet PC (Personal Computer), a laptop, and / or a PC, as described above, and may further include a general-purpose computer and / or data server capable of providing a pre-stored facial image (310).

[0147] Finally, the system for providing information on demodex mites based on the device for providing information on demodex mites according to one embodiment of the present invention includes a device for providing information on demodex mites (100), a medical device (200), and a user device (300), and thus receives a facial image (310) of an individual (user) from the user device (300), and predicts and determines various information (data) about a density probability of demodex mites for the individual, a distribution probability of demodex mites, and a skin treatment method for the individual based on the facial image (310) received from the device for providing information on demodex mites (100), and provides various information (data) about demodex mites derived from the device for providing information on demodex mites (100) to the medical device (200) and / or the user device (300). However, without limitation, the operation of predicting and determining various information (data) about the density probability of demodex mites for an object, the distribution probability of demodex mites, and the skin treatment method for an object based on the received facial image (310) may be performed by a medical device or a user device (300).

[0148]

[0149] Hereinafter, with reference to FIG. 2, the procedure of a method for providing information on a hair follicle mite according to one embodiment of the present invention will be described. At this time, for convenience of explanation, the description will be made with reference to FIGS. 3a to 9.

[0150]

[0151] Figure 2 is a flowchart of a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0152] Referring to FIG. 2, a method for providing information on Demodex mites according to an embodiment of the present invention includes the steps of: receiving a face image of an individual (S210); extracting a first feature for Demodex mites by inputting the face image into a first prediction model trained to extract a first feature for Demodex mites by inputting the face image (S220); generating a saliency map from the first feature (S230); acquiring regions of interest in the face from the saliency map as one or more patches; extracting a second feature for Demodex mites by inputting one or more patches into a second prediction model trained to extract a second feature for Demodex mites by inputting one or more patches (S250); and inputting the first feature and the second feature into a third prediction model trained to predict a density probability of Demodex mites for an individual by inputting the first feature and the second feature. It may include a step (S260) of predicting the final density probability of the hair follicle mite for an object by inputting the feature.

[0153] First, a method for providing information on Demodex mite according to an embodiment of the present invention may include a step (S210) of receiving a facial image of an individual, and in the step (S210) of receiving a facial image of an individual, the facial image of the individual may refer to a facial image of an individual suspected of having a disease caused by Demodex mite. In addition, the facial image of the individual may refer to an image including an upper region of the individual, preferably including a facial region above the neck.

[0154] Furthermore, in the step of receiving a face image of an object (S210), a face image captured from an object, i.e., a user's device, may be received, but is not limited thereto, and an image including a face area of ​​an object pre-stored in a general-purpose computer, laptop, and / or data server, etc. may be received.

[0155] Furthermore, the step of receiving a facial image of an object (S210) may further include a step of receiving clinical information of the object as well as the facial image of the object. More specifically, the step of receiving a facial image of the object (S210) may include receiving clinical information related to a lesion in the face of the object, and such clinical information may include, but is not limited to, data stored in a medical institution and / or data acquired through a question-and-answer session in the device of the object (user), and may include various data related to the object and the lesion in the face.

[0156] Furthermore, clinical information may include, but is not limited to, gender, age, underlying disease, allergy markers including serum eosinophil count, total IgE and ECP, symptoms including itching, intermittent flushing, fever, dryness and swelling, etc., and clinical information (age and / or allergy markers which are continuous variables) may be standardized to have a mean of 0 and a variance of 1 and used in a method for providing information on Demodex mites according to an embodiment of the present invention.

[0157] Next, the method for providing information on a follicle mite according to one embodiment of the present invention may further include a step of de-identifying a landmark within the face from the received face image after the receiving step (S210).

[0158] More specifically, referring to FIG. 3a, a schematic diagram of a de-identification process according to a method for providing information on a hair follicle mite according to one embodiment of the present invention is illustrated.

[0159] First, the received facial image is automatically set to coordinates (10), a facial area is detected based on the set coordinates (10), and based on the detected facial area, only the facial portion of the received facial image is cropped to be enlarged, and a facial landmark (11) in the cropped image can be de-identified.

[0160] Furthermore, the area that serves as a reference for the landmark to be anonymized can be set based on coordinates (10).

[0161] More specifically, referring to FIG. 3b, a schematic diagram of coordinates in a de-identification process according to a method for providing information on a hair follicle mite according to one embodiment of the present invention is illustrated.

[0162] The received facial image can be anonymized by setting coordinates (10) of facial lines, eyebrows, nose, mouth, etc. based on the anatomical structure of the face, and covering up landmarks (11) within the face based on the set coordinates (10). At this time, the facial landmarks (11) may include, but are not limited to, eyes and / or mouth, and may include various areas within the face.

[0163] Furthermore, the de-identification method may be performed by, but is not limited to, blinding or mosaicking, and may include any method that can protect an individual's portrait and personal information.

[0164] Accordingly, in the method for providing information on hair follicle mites according to one embodiment of the present invention, personal portrait and personal information can be safely protected by a de-identification method.

[0165] Moreover, the method for providing information on Demodex mites according to one embodiment of the present invention can minimize interference from areas other than the area of ​​interest where Demodex mites are predicted to exist by providing an anonymized image, and can predict more accurate density and distribution of Demodex mites within the area of ​​interest.

[0166] Accordingly, referring back to FIG. 2, the method for providing information on a hair follicle mite according to an embodiment of the present invention may further include a step of de-identifying a facial landmark from the received facial image after the step (S210) of receiving a facial image of an individual, as including the processes of FIGS. 3a and 3b described above. In this case, the de-identifying step may include a step of extracting coordinates for an anatomical structure within the face from the received facial image, and de-identifying the facial landmark based on the extracted coordinates.

[0167] Next, a method for providing information on a demodex mite according to an embodiment of the present invention may include a step (S220) of extracting a first feature for a demodex mite by inputting a facial image into a first prediction model, and in the step (S220) of extracting the first feature, the first prediction model may be a model trained to extract the first feature for a demodex mite by inputting a facial image.

[0168] More specifically, referring to FIG. 4, an exemplary diagram is shown for the learning and inference process for the first prediction model in the method for providing information on hair follicle mites according to one embodiment of the present invention.

[0169] First, in a method for providing information on a hair follicle mite according to an embodiment of the present invention, a first prediction model (30) may be included in a global module, and the global module may include a process in which a facial image (13) is input to the first prediction model (30), a first feature (31) is extracted, a convolution layer (1x1 conv.) and a sigmoid function are sequentially performed on the extracted first feature (31), thereby generating a significant region map (33), and a fagg layer is applied to the generated significant region map (33) to predict a first density probability (global prediction).

[0170] That is, the first prediction model (30) can not only extract the first feature (31), but also generate (predict) a saliency map and a first density probability.

[0171] More specifically, the face image (13) is adjusted to (2944, 1920, 3) pixels and can be normalized to a value between -1 and 1, but is not limited thereto, and can be adjusted to pixels of various sizes. At this time, the face image (13) may be an image in which landmarks within the face are de-identified, but is not limited thereto, and may also include non-de-identified face images received from medical staff devices, user devices, and medical institution DBs.

[0172] Next, the adjusted face image (13) can be input into the first prediction model (30), and thus, the first feature (feature maps, 31) can be extracted. Furthermore, the extracted first feature (31) can be used as a vector value after performing pooling in the third prediction model, which will be described later.

[0173] At this time, the first prediction model (30) may be a model based on at least one of ResNet, XGBoost, and MLP, but is not limited thereto, and may be at least one model among EfficientNet, AlexNet, GoogLeNet, SENet, GPipe, SqueezeNet, MobileNets, ShuffleNets, ResNet, DenseNet, U-net, and VGG-Net models, and an ensemble model composed of a combination of at least two of the above-described models.

[0174] However, preferably, the first prediction model (30) that inputs a face image (13) may be a convolutional neural network (CNN) that has undergone ImageNet transfer learning, such as ResNet 18 and / or DenseNet.

[0175] That is, the method for providing information on Demodex mite according to one embodiment of the present invention can be sufficiently trained to have high predictive power even with a small amount of facial image data by including the first prediction model (30) of transfer-learned ResNet 18 and / or DenseNet. Accordingly, the method for providing information on Demodex mite according to one embodiment of the present invention can overcome limitations in the amount of learning data.

[0176] Next, a convolution layer (1x1 conv.) and a sigmoid function can be applied to the first feature (31) extracted from the global module to generate (output) a saliency map (33).

[0177] Furthermore, although the 1x1 sized convolution layer is illustrated as reducing the first feature (31) to 1 channel, it is not limited thereto, and various channels desired by the user may be set to be derived depending on the number of classes. However, in the method for providing information on Demodex mites according to one embodiment of the present invention, 1 channel may be most preferable in order to output only the class in which Demodex mites are predicted to be present (positive class). That is, the 1x1 sized convolution layer (1x1 conv layer) may be a layer that can be adjusted to a desired number of channels.

[0178] Accordingly, a method for providing information on demodex mites according to one embodiment of the present invention can generate a significant area map (33), which is visualization data for an area where demodex mites are predicted to be concentrated, from a first feature (31), based on the output result of a convolution layer in a global module.

[0179] At this time, the important area map (33) means a visualization method model that expresses a part with a large result value on an image, and in the method for providing information on hair follicle mites according to one embodiment of the present invention, a saliency map was used, but is not limited thereto, and Grad-cam, which is a visualization method model that multiplies a feature map by a weight and outputs it as a heat map, can be applied, and the weight in Grad-cam can use a differential value (the influence that each element of the feature map has on a specific class).

[0180] Next, a fagg layer for average pooling is applied to the important region map (33) so that the first density probability, i.e., the global prediction, which is the density probability of the follicle mites for the entire facial area, can be predicted.

[0181] More specifically, since the important region map (33) is 1 channel (num_classes), only pooling for dimensionality reduction is performed in the global module so that the data for the important region map (33) can be flattened to 1 dimension. At this time, the fagg layer for pooling is a function that can convert the important region map (33) into image level class prediction, and GAP (global average pooling) or GMP (global max pooling) can be used, but in the method for providing information on Demodex mite according to one embodiment of the present invention, top t % average pooling (t = hyperparameter) may be preferable.

[0182] Accordingly, the first prediction model (30) in the global module may be a model learned to predict the probability for class 0 and class 1 by inputting a facial image, and the first density probability, the global prediction value, may mean the density prediction value for the follicle mites of an individual derived from the facial image, and may be predicted and displayed as the probability for class 0 and class 1.

[0183] For example, class 0 is 1 cm 2 The number of hair follicle mites per cell is 0 or more and less than 5, and class 1 is 1 cm 2 The number of hair follicle mites per colony may be 5 or more, but is not limited thereto.

[0184] Furthermore, the first prediction model (30) in the method for providing information on hair follicle mites according to one embodiment of the present invention may present a prediction result as a binary classification such as the aforementioned class 0 and class 1, but is not limited thereto, and may have two or more output values ​​as a multi classification.

[0185] Finally, referring back to FIG. 2, the method for providing information on a demodex mite according to an embodiment of the present invention may include a step (S220) of inputting a facial image to a first prediction model trained to extract a first feature for a demodex mite by inputting a facial image, and a step (S230) of generating a saliency map from the first feature, as it includes the process of FIG. 4 described above.

[0186] In addition, a method for providing information on demodex mites according to one embodiment of the present invention may further include, after the step of extracting the first feature, a step of inputting the first feature into a first prediction model that is further trained to predict the density probability of demodex mites for an object by inputting the first feature into the first prediction model, thereby predicting the first density probability of demodex mites for the object.

[0187] That is, the step of extracting the first feature (S220), the step of generating a significant area map (S230), and the step of predicting the first density probability may be performed within one global module including the first prediction model, but are not limited thereto.

[0188] Next, a method for providing information on a follicle mite according to one embodiment of the present invention may include a step (S240) of acquiring regions of interest within a face as one or more patches from a significant region map, and the patches may be acquired according to a search algorithm based on the significant region map.

[0189] More specifically, referring to FIG. 5, an exemplary diagram of a patch acquisition process in a method for providing information on a hair follicle mite according to one embodiment of the present invention is illustrated.

[0190] In a method for providing information on Demodex mites according to one embodiment of the present invention, a patch (15) used may be obtained from a face image (13) by retrieving a region of interest (retrieve ROI) from a significant region map (33), generating a patch map (patch map, 35) based on the retrieved region of interest, and then generating location information data from the generated patch map (35). At this time, the significant region map (33) used in the process of obtaining the patch (15) may be a significant region map (33) generated from a global module, but is not limited thereto. That is, regions of interest within the face may be acquired as one or more patches from a face image or significant region map.

[0191] Furthermore, using the patch location information (6, ) obtained by retrieving the region of interest (retrieve ROI), patches (N, 18, 256, 256) can be obtained for each R, G, B (3-channel color image expressed by a combination of red, green, and blue colors) channel from the original face image (13).

[0192] This patch acquisition process can be performed by a patch search algorithm, and the patch search algorithm can mean an algorithm that determines data within a window while a window of a fixed size moves. That is, the patch search algorithm used in the method for providing information on Demodex mites according to one embodiment of the present invention is a sliding window algorithm, in which a sliding window of size (256, 256, 3) moves around the important region map (33) or the face image (13), captures the ROI (region of interest) with the largest sum of values, and crops the data for the corresponding coordinates to acquire (extract) it as a patch.

[0193] Accordingly, the size of the patch (15) used in the method for providing information on hair follicle mites according to one embodiment of the present invention may be (256, 256, 3), which is a sliding window size, but is not limited thereto and may be set to various sizes by the user.

[0194] Furthermore, the number of patches (15) may vary depending on the ROI in the face image and / or important region map of the subject, but in one embodiment of the present invention, it may be 1 to 10, but is not limited thereto, and a preferred number of patches (15) may be 6.

[0195] Ultimately, the patch (15) in the method for providing information on a folliculitis mite according to one embodiment of the present invention may mean a patch obtained by automatically selecting an area where a folliculitis mite is predicted to be distributed by machine learning (artificial intelligence), rather than a patch of an area preset based on the anatomical structure of the face.

[0196] Accordingly, the location and number of patches (15) obtained for each individual may be different, and the method for providing information on demodex mites according to one embodiment of the present invention can predict a more accurate density probability for demodex mites by using only patches in an area where demodex mites are predicted to be distributed.

[0197] Again, referring to FIG. 2, a method for providing information on a demodex mite according to one embodiment of the present invention may include a step (S250) of extracting a second feature on a demodex mite by inputting one or more patches into a second prediction model.

[0198] More specifically, referring to FIG. 6a, an exemplary diagram is shown for the learning and inference process for the second prediction model in the method for providing information on hair follicle mites according to one embodiment of the present invention.

[0199] First, in a method for providing information on a hair follicle mite according to an embodiment of the present invention, a second prediction model (40) may be included in a local module, and the local module may include a process in which a patch (15) obtained from a facial image (13) and / or a significant region map (33) is input into a second prediction model (40), a second feature (41) is extracted, attention is applied to the extracted second feature (41), and a fc layer (Fully Connected Layer, fc layer) and a softmax function are sequentially performed on the second feature (43) to which attention is applied, thereby extracting (predicting) a second density probability (local prediction).

[0200] That is, the second prediction model (40) can generate (predict) not only the second feature (41), but also the second density probability.

[0201] More specifically, patches (patches, 15) of size (256, 256, 3) can be input to the second prediction model (40), and second features (feature maps, 41) of size (n, 6, 512) can be extracted. At this time, since the patch (15) can be the patch (15) obtained in the process of FIG. 5 above, it can be a patch (N, 18, 256, 256) according to the R, G, and B channels.

[0202] Next, before being input to the second prediction model (40), the patch (15) may be subjected to a reshape function to change its shape and structure (rows and columns) to (N*6, 3, 256, 256). That is, one or more patches (15) input to the second prediction model (40) may be input in the form of a single batch, as six patches (15) containing R, G, and B information.

[0203] At this time, in the method for providing information on hair follicle mites according to one embodiment of the present invention, the number of patches input to the second prediction model (40) is shown to be 6, but is not limited thereto, and since the patches obtained may be different depending on the individual, the number of input patches may be different depending on the individual or may be freely set by the user.

[0204] Furthermore, the second prediction model (40) may be a model based on at least one of ResNet, XGBoost, and MLP, similar to the first prediction model (30), but is not limited thereto, and may be at least one model among EfficientNet, AlexNet, GoogLeNet, SENet, GPipe, SqueezeNet, MobileNets, ShuffleNets, ResNet, DenseNet, U-net, and VGG-Net models, and an ensemble model composed of a combination of at least two of the aforementioned models.

[0205] However, preferably, the second prediction model (40) that takes the patch (15) as input may be a convolutional neural network (CNN) with ImageNet transfer learning, such as ResNet 18 and / or DenseNet.

[0206] That is, the method for providing information on Demodex mites according to one embodiment of the present invention can be sufficiently trained to have high predictive power even with a small amount of patch data by including a second prediction model (40) of transfer-learned ResNet 18 and / or DenseNet. Accordingly, the method for providing information on Demodex mites according to one embodiment of the present invention can overcome limitations in the amount of learning data.

[0207] Next, the second feature (41) can be applied with an attention score derived by an attention module.

[0208] More specifically, referring to FIG. 6b, it is a block diagram showing the configuration of an attention module in a method for providing information on a hair follicle mite according to one embodiment of the present invention.

[0209] In a method for providing information on a hair follicle mite according to one embodiment of the present invention, the attention module is a module capable of deriving (calculating) an attention score indicating which patch among one or more patches the second prediction model focused on and learned, and may include, but is not limited to, three fully connected layers (FC layers) and three activation functions.

[0210] At this time, the activation function is a function that can improve the recognition efficiency of complex patterns by increasing nonlinearity, and includes, but is not limited to, sigmoid, hyperbolic tangent (tanh), and softmax, and various functions that can increase the nonlinearity of data or compress and represent data for output efficiency are all included and can be used in the attention module of the present invention.

[0211] Furthermore, the order and placement of the activation functions are not limited to the order of sigmoid, hyperbolic tangent (tanh), and softmax, and can be freely set by the user. However, it may be most desirable to place sigmoid or hyperbolic tangent (tanh) first to activate the neurons of the fc layer, and softmax last to calculate attention.

[0212] Accordingly, the second feature is input to the attention module and outputs 6 logits, and the attention score (attention score = prob, Probability) for each patch can be derived (output) by multiplying the output logits by the softmax output value (softmax output).

[0213] Referring back to FIG. 6a, the attention score derived through the process of FIG. 6b described above can be derived (output) for each patch, and the derived attention score can be used by multiplying it by the second feature (41). Accordingly, the second feature (41) can be output as an output vector value (attention-weighted representation Z vector) to which the attention score is applied.

[0214] Next, the output vector value can be input to the fc layer so that each patch can be classified by label, and the classification value (43) output by the fc layer can be classified into classes again by applying the softmax function. That is, since the local module includes the fc layer and softmax, which are classifiers, the output vector value can be predicted as a local prediction, which is a second density probability. At this time, the classification value (43) output by the fc layer can be used as input data for the third prediction model to be described later.

[0215] Accordingly, the second prediction model (40) in the local module may be a model trained to predict the probability for class 0 and class 1 by inputting a patch, and the second density probability, which is a local prediction value, may mean a density prediction value for the individual hair follicle mites derived from a patch rather than a prediction value for each patch, and may be predicted and displayed as a probability for class 0 and class 1.

[0216] For example, category 0 is 1 cm 2 The number of hair follicle mites per cell is 0 or more but less than 5, and the length is 1 cm 2 The number of hair follicle mites per colony may be 5 or more, but is not limited thereto.

[0217] Furthermore, the second prediction model (40) in the method for providing information on hair follicle mites according to one embodiment of the present invention may present the prediction result as a binary classification such as the aforementioned class 0 and class 1, but is not limited thereto, and may have two or more output values ​​as a multi classification.

[0218] Finally, referring back to FIG. 2, the method for providing information on a demodex mite according to one embodiment of the present invention may include a step (S250) of extracting a second feature on a demodex mite by inputting one or more patches into a second prediction model, as it includes the processes in FIGS. 6a and 6b described above.

[0219] In addition, the method for providing information on demodex mites according to one embodiment of the present invention may further include, after the step of extracting the second feature, a step of inputting the second feature into a second prediction model that is further trained to predict the density probability of demodex mites for an object by inputting the second feature into the second prediction model, thereby predicting the second density probability of demodex mites for an object.

[0220] That is, the step of extracting the second feature (S250) and the step of predicting the second density probability may be performed within one local module including the second prediction model, but are not limited thereto.

[0221] Next, the method for providing information on Demodex mite according to one embodiment of the present invention may include a step (S260) of predicting the final density probability of Demodex mite 1 for an individual by inputting the first feature and the second feature into a third prediction model, and in the step (S260) of predicting the final density probability, the third prediction model may be a model learned to predict the density probability of Demodex mite for an individual by inputting the first feature and the second feature.

[0222] More specifically, referring to FIG. 7, an exemplary diagram of the entire learning and inference process of a method for providing information on a hair follicle mite according to one embodiment of the present invention is illustrated.

[0223] A method for providing information on Demodex mites according to one embodiment of the present invention may include a global module including a first prediction model (30) capable of extracting a first feature (31), a local module including a second prediction model (40) capable of extracting a second feature (41), a fusion module including a third prediction model (50) capable of extracting (predicting) a final density probability (fusion prediction, 51), and a clinical data module including a fourth prediction model (60) capable of extracting a third feature (61).

[0224] First, in the fusion module, the third prediction model (50) can predict the density probability of the hair follicle mite of an individual by using the first feature (31) and the second feature (41) as inputs, but is not limited thereto.

[0225] At this time, the first feature (31) input to the third prediction model (50) may mean a vector value (37) to which pooling is applied for dimensionality reduction, and may be a vector of the size of ( , 512). More specifically, the vector value (37) pooled from the first feature (31) may mean a vector value of an activation layer connected by extracting the maximum value of each feature map from the first feature (31) extracted from the first prediction model. That is, the pooling applied to the first feature (31) may mean global max pooling, but is not limited thereto.

[0226] Furthermore, the second feature (41) input to the third prediction model (50) may mean an output vector value (attention-weighted representation Z vector, 43) of an FC layer (fully connected layer) to which attention is applied, and may be a vector of the size of ( , 512).

[0227] Accordingly, the vector value (37) to which pooling is applied for the first feature (31) and the output vector value (43) of the FC layer to which attention is applied for the second feature (41) are combined into one vector by the concat function, and the combined vector can be trained on the classifier fc layer (fully connected layer) and softmax to output (predict) the final density probability (fusion prediction, 51). That is, the third prediction model (50) may mean a model including a concat function, an fc layer (fully connected layer), and a softmax, but is not limited thereto.

[0228] Furthermore, the third prediction model (50) in the fusion module may be a model trained to predict the probability for class 0 and class 1 using the first feature (31) and the second feature (41) as inputs, and the final density probability may mean the final density prediction value for the hair follicle mites of the individual, and may be predicted and displayed as the probability for class 0 and class 1.

[0229] For example, category 0 is 1 cm 2 The number of hair follicle mites per cell is 0 or more but less than 5, and the length is 1 cm 2 The number of hair follicle mites per colony may be 5 or more, but is not limited thereto.

[0230] Furthermore, the third prediction model (50) in the method for providing information on hair follicle mites according to one embodiment of the present invention may present prediction results as binary classifications such as the aforementioned class 0 and class 1, but is not limited thereto, and may have two or more output values ​​as multi classifications.

[0231] Meanwhile, the method for providing information on Demodex mites according to one embodiment of the present invention may further include a clinical data module. More specifically, the clinical data module may include a fourth prediction model (60), which may extract a third feature (61) from clinical information (clinical features, 16).

[0232] At this time, clinical information (16) can be received from medical staff devices, user devices, and medical institution DBs, and the received clinical information can be adjusted and normalized to a size of (12, 1) and used as clinical features (clinical features, 16).

[0233] Furthermore, the fourth prediction model (60) may be a model based on at least one of ResNet, XGBoost, and MLP, but is not limited thereto, and may be at least one model among EfficientNet, AlexNet, GoogLeNet, SENet, GPipe, SqueezeNet, MobileNets, ShuffleNets, ResNet, DenseNet, U-net, and VGG-Net models, and an ensemble model composed of a combination of at least two of the aforementioned models.

[0234] However, preferably, the fourth prediction model (60) that inputs clinical information (16) may be at least one of a multilayer perceptron (MLP), a random forest, and XGBoost.

[0235] In the clinical information module, a third feature (61) can be extracted from clinical information (clinical features, 16) by the fourth prediction model (60), and the extracted third feature (61) is combined into one vector by the concat function together with the first feature (31) and the second feature (41), and input into the third prediction model (50) to predict the density probability of an object.

[0236] At this time, the size of the third feature may be ( , 32), and the size of the vector in which the first feature (31), the second feature (41), and the third feature (61) are combined may be ( , 1056).

[0237] Accordingly, in the method for providing information on a demodex mite according to one embodiment of the present invention, the accuracy and reliability of prediction of a demodex mite of an individual can be further improved by additionally inputting and using a third feature (61).

[0238] Meanwhile, various learning procedures included in the method for providing information on demodex mites according to one embodiment of the present invention can generalize the model using a cross validation method.

[0239] Furthermore, in the method for providing information on hair follicle mites according to one embodiment of the present invention, the loss can be obtained by adding all individual loss values ​​derived by applying a binary cross entropy loss function, and then applying the result to [Mathematical Formula 1] below.

[0240]

[0241] BCE( ) : Loss function for the first density probability

[0242] BCE( ) : Loss function for the second density probability

[0243] BCE( ) : Loss function for final density probability

[0244] : Regularization formula for model generalization

[0245] At this time, since the aforementioned [Mathematical Formula 1] includes all loss functions obtained from the global module, local module, and fusion module, learning for all of the first feature, second feature, and third feature can be possible.

[0246] Furthermore, the loss function includes a significant area map mean value, such as a penalty ( ) can be added to obtain a generalized model, which is not mandatory and can be freely set by the user.

[0247] Furthermore, the fourth prediction model (60) can provide SHAP-based feature importance for clinical information based on clinical information (16). More specifically, referring to FIG. 8, an exemplary SHAP result is illustrated in a method for providing information on Demodex mites according to one embodiment of the present invention.

[0248] The fourth prediction model (60) can be output as a SHAP (Shapley Additive exPlanations) value by taking clinical information (16) as input and showing the influence (feature importance) of the clinical information (16) on the output value (density probability for Demodex mites).

[0249] Accordingly, the method for providing information on Demodex mites according to one embodiment of the present invention further includes a step of outputting characteristic importance based on clinical information, thereby making it possible to more easily and visually confirm the contribution of clinical information of an individual to the probability of Demodex mites density.

[0250] Finally, referring back to FIG. 2, the method for providing information on a demodex mite according to one embodiment of the present invention may include a step (S260) of predicting the final density probability of a demodex mite for the object by inputting the first feature and the second feature into the third prediction model, as the method includes the processes of FIGS. 7 and 8 described above.

[0251] In addition, the method for providing information on demodex mites according to one embodiment of the present invention may further include a step of receiving clinical information of an individual by using clinical information.

[0252] Furthermore, a method for providing information on Demodex mites according to one embodiment of the present invention may further include, after the step of receiving clinical information, a step of inputting clinical information into a fourth prediction model trained to extract features of Demodex mites by inputting clinical information, thereby extracting a third feature for Demodex mites.

[0253] Accordingly, in the method for providing information on demodex mites according to one embodiment of the present invention, the third prediction model may be further trained to predict the density probability of demodex mites for an individual by inputting the first feature, the second feature, and the third feature, and the step of predicting the final density probability (S260) may include a step of predicting the final density probability of demodex mites for the individual by inputting the first feature, the second feature, and the third feature into the third prediction model described above.

[0254] In addition, a method for providing information on demodex mites according to one embodiment of the present invention may further include a step of outputting characteristic importance based on clinical information.

[0255] Next, the method for providing information on demodex mites according to one embodiment of the present invention may further include a step of providing a treatment guideline based on the predicted final density probability.

[0256] More specifically, referring to FIG. 9, an exemplary diagram for the final density probability of a demodex mite for a method for providing information on a demodex mite according to one embodiment of the present invention is shown.

[0257] A method for providing information on Demodex mites according to one embodiment of the present invention can predict the final density probability (51) for Demodex mites through a third prediction model (50). That is, the final density probability (51) for Demodex mites of a predicted individual can be output as the probabilities of class 0 and class 1, and treatment guidelines can be displayed and provided for each classification.

[0258] At this time, the guidelines may include, but are not limited to, domestic and international disease guidelines, including guidelines for the use of antibiotics for skin and soft tissue infections, clinical practice guidelines, etc., and all treatment guidelines published online and offline may be included.

[0259] For example, in a method for providing information on demodex mites according to one embodiment of the present invention, if the predicted probability is class 1, a recommendation for the use of antibiotics (topical antiparasitic agents) or oral antiparasitic agents may be provided, and in addition, improvements in related lifestyle habits, etc. may be additionally provided based on rosacea disease guidelines.

[0260] According to the method for providing information on Demodex folliculitis according to one embodiment of the present invention, the present invention can objectively and quickly perform a diagnosis and prediction of Demodex folliculitis non-invasively, thereby reducing the psychological and financial burden on patients due to biopsies and further improving the workflow of medical staff in actual clinical practice. Furthermore, the medical staff can quickly select an appropriate treatment method based on the results, thereby contributing to a positive treatment outcome in the prognosis of skin diseases caused by Demodex folliculitis, including the injection of the present invention.

[0261]

[0262] Evaluation: Evaluation of the predictive models used in various embodiments of the present invention.

[0263]

[0264] Below, the evaluation results for the prediction model used in the method for providing information on demodex mites according to one embodiment of the present invention will be described.

[0265] FIG. 10 illustrates various image results derived from a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0266] According to one embodiment of the present invention, a method for providing information on a hair follicle mite may be performed by inputting a facial image, generating a saliency map based on a first feature, a patch map in which a region of interest is retrieved (retrieve ROI) from the saliency map, and obtaining patches based on the patch map. In this case, the saliency map

[0267] According to an embodiment of the present invention, a method for providing information on Demodex mites is not a method of extracting only a preset area based on an anatomical structure, but rather, an area predicted to be distributed by Demodex mites can be automatically selected, displayed, and presented by machine learning (artificial intelligence). Therefore, the displayed area of ​​interest in a saliency map and a patch map and the location of a patch obtained from the area of ​​interest may differ for each individual.

[0268] Accordingly, referring to FIG. 10, since the input facial image is different for each object, the region of interest displayed in the saliency map and patch map and the patch obtained from the region of interest appear to be different.

[0269] Furthermore, a method for providing information on hair follicle mites according to one embodiment of the present invention can derive (calculate) an attention score for each patch using an attention module. Accordingly, the attention score for each patch appears to be derived differently depending on the skin condition on the patch.

[0270] That is, the method for providing information on demodex mites according to one embodiment of the present invention can provide skin condition and related information on demodex mites tailored to an individual based on the skin condition of the individual, and thus, the present invention can predict the density and distribution of demodex mites with high accuracy.

[0271] Figures 11a and 11b are evaluation results for various prediction models used in a method for providing information on hair follicle mites according to one embodiment of the present invention.

[0272] At this time, the prediction models included in various embodiments of the present invention were trained based on facial images and clinical information of 1,024 patients collected in a clinical setting, and the trained patient data used were 769 patients as class 0 and 255 patients as class 1. In addition, the ratio of training data and test data was 9:1, and 924 training data (0:719, 1:205) were used, and 100 test data (0:50, 1:50) were used. In addition, ResNet18 was used for the first and second prediction models because the input data was face images and patch image data, MLP was used for the fourth prediction model because the input data was clinical data, and a model including a concat function, a fully connected layer, and a softmax that can combine and classify the input data as vector values ​​was used for the third prediction model, but the evaluation results should not be interpreted as being limited to the corresponding models. Furthermore, each fold is a data set that is combined by separating the input data into a training set (Training Set / Train) and a validation set (Validation Set / Valid), and then performing cross-validation that repeatedly learns in parallel multiple times, and the ratio of training set (train): validation set (valid): test set (test) can be 8:2:1, but is not limited thereto.

[0273] First, referring to (a) of FIG. 11a, an evaluation result for an image-based model used in a method for providing information on a hair follicle mite according to one embodiment of the present invention is illustrated, and the image-based model may include a first prediction model and a second prediction model that input image data such as a facial image and a patch.

[0274] The accuracy and AUC for the validation prediction values ​​of the image-based model are 0.775 and 0.8523 for fold 1, 0.8 and 0.8355 for fold 2, 0.65 and 0.7031 for fold 3, 0.775 and 0.8517 for fold 4, 0.7875 and 0.8634 for fold 5, 0.7375 and 0.8462 for fold 6, 0.85 and 0.8703 for fold 7, 0.6875 and 0.7987 for fold 8, and 0.8125 and 0.8125 for fold 9. It appears to be 0.8410, and for fold 10, it appears to be 0.725 and 0.789.

[0275] That is, since the accuracy and AUC for the verification prediction value of the image based model used in the method for providing information on demodex mites according to one embodiment of the present invention are 0.65 and 0.7031 or higher, it can be meant that it has a predictive power higher than normal.

[0276] Furthermore, the accuracy and AUC for the test prediction values ​​of the image based model are 0.64 and 0.7286 for fold 1, 0.675 and 0.762 for fold 2, 0.695 and 0.7453 for fold 3, 0.665 and 0.7288 for fold 4, 0.69 and 0.7517 for fold 5, 0.635 and 0.7399 for fold 6, 0.65 and 0.7867 for fold 7, 0.665 and 0.7597 for fold 8, and 0.655 and 0.7597 for fold 9. It appears to be 0.769, and for fold 10, it appears to be 0.66 and 0.7637.

[0277] That is, since the accuracy and AUC for the test prediction values ​​of the image based model used in the method for providing information on demodex mites according to one embodiment of the present invention are 0.635 and 0.7453 or higher, it can be meant that it has a predictive power higher than normal.

[0278] Next, referring to (b) of FIG. 11a, an evaluation result for a clinical data based model used in a method for providing information on demodex mites according to one embodiment of the present invention is illustrated, and the clinical data based model may include a fourth prediction model that uses clinical information data as input.

[0279] The accuracy and AUC for the validation prediction value of the clinical data-based model are 0.7875 and 0.8441 for fold 1, 0.725 and 0.8243 for fold 2, 0.725 and 0.7703 for fold 3, 0.8 and 0.8697 for fold 4, 0.7125 and 0.7641 for fold 5, 0.7875 and 0.93 for fold 6, 0.775 and 0.9048 for fold 7, 0.775 and 0.9048 for fold 8, and 0.7875 for fold 9. and 0.9199, and for fold 10, they are 0.8 and 0.9037.

[0280] That is, since the accuracy and AUC for the verification prediction value of the clinical data based model used in the method for providing information on demodex mites according to one embodiment of the present invention are 0.725 and 0.7641 or higher, it can be meant that it has a predictive power higher than normal.

[0281] Furthermore, the accuracy and AUC for the test prediction value of the clinical data based model are 0.665 and 0.783 for fold 1, 0.735 and 0.7972 for fold 2, 0.745 and 0.7873 for fold 3, 0.705 and 0.7913 for fold 4, 0.67 and 0.7582 for fold 5, 0.7 and 0.7898 for fold 6, 0.635 and 0.781 for fold 7, 0.755 and 0.7908 for fold 8, and 0.675 and 0.7908 for fold 9. It appears to be 0.786, and for fold 10, it appears to be 0.675 and 0.7956.

[0282] That is, since the accuracy and AUC of the test prediction value of the clinical data based model used in the method for providing information on demodex mites according to one embodiment of the present invention are 0.635 and 0.7582 or higher, it can be meant that it has a predictive power higher than normal.

[0283] Ultimately, in the method for providing information on Demodex mites according to one embodiment of the present invention, the image-based model and the clinical data-based model can predict the density and distribution of Demodex mites with an accuracy higher than average even before predicting the final density of Demodex mites.

[0284] Next, referring to FIG. 11b, an evaluation result for a combined model used in a method for providing information on a demodex mite according to one embodiment of the present invention and a final average of models used in the present invention are illustrated, and the combined model may include a third prediction model that inputs vector values ​​derived from the first prediction model, the second prediction model, and the fourth prediction model.

[0285] First, referring to (a) of Fig. 11b, the accuracy and AUC for the validation prediction value of the combined model are 0.7875 and 0.8145 for fold 1, 0.75 and 0.8047 for fold 2, 0.775 and 0.7383 for fold 3, 0.8125 and 0.9015 for fold 4, 0.8 and 0.7764 for fold 5, 0.75 and 0.8318 for fold 6, 0.8125 and 0.8528 for fold 7, and 0.6875 and 0.8574 for fold 8. For fold 9, they are 0.8375 and 0.8593, and for fold 10, they are 0.7375 and 0.8862.

[0286] That is, since the accuracy and AUC for the verification prediction value of the combined model used in the method for providing information on demodex mites according to one embodiment of the present invention are 0.6875 and 0.7383 or higher, it can be meant that it has a predictive power higher than normal.

[0287] Furthermore, the accuracy and AUC for the test prediction value of the combined model are 0.708 and 0.8185 for fold 1.

[0288] For fold 2, it appears to be 0.74 and 0.8016, for fold 3, it appears to be 0.675 and 0.745, for fold 4, it appears to be 0.735 and 0.8162, for fold 5, it appears to be 0.65 and 0.7783, for fold 6, it appears to be 0.73 and 0.7961, for fold 7, it appears to be 0.68 and 0.7973, for fold 8, it appears to be 0.685 and 0.7669, for fold 9, it appears to be 0.615 and 0.8182, and for fold 10, it appears to be 0.74 and 0.8092.

[0289] That is, since the accuracy and AUC for the test prediction values ​​of the combined model used in the method for providing information on demodex mites according to one embodiment of the present invention are 0.615 and 0.745 or higher, it can be meant that it has a predictive power higher than normal.

[0290] Ultimately, the combined model used in the method for providing information on demodex mites according to one embodiment of the present invention is shown to ultimately predict (classify) the density and distribution of demodex mites of an individual with high accuracy.

[0291] As a result, referring to (b) of FIG. 11B, the average value for the performance of 10 folds in the model used in the method for providing information on Demodex according to one embodiment of the present invention is shown, and the accuracy and AUC for the test prediction value of 10 folds in the image based model are shown to be 0.663 and 0.75354, the accuracy and AUC for the test prediction value of 10 folds in the clinical data based model are shown to be 0.696 and 0.78602, and the accuracy and AUC for the test prediction value of 10 folds in the combined model are shown to be 0.6955 and 0.79473.

[0292] That is, it can mean that all models used in the present invention have excellent predictive power above normal. Ultimately, since the density prediction model according to various embodiments of the present invention can predict the density and distribution of Demodex mites with high accuracy, it can be applied to a diagnostic system for Demodex mites in place of a biopsy in actual clinical practice.

[0293] Although the embodiments of the present invention have been described in more detail with reference to the attached drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be implemented without departing from the technical spirit of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain it, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are exemplary in all aspects and not restrictive. The protection scope of the present invention should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

[0294]

[0295] [National Research and Development Project Supporting This Invention]

[0296] [Project ID] 1425174368

[0297] [Assignment Number] S3282903

[0298] [Ministry Name] Ministry of SMEs and Startups

[0299] [Name of Project Management (Specialist) Agency] Small and Medium Business Technology Information Promotion Agency

[0300] [Research Project Name] Small and Medium Enterprise Technology Development Support Project

[0301] [Research Project Name] Development of a Device for Atopic Dermatitis Improvement and Recovery Using Biomicrocurrent

[0302] [Name of the project performing organization] Barun Bio Co., Ltd.

[0303] [Research Period] January 1, 2023 - December 31, 2023

Claims

1. A method for providing information on a folliculitis mite performed by a processor of a folliculitis prediction device, A step of receiving a face image of an object; A step of extracting a first feature for a folliculitis mite by inputting a facial image into a first prediction model trained to extract a first feature for a folliculitis mite by inputting the facial image; A step of generating a saliency map from the first feature; A step of obtaining regions of interest within the face as one or more patches from the above important region map; A step of extracting a second feature for a demodex mite by inputting one or more patches into a second prediction model trained to extract a second feature for a demodex mite by inputting one or more patches, and A method for providing information on demodex mites, comprising: a step of predicting a final density probability of demodex mites for an object by inputting the first feature and the second feature into a third prediction model trained to predict a density probability of demodex mites for an object by inputting the first feature and the second feature; 2. In paragraph 1, After the above receiving step, A method for providing information on a follicle mite, further comprising a step of de-identifying landmarks within a face from the received facial image.

3. In paragraph 4, The above de-identification step is, Extract coordinates for anatomical structures within the face from the received facial image, A method for providing information on a follicle mite, comprising a step of de-identifying landmarks within a face based on the extracted coordinates.

4. In paragraph 1, After the step of extracting the first feature, A method for providing information on demodex mites, further comprising the step of predicting a first density probability of demodex mites for an object by inputting the first feature into a first prediction model that is further trained to predict a density probability of demodex mites for an object by inputting the first feature.

5. In paragraph 4, The above first density probability is, A method for providing information on a follicle mite, which is a global prediction value for the above facial image.

6. In paragraph 1, After the step of extracting the second feature, A method for providing information on demodex mites, further comprising the step of predicting a second density probability of demodex mites for an object by inputting the second feature into a second prediction model that is further trained to predict a density probability of demodex mites for an object by inputting the second feature.

7. In paragraph 6, The second density probability is, A method for providing information on a follicle mite, which is a local prediction value for the above patch image.

8. In paragraph 1, The above receiving step is, Further comprising the step of receiving clinical information of the object, After the above receiving step, The fourth prediction model was trained to extract features of Demodex mites using clinical information as input. A method for providing information on demodex mites, further comprising a step of extracting a third feature for demodex mites by inputting the above clinical information.

9. In paragraph 8, The third prediction model above is, It is further trained to predict the density probability of hair follicle mites for an object using the first feature, the second feature, and the third feature as input. The step of predicting the final density probability is: A method for providing information on demodex mites, comprising: a step of predicting the final density probability of demodex mites for the object by inputting the first feature, the second feature, and the third feature into the third prediction model.

10. In paragraph 8, A method for providing information on demodex mites, further comprising a step of outputting characteristic importance based on the above clinical information.

11. In paragraph 1, The first feature of the step of predicting the final density probability is, A method for providing information on hair follicle mites as pooled vector values.

12. In paragraph 1, The second feature of the step of predicting the final density probability is, A method for providing information on a hair follicle, which is an attention-weighted representation Z vector.

13. In paragraph 12, The step of extracting the above second feature is: A method for providing information on a hair follicle mite, further comprising the step of applying an attention score to the extracted second feature.

14. In paragraph 13, The step of applying the above attention score is: A step in which one or more of the above patches are input into one or more FC layers as a batch; A step of outputting an attention score based on the logit output from the above FC layer, and A method for providing information on a hair follicle mite, comprising a step of multiplying the output attention score by the second feature.

15. In paragraph 1, The above first prediction model is, A method for providing information on a demodex mite, wherein the model is trained to predict the probability for class 0 and class 1 by inputting the above facial image.

16. In paragraph 1, The second prediction model is, A method for providing information on Demodex mites, wherein the model is trained to predict the probability for class 0 and class 1 using the above patch as input.

17. In paragraph 1, The third prediction model above is, A method for providing information on Demodex mites, wherein the model is trained to predict the probability for class 0 and class 1 using the first feature and the second feature as inputs.

18. In any one of paragraphs 15 to 17, The above category 0 is, 1 cm 2 The number of hair follicle mites in the sugar is 0 or more and less than 5, The above category 1 is, 1 cm 2 A method for providing information on Demodex mites, wherein the number of Demodex mites per party is 5 or more.

19. A communication unit configured to receive an image of the face of an object, and including a processor connected to communicate with the communication unit; The above processor, A first prediction model trained to extract a first feature for a demodex mite by inputting a facial image as input is input to the first prediction model, and a first feature for a demodex mite is extracted by inputting the facial image as input, Generate a saliency map from the first feature, Obtaining regions of interest within the face as one or more patches from the above important region map, Extracting a second feature for a demodex mite by inputting one or more patches into a second prediction model trained to extract a second feature for a demodex mite by inputting one or more patches, A device for providing information on a demodex, configured to predict a final density probability of a demodex for an object by inputting the first feature and the second feature into a third prediction model trained to predict a density probability of a demodex for an object by inputting the first feature and the second feature.

20. In paragraph 19, The above processor, A device for providing information on a follicle mite, further configured to de-identify landmarks within a face from the received facial image.

21. In paragraph 20, The above processor, Extract coordinates for anatomical structures within the face from the received facial image, A device for providing information on a follicle mite, configured to de-identify landmarks within a face based on the extracted coordinates.

22. In paragraph 19, The above processor, After extracting the above first feature, A device for providing information on a demodex, further configured to predict a first density probability of a demodex mite for an object by inputting the first feature into a first prediction model further trained to predict a density probability of a demodex mite for an object by inputting the first feature.

23. In paragraph 18, The above first density probability is, A device for providing information on a follicle mite, which is a global prediction for the above facial image.

24. In paragraph 19, The above processor, After the step of extracting the second feature, A device for providing information on a demodex, further configured to predict a second density probability of a demodex for an object by inputting the second feature into a second prediction model further trained to predict a density probability of a demodex for an object by inputting the second feature.

25. In paragraph 24, The second density probability is, A device for providing information on hair follicle mites, which is a global prediction for the above patch image.

26. In paragraph 19, The above communication department, Further configured to receive clinical information of the subject, The above processor, A device for providing information on Demodex mites, further configured to extract a third feature for Demodex mites by inputting clinical information into a fourth prediction model trained to extract features of Demodex mites by inputting clinical information.

27. In paragraph 26, The third prediction model above is, It is further trained to predict the density probability of hair follicle mites for an object using the first feature, the second feature, and the third feature as input. The above processor, In the above third prediction model, A device for providing information on a demodex mite, configured to predict a final density probability of a demodex mite for the object using the first feature, the second feature, and the third feature as inputs.

28. In paragraph 26, The above processor, A device for providing information on demodex mites, further configured to output characteristic importance based on the above clinical information.

29. In paragraph 19, The first feature input to the third prediction model is, A device for providing information on hair follicle mites, which is a pooled vector value.

30. In paragraph 19, The second feature input to the third prediction model is, A device for providing information on a hair follicle, which is an attention-weighted representation Z vector.

31. In paragraph 30, The above processor, A device for providing information on a hair follicle, further configured to apply an attention score to the extracted second feature.

32. In paragraph 31, The above processor, The above one or more patches are input to one or more FC layers as a batch, An attention score is output based on the logit output from the above FC layer, A device for providing information on a hair follicle mite, further configured to multiply the output attention score by the second feature.

33. In paragraph 19, The above first prediction model is, A device for providing information on demodex mites, which is a model trained to predict the probability of class 0 and class 1 by inputting the above facial image.

34. In paragraph 19, The second prediction model is, A device for providing information on Demodex mites, which is a model trained to predict probabilities for class 0 and class 1 using the above patch as input.

35. In paragraph 19, The third prediction model above is, A device for providing information on a demodex, which is a model trained to predict the probability for class 0 and class 1 using the first feature and the second feature as inputs.

36. In any one of paragraphs 33 to 35, The above category 0 is, 1 cm 2 The number of hair follicle mites in the sugar is 0 or more and less than 5, The above category 1 is, 1 cm 2 A device for providing information on demodex mites, in which the number of demodex mites per party is 5 or more.

Images