Method and apparatus for determining skin disease diagnosis information

Abstract

According to an embodiment of the present invention, the method for determining skin disease diagnosis information by an electronic device comprises the steps of: acquiring an image of a diagnosis target; obtaining at least one atypical mask from a first artificial intelligence model outputting the atypical mask for the at least one lesion area when the image of the diagnosis target is inputted on the basis of a prompt designating the atypical mask for the at least one lesion area included in the image of the diagnosis target; and tokenizing attribute data of the at least one atypical mask obtained from the first artificial intelligence model, and when the tokenized attribute data and the image of the diagnosis target are inputted, acquiring diagnosis information including the skin disease severity for each lesion area of the image of the diagnosis target from a second artificial intelligence model outputting same.

Description

Method and device for determining skin disease diagnostic information

[0001] The present disclosure relates to a method and device for determining skin disease diagnosis information. More specifically, the present disclosure relates to a method and device for providing disease diagnosis information for a plurality of lesion areas included in a diagnosis target image using an artificial intelligence model.

[0002] Atopic dermatitis is a chronic skin condition that affects 220 million people worldwide. It causes symptoms such as skin inflammation, itching, and dryness, and can cause significant discomfort in daily life.

[0003] In particular, skin conditions like atopic dermatitis are not self-limiting; they often develop into respiratory allergies like asthma, allergies, and rhinitis. Therefore, accurate diagnosis, identification of severity, and appropriate management and treatment are essential.

[0004] Traditional skin disease diagnosis relied on individual physicians' assessments, or attempts at diagnosing skin diseases using conventional artificial intelligence technology were attempted. However, conventional skin disease diagnosis and severity assessment methods had limitations in diagnostic accuracy for multiple lesion regions within the target image, and accurate identification of lesion regions was difficult.

[0005] Therefore, there is a need for the development of a technology to extract atypical masks for multiple lesion areas within a diagnosis target image and to accurately diagnose multiple lesion areas within a diagnosis target image using the extracted atypical masks.

[0006] According to one embodiment, a method for determining skin disease diagnosis information and an electronic device performing the same may be provided.

[0007] According to one embodiment, a method for training an artificial intelligence model used to determine skin disease diagnosis information and an electronic device performing the same can be provided.

[0008] As a technical means for achieving the above-described technical problem, according to one embodiment, a method for an electronic device to determine skin disease diagnosis information may be provided, comprising: acquiring a diagnosis target image; acquiring a prompt for designating an atypical mask for at least one lesion area included in the diagnosis target image; based on the prompt, when the diagnosis target image is input, inputting the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for the at least one lesion area, thereby acquiring at least one atypical mask from the first artificial intelligence model; tokenizing attribute data of the at least one atypical mask acquired from the first artificial intelligence model; and when the tokenized attribute data and the diagnosis target image are input, acquiring the diagnosis information from a second artificial intelligence model that outputs diagnosis information regarding at least one of a type of skin disease appearing in the diagnosis target image or a skin disease severity value for each lesion area of ​​the diagnosis target image.

[0009] As a technical means for achieving the above-described technical problem, according to another embodiment, an electronic device for determining skin disease diagnosis information is provided, comprising: a memory for storing one or more instructions; at least one processor for executing the one or more instructions; wherein the at least one processor obtains a diagnosis target image by executing the one or more instructions, obtains a prompt for designating an atypical mask for at least one lesion area included in the diagnosis target image, and, based on the prompt, inputs the diagnosis target image into a first artificial intelligence model for outputting an atypical mask for the at least one lesion area when the diagnosis target image is input, thereby obtaining at least one atypical mask from the first artificial intelligence model, tokenizing attribute data of the at least one atypical mask obtained from the first artificial intelligence model, and, when the tokenized attribute data and the diagnosis target image are input, outputting diagnostic information regarding at least one of a type of skin disease appearing in the diagnosis target image or a skin disease severity value for each lesion area of ​​the diagnosis target image, the electronic device may be provided.

[0010] As a technical means for achieving the above-described technical problem, according to another embodiment, a method for an electronic device to determine skin disease diagnosis information is provided, comprising: acquiring a diagnosis target image; acquiring a prompt for designating an atypical mask for at least one lesion area included in the diagnosis target image; based on the prompt, when the diagnosis target image is input, inputting the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for the at least one lesion area, thereby acquiring at least one atypical mask from the first artificial intelligence model; tokenizing attribute data of the at least one atypical mask acquired from the first artificial intelligence model; and when the tokenized attribute data and the diagnosis target image are input, acquiring the diagnosis information from a second artificial intelligence model that outputs diagnosis information regarding at least one of a type of skin disease appearing in the diagnosis target image or a skin disease severity value for each lesion area of ​​the diagnosis target image. A computer-readable recording medium storing a program for causing the method to be performed may be provided.

[0011] In one embodiment, the type and numerical value of a skin disease in a diagnostic target image can be accurately determined.

[0012] In one embodiment, the severity scores for each of multiple lesion areas within a diagnostic target image can be accurately identified.

[0013] FIG. 1 is a schematic diagram illustrating a process by which an electronic device determines skin disease diagnosis information according to one embodiment.

[0014] FIG. 2 is a diagram illustrating a process in which an electronic device extracts an amorphous mask from a diagnosis target image according to one embodiment.

[0015] FIG. 3 is a flowchart of a method for an electronic device to determine skin disease diagnosis information according to one embodiment.

[0016] FIG. 4 is a flowchart of a method for an electronic device to train an artificial intelligence model according to an embodiment.

[0017] FIG. 5 is a diagram showing the structure of an artificial intelligence model used by an electronic device for extracting an atypical mask according to one embodiment.

[0018] Figure 6 is a block diagram of an electronic device according to one embodiment.

[0019] Figure 7 is a block diagram of an electronic device according to another embodiment.

[0020] Figure 8 is a block diagram of a server according to one embodiment.

[0021] According to one embodiment, a method for determining skin disease diagnosis information by an electronic device may be provided, comprising: acquiring a diagnosis target image; acquiring a prompt for designating an atypical mask for at least one lesion area included in the diagnosis target image; based on the prompt, when the diagnosis target image is input, inputting the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for the at least one lesion area, thereby acquiring at least one atypical mask from the first artificial intelligence model; tokenizing attribute data of the at least one atypical mask acquired from the first artificial intelligence model; and when the tokenized attribute data and the diagnosis target image are input, acquiring the diagnosis information from a second artificial intelligence model that outputs diagnosis information regarding at least one of a type of skin disease appearing in the diagnosis target image or a skin disease severity value for each lesion area of ​​the diagnosis target image.

[0022] According to one embodiment, an electronic device for determining skin disease diagnosis information may be provided, comprising: a memory for storing one or more instructions; at least one processor for executing the one or more instructions; wherein the at least one processor obtains a diagnosis target image by executing the one or more instructions, obtains a prompt for designating an atypical mask for at least one lesion area included in the diagnosis target image, and, based on the prompt, inputs the diagnosis target image into a first artificial intelligence model for outputting an atypical mask for the at least one lesion area when the diagnosis target image is input, thereby obtaining at least one atypical mask from the first artificial intelligence model, tokenizes attribute data of the at least one atypical mask obtained from the first artificial intelligence model, and, when the tokenized attribute data and the diagnosis target image are input, outputs diagnostic information regarding at least one of a type of skin disease appearing in the diagnosis target image or a skin disease severity value for each lesion area of ​​the diagnosis target image, thereby obtaining the diagnostic information from a second artificial intelligence model.

[0023] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0024] The terms used in this disclosure have been selected from widely used, current terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0025] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used throughout the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0026] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description of the present disclosure, and similar parts are designated with similar reference numerals throughout the specification.

[0027]

[0028] FIG. 1 is a schematic diagram illustrating a process by which an electronic device determines skin disease diagnosis information according to one embodiment.

[0029] Referring to FIG. 1, a process is illustrated in which an electronic device (1000) acquires a diagnosis target image (110) and outputs diagnosis information (120) including information on the type of skin disease (122) and severity (124) for each lesion area appearing in the diagnosis target image. According to one embodiment, the electronic device (1000) may perform a method of providing skin disease diagnosis information by connecting with a server (2000) via a network (3000). According to one embodiment, the electronic device (1000) may extract at least one atypical mask corresponding to at least one lesion area appearing in the diagnosis target image, and determine the type of skin disease and the severity value for at least one lesion area in the diagnosis target image based on attribute data of the extracted atypical mask.

[0030] According to one embodiment, the electronic device (1000) may accurately identify atopic dermatitis among types of skin diseases and the severity of each of a plurality of atopic dermatitis lesion areas appearing in a diagnosis target image. The electronic device (1000) according to the present disclosure may diagnose atopic dermatitis accurately at an early stage by using at least one of a deep learning or machine learning model trained to identify a severity score for each of a plurality of lesion areas based on image resolution conversion, atypical mask extraction, and atypical mask attribute data, thereby preventing the development of respiratory allergic diseases such as asthma and allergic rhinitis in the future. According to another embodiment, the electronic device (1000) may determine a type of skin disease and a skin disease severity score for all lesion areas in a diagnosis target image.

[0031] Although not illustrated in FIG. 1, the electronic device (1000) according to the present disclosure may determine a severity score for all lesions in an image to be diagnosed, may determine a severity score for each lesion region in the image to be diagnosed, or may determine a severity score for all lesions and each lesion region in the image to be diagnosed together. In addition, the electronic device (1000) may determine a severity score (or index) of a skin disease for each lesion region in the image to be diagnosed as a first level (mild, MILD), a second level (moderate, MODERATE), a third level (severe, SEVERE), and a fourth level (very severe, MOST SEVERE), but is not limited thereto, and may determine a severity level for each lesion region in the image to be diagnosed as at least one severity score.

[0032] In addition, according to one embodiment, the electronic device (1000) may identify lesion areas in which a severity value is not identified for each lesion area in the diagnosis target image as a result of identifying a severity value for each lesion area in the diagnosis target image, and may first output diagnostic information for lesion areas in which a severity value is identified for each lesion area in the diagnosis target image, and then output content that inquires whether to re-perform a severity identification process for lesion areas in which a severity value is not identified.

[0033] In addition, according to one embodiment, the electronic device (1000) can identify the severity level of each lesion region in the diagnosis target image, and also identify the progression stage (e.g., early, middle, late, terminal) according to the corresponding skin disease for each lesion region. According to one embodiment, the progression stage according to the corresponding skin disease for each lesion region can be determined based on the type of skin disease identified for each lesion region, the degree of redness according to the type of skin disease, the size of the lesion region, the location of the lesion region, and representative color information. According to one embodiment, the electronic device (1000) can also determine the representative skin disease and skin disease severity level appearing throughout the diagnosis target image based on the severity level, degree of redness, size of the lesion region, location of the lesion region, and representative color information of the remaining lesion regions in which the severity level is identified, excluding lesion regions in which the severity level is not identified.

[0034] Although the electronic device (1000) and the server (2000) are illustrated separately in the following specification, it is to be understood that the entire or at least a part of the operations of the method for determining skin disease diagnosis information may be performed by one of the electronic device (1000) or the server (2000), or by linkage of one or more devices. According to one embodiment, the skin disease diagnosis system (10) may include the electronic device (1000) and the server (2000). According to another example, the skin disease diagnosis system (10) may include the electronic device (1000), the server (2000), and other electronic devices (not shown), and may perform a method for determining skin disease diagnosis information for a diagnosis target image by linkage of at least one or more devices.

[0035] According to one embodiment, the electronic device (1000) may be connected to a server (2000) via a network (3000). The electronic device (1000) may also be connected to another electronic device (not shown) via the network (3000). For example, the electronic device (1000) may transmit a method for determining skin disease diagnosis information, or diagnostic information obtained as a result of performing the method for determining skin disease diagnosis information, to another electronic device (e.g., a terminal used by a patient or a doctor) connected to the electronic device (1000).

[0036]

[0037] FIG. 2 is a diagram illustrating a process in which an electronic device extracts an amorphous mask from a diagnosis target image according to one embodiment.

[0038] Referring to Figures (210) and (220) of FIG. 2, the results of extracting a plurality of lesion regions from a diagnosis target image using a learned first artificial intelligence model by an electronic device (1000) are illustrated. For example, the electronic device (1000) may obtain a diagnosis target image including at least one lesion region, and separate at least one lesion region (LESION) and a background (BACKGROUND) within the obtained diagnosis target image. The electronic device (1000) may also separate an atypical mask for at least one lesion region identified within the diagnosis target image as shown in Figure (220).

[0039]

[0040] FIG. 3 is a flowchart of a method for an electronic device to determine skin disease diagnosis information according to one embodiment.

[0041] In S310, the electronic device (1000) can acquire a diagnostic target image. For example, the diagnostic target image acquired by the electronic device (1000) may include at least one lesion area. The electronic device (1000) may acquire the diagnostic target image from another electronic device or server connected to the electronic device, or may acquire the diagnostic target image using a camera device included in the electronic device.

[0042] Although not shown in FIG. 3, according to another embodiment, at S310, the electronic device (1000) may acquire a diagnosis target image and then input the diagnosis target image into a third artificial intelligence model that converts the diagnosis target image into a high-resolution image, thereby acquiring a resolution-converted diagnosis target image from the third artificial intelligence model. At S330, the electronic device (1000) may also acquire at least one atypical mask from the first artificial intelligence model by inputting a resolution-converted diagnosis target image, instead of the diagnosis target image, into the first artificial intelligence model.

[0043] In S320, the electronic device (1000) may obtain a prompt for specifying an atypical mask for at least one lesion area included in the diagnosis target image. For example, the electronic device (1000) may obtain, as a prompt, at least one of tokenized text for specifying an atypical mask or a position value of at least one point on the diagnosis target image based on a user input to the electronic device.

[0044] Although not illustrated in FIG. 3, after step S320, when both a prompt in the form of a position value indicating at least one point (e.g., a point within a specific lesion area) on a diagnosis target image for specifying an atypical mask for at least one lesion area and a prompt in the form of a text for specifying a predetermined atypical mask are obtained among the prompts, the electronic device (1000) may determine the reliability of a token sequence obtained by applying a tokenizer to each prompt, and may obtain at least one atypical mask from the first artificial intelligence model based on one type of prompt selected based on the determined reliability value. According to one embodiment, the reliability is an index indicating the extraction performance of an atypical mask obtained based on a prompt in the form of a position value or a text type, and may indicate an evaluation result of an output result of the first artificial intelligence model learned using the JACCARD index and the DICE index.

[0045] In addition, although not shown in FIG. 3, the electronic device (1000) may obtain a text for specifying an atypical mask, tokenize the obtained text using a tokenizer to obtain a token sequence (=tokenized text), and input a prompt in the form of a token sequence into the first artificial intelligence model, thereby obtaining at least one prompt-based atypical mask from the first artificial intelligence model. The tokenizer used by the electronic device (1000) may be a Bert tokenizer, but is not limited thereto, and may include various forms of tokenizers for converting the input text into a token sequence that can be processed by the artificial intelligence model.

[0046] In S330, the electronic device (1000) inputs the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for at least one lesion area based on the prompt, thereby obtaining at least one atypical mask from the first artificial intelligence model. According to one embodiment, the first artificial intelligence model may be a fine-tuned neural network model based on a pre-learned GENERALIZED SAM (SEGMENT ANYTHING MODEL).

[0047] According to one embodiment, at least one atypical mask acquired by the electronic device (1000) from the first artificial intelligence model may correspond to at least one lesion segment obtainable from the diagnosis target image. For example, the atypical mask acquired by the electronic device (1000) using the first artificial intelligence model may include attribute data regarding at least one of the area, position, dominant color of the atypical mask, and the type of body part (BODY PART) to which the corresponding lesion area or the entire diagnosis target image including the corresponding lesion areas belongs.

[0048] In S340, the electronic device (1000) may tokenize attribute data of at least one atypical mask obtained from the first artificial intelligence model. For example, the electronic device (1000) may identify attribute data of at least one atypical mask (e.g., at least one lesion segment) obtainable from the diagnosis target image from the first artificial intelligence model, and tokenize the identified attribute data using a tokenizer. For example, the electronic device (1000) may tokenize attribute data regarding at least one of an area, a location, a donor color, and a redness degree of the at least one atypical mask obtained from the first artificial intelligence model.

[0049] According to one embodiment, the electronic device (1000) may tokenize attribute data of an atypical mask output from the first artificial intelligence model using a separately trained tokenizer different from the tokenizer used to tokenize a prompt to be applied to the first artificial intelligence model. The electronic device (1000) according to the present disclosure may use different tokenizers for a first artificial intelligence model that extracts an atypical mask from an image to be diagnosed, and a second artificial intelligence model that identifies a type of skin disease and a severity level for the entire diagnostic image or each lesion area in the diagnostic image based on the mask attribute data, thereby allowing an optimized token sequence to be applied for each artificial intelligence model.

[0050] In S350, when tokenized attribute data and the diagnosis target image are input, the electronic device (1000) may obtain the diagnosis information from a second artificial intelligence model that outputs diagnosis information regarding at least one of the type of skin disease appearing in the diagnosis target image or the skin disease severity value for each lesion area of ​​the diagnosis target image. According to one embodiment, the diagnosis information may include at least one of the type of skin disease appearing in the diagnosis target image, the skin disease severity value for each lesion area in the diagnosis target image, or the skin disease severity value appearing in the entire diagnosis target image.

[0051] In addition, although not shown in FIG. 3, according to another embodiment, the electronic device (1000) may preprocess the diagnosis target image to normalize the size of the diagnosis target image after acquiring the diagnosis target image in S310. According to one embodiment, the operation of preprocessing the diagnosis target image by the electronic device (1000) after step S310 may correspond to the operation of preprocessing a plurality of training images acquired for training the first artificial intelligence model and the second artificial intelligence model described below. The electronic device (1000) may acquire at least one atypical mask by inputting the preprocessed diagnosis target image into the first artificial intelligence model, and may post-process at least one atypical mask acquired from the first artificial intelligence model by converting at least one of the area, the location, the representative color, or the degree of redness among the attribute data of the acquired at least one atypical mask.

[0052] In one embodiment, the step of post-processing at least one amorphous mask by the electronic device (1000) may include the step of applying a sharpening and noise removal technique to the at least one amorphous mask. In one embodiment, the electronic device (1000) may also perform a noise removal process on the diagnosis target image by using an image filter such as a Gaussian filter, a median filter, or a bilateral filter.

[0053] In addition, the electronic device (1000) tokenizes attribute data (e.g., mask area, location, representative color, degree of redness, etc.) obtained from at least one post-processed atypical mask, and inputs the tokenized, post-processed attribute data obtained from at least one atypical mask into a second artificial intelligence model, thereby obtaining more accurate skin disease diagnosis information. The electronic device (1000) according to the present disclosure can accurately determine diagnostic information for a skin disease on a diagnosis target image through a process of preprocessing an acquired diagnosis target image and a process of post-processing at least one atypical mask obtained by inputting the acquired diagnosis target image into a first artificial intelligence model.

[0054] In addition, according to one embodiment, the electronic device (1000) may obtain a resolution-converted diagnosis target image by inputting the diagnosis target image into a third artificial intelligence model (e.g., SRGAN), and may determine more accurate skin disease diagnosis information by performing the above-described preprocessing process on the resolution-converted diagnosis target image converted to high resolution. According to another embodiment, the electronic device (1000) may determine the type of skin disease and the skin disease severity level for lesion areas in the diagnosis target image or the entire diagnosis target image by using not only a neural network-based artificial intelligence model but also at least one network for performing machine learning operations.

[0055] According to one embodiment, a network for performing machine learning operations used by the electronic device (1000) may include an algorithm operation model such as a support vector machine or a random forest, and may use a gradient boosting-based model (e.g., XGB BOOST, LIGHT GBM, etc.) as an ensemble learning method that combines multiple weak learners to create a strong learner to determine a severity value for each lesion area based on an atypical mask attribute acquired from a diagnosis target image.

[0056]

[0057] FIG. 4 is a flowchart of a method for an electronic device to train an artificial intelligence model according to an embodiment.

[0058] In S410, the electronic device (1000) may acquire a plurality of training images. For example, the electronic device (1000) may acquire a plurality of training images labeled into two classes: those with actual skin diseases and those without actual skin diseases. Furthermore, although not illustrated in FIG. 4 , the electronic device (1000) may preprocess the images to normalize the sizes of the labeled training images. For example, the electronic device (1000) may preprocess the acquired training images so that pixel values ​​within the training images fall within a range of 0 to 1 by normalizing the acquired training images.

[0059] In S420, the electronic device (1000) may augment a plurality of training images. According to one embodiment, the electronic device (1000) may augment the training images so that each of the augmented training images generated by augmenting the training images, upon acquiring the diagnostic information for each of the plurality of training images, includes lesion areas representing different severity levels.

[0060] For example, in the case of lesions such as atopic dermatitis, the lesions are characterized by swelling and growing larger as they progress in the early stages. As the disease progresses, lesions related to atopic dermatitis are characterized by festering or appearing as wounds. Even lesions related to the same skin disease appearing on the same body part of the same patient can exhibit different severities. Therefore, a single diagnostic target image often includes lesion areas with different severity levels.

[0061] Accordingly, the electronic device (1000) according to the present disclosure may obtain diagnostic information by applying the first artificial intelligence model and the second artificial intelligence model to each of a plurality of augmented learning images, and may augment the learning images so that each of the augmented learning images includes all or one or more lesion areas representing four different severity levels.

[0062] For example, the skin disease severity score determined by the electronic device according to the present disclosure may include four severity ranges according to the severity level of the skin disease, and the augmented learning image augmented by the electronic device (1000) may include one or more lesion areas indicating a first level of severity score (e.g., MILD), one or more lesion areas indicating a second level of severity score (e.g., MODERATE), one or more lesion areas indicating a third level of severity score (e.g., SEVERE), and one or more lesion areas indicating a fourth level of severity score (e.g., MOST SEVERE). In S430, the electronic device (1000) may train the first artificial intelligence model and the second artificial intelligence model based on the augmented learning images.

[0063] According to one embodiment, the step of augmenting the learning images by the electronic device (1000) may include a step of randomly horizontally or vertically flipping (FLIPing) some of the learning images so that each of the augmented learning images includes all lesion regions (e.g., lesion regions of the first level, the second level, the third level, and the fourth level) representing different severity levels, and a step of randomly rotating some of the learning images within a predetermined angular range (e.g., -15 degrees to +15 degrees) so that each of the augmented learning images includes all lesion regions representing different severity levels.

[0064] Also, although not illustrated in FIG. 4, the process by which the electronic device (1000) augments training images will be described in more detail. According to one embodiment, the electronic device (1000) can acquire multiple training images and identify severity scores for each lesion region in the acquired training images. The severity scores for each lesion region in the training images may be pre-labeled.

[0065] When the electronic device (1000) identifies the severity level of each lesion region of the learning images and the number of lesions identified with a specific severity level (e.g., a second level severity level) exceeds 50% of the total number of lesion regions in the learning images, the electronic device may augment the learning images until the number of lesions identified with the specific severity level becomes less than 50% of the total number of lesion regions in the learning images.

[0066] Although not illustrated in FIG. 4, in S430, the process of training the first artificial intelligence model and the second artificial intelligence model by the electronic device (1000) based on the augmented training images will be described in detail below. For example, the electronic device (1000) may obtain learning atypical masks from the first artificial intelligence model by inputting the augmented training images into the first artificial intelligence model. The electronic device (1000) may label the learning atypical masks with skin disease types and severity label values ​​for each skin disease type determined based on attribute data of the obtained learning atypical masks. The electronic device (1000) may train the second artificial intelligence model based on the labeled learning atypical masks. According to an embodiment, the process of training the first artificial intelligence model and the second artificial intelligence model by the electronic device (1000) may correspond to an operation of modifying and updating layers including at least one node in the artificial intelligence model and weights regarding connection strengths of the layers.

[0067] Hereinafter, an operation of an electronic device (1000) according to the present disclosure for labeling learning atypical masks will be described in detail. According to one embodiment, the electronic device (1000) may label learning atypical masks based on at least one of the size of an area of ​​learning atypical masks obtained from each of the augmented learning images, the number of atypical masks obtained from each of the augmented learning images, and the channel color values ​​of the atypical masks obtained from each of the augmented learning images.

[0068] According to one embodiment, the electronic device (1000) may label a severity score as a first level (e.g., MILD) if a lesion area corresponding to the learning atypical mask shows only a mild rash, may label it as a second level (e.g., MODERATE) if the lesion area corresponding to the learning atypical mask shows a rash accompanied by severe inflammation on less than 10% of the total body surface area, may label it as a third level (e.g., SEVERE) if a lesion area corresponding to the learning atypical mask shows a rash accompanied by severe inflammation on more than 10% or less than 30% of the body surface area, may label the severity score of the learning atypical mask as a third level (e.g., SEVERE), and may label the learning atypical mask as a fourth level (e.g., MOST SEVERE) if the lesion area corresponding to the learning atypical mask shows a rash accompanied by severe inflammation on more than 30% of the body surface area.

[0069] According to another embodiment, the electronic device (1000) labels the learning image as a third level (e.g., SEVERE) if the size of the total lesion area corresponding to the learning atypical mask is 50% or more of the total body surface area in a unit learning image including a plurality of learning atypical masks rather than a learning atypical mask unit, and labels the learning image as a second level (e.g., MODERATE) if the size of the total lesion area in the learning image is greater than 10% of the total area, the number of lesion areas in the learning image is 3 or more, and the ratio of the values ​​of the red (REDNESS) channel greater than 200 is greater than 0.5, and labels other learning images as a first level (e.g., MILD).

[0070] In addition, although not shown in FIG. 4, an operation of training a third artificial intelligence model used by the electronic device (1000) to convert the resolution of an image to be diagnosed will be described in detail. According to one embodiment, before acquiring the image to be diagnosed, the electronic device (1000) may acquire ultra-high-resolution learning-use upscaled images and low-resolution learning-use upscaled images as a resolution-converted training data set. The electronic device (1000) may acquire diagnostic information about the training data set by inputting the acquired training data set into the first artificial intelligence model and the second artificial intelligence model, and may augment the training data set so that each of the augmented training data sets generated by augmenting the training data set includes lesion areas representing different severity levels. The electronic device (1000) may train the third artificial intelligence model based on the augmented training data set.

[0071] Accordingly, the electronic device (1000) according to the present disclosure not only augments the training image data for training the first artificial intelligence model and the second artificial intelligence model in the above manner to accurately identify the diagnosis target image including lesion regions at different levels, but also augments the training data set for training the third artificial intelligence model for resolution conversion of the diagnosis target image so that lesion regions indicating different severity levels are included, thereby more accurately determining the severity level for each lesion in the diagnosis target image including lesion regions at different levels.

[0072] In one embodiment, the first artificial intelligence model is a Segment Anything Model (SAM) model, the second artificial intelligence model is a network model that performs a neural network or machine learning operation that provides the diagnostic information based on the attribute data of the atypical mask, and the third artificial intelligence model may be a Super Resolution GAN (SRGAN) model, but is not limited thereto. In another embodiment, the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model disclosed in the present specification may include at least one network that performs a machine learning operation or may include at least one network that performs a neural network operation. For example, the artificial intelligence models disclosed in the present specification may be, but are not limited to, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Network.

[0073] In addition, according to one embodiment, the electronic device (1000) can use transfer learning using a pre-learned model to learn a second artificial intelligence model for severity score identification, and can effectively retrain the second artificial intelligence model based on a pre-learned GENERALIZED SAM model for new training images or augmented training images. In addition, according to one embodiment, the electronic device (1000) can also determine the type and severity of a skin disease for the entire diagnosis target image or for each lesion area using an ensemble model that combines prediction values ​​of a plurality of neural network-based network models.

[0074] In addition, although not shown in FIG. 4, after the electronic device (1000) trains the first artificial intelligence model and the second artificial intelligence model in S430, the electronic device may evaluate the extraction performance of the atypical mask (e.g., lesion segmentation) of the first artificial intelligence model using the JACCARD index and the DICE index, and retrain the artificial intelligence model based on the evaluation results.

[0075]

[0076] FIG. 5 is a diagram showing the structure of an artificial intelligence model used by an electronic device for extracting an atypical mask according to one embodiment.

[0077] Referring to Figure (510), the structure of the GENERALIZED SAM model used to train the second artificial intelligence model used by the electronic device (1000) is illustrated. According to one embodiment, the TRANSFORMER of the GENERALIZED SAM model used by the electronic device can use 256 as the Embedding Dimension, and the Transformer MLP Block can use an Internal Dimension of 2048. In one example, this may be applied only to a prompt token. In addition, the Cross-attention Layer of the GENERALIZED SAM model can use 64x64 image embedding, the channels of Query / Key / Value can be doubled to 128 for computational efficiency, and all Attention Layers can use 8 Heads. In addition, the Transposed Convolution applied to the Output Image Embedding is 2x2, stride 2, and the Output Channel can use 64 and 32 (GELU activation, Layer Normalization).

[0078]

[0079] Figure 6 is a block diagram of an electronic device according to one embodiment.

[0080] Figure 7 is a block diagram of an electronic device according to another embodiment.

[0081] As illustrated in FIG. 6, an electronic device (1000) according to an embodiment may include a processor (1300), a network interface (1500), and a memory (1700). However, not all of the illustrated components are essential components. The electronic device (1000) may be implemented with more components than the illustrated components, or may be implemented with fewer components.

[0082] For example, as illustrated in FIG. 7, the electronic device (1000) may further include a user input interface (1100), an output unit (1200), a sensing unit (1400), a network interface (1500), an A / V input unit (1600), and a memory (1700) in addition to a processor (1300), a network interface (1500), and a memory (1700).

[0083] The user input interface (1100) refers to a means for a user to input data for controlling an electronic device (1000). For example, the user input interface (1100) may include, but is not limited to, a key pad, a dome switch, a touch pad (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, a jog switch, etc.

[0084] The user input interface (1100) can acquire user input for obtaining text for specifying an atypical mask on a diagnosis target image or for specifying an atypical mask position value. The output unit (1200) can output an audio signal, a video signal, or a vibration signal, and the output unit (1200) can include a display unit (1210), an audio output unit (1220), and a vibration motor (1230).

[0085] The display unit (1210) includes a screen for displaying and outputting information processed in the electronic device (1000). In addition, the screen may output diagnostic information regarding at least one of the following: a type of skin disease, a severity score for the entire diagnostic image, and a severity score for each lesion area within the diagnostic image, as skin disease diagnostic information determined by the electronic device (1000).

[0086] The audio output unit (1220) outputs audio data received from the network interface (1500) or stored in the memory (1700). In addition, the audio output unit (1220) outputs audio signals related to functions performed in the electronic device (1000) (e.g., call signal reception sound, message reception sound, notification sound).

[0087] The processor (1300) typically controls the overall operation of the electronic device (1000). For example, the processor (1300) can control the user input interface (1100), the output unit (1200), the sensing unit (1400), the network interface (1500), the A / V input unit (1600), etc., by executing programs stored in the memory (1700). In addition, the processor (1300) can perform the functions of the electronic device (1000) described in FIGS. 1 to 5 by executing programs stored in the memory (1700).

[0088] According to one embodiment, at least one processor (1300) may obtain a diagnosis target image by executing the one or more instructions, obtain a prompt for designating an atypical mask for at least one lesion area included in the diagnosis target image, and, based on the prompt, input the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for at least one lesion area when the diagnosis target image is input, thereby obtaining at least one atypical mask from the first artificial intelligence model, tokenizing attribute data of the at least one atypical mask obtained from the first artificial intelligence model, and, when the tokenized attribute data and the diagnosis target image are input, outputting diagnostic information regarding at least one of a type of skin disease appearing in the diagnosis target image or a skin disease severity value for each lesion area of ​​the diagnosis target image, thereby obtaining the diagnostic information from a second artificial intelligence model.

[0089] According to one embodiment, at least one processor (1300) may obtain a resolution-converted diagnosis target image from the third artificial intelligence model by inputting the diagnosis target image into a third artificial intelligence model that converts the diagnosis target image into a high-resolution image, and may obtain the at least one amorphous mask from the first artificial intelligence model by inputting the resolution-converted diagnosis target image into the first artificial intelligence model.

[0090] Additionally, according to one embodiment, the processor (1300) may tokenize attribute data regarding at least one of the area, location, representative color, or degree of redness of at least one atypical mask obtained from the first artificial intelligence model.

[0091] According to one embodiment, the processor (1300) may, after the step of acquiring the diagnosis target image, preprocess the diagnosis target image to normalize the size of the acquired diagnosis target image, input the preprocessed diagnosis target image into the first artificial intelligence model, thereby acquiring the at least one atypical mask, and post-process the at least one atypical mask acquired from the first artificial intelligence model by converting at least one of an area, a location, a representative color, or a degree of redness among attribute data of the acquired at least one atypical mask, and tokenize attribute data acquired from the post-processed at least one atypical mask.

[0092] In addition, according to one embodiment, the processor (1300) may acquire a plurality of learning images, and, based on the diagnostic information acquisition result for each of the plurality of learning images, augment the learning images so that each of the augmented learning images includes lesion areas representing different severity levels, and train the first artificial intelligence model and the second artificial intelligence model based on the augmented learning images.

[0093] In addition, according to one embodiment, the processor (1300) may augment the learning images by randomly flipping some of the learning images horizontally or vertically so that each of the augmented learning images includes all lesion regions representing different severity levels, and by randomly rotating some of the learning images within a predetermined angular range so that each of the augmented learning images includes all lesion regions representing different severity levels.

[0094] The sensing unit (1400) can detect the status of the electronic device (1000) or the status around the electronic device (1000) and transmit the detected information to the processor (1300). The sensing unit (1400) can sense specification information of the electronic device (1000), temperature, humidity, and air pressure information for the monitoring target space, etc.

[0095] For example, the sensing unit (1400) may include at least one of a magnetic sensor (1410), an acceleration sensor (1420), a temperature / humidity sensor (1430), an infrared sensor (1440), a gyroscope sensor (1450), a position sensor (e.g., GPS) (1460), a pressure sensor (1470), a proximity sensor (1480), and an RGB sensor (illuminance sensor) (1490), but is not limited thereto. Since the function of each sensor can be intuitively inferred from its name by those skilled in the art, a detailed description thereof will be omitted.

[0096] The network interface (1500) may include one or more components that enable the electronic device (1000) to communicate with other devices (not shown) and a server (2000). The other devices (not shown) may be computing devices such as the electronic device (1000) or sensing devices, but are not limited thereto. For example, the network interface (1500) may include a wireless communication interface (1510), a wired communication interface (1520), and a mobile communication unit (1530). The wireless communication interface (1510) may include, but is not limited to, a short-range wireless communication unit, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, a UWB (ultra wideband) communication unit, etc.

[0097] The wired communication interface (1520) may include at least one wired interface for exchanging data with an external device connected to the electronic device via wired communication. The mobile communication unit (1520) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signals may include various types of data resulting from the transmission and reception of voice call signals, video call signals, or text / multimedia messages.

[0098] The A / V (Audio / Video) input unit (1600) is for inputting audio signals or video signals, and may include a camera (1610) and a microphone (1620), etc. The camera (1610) can obtain image frames, such as still images or moving images, through an image sensor in video call mode or shooting mode. Images captured through the image sensor can be processed through a processor (1300) or a separate image processing unit (not shown).

[0099] The microphone (1620) receives an external acoustic signal and processes it into electrical voice data. For example, the microphone (1620) can receive an acoustic signal from an external device or a user. The microphone (1620) can receive a user's voice input. The microphone (1620) can utilize various noise removal algorithms to remove noise generated during the process of receiving an external acoustic signal.

[0100] The memory (1700) can store a program for processing and controlling the processor (1300), and can also store data input to or output from the electronic device (1000). In addition, the memory (1700) can store learning images used by the electronic device (1000), augmented learning images, structural information of the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model, and weight information.

[0101] For example, the memory (1700) may store weight values ​​regarding layers, nodes, and connection strengths of at least one neural network model. Furthermore, the electronic device (1000) may additionally store learning data generated by the electronic device (1000) to train the neural network model.

[0102] The memory (1700) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0103] Programs stored in memory (1700) can be classified into multiple modules according to their functions, for example, a UI module (1710), a touch screen module (1720), a notification module (1730), etc.

[0104] The UI module (1710) can provide a UI, GUI, etc. for the electronic device (1000) to provide a skin disease diagnosis service. The touch screen module (1720) can detect a touch gesture on a touch screen for designating an atypical mask and transmit information about the touch gesture to the processor (1300).

[0105] The notification module (1730) may generate a signal to notify the occurrence of an event in the electronic device (1000). For example, if the diagnostic information provided by the electronic device (1000) includes lesions with a very serious severity level (e.g., index), the electronic device (1000) may output alarm content by executing an instruction corresponding to the notification module (1730). Examples of events according to one embodiment include reception of a call signal, reception of a message, input of a key signal, schedule notification, etc. The notification module (1730) may output the notification signal in the form of a video signal through the display unit (1210), output the notification signal in the form of an audio signal through the audio output unit (1220), or output the notification signal in the form of a vibration signal through the vibration motor (1230).

[0106]

[0107] Figure 8 is a block diagram of a server according to one embodiment.

[0108] According to one embodiment, the server (2000) may include a network interface (2100), a database (2200), and a processor (2300). The configuration of the server (2000) illustrated in FIG. 8 may correspond to the configuration of the electronic device (1000) described in FIGS. 6 to 7. The network interface (2100) may correspond to the network interface (not illustrated) of the electronic device (1000) described above. According to one embodiment, the network interface (2100) may transmit or receive information about artificial intelligence models learned by the electronic device or server, or information about a neural network model (e.g., weight values ​​regarding layers and connection strengths between layers). According to another embodiment, the network interface (2100) may transmit information about an artificial intelligence model for diagnosing skin diseases learned by the server, information about layers of an artificial neural network and nodes included in the layers, or weight values ​​about connection strengths of layers within the neural network to the electronic device (1000).

[0109] Additionally, according to one embodiment, the database (2200) may correspond to the memory described above in FIGS. 6 and 7. For example, the database (2200) may store diagnostic target images acquired from the electronic device (1000), information on artificial intelligence models, images for learning, information on augmented images for learning, and the like.

[0110] In one embodiment, the processor (2300) can control the overall operation of the server (2000). For example, the processor (2300) can perform all or at least a part of the method for determining skin disease diagnosis information described in FIGS. 1 to 7 by controlling the network interface (2100) and the database (2200) in conjunction with the electronic device (1000).

[0111] The method for determining skin disease diagnosis information according to the present disclosure and the method for training an artificial intelligence model for determining skin disease diagnosis information may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the medium may be those specifically designed and configured for the present invention or may be known and usable by those skilled in the art of computer software.

[0112] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0113] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. In a method for an electronic device to determine skin disease diagnosis information, A step of acquiring a diagnostic target image; Obtaining a prompt for specifying an atypical mask for at least one lesion area included in the above diagnostic target image; Based on the above prompt, when the diagnosis target image is input, a step of obtaining at least one atypical mask from the first artificial intelligence model by inputting the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for at least one lesion area; A step of tokenizing the attribute data of at least one atypical mask obtained from the first artificial intelligence model; and A method comprising: a step of obtaining diagnostic information from a second artificial intelligence model that outputs diagnostic information regarding at least one of the type of skin disease appearing in the diagnostic image or the skin disease severity value for each lesion area of ​​the diagnostic image, when the tokenized attribute data and the diagnostic target image are input; 2. In the first paragraph, the method A step of obtaining a resolution-converted diagnosis target image from the third artificial intelligence model by inputting the diagnosis target image into a third artificial intelligence model that converts the diagnosis target image into a high-resolution image; and A method comprising: a step of obtaining at least one atypical mask from the first artificial intelligence model by inputting the image to be diagnosed as a resolution conversion target into the first artificial intelligence model; 3. In the second paragraph, the step of obtaining the prompt is A method comprising: obtaining, as a prompt, at least one of tokenized text for specifying the atypical mask or a position value of at least one point on the diagnostic target image based on a user input to the electronic device; 4. In the second paragraph, the step of tokenizing the attribute data is A method comprising: a step of tokenizing attribute data regarding at least one of an area, a location, a representative color, or a degree of redness of at least one atypical mask obtained from the first artificial intelligence model; 5. In the fourth paragraph, the method After the step of acquiring the above diagnostic target image, a step of preprocessing the diagnostic target image to normalize the size of the acquired diagnostic target image; A step of obtaining at least one atypical mask by inputting the preprocessed diagnosis target image into the first artificial intelligence model; A step of post-processing at least one atypical mask obtained from the first artificial intelligence model by converting at least one of the attribute data of the at least one atypical mask obtained from the first artificial intelligence model, including the area, location, representative color, or degree of redness; and A method comprising: a step of tokenizing attribute data obtained from at least one post-processed amorphous mask; 6. In the fifth paragraph, the step of post-processing at least one amorphous mask A method comprising: applying a sharpening and noise removal technique to at least one of the above-described irregular masks; 7. In the second paragraph, before acquiring the diagnostic target image, A step of acquiring multiple training images; A step of augmenting the learning images so that each of the augmented learning images, which is generated by augmenting the learning images as a result of acquiring the diagnostic information for each of the plurality of learning images, includes lesion areas representing different severity levels; and A method comprising: a step of training the first artificial intelligence model and the second artificial intelligence model based on the augmented learning images; 8. In the 7th paragraph, the step of enhancing the learning images is A step of randomly horizontally or vertically flipping (FLIP) some of the training images so that each of the augmented training images includes all lesion areas representing different severity levels; and A method comprising: a step of randomly rotating some of the training images within a predetermined angular range so that each of the augmented training images includes lesion areas representing different severity levels; 9. In paragraph 7, The above skin disease severity scale includes four severity ranges according to the severity level of the skin disease. The step of augmenting the above learning images is A method comprising: a step of enhancing the learning images so that each of the enhanced learning images includes at least one lesion area corresponding to the four severity ranges, based on the diagnostic information acquisition result for each of the acquired learning images; 10. In the 7th paragraph, the step of training the first artificial intelligence model and the second artificial intelligence model is A step of obtaining learning atypical masks from the first artificial intelligence model by inputting the augmented learning images into the first artificial intelligence model; A step of labeling the learning atypical masks with a skin disease type and a severity label value for each skin disease type determined based on the attribute data of the acquired learning atypical masks; and A method comprising: a step of training the second artificial intelligence model based on the labeled learning atypical masks; 11. In the 10th paragraph, the method Before acquiring the above diagnostic target image, A step of acquiring ultra-high resolution learning upscaled images and low resolution learning upscaled images as a resolution conversion training data set; A step of augmenting the training data set so that each of the augmented training data sets generated by augmenting the training data set includes lesion areas indicating different severity levels, as a result of acquiring diagnostic information for the training data set obtained by inputting the acquired training data set into the first artificial intelligence model and the second artificial intelligence model; and A method comprising: a step of training the third artificial intelligence model based on the augmented training data set; 12. In paragraph 2, A method characterized in that the first artificial intelligence model is a SAM (Segment Anything Model) model, the second artificial intelligence model is a neural network or network model that performs machine learning operations to provide the diagnostic information based on attribute data of the atypical mask, and the third artificial intelligence model is a SRGAN (Super Resolution GAN) model.

13. In the 10th paragraph, the labeling step A method comprising: labeling the learning amorphous masks based on at least one of the size of the area of ​​the learning amorphous masks obtained from each of the augmented learning images, the number of the amorphous masks obtained from each of the augmented learning images, and the channel color values ​​of the amorphous masks obtained from each of the augmented learning images; 14. In an electronic device for determining skin disease diagnosis information, Memory that stores one or more instructions; At least one processor executing one or more of the instructions; The at least one processor executes the one or more instructions, Acquire the diagnostic target image, Obtain a prompt for specifying an atypical mask for at least one lesion area included in the above diagnostic target image, Based on the above prompt, when the diagnosis target image is input, at least one atypical mask is obtained from the first artificial intelligence model by inputting the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for at least one lesion area, Tokenizing the attribute data of at least one atypical mask obtained from the first artificial intelligence model, An electronic device that obtains diagnostic information from a second artificial intelligence model that outputs diagnostic information regarding at least one of the type of skin disease appearing in the diagnostic image or the skin disease severity value for each lesion area of ​​the diagnostic image when the tokenized attribute data and the diagnostic target image are input.

15. In a method for determining skin disease diagnosis information by an electronic device, A step of acquiring a diagnostic target image; Obtaining a prompt for specifying an atypical mask for at least one lesion area included in the above diagnostic target image; Based on the above prompt, when the diagnosis target image is input, a step of obtaining at least one atypical mask from the first artificial intelligence model by inputting the diagnosis target image into a first artificial intelligence model that outputs an atypical mask for at least one lesion area; A step of tokenizing the attribute data of at least one atypical mask obtained from the first artificial intelligence model; and A computer-readable recording medium storing a program for performing a method, comprising: obtaining diagnostic information from a second artificial intelligence model that outputs diagnostic information regarding at least one of the type of skin disease appearing in the diagnostic image or the skin disease severity value for each lesion area of ​​the diagnostic image when the tokenized attribute data and the diagnostic target image are input;

Images