Spectral image-based melanoma determination method, spectral image-based melanoma detection method, and apparatus for supporting same

Abstract

The present invention relates to a spectral image-based melanoma determination method, a spectral image-based melanoma detection method, and a device for supporting the methods. The present invention may provide a spectral image-based melanoma detection method and an operating device therefor, the method comprising the steps of: controlling a spectral camera and acquiring a current spectral image of an examination subject in which at least one skin lesion has occurred; the processor separates a foreground region in which at least one skin lesion appears and a background region except the foreground region from the current spectral image; and the processor compares a pre-stored reference model with at least a part of the separated foreground region and background region, and outputs a melanoma determination result of the examination subject.

Description

Technical Field

[0001] The present invention relates to melanoma identification or detection, and more particularly, to a melanoma identification or detection technology using a non-destructive spectral image. Background Art

[0002] Melanoma is defined as the most dangerous type of skin cancer among skin diseases. In most cases, melanoma can be cured if detected early, but if not detected early, melanoma will spread to other parts, making it difficult to treat and causing the risk of death. There are self-diagnosis methods for melanoma, such as checking the pattern changes of spots on the skin and the shape of newly appeared spots, but it is not easy for ordinary people to distinguish between normal spots and abnormal spots. In recent years, due to technological advances and continuous research, various methods for diagnosing melanoma have been proposed, and the ABCD method has been suggested as a clinical method. The ABCD method is a method in which medical professionals directly observe patient lesions and identify suspicious areas by observing the asymmetry, border irregularity, color irregularity and diameter of spots or moles. Once a suspicious area is identified, the skin is cut and a tissue biopsy is performed to confirm whether it is melanoma.

[0003] Therefore, since the method of separating skin tissue suspected of melanoma from the skin and examining it is mainly used to diagnose the occurrence of melanoma, it takes a lot of time to determine whether melanoma exists, and the person being examined may feel discomfort due to the incision of the skin tissue.

[0004] The number of patients with melanoma lesions has been steadily increasing, but the number of medical professionals who can observe and examine the lesions has not increased easily. As a result, there is a shortage of personnel to identify melanoma lesions, and if medical professionals conduct regular examinations for melanoma, there is a problem of spending a lot of money and a lot of time. Summary of the invention

[0005] Technical issues

[0006] The present invention aims to provide a melanoma identification method and device based on spectral images, which can efficiently and accurately separate the foreground and background of spectral images in melanoma identification to clearly distinguish the area of ​​melanoma identification, thereby increasing the amount of calculation and producing more accurate identification results.

[0007] In addition, the present invention aims to provide a melanoma inspection method and device based on spectral images, which can use spectral images to quickly and accurately inspect patients for melanoma during melanoma inspection, and adopt a non-destructive solution to eliminate processes such as skin incisions.

[0008] However, the objects of the present invention are not limited to the above objects, and other objects not mentioned can be clearly understood from the following description.

[0009] Technical Solution

[0010] In order to achieve the above-mentioned object, the melanoma identification device according to the present invention may include: a spectral camera, which acquires a spectral image of an inspection target including at least one skin lesion; and a processor, which is functionally connected to the spectral camera. The processor may be configured to control the spectral camera to acquire a current spectral image of the inspection target, separate a foreground area where at least one skin lesion has occurred and a background area other than the foreground area from the current spectral image, and compare at least a portion of the separated foreground area and background area with a pre-stored reference model to output a melanoma identification result of the inspection target.

[0011] Specifically, the processor can be configured to detect an adjacency matrix based on multiple dimensional vectors and edges corresponding to distance values ​​between the multiple dimensional vectors by applying a nearest neighbor technique to the current spectral image, and extract the foreground area based on edges in the adjacency matrix that are greater than or equal to a predefined reference value.

[0012] Specifically, the processor may be configured to extract clusters having a similarity with clusters corresponding to the spectrum of the foreground area higher than or equal to a predefined reference value from the reference model, and output a melanoma identification result of the inspection target based on the clusters extracted from the reference model.

[0013] Specifically, the processor can be configured to detect the directionality of the cluster corresponding to the spectrum of the foreground area; extract clusters having similar directionality within a certain range to the detected directionality from the reference model; and output a melanoma identification result of the inspection target based on the clusters extracted from the reference model.

[0014] In particular, the processor may be configured to output the adjacency matrix to a display after visualization.

[0015] Specifically, the processor may be configured to output the melanoma identification result through cross-validation between a melanoma identification result based on the foreground region and a melanoma identification result based on both the foreground region and the background region.

[0016] A melanoma identification method according to an embodiment of the present invention may include: controlling a spectral camera by a processor of a melanoma identification device to acquire a current spectral image of an inspection target including at least one skin lesion; separating a foreground area where at least one skin lesion has occurred and a background area other than the foreground area from the current spectral image by the processor; and comparing at least a portion of the separated foreground area and background area with a pre-stored reference model by the processor to output a melanoma identification result of the inspection target.

[0017] Specifically, the step of separating the foreground area and the background area may include: detecting an adjacency matrix based on multiple dimensional vectors and edges corresponding to distance values ​​between the multiple dimensional vectors by applying the nearest neighbor technique to the current spectral image; and extracting the foreground area based on edges in the adjacency matrix that are greater than or equal to a predefined reference value.

[0018] Specifically, the step of outputting the melanoma identification result may include one of the following steps: extracting from a reference model a cluster whose similarity to the cluster corresponding to the spectrum of the foreground area is higher than or equal to a predefined reference value, and outputting the melanoma identification result of the inspection target based on the cluster extracted from the reference model; or, detecting the directionality of the cluster corresponding to the spectrum of the foreground area, extracting from the reference model a cluster having a similar directionality to the detected directionality within a certain range, and outputting the melanoma identification result of the inspection target based on the cluster extracted from the reference model.

[0019] Specifically, the step of outputting the melanoma identification result may include: outputting the melanoma identification result through cross-validation between a melanoma identification result based on the foreground region and a melanoma identification result based on both the foreground region and the background region.

[0020] A server device supporting melanoma identification based on spectral images according to an embodiment of the present invention may include: a server communication circuit that establishes a communication channel with a melanoma identification device; and a server processor that is functionally connected to the server communication circuit. The server processor may be configured to: receive a current spectral image of an inspection target including at least one skin lesion from the melanoma identification device; separate a foreground area where at least one skin lesion has occurred and a background area other than the foreground area from the current spectral image; perform melanoma identification on the inspection target by comparing at least a portion of the separated foreground area and background area with a pre-stored reference model; and send a melanoma identification result to the melanoma identification device.

[0021] Specifically, the server processor may be configured to perform the melanoma identification through cross-validation between a melanoma identification result based on the foreground region and a melanoma identification result based on both the foreground region and the background region.

[0022] The melanoma inspection device according to the present invention may include: a spectral camera that acquires a spectral image of an inspection target including at least one skin lesion; and a processor that is functionally connected to the spectral camera. The processor may be configured to control the spectral camera to acquire a current spectral image of the inspection target, detect a latent vector by applying the current spectral image to a pre-stored reference model, and determine whether the inspection target is melanoma based on the directionality and form of the latent representation corresponding to the latent vector.

[0023] Specifically, the processor may be configured to control the spectral camera so that a shooting angle and a distance of the spectral camera relative to the inspection target become a predefined shooting angle and a distance.

[0024] Specifically, the processor may be configured to determine a melanoma type of the inspection target based on the directionality and form of the potential representation, and output information about the determined melanoma type.

[0025] Specifically, the processor may be configured to determine that the skin lesions in the current spectral image are represented as being farther from the center of the potential representation, the stronger the intensity of the lesions.

[0026] Specifically, the processor can be configured to: when the current spectral image contains multiple skin lesions, divide the image into regions including the multiple skin lesions; separate each skin lesion and a background region of a predetermined size surrounding each skin lesion in the divided regions; and sequentially perform melanoma inspection on each of the separated skin lesions and background regions.

[0027] Specifically, the processor can be configured to: extract clusters of spectra of a current spectral image; detect clusters of potential representations that are most similar to the clusters corresponding to the current spectral image from a reference model; determine whether the cluster corresponding to the current spectral image is melanoma and the type of melanoma based on the clusters detected from the reference model; and output determined information.

[0028] A melanoma inspection method based on spectral images according to an embodiment of the present invention may include: controlling a spectral camera through a processor of a melanoma inspection device to acquire a current spectral image of an inspection target including skin lesions; detecting a latent vector by applying the current spectral image to a pre-stored reference model; and determining whether the inspection target is melanoma and at least one of the types of melanoma based on the directionality and form of the latent representation corresponding to the latent vector.

[0029] Specifically, the method may further include outputting information including at least one of whether the determined inspection target is melanoma and a type of melanoma.

[0030] Specifically, the step of detecting the latent vector may include: when the current spectral image contains multiple skin lesions, dividing the image into regions including the multiple skin lesions; separating each skin lesion and a background region of a predetermined size surrounding each skin lesion in the divided regions; and detecting the latent vector regarding each of the separated skin lesions and background regions in turn.

[0031] Specifically, the step of determining whether the inspection target is melanoma and at least one of the types of melanoma may include: extracting clusters of the spectrum of the current spectral image; detecting clusters of potential representations that are most similar to the clusters corresponding to the current spectral image from a reference model; and determining whether the clusters corresponding to the current spectral image are melanoma and the type of melanoma based on the clusters detected from the reference model.

[0032] A server device supporting melanoma inspection based on spectral images according to an embodiment of the present invention may include: a server communication circuit that establishes a communication channel with a melanoma inspection device; and a server processor that is functionally connected to the server communication circuit. The server processor may be configured to: receive a current spectral image of an inspection target including skin lesions from the melanoma inspection device; detect a latent vector by applying the current spectral image to a pre-stored reference model; determine whether the inspection target is a melanoma and at least one of a melanoma type based on the directionality and form of the latent representation corresponding to the latent vector; and send the determined information to the melanoma inspection device.

[0033] Specifically, the server processor can be configured to: extract clusters of spectra of a current spectral image; detect clusters of potential representations that are most similar to the clusters corresponding to the current spectral image from a reference model; and determine whether the cluster corresponding to the current spectral image is melanoma and the type of melanoma based on the clusters detected from the reference model.

[0034] Beneficial Effects

[0035] According to the present invention, the method and apparatus for melanoma identification based on spectral images can more accurately identify melanoma by precisely and accurately separating the foreground and background corresponding to skin lesions from the spectral images.

[0036] In addition, the method and device for melanoma inspection based on spectral images according to the present invention can reduce the manpower, time and cost consumption of the method of direct inspection of lesions by medical professionals, can quickly and accurately provide inspection results of lesions, and can minimize the burden on patients through non-destructive inspection schemes without skin tissue incisions.

[0037] In addition, according to the embodiments of the present invention to be described later, various effects other than the above-mentioned effects may be disclosed directly or implicitly in the detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a diagram showing an example of a system environment supporting a melanoma identification function according to the first embodiment of the present invention.

[0039] Figure 2 : is a diagram showing an example of components of the melanoma identification device according to the first embodiment of the present invention.

[0040] Figure 3 It is shown Figure 2 A diagram of an example of components of a processor.

[0041] Figure 4 : is a diagram showing an example of a spectrum image of an inspection target according to the first embodiment of the present invention.

[0042] Figure 5 It is shown Figure 4 FIG. 1 shows an example of separation between foreground and background in a spectral image.

[0043] Figure 6 is a diagram showing an example of components of a server device according to the first embodiment of the present invention.

[0044] Figure 7 : is a diagram showing an example of a method for operating a melanoma identification apparatus in relation to the melanoma identification method according to the first embodiment of the present invention.

[0045] Figure 8 is a diagram showing another example of a method for operating a melanoma identification apparatus in relation to the melanoma identification function according to the first embodiment of the present invention.

[0046] Fig. 9 : is a diagram showing an example of a system environment supporting a melanoma inspection function according to a second embodiment of the present invention.

[0047] Fig.10 : is a diagram showing an example of components of a melanoma inspection device according to a second embodiment of the present invention.

[0048] Fig.11It is shown Fig.10 A diagram of an example of components of a processor.

[0049] Fig.12 is a diagram showing an example of latent vector representation of melanoma data according to the second embodiment of the present invention.

[0050] Fig.13 : is a diagram showing an example of a melanoma inspection function of an inspection target according to the second embodiment of the present invention.

[0051] Fig.14 is a diagram showing an example of components of a server device according to a second embodiment of the present invention.

[0052] Fig.15 : is a diagram showing an example of a method for operating a melanoma inspection apparatus related to a melanoma inspection method according to a second embodiment of the present invention.

[0053] Fig.16 is a diagram showing another example of a method for operating a melanoma inspection device in relation to the melanoma inspection function according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0054] Now, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0055] However, in the following description and the accompanying drawings, in order to avoid obscuring the subject matter of the present invention, well-known functions and components may not be described and explained in detail. In addition, the same components are represented by the same reference numerals as much as possible throughout the accompanying drawings.

[0056] The terms or words used in the following description and drawings should not be interpreted as limited to their usual or dictionary meanings, but should be interpreted as meanings and concepts that conform to the technical ideas of the present invention based on the principle that the inventor can appropriately define the concepts of the terms to best describe his or her invention. Therefore, the embodiments described herein are only the most preferred embodiments of the present invention and do not represent all technical ideas of the present invention. Therefore, it should be understood that when submitting this application, there may be various equivalents and modified examples that can replace the embodiments.

[0057] In addition, the terms including ordinal numbers such as first, second, etc. are used only to describe various elements for the purpose of distinguishing one element from another element, and are not used to limit such elements. For example, without departing from the scope of the present invention, the second element can be named as the first element, and similarly, the first element can also be named as the second element.

[0058] In addition, the terms used herein are only used to describe specific embodiments and do not limit the present disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, terms such as "comprises" and "includes" used herein are intended to specify the presence of features, numbers, steps, operations, elements, components, or combinations thereof disclosed herein, and should not be interpreted as precluding the possibility of the presence or addition of other features, numbers, steps, operations, elements, components, or combinations thereof.

[0059] In addition, terms such as "unit" and "module" used herein refer to a unit that processes at least one function or operation and can be implemented by hardware, software, or a combination of hardware and software. In addition, the terms "a", "an", "one", "the" and similar terms can be used as both singular and plural meanings in the context of describing the present invention (especially in the context of the appended claims), unless the context clearly indicates otherwise.

[0060] In addition to the above-mentioned terms, specific terms used in the following description are provided to help understand the present invention, and the use of these specific terms may be changed to other forms without departing from the technical meaning of the present invention.

[0061] In addition, embodiments within the scope of the present invention include computer-readable media having computer-executable instructions or data structures stored on computer-readable media. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. For example, such computer-readable media can include, but are not limited to, RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, or any other physical storage medium, which can be used to store or transfer certain program codes formed by computer-executable instructions, computer-readable instructions or data structures, and can be accessed by a general-purpose or special-purpose computer system.

[0062] [First embodiment]

[0063] Hereinafter, in the present invention, a melanoma identification function, a melanoma identification system environment supporting the melanoma identification function, and the type and role of each component included in the melanoma identification system environment will be described.

[0064] Figure 1 is a diagram showing an example of a system environment supporting a melanoma identification function according to the first embodiment of the present invention.

[0065] Reference Figure 1, the system environment 10 supporting the melanoma identification function of the present invention may include an inspection target 50 (e.g., at least a portion of the skin of a patient having a lesion (or skin lesion)), a melanoma identification device 100 capable of photographing at least a portion of the area of ​​the inspection target 50, and a server device 200 capable of supporting the melanoma identification device 100. In the illustrated system environment 10, a communication channel is formed between the melanoma identification device 100 and the server device 200, and the server device 200 supports the melanoma identification device 100. However, this is merely exemplary, and the present invention is not limited thereto. In an alternative example, the system environment 10 may be configured based on an electronic device (e.g., the melanoma identification device 100) capable of executing an embedded program for performing spectral image collection and melanoma determination on the inspection target 50, without using both the melanoma identification device 100 and the server device 200. In this case, the server device 200 may be omitted from the configuration of the system environment 10.

[0066] The inspection target 50 may include at least a portion of the patient's skin having at least one skin lesion. Although the patient's back is exemplarily shown as the inspection target 50, the present invention is not limited thereto. For example, the inspection target 50 may include any area where melanoma may occur. For example, the inspection target 50 may be any skin area where melanoma may occur, such as the soles of the feet, the palms of the hands, the face, the chest, the abdomen, the buttocks, the genitals or surrounding areas, the scalp, etc., and may also be at least a portion of an internal organ.

[0067] The melanoma identification device 100 may obtain a spectral image about the inspection target 50. For example, the melanoma identification device 100 may be configured to approach the inspection target 50 within a certain distance and capture a spectral image of the inspection target 50 at a certain level or higher of resolution. In this regard, the melanoma identification device 100 may include at least one spectral camera 120 and a mounting structure 129 that mounts the spectral camera 120 and enables the spectral camera 120 to perform at least one of a linear motion and a rotational motion. The spectral camera 120 may capture a spectral image of the inspection target 50 located at a certain angle and within a certain distance. The mounting structure 129 is configured to at least temporarily mount the spectral camera 120 and allow the spectral camera 120 to perform a linear motion in at least one direction of a front-to-back direction, a left-to-right direction, and an up-and-down direction and / or a rotational motion at a specific angle. Therefore, the melanoma identification device 100 may adjust the shooting range of the inspection target 50 as needed. The melanoma identification device 100 may capture at least one region of the inspection target 50 including at least one lesion to obtain a spectral image corresponding to the region. In addition, the melanoma identification device 100 can separate the foreground (e.g., the region of interest which is the skin lesion area) and the background (e.g., the skin area without lesions except the region of interest) of the acquired spectral image of the inspection target 50, compare the separated data with a pre-stored reference model, and then perform melanoma identification based on the comparison result.

[0068] Alternatively, the melanoma identification device 100 may obtain the melanoma identification result for the inspection target 50 through the server device 200. In this case, the melanoma identification device 100 does not perform foreground and background separation and model comparison analysis based on the separation data on the spectral image. Instead, the melanoma identification device 100 may be configured to send the acquired spectral image to the server device 200, receive the melanoma identification result from the server device 200, and output the received result. In addition, when the melanoma identification device 100 is configured to independently perform melanoma identification and classification and result output without the server device 200, the melanoma identification device 100 may pre-store a reference model received from an external server device. The reference model may be a model generated by performing machine learning or unsupervised learning on various melanoma spectral images. The reference model may be generated based on various spectral images. The reference model may be generated based on data for separating the foreground and background of various spectral images, and may include at least one of a foreground reference model generated based only on the foreground and an integrated reference model based on the separated foreground and background. For example, when the spectral image acquired by the spectral camera 120 and the corresponding inspection record of the inspection target 50 are cumulatively stored by a predetermined number or more, the melanoma identification device 100 can separate the foreground and background of the spectral image, and directly generate a reference model based on unsupervised learning of at least a portion of the separated data. Here, in order to protect the information of the individual inspection record, the information used to generate the reference model may include only the spectral image of the inspection target 50 and the inspection result of the spectral image. Additionally or alternatively, the information used to generate the reference model may include only information other than information that can identify an individual, such as the age, gender, and weight of the inspection target 50.

[0069] The server device 200 may establish a communication channel with the melanoma identification device 100. The server device 200 may receive at least one spectral image of the inspection target 50 from the melanoma identification device 100, and perform melanoma identification on the received at least one spectral image. In this process, the server device 200 may pre-store a reference model for comparative analysis of the currently acquired spectral image. The reference model may be generated by separating the foreground and the background from the spectral image collected and provided by the melanoma identification device 100 and performing unsupervised learning on the separated data, or may be received from a separate external server device that provides the reference model. The server device 200 may provide the melanoma identification device 100 with the result of the comparative analysis with the reference model. At the same time, if the acquired spectral image has insufficient resolution or a spectral image with higher resolution is required, the server device 200 may request the melanoma identification device 100 to adjust the resolution of the inspection target 50 having the corresponding lesion, and obtain a spectral image with improved resolution. In response, the melanoma identifying apparatus 100 may adjust the photographing angle and photographing distance of the spectral camera 120 to capture a new spectral image of the inspection target 50 and provide it to the server apparatus 200 .

[0070] As described above, in the system environment 10 supporting the melanoma identification function according to the first embodiment of the present invention, the melanoma identification device 100 can acquire a spectral image of the inspection target 50, separate the foreground and background of the spectral image, and perform a comparison with a pre-learned reference model based on the separated data, thereby obtaining a melanoma identification result of the inspection target 50 more accurately and quickly through more accurate information comparison.

[0071] Figure 2 is a diagram showing an example of components of a melanoma identification device according to a first embodiment of the present invention, and Figure 3 It is shown Figure 2 A diagram of an example of components of a processor.

[0072] First, refer to Figure 2 , the melanoma identification device 100 according to the first embodiment of the present invention may include a communication circuit 110, a spectral camera 120, a memory 130, an input unit 140, a display 160, and a processor 150. In addition, the melanoma identification device 100 may further include a mounting structure 129 for mounting the spectral camera 120, so that at least one spectral camera 120 can capture a spectral image of the inspection target 50. In addition, the melanoma identification device 100 may further include a power source (e.g., a permanent power source or a battery) required for the operation of at least one of the above-mentioned components (e.g., the communication circuit 110, the spectral camera 120, the memory 130, the input unit 140, the display 160, and the processor 150).

[0073] The communication circuit 110 may support the communication function of the melanoma identification device 100. For example, if the server device 200 is designed to perform the calculation required for the melanoma identification function according to the first embodiment of the present invention, the communication circuit 110 may send at least one spectral image collected by the spectral camera 120 to the server device 200. In addition, the melanoma identification function may be independently performed by the melanoma identification device 100. In this case, the communication circuit 110 may send a message including the inspection result according to the melanoma identification function to a terminal of an administrator of the melanoma identification device 100 or a designated user terminal device under the control of the processor 150. Alternatively, the communication circuit 110 may output (or send) the message to the server device 200 under the control of the processor 150. If the melanoma identification device 100 includes a separate output device (e.g., a display 160 or an audio device), the message may be output through the output device. The communication circuit 110 may establish a communication channel with an external server device and receive a reference model 131 from the external server device. The reference model 131 may include a model generated by learning data in which the foreground and background of various melanoma-related spectral images are separated. In this regard, the communication circuit 110 may periodically establish a communication channel with the external server device under the control of the processor 150, and if there is a newly updated reference model 131, the communication circuit 110 may receive the updated reference model 131 from the external server device and store (or update) it in the memory 130.

[0074] The spectral camera 120 may be configured to capture a spectral image of the inspection target 50. At least one spectral camera 120 may be provided, and when a plurality of spectral cameras are provided, the spectral cameras may be arranged to capture the inspection target 50 by dividing the inspection target 50 into regions or to capture the inspection target 50 from different angles. The spectral camera 120 may be activated in response to the control of the processor 150, and when acquiring a spectral image of the inspection target 50, the spectral camera may send the acquired image to the processor 150. Alternatively, in response to the control of the processor 150, the spectral image acquired by the spectral camera 120 may be sent to the server device 200 via the communication circuit 110.

[0075] The memory 130 may store at least one program or data required for the operation of the melanoma identification device 100. For example, the memory 130 may temporarily or semi-permanently store a control program required for operating at least one spectral camera 120 and a spectral image 133 acquired by the at least one spectral camera 120. For example, the memory 130 may store a reference model 131 for comparative analysis with a currently captured spectral image for the inspection target 50. The reference model 131 may be received from an external server device as described above. Alternatively, if a plurality of spectral images 133 are cumulatively stored in a predetermined number or more, the reference model 131 may be generated by the operation of the processor 150. In this regard, the memory 130 may store a diagnosis result matched with the spectral image 133 (e.g., whether the spectral image 133 is melanoma, information about the type of melanoma, at least part of the information about the patient's age, weight, and place of residence, etc.). For example, the reference model 131 may be generated by performing unsupervised learning on separation data in which the foreground and background of the plurality of spectral images 133 are separated.

[0076] The input unit 140 may include various input tools for operating the melanoma identification device 100. For example, the input unit 140 may generate at least one of an input signal for activating the spectral camera 120, an input signal for operating the spectral camera 120 to acquire the spectral image 133, an input signal for inputting a diagnosis result of the acquired spectral image 133, and an input signal for requesting output of an analysis result of the spectral image 133 in response to a user's manipulation. The input unit 140 may include at least one of various tools, such as a soft key (or an input tool based on a touch screen or a touch pad), a physical key, a voice input device, a gesture input device, and a shuttle knob.

[0077] The display 160 may output at least one screen required for operating the melanoma identification apparatus 100. For example, the display 160 may output at least one of a screen indicating whether at least one device (e.g., at least one of the spectral camera 120, the mounting structure 129, the communication circuit 110, and the input unit 140) connected to the melanoma identification apparatus 100 is in a normal state, a screen related to activation of the spectral camera 120, a screen related to acquisition of the spectral image 133 by the spectral camera 120, and a screen related to a melanoma identification result according to analysis of the spectral image 133. In addition, when the melanoma identification apparatus 100 is operated in conjunction with the server apparatus 200, the display 160 may output at least one of a screen related to accessing the server apparatus 200, a screen related to melanoma identification of the spectral image 133 and a screen related to a request to receive a melanoma identification result of the spectral image 133.

[0078] The processor 150 may perform at least one of transmission and processing of signals required for the operation of the melanoma identification device 100 and storage and output of processing results. For example, the processor 150 may control the spectral camera 120 to acquire the spectral image 133, separate the foreground and background of the acquired spectral image 133, compare and analyze the separated data with the reference model 131 pre-stored in the memory 130, and output the melanoma identification result based on the comparison and analysis. In this regard, the processor 150 may include, for example, Figure 3 Components shown.

[0079] Reference Figure 3 , the processor 150 may include at least one of a spectral camera controller 151 , a foreground separator 152 , a reference model learner 153 , and a melanoma identifier 154 .

[0080] The spectral camera controller 151 may control at least a portion of the spectral camera 120 or the mounting structure 129 so that the spectral camera 120 can capture the inspection target 50 at a certain resolution or higher. In this regard, the mounting structure 129 may be configured to perform at least one of a linear motion in at least one of a front / rear direction, a left / right direction, and an up / down direction and a rotational motion at a specific angle relative to the inspection target 50 under the control of the spectral camera controller 151. For example, the spectral camera controller 151 may control the spectral camera 120 to obtain a preview image of the inspection target 50 in which a lesion has occurred, and detect the resolution of the lesion in the preview image. If the resolution of the lesion is lower than a predefined reference value, the spectral camera controller 151 may adjust the mounting structure 129 to change at least one of a shooting distance and a shooting angle between the spectral camera 120 and the inspection target 50. For example, the spectral camera controller 151 may first adjust the position of the spectral camera 120 to shoot in a direction perpendicular to the center point of the inspection target 50, and then automatically adjust the mounting structure 129 so that the resolution of the inspection target 50 is higher than or equal to the predefined reference value. During this process, the spectral camera controller 151 may further adjust at least one of the shooting angle and shooting distance of the spectral camera 120 relative to the inspection target 50 so that the ratio of the lesion size on the inspection target 50 where the lesion occurs to the background size of the lesion is higher than a predetermined value.

[0081] When the spectral camera controller 151 completes the preparation work for photographing the inspection target 50, the spectral camera controller 151 may control the spectral camera 120 to acquire at least one spectral image. For example, when the condition that the ratio of the lesion size to the background size in the inspection target 50 is within a predetermined value is met, the spectral camera controller 151 may control to acquire a spectral image of the inspection target 50. Alternatively, when the resolution of the lesion in the inspection target 50 is greater than or equal to a predefined value, the spectral camera controller 151 may control to acquire a spectral image of the inspection target 50. When the spectral images 133 that meet the above conditions are accumulated and stored, the melanoma identification device 100 may perform unsupervised learning to generate a reference model 131 based on unified data, thereby obtaining a more reliable learning result.

[0082] The foreground separator 152 can separate the foreground and the background from the spectral image collected by the spectral camera controller 151. By using a spectral image such as M1 to separate the foreground (region of interest) and the background (general skin area outside the region of interest) to perform melanoma identification, the foreground separator 152 can achieve higher accuracy in the melanoma identification result. The foreground separator 152 can apply a machine learning (e.g., K-nearest neighbor (KNN)) algorithm as a method for separating the foreground and the background in the spectral image. In the spectral image, each spatial coordinate can be vector data as a spectrum with multiple wavelengths. For example, the wavelength of a single spectral image spectrum can be represented as a vector of dimension C, and the number of vectors corresponds to H (height) × W (width). The foreground separator 152 can obtain an adjacency matrix by performing KNN on H×W C-dimensional vectors (spectra) to calculate the index of the vector that is most adjacent to each vector. In the adjacency matrix, a vertex (or node) corresponds to each C-dimensional vector (spectrum), and an edge can correspond to the distance between C-dimensional vectors. The distance between C-dimensional vectors can be measured by at least one deep learning type index of L2 norm (e.g., the straight-line distance from the origin of a particular vector to the origin of another vector), L1 norm (the sum of the absolute values ​​of the differences between the individual vectors), and cosine similarity (e.g., the similarity of the cosine angle between two vectors from -1 to 1, the closer to 1, the greater the similarity).

[0083] The reference model learner 153 can accumulate and store data in which the foreground and background are separated by the foreground separator 152 for the acquired spectral image 133. If the foreground and background separation data of the acquired spectral image 133 accumulate to a predetermined amount or more, the reference model learner 153 can use the foreground and background separation data of the spectral image 133 for modeling by unsupervised learning. For example, the reference model learner 153 can learn the spectrum in which the foreground and background are separated individually or collectively in an unsupervised manner (such as a variational autoencoder). For example, the reference model learner 153 can perform unsupervised learning on data consisting only of spectra related to the foreground, thereby generating a reference model related to the foreground. Alternatively, the reference model learner 153 can perform unsupervised learning on all spectra related to the foreground and background, thereby generating a reference model related to the foreground and background. That is, the reference model learner 153 can model various spectral images related to melanoma and generate a reference model corresponding to the foreground data and the foreground background comprehensive data. Furthermore, if the melanoma identifying apparatus 100 is designed to receive the reference model 131 from the external server apparatus 200 , the reference model generating operation of the reference model learner 153 may be omitted.

[0084] When the current spectral image for the patient examination target 50 is received from the spectral camera controller 151 or the spectral image 133 is stored in the memory 130, the melanoma identifier 154 may collect the spectral image 133 stored in the memory 130 for melanoma identification. In addition, the melanoma identifier 154 may transfer the spectral image 133 to the foreground separator 152 and obtain data in which the foreground and the background are separated from the foreground separator 152. The melanoma identifier 154 may perform clustering by analyzing at least one of the separated foreground spectrum of the current spectral image and the spectrum including both the foreground and the background, and compare the clustering result of the spectrum with the reference model 131. In this process, the melanoma identifier 154 may perform more accurate melanoma identification by comparing and analyzing the foreground spectrum with the foreground reference model while excluding the background spectrum that may become a noise factor in the foreground analysis, and also reduce the amount of calculation in the model comparison. Furthermore, the melanoma identifier 154 can perform more accurate melanoma identification on the inspection target 50 by comparing and analyzing a foreground spectrum with a foreground reference model, comparing and analyzing a spectrum including both foreground and background with a reference model including both foreground and background, and synthesizing the results.

[0085] In the above operation, the melanoma identifier 154 can detect the cluster (or cluster of potential representation) most similar to the cluster (or cluster of potential representation) corresponding to the foreground (or foreground and background) of the current spectral image from the reference model 131, and identify the diffusion direction of the detected cluster. By referring to the result of the diffusion direction and the inspection result pre-recorded in the memory 130, the melanoma identifier 154 can output at least one of the melanoma identification result and the melanoma type of the current spectral image. In addition, if the melanoma identifier 154 does not detect a cluster in the reference model 131 whose similarity to the cluster corresponding to the foreground (or foreground and background) spectrum of the current spectral image is within a predefined range, it can detect the cluster with the highest similarity (or the cluster with the shortest distance value between the clusters of the reference model), and provide melanoma probability information corresponding to the cluster according to the similarity (or according to the distance value), or recommend a detailed tissue examination.

[0086] The above-mentioned melanoma identification device 100 of the present invention can perform non-destructive melanoma inspection (e.g., inspection without skin incision) by separating the foreground and background of the spectral image acquired by the spectral camera and comparing it with a reference model, and can also perform more effective melanoma identification by reducing the time and calculation required for melanoma identification.

[0087] Figure 4 is a diagram showing an example of a spectrum image of an inspection target according to the first embodiment of the present invention, and Figure 5 It is shown Figure 4 FIG. 1 shows an example of separation between foreground and background in a spectral image.

[0088] First, refer to Figure 4 , the spectral image obtained by photographing the inspection target 50 with the spectral camera 120 may include spectra of various wavelengths, and when only wavelengths of the visible light band (e.g., 400nm to 700nm) are extracted, an image as shown in the figure may be displayed. The image of the inspection target 50 shown may have, for example, a foreground area 50a' (or an area where skin lesions have occurred, or an area of ​​interest) and a background area 50b'. The background area 50b' may be determined according to the definition of the foreground area 50a'.

[0089] As described above, the foreground separator 152 of the melanoma identifying apparatus 100 may convert the spectral wavelengths of the spectral image of the inspection target 50 into respective dimensional vectors by using a suitable algorithm (eg, KNN algorithm) and detect the adjacency matrix 51 ′ by using the converted dimensional vectors. Figure 5The screens 501 and 503 of FIG. 5 visualize the adjacency matrix 51' detected by the foreground separator 152. The adjacency matrix 51' can be represented as a plurality of vertices corresponding to the C-dimensional vectors (or spectra) and a plurality of edges corresponding to the distance values ​​between the respective dimensional vectors. The adjacency matrix 51' represented by vertices and edges may include a foreground matrix region 51a' and a background matrix region 51b' corresponding to the foreground region 50a' and the background region 50b', respectively, as shown above. Figure 4 shown.

[0090] Furthermore, as in the screen 501, if the distance between the dimension vectors (or vertices) represented by the vertex and the edge in the adjacency matrix 51 is shorter, the edge can be displayed as shorter, and if the distance between the dimension vectors is longer, the edge can be displayed as longer. The central part of the figure corresponding to the adjacency matrix 51 shown in the screen 501 is compared with Figure 4 When comparing the image of the visible light region shown in FIG. 503 , it can be seen that the boundary edge between the foreground matrix region 51a' corresponding to the melanoma region and the background matrix region 51b' corresponding to the normal skin region is displayed longer. Based on this characteristic, the foreground separator 152 can effectively separate the foreground matrix region 51a' corresponding to the melanoma region from the background matrix region 51b' by thresholding the edge far beyond the predefined reference value. When separating the foreground matrix region 51a' through the above operation, the foreground separator 152 can define the boundary between the foreground matrix region 51a' and the background matrix region 51b' as shown in the screen 503, and effectively separate the foreground matrix region 51a' using the defined boundary.

[0091] The reference value (or threshold) in the above thresholding can be specified by the administrator of the melanoma identification device 100. If the administrator fails or does not specify a separate reference value, the foreground separator 152 can determine a certain reference value (for example, μ (mean) ± 1.5σ (standard deviation)) based on the mean and standard deviation of all edges contained in the adjacency matrix 51. Here, the constant 1.5 in front of the standard deviation value can be changed according to user specifications. The foreground separator 152 can convert the information about the foreground matrix area 51a' and the background matrix area 51b' into a binary mask form after thresholding. In this operation, the foreground separator 152 can selectively apply morphological operations as a method of trimming the edges of the foreground matrix area 51a'.

[0092] Figure 6 200 is a diagram showing an example of components of a server device according to the first embodiment of the present invention. As described above, if the melanoma identification device 100 is designed to independently perform a melanoma identification function according to the first embodiment of the present invention, the server device 200 may be omitted.

[0093] Reference Figure 6 , the server device 200 may include a server communication circuit 210 , a server memory 230 , and a server processor 250 .

[0094] The server communication circuit 210 may establish a communication channel with the melanoma identification device 100. The server communication circuit 210 may receive at least one current spectrum image from the melanoma identification device 100 periodically or in response to the occurrence of a predefined event. The server communication circuit 210 may receive a server reference model 231 from an external server device. The server communication circuit 210 may send the analysis result of the received at least one current spectrum image 233 to the melanoma identification device 100 (or a designated user terminal) under the control of the server processor 250.

[0095] The server memory 230 may store at least one program or data required for the operation of the server device 200. For example, the server memory 230 may store at least one of a current spectral image 233 collected and transmitted by the melanoma identification device 100 and a server reference model 231 for comparison with the current spectral image 233. The current spectral image 233 may be an image corresponding to the above-mentioned spectral image 133 stored in the memory 130 of the melanoma identification device 100. For example, the current spectral image 233 may include a spectral image currently acquired for the inspection target 50. The server reference model 231 may correspond to the above-mentioned reference model 131 stored in the memory 130 of the melanoma identification device 100. For example, the server reference model 231 may be generated by the melanoma identification device 100 and provided to the server device 200. Alternatively, the server reference model 231 may be generated by the server processor 250 based on a predetermined number or more of the current spectral images 233 stored in the server memory 230.

[0096] The server processor 250 may control transmission and processing of signals required for the operation of the server device 200 , storage or transmission of results, or transmission of messages corresponding to the results. In this regard, the server processor 250 may include a data collector 251 and a melanoma detector 252 .

[0097] The data collector 251 may establish a communication channel with the melanoma identification apparatus 100, and receive the current spectral image 233 of the inspection target 50 from the melanoma identification apparatus 100. Here, the server apparatus 200 may be in a state where the server reference model 231 has been pre-stored, and if there is no server reference model 231, the data collector 251 may receive the server reference model 231 from an external server apparatus. As described above, the server reference model 231 may also be generated by model learning in the server apparatus 200. In this regard, when the server apparatus 200 operates in a mode of learning the server reference model 231, the data collector 251 may collect various spectral images required for generating the server reference model 231 from an external server apparatus or the melanoma identification apparatus 100.

[0098] The data collector 251 may receive the current spectrum image 233 from the melanoma identification apparatus 100 in association with a request for melanoma identification of the patient's examination target 50. In this case, the data collector 251 may store the current spectrum image 233 together with the identifier information of the melanoma identification apparatus 100 in the memory 130 and request the melanoma detector 252 to analyze the current spectrum image 233.

[0099] When the melanoma detector 252 is notified of receiving the current spectrum image 233 from the data collector 251, the melanoma detector 252 may perform analysis on the current spectrum image 233 stored in the server memory 230. For example, the melanoma detector 252 may separate the foreground region and the background region of the current spectrum image 233, compare the separated foreground and background spectra with the server reference model 231, and thereby perform melanoma identification on the foreground spectrum (or foreground and background spectra) of the inspection target 50. In this operation, the melanoma detector 252 may detect, for example, the directionality of the latent vector of the foreground region, and collect the result of whether there is a melanoma corresponding to the detected directionality of the latent vector of the foreground region and the type of the melanoma.

[0100] The melanoma detector 252 may provide the melanoma identification apparatus 100 with a picture indicating the directionality of the potential vector of at least a portion (e.g., a foreground region or a region including both the foreground and the background) of the current spectral image 233. Alternatively, the melanoma detector 252 may detect a cluster in the server reference model 231 that is most similar to the cluster corresponding to the foreground spectrum, and based on this, perform melanoma identification on the cluster corresponding to the foreground spectrum. If there is no cluster corresponding to at least some of the spectra of the current spectral image in the server reference model 231, the melanoma detector 252 may output a melanoma probability value based on a distance value to the most similar cluster.

[0101] As described above, the server device 200 according to the first embodiment of the present invention can provide a more accurate melanoma identification result by comparing the spectrum of a partial wavelength band of the foreground area where skin lesions have occurred in the spectral image of the inspection target 50 with a pre-stored reference model and then performing melanoma identification, and when necessary, the server device 200 can improve the reliability of the result by cross-identification (for example, by integrating melanoma identification of only the foreground area and melanoma identification of both the foreground area and the background area).

[0102] Figure 7 : is a diagram showing an example of a method for operating a melanoma identification apparatus in relation to the melanoma identification method according to the first embodiment of the present invention.

[0103] Reference Figure 7 In the operation method of the melanoma identification apparatus 100 related to the melanoma determination method according to the first embodiment of the present invention, the processor 150 of the melanoma identification apparatus 100 may check whether an event of requesting the creation of a reference model occurs in step 701. The event of requesting the creation of a reference model may include, for example, an event of inputting a request through the input unit 140 by an administrator of the melanoma identification apparatus 100 or an event of receiving a request for generating and providing a reference model from the server apparatus 200. Alternatively, the melanoma identification apparatus 100 may be designed to collect various spectral images related to melanoma and generate corresponding reference models according to predefined scheduling information.

[0104] If an event related to creating a reference model does not occur, the processor 150 of the melanoma identification apparatus 100 may perform a specified function in step 703. For example, if the reference model has been stored in the memory 130, or the reference model has been received from an external server device, the processor 150 of the melanoma identification apparatus 100 may provide a melanoma identification function based on the reference model stored in the memory 130.

[0105] If an event related to creating a reference model occurs, the processor 150 of the melanoma identification device 100 may collect spectral images related to melanoma in step 705. In this operation, the processor 150 may access an external server device for providing spectral images related to melanoma, and collect spectral images from the external server device. Alternatively, when the spectral camera 120 connected to the melanoma identification device 100 photographs the inspection target 50 having a lesion, the processor 150 may store and manage the photographed spectral image of the inspection target 50 as one of the spectral images related to melanoma in the memory 130. Additionally or optionally, the processor 150 may collect the inspection results of the acquired spectral images together, perform information matching, and store the matched information in the memory 130.

[0106] The processor 150 may perform foreground separation of a melanoma-related spectral image. With respect to foreground separation, the processor 150 may apply a KNN algorithm or the like to the spectral image, and configure an adjacency matrix using a dimension vector and a distance value between the dimension vectors. The processor 150 may define a foreground matrix region using an edge in the adjacency matrix whose length is greater than or equal to a reference value, define a boundary of the foreground matrix region, and then extract the foreground matrix region. When the foreground matrix region is obtained in this manner, the processor 150 may perform unsupervised learning on the obtained foreground matrix in step 709 to generate a reference model based on the foreground matrix. Thereafter, in step 711, the processor 150 may check whether the learning is completed. For example, the processor 150 may determine whether the learning is completed by checking whether a predetermined number of times or more of learning is performed, or whether a predetermined type or more of learning of spectral images associated with various melanomas is performed. If the learning is not completed, the processor 150 may return to step 705 to perform a predetermined number of times or a predetermined type or more times or types of reference model learning for the foreground matrix region after foreground separation of the spectral image.

[0107] In addition, in addition to modeling the foreground matrix area, the processor 150 can also perform reference modeling learning on the spectral image including both the foreground and the background. Therefore, the created reference model can include, for example, at least one of a reference model created based on the foreground area and a reference model based on the spectral image including both the foreground and the background.

[0108] When the learning is completed, the processor 150 may store the reference model in the memory 130 or update the previously stored reference model in step 711. Alternatively, the processor 150 may provide the reference model to the server device 200 requesting the creation of the reference model. Thereafter, the processor 150 may return to step 701 and re-execute subsequent operations, or return to step 703 and execute a specified function.

[0109] Figure 8 is a diagram showing another example of a method for operating a melanoma identification apparatus in relation to the melanoma identification function according to the first embodiment of the present invention.

[0110] Reference Figures 1 to 8 Regarding the melanoma identification function, the processor 150 of the melanoma identification device 100 may check in step 801 whether spectral image collection is requested for melanoma identification. In this regard, the melanoma identification device 100 may activate a spectral image-based melanoma identification application in response to a user input, and activate the spectral camera 120 after activating the application. If there is no spectral image collection, the processor 150 may perform a designated function in step 803. For example, the processor 150 may input identification information of the inspection target 50 or basic information of the patient (e.g., age, sex, weight, lesion occurrence time, etc.) in response to a user input.

[0111] When requesting spectral image collection, the processor 150 may adjust the distance and angle between the spectral camera 120 and the inspection target 50 so that the inspection target 50 has a shooting distance or angle for shooting with a predefined size (or a distance or angle that makes the resolution of a certain area including the lesion become a predefined resolution). In order to adjust the predefined distance and angle, the processor 150 may control the spectral camera 120 to acquire a preview image, and adjust at least one of the distance and angle between the spectral camera 120 and the inspection target 50 by analyzing the acquired preview image so that the size of the background area including the lesion becomes the predefined size. When the preparation is completed, the processor 150 may control the spectral camera 120 to acquire a spectral image of the inspection target 50. Thereafter, in step 805, the processor 150 may perform foreground separation of the spectral image. For example, the processor 150 may apply machine learning (e.g., a K nearest neighbor (KNN) algorithm) to a spectral image such as M1 to generate vector data corresponding to the spectra of multiple wavelengths contained in the spectral image. For example, the processor 150 may perform KNN on H (height) × W (width) C-dimensional vectors (spectra) to calculate the index of the vector closest to each vector, thereby obtaining an adjacency matrix. In the adjacency matrix, a vertex (or node) corresponds to each C-dimensional vector (spectrum), an edge corresponds to the distance between the C-dimensional vectors, and the distance can be measured by an indicator such as L2, L1, cosine similarity. The processor 150 may generate a foreground region (or foreground matrix region 51a') by connecting edges whose length is greater than or equal to a predefined reference value.

[0112] In step 807, the processor 150 may apply at least some of the separation data to a reference model pre-stored in the memory 130, and in step 809, the processor 150 may identify whether the lesion is a melanoma based on the result applied to the reference model. Thereafter, the processor 150 may output the melanoma identification result to a designated device (e.g., the display 160). In this operation, the processor 150 may obtain a melanoma identification result for the foreground area by comparing only the foreground area with the reference model. Alternatively, the processor 150 may perform melanoma identification on the foreground area, perform melanoma identification by applying both the foreground area and the background area to the reference model, and then output a melanoma determination result by cross-validation. In this operation, the reference model used to perform melanoma identification on the foreground area and the reference model used to perform melanoma identification on the foreground area and the background area may be different.

[0113] Additionally or alternatively, the processor 150 may output a screen interface on the display 160, through which the medical professional may input an inspection result of at least one of information on whether or not melanoma exists and the type of melanoma, and the processor 150 may determine whether or not melanoma exists and the type of melanoma in the inspected spectral image in response to the input of the medical professional. The melanoma identification device 100 may maintain the result of the current spectral image as a potential result until the medical professional makes a determination.

[0114] In step 811, the processor 150 of the melanoma identification apparatus 100 may check whether melanoma identification is completed. If an event corresponding to completion of melanoma identification occurs, such as input of completion of examination by a medical professional or a request to deactivate the spectral camera 120, melanoma identification may be completed. If no event related to completion of melanoma identification occurs, the process may return to step 801 and re-execute subsequent operations.

[0115] In addition, although the above description describes that the melanoma identification device 100 performs the operation of acquiring a spectral image of the inspection target 50 and determining whether the lesion is a melanoma and / or the type of melanoma, the present invention is not limited thereto. Figure 8The operations described in step 804 may be performed by the server device 200. In this case, the server device 200 may establish a communication channel with the melanoma identification device 100 before step 801, and then check whether the spectral image is received in step 801. When the spectral image is received, the server device 200 may perform the operation of step 805 on the received spectral image, determine whether the lesion is a melanoma and / or the type of melanoma in step 809, and send the determination result back to the melanoma identification device 100. In this process, the server device 200 may store and manage the identification information of the melanoma identification device 100 that provides the spectral image and the received spectral image, so that the determination result can be sent to the corresponding melanoma identification device 100.

[0116] As described above, the melanoma identification function based on spectral images according to the first embodiment of the present invention can determine whether melanoma exists based on the spectral image of suspected melanoma skin, and in the melanoma identification process, the reliability of the melanoma identification result can be improved by separating the foreground and background of the spectral image and utilizing the separated data in whole or in part.

[0117] [Second embodiment]

[0118] Hereinafter, in the present invention, a melanoma inspection function, a melanoma inspection system environment supporting the melanoma inspection function, and the type and role of each component included in the melanoma inspection system environment will be described.

[0119] Fig. 9 : is a diagram showing an example of a system environment supporting a melanoma inspection function according to a second embodiment of the present invention.

[0120] Reference Fig. 9 , the system environment 10 supporting the melanoma inspection function of the present invention may include an inspection target 50 (e.g., at least a portion of the skin of a patient having a lesion (or skin lesion)), a melanoma inspection device 100 capable of photographing at least a portion of the inspection target 50, and a server device 200 capable of supporting the melanoma inspection device 100. In the system environment 10 shown, a communication channel is formed between the melanoma inspection device 100 and the server device 200, and the server device 200 supports the melanoma inspection device 100. However, this is merely exemplary, and the present invention is not limited thereto. In an alternative example, it may be configured to be capable of performing spectral image collection and melanoma inspection on the inspection target 50 based on an embedded program without using both the melanoma inspection device 100 and the server device 200. In this case, the server device 200 may be omitted from the configuration of the system environment 10.

[0121] The inspection target 50 may include the skin of a patient having at least one lesion that may be suspected of being a melanoma. Although the back of the patient is exemplarily shown as the inspection target 50, the present invention is not limited thereto. For example, the inspection target 50 may include any area where a melanoma may appear. For example, the inspection target 50 may be any skin area where a melanoma may appear, such as the soles of the feet, the palms of the hands, the face, the chest, the abdomen, the buttocks, the genitals or surrounding areas, the scalp, etc.

[0122] The melanoma inspection device 100 can obtain a spectral image about the inspection target 50. For example, the melanoma inspection device 100 can be configured to approach the inspection target 50 within a certain distance and capture a spectral image of the inspection target 50 at a certain level or higher level of resolution. In this regard, the melanoma inspection device 100 can include at least a spectral camera 120 and a mounting structure 129 to mount the spectral camera 120. The spectral camera 120 can capture a spectral image about the inspection target 50. The mounting structure 129 is configured to at least temporarily mount the spectral camera 120 and allow the spectral camera 120 to perform linear movement in at least one direction of the front-back direction, the left-right direction, and the up-down direction and / or to perform rotational movement at a specific angle. Therefore, the melanoma inspection device 100 can adjust the shooting range of the inspection target 50 as needed. For example, the melanoma inspection device 100 can obtain a spectral image of the inspection target 50 including a plurality of lesions by shooting the inspection target 50 at a certain distance. Alternatively, the melanoma inspection device 100 may obtain a spectral image of the inspection target 50 including only one of the plurality of lesions by approaching the inspection target 50 within a certain distance and then photographing the inspection target 50. The melanoma inspection device 100 may analyze the obtained inspection target 50, compare the analysis data with a pre-stored reference model, and output information including whether it is a melanoma and the type of melanoma based on the result. In addition, the melanoma inspection device 100 may obtain the result of the inspection target 50 through the server device 200. In this case, the melanoma inspection device 100 may be configured to send the obtained spectral image to the server device 200 without performing a separate analysis and result derivation on the obtained spectral image, then receive the result from the server device 200, and output the received result. In addition, when the melanoma inspection device 100 is configured to independently perform melanoma inspection and classification and result output without the server device 200, the melanoma inspection device 100 may pre-store the reference model received from the external server device. Alternatively, when the spectral images acquired by the spectral camera 120 and the corresponding inspection records of the inspection target 50 are cumulatively stored by a predetermined number or more, the melanoma inspection device 100 may directly generate a reference model based on unsupervised learning of the stored data. Here, in order to protect the information of the individual inspection records, the information used to generate the reference model may include only the spectral image of the inspection target 50 and the inspection result of the spectral image. Additionally or alternatively, the information used to generate the reference model may include only information other than information that can identify an individual, such as the age, gender, and weight of the inspection target 50.

[0123] The server device 200 may establish a communication channel with the melanoma inspection device 100. The server device 200 may collect at least one spectral image of the inspection target 50 from the melanoma inspection device 100, and analyze the collected at least one spectral image. In this process, the server device 200 may pre-store a reference model for comparative analysis of the currently acquired spectral image. The reference model may be generated by performing unsupervised learning on the spectral image collected and provided by the melanoma inspection device 100, or may be received from a separate external server device that provides the reference model. The server device 200 may provide the melanoma inspection device 100 with the result of the comparative analysis with the reference model. In addition, if the acquired spectral image resolution is insufficient or a spectral image with a higher resolution is required, the server device 200 may request the melanoma inspection device 100 to adjust the resolution of the inspection target 50 having the corresponding lesion, and obtain a spectral image with an improved resolution. In addition, if the melanoma inspection device 100 is designed to directly output the result of whether the inspection target 50 is melanoma and its type, the server device 200 may be omitted from the configuration of the system environment 10.

[0124] As described above, the system environment 10 supporting the melanoma inspection function according to the second embodiment of the present invention can obtain a spectral image of the inspection target 50, and quickly calculate the result of whether the inspection target 50 is a melanoma by comparing it with the pre-learned reference model. In addition, the system environment 10 according to the second embodiment of the present invention can quickly and accurately provide information about the type of melanoma and the corresponding treatment plan according to the learning form of the reference model, and can allow accurate tissue inspection of the lesion as needed, thereby supporting saving time and labor costs for lesion identification.

[0125] Fig.10 is a diagram showing an example of components of a melanoma inspection device according to a second embodiment of the present invention, and Fig.11 It is shown Fig.10 A diagram of an example of components of a processor. Fig.12 is a diagram showing an example of latent vector representation of melanoma data according to the second embodiment of the present invention.

[0126] First, refer to Fig.10, the melanoma inspection device 100 according to the second embodiment of the present invention may include a communication circuit 110, a spectral camera 120, a memory 130, an input unit 140, a display 160, and a processor 150. In addition, the melanoma inspection device 100 may further include a mounting structure 129 for mounting the spectral camera 120, so that at least one spectral camera 120 can capture a spectral image of the inspection target 50. In addition, the melanoma inspection device 100 may further include a power source (e.g., a permanent power source or a battery) required for the operation of at least one of the above-mentioned components (e.g., the communication circuit 110, the spectral camera 120, the memory 130, the input unit 140, the display 160, and the processor 150).

[0127] The communication circuit 110 may establish a communication channel with the server device 200. If the server device 200 is designed to perform the calculations required for the melanoma inspection function according to the second embodiment of the present invention, the communication circuit 110 may send at least one spectral image collected by the spectral camera 120 to the server device 200. In addition, the melanoma inspection function may be independently performed by the melanoma inspection device 100. In this case, the communication circuit 110 may send a message including the inspection result according to the melanoma inspection function to a terminal of an administrator of the melanoma inspection device 100 or a designated user terminal device under the control of the processor 150. Alternatively, the communication circuit 110 may output (or send) the message to the server device 200 under the control of the processor 150. If the melanoma inspection device 100 includes a separate output device (e.g., a display 160 or an audio device), the message may be output through the output device.

[0128] The communication circuit 110 may establish a communication channel with an external server device and receive the first reference model 131a from the external server device. In this regard, the communication circuit 110 may periodically establish a communication channel with the external server device under the control of the processor 150, and if there is a newly updated first reference model 131a, the updated first reference model 131a may be received from the external server device and stored (or updated) in the memory 130.

[0129] The spectral camera 120 may be configured to capture a spectral image of the inspection target 50. At least one spectral camera 120 may be provided, and when a plurality of spectral cameras are provided, the spectral cameras may be arranged to capture the inspection target 50 by dividing the inspection target 50 into regions or to capture the inspection target 50 from different angles. The spectral camera 120 may be activated in response to the control of the processor 150, and when acquiring a spectral image of the inspection target 50, the acquired image may be sent to the processor 150. Alternatively, in response to the control of the processor 150, the spectral image acquired by the spectral camera 120 may be sent to the server device 200 via the communication circuit 110.

[0130] The memory 130 may store at least one program or data required for the operation of the melanoma inspection device 100. For example, the memory 130 may temporarily or semi-permanently store a control program required for operating the at least one spectral camera 120 and a first spectral image 133a acquired by the at least one spectral camera 120. For example, the memory 130 may store a first reference model 131a for comparative analysis with the spectral image of the inspection target 50. The first reference model 131a may be received from an external server device as described above. Alternatively, if a plurality of first spectral images 133a are cumulatively stored in a predetermined number or more, the first reference model 131a may be generated by the operation of the processor 150. In this regard, the memory 130 may store a diagnosis result matched with the first spectral image 133a (e.g., whether the first spectral image 133a is melanoma, information about the type of melanoma, information about at least part of the patient's age, weight, and place of residence, etc.). For example, the first reference model 131a may be generated by performing unsupervised learning on a plurality of first spectral images 133a.

[0131] The input unit 140 may include various input tools for operating the melanoma inspection device 100. For example, the input unit 140 may generate at least one of an input signal for activating the spectral camera 120, an input signal for acquiring the first spectral image 133a through the spectral camera 120, an input signal for inputting a diagnosis result of the acquired first spectral image 133a, and an input signal for requesting output of an analysis result of the first spectral image 133a in response to a user's manipulation. The input unit 140 may include at least one of various tools, such as a soft key (or an input tool based on a touch screen or a touch pad), a physical key, a voice input device, a gesture input device, and a shuttle knob.

[0132] The display 160 may output at least one screen required for operating the melanoma inspection device 100. For example, the display 160 may output at least one of a screen indicating whether at least one device (e.g., at least one of the spectral camera 120, the mounting structure 129, the communication circuit 110, and the input unit 140) connected to the melanoma inspection device 100 is in a normal state, a screen related to activation of the spectral camera 120, a screen related to acquisition of the first spectral image 133a by the spectral camera 120, and a screen related to a melanoma analysis result of the first spectral image 133a. In addition, when the melanoma inspection device 100 is operated in conjunction with the server device 200, the display 160 may output at least one of a screen related to accessing the server device 200, a screen related to an analysis request for the first spectral image 133a, and a screen related to reception of an analysis result of the first spectral image 133a.

[0133] The processor 150 may perform at least one of transmission and processing of signals required for the operation of the melanoma inspection device 100 and storage and output of processing results. For example, the processor 150 may control the spectral camera 120 to acquire the first spectral image 133a, perform comparative analysis between the acquired first spectral image 133a and the first reference model 131a, and output a melanoma inspection result based on the comparative analysis. In this regard, the processor 150 may include, for example, Fig.11 Components shown.

[0134] Reference Fig.11 , the processor 150 may include at least one of a spectral camera controller 151, a spectral image collector 152a, a reference model learner 153a, and a melanoma identifier 154a.

[0135] The spectral camera controller 151a may control at least a portion of the spectral camera 120 or the mounting structure 129 so that the spectral camera 120 can capture the inspection target 50 at a certain resolution or higher. In this regard, the mounting structure 129 may be configured to perform at least one of a linear motion in at least one of a front / rear direction, a left / right direction, and an up / down direction relative to the inspection target 50 and a rotational motion at a specific angle under the control of the spectral camera controller 151a. The spectral camera controller 151a may control the spectral camera 120 to obtain a preview image of the inspection target 50 in which a lesion has occurred, and detect the resolution of the lesion in the preview image. If the resolution of the lesion is lower than a predefined reference value, the spectral camera controller 151a may adjust the mounting structure 129 to change at least one of a distance and a shooting angle between the spectral camera 120 and the inspection target 50. For example, the spectral camera controller 151a may first adjust the position of the spectral camera 120 to shoot in a direction perpendicular to the center point of the inspection target 50, and then automatically adjust the mounting structure 129 so that the resolution of the inspection target 50 is higher than or equal to the predefined reference value. During this process, the spectral camera controller 151a may further adjust at least one of the shooting angle and distance of the spectral camera 120 relative to the inspection target 50 so that the ratio of the size of the lesion on the inspection target 50 where the lesion has occurred to the background size of the lesion is higher than a predetermined value.

[0136] When the spectral camera controller 151a completes the preparation work for photographing the inspection target 50, the spectral image collector 152a may control the spectral camera 120 to acquire at least one spectral image. For example, when the condition that the ratio of the lesion size to the background size in the inspection target 50 is within a predefined value is satisfied, the spectral image collector 152a may control to acquire the spectral image of the inspection target 50. Alternatively, when the resolution of the lesion in the inspection target 50 is greater than or equal to a predefined value, the spectral image collector 152a may control to acquire the spectral image of the inspection target 50. When the first spectral image 133a satisfying the above conditions is accumulated and stored, the melanoma inspection device 100 may perform unsupervised learning to generate the first reference model 131a based on the unified data, thereby obtaining a more reliable learning result.

[0137] The reference model learner 153a can cumulatively store the acquired first spectral images 133a. If the acquired first spectral images 133a accumulate to a predetermined number or more, the reference model learner 153a can use the first spectral images 133a to model unsupervised learning. For example, when the first spectral image 133a is acquired, the reference model learner 153a can separate the background and foreground from the acquired first spectral image 133a (roughly using the K nearest neighbor (KNN) algorithm, etc.). The reference model learner 153a can perform learning of separated spectra in an unsupervised manner (such as a variational autoencoder). The reference model learner 153a can input the spectrum corresponding to the test set (e.g., the MINST dataset) into the learning model and check the representation of the latent vector. At this time, if the dimension of the latent vector exceeds three dimensions, for visualization, the reference model learner 153a can reduce the dimension to three or two dimensions by using methods such as PCA (principal component analysis, t-SNE (t-distributed stochastic neighbor embedding), UMAP (uniform manifold approximation), etc. Fig.12 As shown, the reference model learner 153a can represent the displayed latent vector with different colors according to the predefined lesion information. Here, the representation of different colors can be replaced by the representation of different shapes, different shadows, or different shapes of lines. In relation to the latent vector information, the center point of the latent representation can correspond to the background in the lesion image, and the latent representation displayed by expanding based on the center point can indicate each lesion or normal state. In this case, similar lesions also have similar diffusion directions of latent representations, and lesions close to normal may present a situation similar to the diffusion direction of normal latent representations. Based on this characteristic, the reference model learner 153a can indicate that the farther the lesion is represented from the center, the stronger the intensity of the lesion, and the diagnosis of melanoma or skin lesions can be performed by utilizing the inclusion of known lesion information in the diffusion direction or range. In addition, even if an unknown lesion is input, the reference model learner 153a can provide the possibility of a specific lesion by utilizing the distance (cosine similarity) to the nearest cluster among the clusters of existing lesion latent representations. In connection with performing the above-mentioned operations, by referring to the inspection results of the first spectral image 133a of the diffusion direction for clustering according to the potential characterization results of the latent vector, the reference model learner 153a can input information on whether melanoma exists and its type for the corresponding diffusion direction and generate a first reference model 131a.

[0138] The melanoma identifier 154a may receive the current spectral image of the patient examination target 50 from the spectral image collector 152a. The melanoma identifier 154a may perform clustering by analyzing the spectrum of the current spectral image and compare the clustering result of the spectrum with the first reference model 131a. The melanoma identifier 154a may detect the cluster (or cluster of potential representation) that is most similar to the cluster (or cluster of potential representation) corresponding to the current spectral image from the first reference model 131a, and identify the diffusion direction of the detected cluster. By referring to the result of the diffusion direction and the recorded diagnosis result, the melanoma identifier 154a may output whether the current spectral image is at least one of melanoma and the type of melanoma. In addition, if the melanoma identifier 154a does not detect a cluster whose similarity is within a predefined range in the first reference model 131a, it can detect a cluster with the highest similarity (or a cluster with the shortest distance value between clusters of the reference model) and provide melanoma probability information corresponding to the cluster based on the similarity (or based on the distance value), or recommend a detailed tissue examination.

[0139] The melanoma inspection device 100 of the present invention can non-destructively inspect melanoma more accurately by using a spectral camera. In addition, the melanoma inspection function according to the present invention can reduce the time required for melanoma inspection and eliminate the discomfort caused to the inspected person by skin tissue incision.

[0140] Fig.13 : is a diagram showing an example of a melanoma inspection function of an inspection target according to the second embodiment of the present invention.

[0141] Reference Figures 10 to 13, the processor 150 of the melanoma inspection device 100 according to the second embodiment of the present invention can obtain a spectral image corresponding to the inspection target 50 as shown in the figure. Once the spectral image is obtained, the melanoma inspection device 100 can analyze the inspection target 50 by using at least some wavelengths of the spectrum of the spectral image. For example, the processor 150 of the melanoma inspection device 100 can extract an image by applying some wavelengths of the spectrum of the spectral image of the inspection target 50 (for example, at least some wavelengths within the visible light wavelength range). The processor 150 of the melanoma inspection device 100 can divide the extracted image corresponding to the inspection target 50 into a plurality of regions, including regions where lesions 51a, 52a, 55a, and 56a are located. For example, the processor 150 can divide the image extracted from the inspection target 50 into six regions 51, 52, 53, 54, 55, and 56. The processor 150 may extract some regions 51, 52, 55, and 56 where the lesions 51a, 52a, 55a, and 56a are located among the six regions 51, 52, 53, 54, 55, and 56, and separate the lesion region and the background region for each of the extracted regions 51, 52, 55, and 56. Referring to the illustrated drawings, the processor 150 may separate the first lesion region 51a and the first background region 51b in the first region 51, separate the second lesion region 52a and the second background region 52b in the second region 52, separate the fifth lesion region 55a and the fifth background region 55b in the fifth region 55, and separate the sixth lesion region 56a and the sixth background region 56b in the sixth region 56. In addition, in the above description, the third lesion region and the third background region and the fourth lesion region and the fourth background region are skipped because the regions are numbered in the order of arrangement. However, the present invention is not limited to such numbering of the lesion region and the background region. When numbering is only for lesion areas and background areas, the fifth lesion area and the fifth background area can be referred to as the third lesion area and the third background area, respectively, and the sixth lesion area and the sixth background area can be referred to as the fourth lesion area and the fourth background area, respectively.

[0142] The processor 150 may perform latent vector transformation on each of the lesion regions 51a, 52a, 55a, and 56a and each of the background regions 51b, 52b, 55b, and 56b arranged in the first region 51, the second region 52, the fifth region 55, and the sixth region 56, respectively. The processor 150 may compare the latent vector transformation values ​​of the regions 51, 52, 55, and 56 having lesions with the above Fig.12131a described in order to detect latent vectors having the same or similar directionality. Here, based on the diagnosis result stored in memory 130, the directionality of the latent vectors contained in first reference model 131a can be matched with the presence or absence of melanoma and the type of melanoma. Processor 150 can output at least some information about whether melanoma exists and the type of melanoma for regions 51, 52, 55, and 56 having lesions based on the information previously stored in memory 130 and based on the value compared with first reference model 131a.

[0143] As described above, the processor 150 of the melanoma inspection device 100 can divide the inspection target 50 into multiple areas, distinguish between areas with lesions and areas without lesions, and perform spectral analysis only on areas with lesions among the multiple areas, thereby reducing the computational burden of the inspection target 50, and also providing information about which lesion is melanoma or what type of melanoma is by performing more accurate latent vector comparison on the lesion area.

[0144] Fig.14 200 is a diagram showing an example of components of a server device according to the second embodiment of the present invention. As described above, if the melanoma inspection device 100 is designed to independently perform a melanoma inspection function according to the second embodiment of the present invention, the server device 200 may be omitted.

[0145] Reference Fig.14 , the server device 200 may include a server communication circuit 210 , a server memory 230 , and a server processor 250 .

[0146] The server communication circuit 210 may establish a communication channel with the melanoma inspection device 100. The server communication circuit 210 may receive at least one second spectral image from the melanoma inspection device 100 periodically or in response to the occurrence of a predefined event. The server communication circuit 210 may receive a second reference model 231a from an external server device. The server communication circuit 210 may send the analysis result of the received at least one second spectral image to the melanoma inspection device 100 (or a designated user terminal) under the control of the server processor 250.

[0147] The server memory 230 may store at least one program or data required for the operation of the server device 200. For example, the server memory 230 may store at least one of the second spectral image 233a collected and transmitted by the melanoma inspection device 100 and the second reference model 231a for comparison with the second spectral image 233a. The second spectral image 233a may be an image corresponding to the first spectral image 133a described in the above melanoma inspection device 100. For example, the second spectral image 233a may include a spectral image currently acquired for the inspection target 50. The second reference model 231a may correspond to the first reference model 131a of the above melanoma inspection device 100. For example, the second reference model 231a may be generated by the melanoma inspection device 100 and provided to the server device 200. Alternatively, the second reference model 231a may be generated by the server processor 250 based on a predetermined number or more of the second spectral images 233a stored in the server memory 230.

[0148] The server processor 250 may control transmission and processing of signals required for the operation of the server device 200 , storage or transmission of results, or transmission of messages corresponding to the results. In this regard, the server processor 250 may include a data collector 251 and a melanoma detector 252 .

[0149] The data collector 251 may establish a communication channel with the melanoma inspection device 100 and receive the second spectral image 233a of the inspection target 50 from the melanoma inspection device 100. Here, the server device 200 may be in a state where the second reference model 231a has been pre-stored, and if there is no second reference model 231a, the data collector 251 may receive the second reference model 231a from an external server device. As described above, the second reference model 231a may also be generated by model learning in the server device 200. In this regard, when the server device 200 operates in a mode of learning the second reference model 231a, the data collector 251 may collect various spectral images required for generating the second reference model 231a from the external server device or the melanoma inspection device 100.

[0150] The data collector 251 may receive the second spectral image 233a from the melanoma examination device 100 in association with a request for melanoma examination of the patient examination target 50. In this case, the data collector 251 may store the second spectral image 233a together with identification information of the melanoma examination device 100 in the memory 130 and request the melanoma detector 252 to analyze the second spectral image 233a.

[0151] When the melanoma detector 252 is notified of receiving the second spectral image 233a from the data collector 251, the melanoma detector 252 may perform analysis on the second spectral image 233a stored in the server memory 230. For example, the melanoma detector 252 may compare the spectrum of the second spectral image 233a with the second reference model 231a, detect the directionality of the latent vector for the spectrum of the spectral image of the inspection target 50, and collect the results of whether there is a melanoma corresponding to the detected latent vector directionality and the type of the melanoma. In this process, the melanoma detector 252 may perform the same analysis as described above. Fig.11 The melanoma detector 252 may perform operations similar to the operations of the melanoma identifier 154a of the processor 150 in the above embodiment. For example, when the second spectral image 233a of the inspection target 50 includes a single lesion image, the melanoma detector 252 may perform separation of the background and the foreground (lesion area) in the acquired image. Alternatively, when the second spectral image 233a includes a plurality of lesion images, the melanoma detector 252 may perform separation of the background and the foreground (lesion area) in the acquired image. Fig.13 As shown, the image is divided into individual lesion units, and a separation of the background from the foreground (lesion area) is performed in each area of ​​the individual lesion image. The melanoma detector 252 can apply the separated spectrum to the second reference model 231a to identify the representation of the latent vector. In this operation, if the dimension of the latent vector exceeds three dimensions, for visualization, the melanoma detector 252 can reduce the dimension to three or two dimensions using methods such as PCA, t-SNE, UMAP, etc., and can represent the displayed latent vector in different colors according to the predefined lesion information.

[0152] The melanoma detector 252 may provide the melanoma inspection device 100 with a screen indicating the directionality of the potential vector of the second spectral image 233a. In the screen provided to the melanoma inspection device 100, the center point of the potential representation may correspond to the background in the lesion image, and the potential representation displayed by expanding based on the center point may indicate each lesion or normal state. Here, the melanoma detector 252 may provide a diagnosis result (e.g., text corresponding to information about whether it is a melanoma and the type of melanoma) matching the direction of each potential representation and an inspection result screen. In addition, the melanoma detector 252 may display a numerical value indicating the degree of the lesion according to the distance from the center (e.g., the farther from the center, the stronger the lesion intensity), and when there is no cluster matching the second reference model 231a, a melanoma probability value may be output based on the distance value to the most similar cluster.

[0153] Fig.15 : is a diagram showing an example of a method for operating a melanoma inspection apparatus related to a melanoma inspection method according to a second embodiment of the present invention.

[0154] Reference Fig.15 In the operating method of the melanoma inspection device 100 related to the melanoma inspection method according to the second embodiment of the present invention, the processor 150 of the melanoma inspection device 100 may check in step 701a whether an event of requesting the creation of a reference model occurs. The event of requesting the creation of a reference model may include, for example, an event of inputting a request through the input unit 140 by an administrator of the melanoma inspection device 100 or an event of receiving a request for generating and providing a reference model from the server device 200. Alternatively, the melanoma inspection device 100 may be designed to collect various spectral images related to melanoma and generate corresponding reference models according to predefined scheduling information. If no event related to the creation of a reference model occurs, the processor 150 of the melanoma inspection device 100 may perform a specified function in step 703a. For example, if the reference model has been stored in the memory 130, or if the reference model has been received from an external server device, the processor 150 of the melanoma inspection device 100 may provide a melanoma inspection function based on the reference model stored in the memory 130. The following will be described according to Fig.16 Describe the operations related to the above-mentioned melanoma detection function.

[0155] If an event related to creating a reference model occurs, the processor 150 of the melanoma inspection device 100 may collect various spectral images related to melanoma in step 705a. In this operation, the processor 150 may access an external server device for providing various spectral images related to melanoma, and collect spectral images from the external server device. Alternatively, when the spectral camera 120 connected to the melanoma inspection device 100 photographs the inspection target 50 having a lesion, the processor 150 may store and manage the photographed spectral image of the inspection target 50 as one of the spectral images related to melanoma in the memory 130. Additionally or optionally, the processor 150 may collect the diagnostic results of the acquired spectral images together, perform information matching, and store the matched information in the memory 130.

[0156] The processor 150 may collect a predetermined number or more of various spectral images related to melanoma. Alternatively, the processor 150 may perform the melanoma reference model learning in step 707a each time a spectral image related to melanoma is collected. The melanoma reference model learning may be performed by, for example, separating the lesion area and the background area for each of the various spectral images, and performing unsupervised learning (e.g., variational self-encoder) on the spectrum of each separated area to perform modeling. The processor 150 may check whether the learning is completed in step 709a. If the learning is not completed, the processor 150 may return to step 705a and re-execute subsequent operations. Alternatively, once a new spectral image for learning is acquired, the processor 150 may re-execute steps 705a to 707a. When the learning is completed, the processor 150 may store the reference model in the memory 130 or update the previously stored reference model in step 711a. Alternatively, the processor 150 may provide the reference model to the server device 200 requesting to create the reference model. Thereafter, the processor 150 may return to step 701a and re-execute subsequent operations, or return to step 703a and execute a designated function.

[0157] Fig.16 is a diagram showing another example of a method for operating a melanoma inspection device in relation to the melanoma inspection function according to the second embodiment of the present invention.

[0158] Reference Figures 9 to 16 , in relation to the melanoma inspection function, the processor 150 of the melanoma inspection device 100 may check in step 801 whether spectral image collection is requested for melanoma inspection. In this regard, the melanoma inspection device 100 may activate a spectral image-based melanoma inspection application in response to user input, and activate the spectral camera 120 after activating the application. If spectral image collection is not requested, the processor 150 may perform the specified function in step 803a. For example, the processor 150 may input identification information of the inspection target 50 or basic information of the patient (e.g., age, gender, weight, lesion occurrence time, etc.) in response to user input.

[0159] When requesting to collect a spectral image, the processor 150 may adjust the distance and angle between the spectral camera 120 and the inspection target 50 so that the inspection target 50 has a shooting distance or angle for shooting with a predefined size (or a distance or angle that makes the resolution of a certain area including the lesion become a predefined resolution). In order to adjust the predefined distance and angle, the processor 150 may control the spectral camera 120 to acquire a preview image, and by analyzing the acquired preview image, adjust at least one of the distance and angle between the spectral camera 120 and the inspection target 50 so that the size of the background area including the lesion becomes a predefined size. When the preparation work is completed, the processor 150 may control the spectral camera 120 to acquire a spectral image of the inspection target 50. Subsequently, in step 805a, the processor 150 may analyze the spectral image and determine whether it is a single lesion.

[0160] If the spectral image includes a single lesion in step 805a, the processor 150 may perform latent vector analysis on the single lesion in step 807a. In this regard, the processor 150 may apply the acquired current spectral image to a reference model pre-stored in the memory 130 to detect latent representations.

[0161] In step 809a, the processor 150 may analyze the detected potential signature and output a determination result as to whether the lesion included in the current spectral image is a melanoma and / or the type of melanoma.

[0162] On the other hand, if the spectral image includes a plurality of lesions in step 805a, the processor 150 may process lesion segmentation in step 811a. In this process, the processor 150 may divide the spectral image into a plurality of regions so that a plurality of lesions distributed in the spectral image have predefined sizes. In this process, the processor 150 may divide the spectral image in a grid shape, or divide the spectral image based on a mosaic including a lesion region and a background region of a constant size containing the lesions.

[0163] When the lesion segmentation is completed, the processor 150 may perform a latent vector analysis on the segmented lesions in step 813a. In this process, the processor 150 may call a reference model pre-stored in the memory 130, and apply the data about the plurality of lesions to the reference model in sequence. Alternatively, the processor 150 may extract the latent vector by applying each data to the reference model in the order from relatively small lesions to relatively large lesions among the plurality of lesions. Thereafter, the processor 150 may return to step 809a to determine whether each of the plurality of lesions is a melanoma and the type of melanoma, and output the determined result. In this regard, the processor 150 may output the determined result to a display 160 connected to the melanoma inspection device 100, or output the result to a predefined display device or user terminal.

[0164] Additionally or alternatively, the processor 150 may output a screen interface on the display 160, through which the medical professional may input an inspection result of at least one of information on whether or not melanoma exists and the type of melanoma, and in response to the input of the medical professional, it may be determined whether or not melanoma exists in the inspected spectral image and the type of melanoma. The melanoma inspection device 100 may maintain the result of the current spectral image as a potential result until the medical professional makes a determination.

[0165] In step 815a, the processor 150 of the melanoma inspection device 100 may check whether the melanoma inspection is completed. If an event corresponding to the completion of the melanoma inspection occurs, such as an input of completion of the inspection by a medical professional or a request to deactivate the spectral camera 120, the melanoma inspection may be completed. If no event related to the completion of the melanoma inspection occurs, the process may return to step 801a and re-execute subsequent operations.

[0166] In addition, although the above description describes that the melanoma inspection device 100 performs the operation of acquiring a spectral image of the inspection target 50 and determining whether the lesion is a melanoma and / or the type of melanoma, the present invention is not limited thereto. Fig.16The operations described in step 801a may be performed by the server device 200. In this case, the server device 200 may establish a communication channel with the melanoma inspection device 100 before step 801a, and then check whether the spectral image is received in step 801a. When the spectral image is received, the server device 200 may perform the operation of step 805a on the received spectral image, determine whether the lesion is melanoma and / or the type of melanoma in step 809a, and send the determined result back to the melanoma inspection device 100. In this process, the server device 200 may store and manage the identification information of the melanoma inspection device 100 that provides the spectral image and the received spectral image, so that the determined result can be sent to the corresponding melanoma inspection device 100.

[0167] As described above, the melanoma inspection function based on spectral images according to the second embodiment of the present invention can determine whether melanoma exists based on the spectral image of the skin suspected of melanoma. When inspecting a single lesion image, the background and foreground are separated from the acquired image (roughly using KNN, etc.), and when inspecting multiple lesion images, the image is divided into single lesion units, and the single lesion image is separated into background and foreground, and then the separated spectrum (spectral image) can be learned in an unsupervised manner. The melanoma inspection function according to the present invention can use the reference model that has completed learning to input the spectrum of the currently acquired spectral image, check the representation of the latent vector, and provide at least one result of whether melanoma exists and the type of melanoma. Based on this operation, the melanoma inspection function of the present invention can provide melanoma detection results for lesions in a non-destructive manner, thereby reducing patient rejection reactions and providing faster and more accurate inspection results.

[0168] While the specification contains many specific implementation details, these should not be construed as limitations on the scope of the disclosure or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular disclosures.

[0169] In addition, although the description describes the operations being performed in a predetermined order with reference to the accompanying drawings, it should not be interpreted as requiring the operations to be performed sequentially or in a predetermined order shown to obtain a preferred result, or requiring all operations shown to be performed. In some cases, multitasking and parallel processing may be advantageous. In addition, it should not be interpreted as requiring the various system components to be divided in all types of implementations. It should be understood that the described program components and systems are generally integrated into a single software product or packaged as multiple software products.

[0170] The description shows the best mode of the present disclosure, and provides examples to illustrate the present disclosure and enable those skilled in the art to make and use the present disclosure. The present disclosure is not limited by the specific terms used herein. Based on the above embodiments, those of ordinary skill in the art can modify, change or alter the embodiments without departing from the scope of the present disclosure.

[0171] Therefore, the scope of the present disclosure should not be limited by the described embodiments, but should be defined by the appended claims.

[0172] Description of Reference Numerals

[0173] 10: System environment

[0174] 50: Check target

[0175] 100: Melanoma Identification Device

[0176] 110: Communication circuit

[0177] 120: Spectral Camera

[0178] 130: Memory

[0179] 140: Input unit

[0180] 150: Processor

[0181] 160: Display

[0182] 200: Server device

[0183] 210: Server communication circuit

[0184] 230: Server storage

[0185] 250: Server processor

Claims

1. A melanoma identification device based on spectral images, the device comprising: a spectral camera that acquires a spectral image of an inspection target including at least one skin lesion; as well as A processor functionally connected to the spectral camera and configured to: controlling the spectral camera to acquire a current spectral image of the inspection target, Separating a foreground area where the at least one skin lesion has occurred and a background area other than the foreground area from the current spectral image, and At least a portion of the separated foreground area and background area is compared with a pre-stored reference model to output a melanoma identification result of the inspection object.

2. The device according to claim 1, wherein: The processor is configured to: By applying a nearest neighbor technique to the current spectral image, an adjacency matrix is ​​detected based on a plurality of dimensional vectors and edges corresponding to distance values ​​between the plurality of dimensional vectors, and The foreground area is extracted based on edges in the adjacency matrix that are greater than or equal to a predefined reference value.

3. The device according to claim 2, wherein: The processor is configured to: extracting from the reference model a cluster whose similarity to the cluster corresponding to the spectrum of the foreground area is higher than or equal to a predefined reference value, and The melanoma identification result of the inspection target is output based on the cluster extracted from the reference model.

4. The device according to claim 3, wherein: The processor is configured to: detecting directionality of the cluster corresponding to the spectrum of the foreground region; Extracting from the reference model a cluster having a directionality similar to the detected directionality within a certain range; and The melanoma identification result of the inspection target is output based on the cluster extracted from the reference model.

5. The device according to claim 3, wherein: The processor is configured to: After visualization the adjacency matrix is ​​output to a display.

6. The device according to claim 1, wherein: The processor is configured to: The melanoma identification result is output through cross-validation between a melanoma identification result based on the foreground region and a melanoma identification result based on both the foreground region and the background region.

7. A melanoma identification method based on spectral images, the method comprising the following steps: controlling the spectral camera by a processor of the melanoma identification device to acquire a current spectral image of an inspection target including at least one skin lesion; Separating, by the processor, from the current spectral image a foreground area where the at least one skin lesion has occurred and a background area other than the foreground area; as well as The processor compares at least a portion of the separated foreground area and background area with a pre-stored reference model to output a melanoma identification result of the inspection target.

8. The method according to claim 7, wherein: The step of separating the foreground area and the background area comprises the following steps: detecting an adjacency matrix based on a plurality of dimensional vectors and edges corresponding to distance values ​​between the plurality of dimensional vectors by applying a nearest neighbor technique to the current spectral image; and The foreground area is extracted based on edges in the adjacency matrix that are greater than or equal to a predefined reference value.

9. The method according to claim 8, wherein: The step of outputting the melanoma identification result comprises one of the following steps: extracting from the reference model a cluster whose similarity to the cluster corresponding to the spectrum of the foreground area is higher than or equal to a predefined reference value, and outputting the melanoma identification result of the inspection target based on the cluster extracted from the reference model; or Directivity of a cluster corresponding to the spectrum of the foreground area is detected, clusters having similar directivity within a certain range to the detected directivity are extracted from the reference model, and a melanoma identification result of the inspection target is output based on the cluster extracted from the reference model.

10. The method according to claim 7, wherein: The step of outputting the melanoma identification result comprises: The melanoma identification result is output through cross-validation between a melanoma identification result based on the foreground region and a melanoma identification result based on both the foreground region and the background region.

11. A server device for supporting melanoma identification based on spectral images, the server device comprising: a server communication circuit that establishes a communication channel with the melanoma identification device; as well as A server processor functionally connected to the server communication circuit and configured to: receiving a current spectral image of an inspection target including at least one skin lesion from the melanoma identification device; Separating a foreground area where the at least one skin lesion has occurred and a background area other than the foreground area from the current spectral image; By comparing at least a portion of the separated foreground region and background region with a pre-stored reference model, the inspection object is subjected to melanoma identification; and The melanoma identification result is sent to the melanoma identification device.

12. The device according to claim 11, wherein The server processor is configured to: The melanoma identification is performed by cross-validation between a melanoma identification result based on the foreground region and a melanoma identification result based on both the foreground region and the background region.

13. A melanoma detection device supporting a melanoma detection function based on spectral images, the device comprising: a spectral camera that acquires a spectral image of an inspection target including at least one skin lesion; as well as A processor functionally connected to the spectral camera and configured to: controlling the spectral camera to acquire a current spectral image of the inspection target, detecting a latent vector by applying the current spectral image to a pre-stored reference model, and Whether the inspection target is melanoma is determined based on the directionality and form of the latent representation corresponding to the latent vector.

14. The device according to claim 13, wherein: The processor is configured to: The spectral camera is controlled so that a shooting angle and a distance of the spectral camera relative to the inspection target are predefined shooting angles and distances.

15. The device according to claim 13, wherein: The processor is configured to: The type of melanoma of the inspection target is determined based on the directionality and form of the potential representation, and information on the determined type of melanoma is output.

16. The device according to claim 15, wherein: The processor is configured to: It is determined that the skin lesions in the current spectral image are represented as being farther from the center of the potential representation, the stronger the intensity of the skin lesions.

17. The device according to claim 13, wherein: The processor is configured to: When the current spectral image contains multiple skin lesions, the image is divided into regions including the multiple skin lesions, each skin lesion in the divided region and a background region of a predetermined size surrounding each skin lesion are separated, and melanoma inspection is performed on each of the separated skin lesions and background regions in turn.

18. The device according to claim 13, wherein: The processor is configured to: Extract clusters of the spectrum of the current spectral image, detect from the reference model a cluster of potential representations that is most similar to the cluster corresponding to the current spectral image, determine whether the cluster corresponding to the current spectral image is melanoma and the type of melanoma based on the cluster detected from the reference model, and output the determined information.

19. A melanoma detection method based on spectral images, the method comprising the following steps: controlling the spectral camera by a processor of the melanoma inspection device to acquire a current spectral image of an inspection target including a skin lesion; detecting a latent vector by applying the current spectral image to a pre-stored reference model; and Based on the directionality and the form of the latent representation corresponding to the latent vector, it is determined whether the inspection target is at least one of a melanoma and a type of the melanoma.

20. The method according to claim 19, further comprising the steps of: The output includes information of at least one of whether the determined inspection target is a melanoma and a type of the melanoma.

21. The method according to claim 19, wherein: The step of detecting the potential vector comprises: When the current spectral image includes a plurality of skin lesions, dividing the image into regions including the plurality of skin lesions; separating each skin lesion in the divided area and a background area of ​​a predetermined size surrounding each skin lesion; and The latent vector is detected for each of the separated skin lesions and background regions in turn.

22. The method according to claim 19, wherein: The step of determining whether the inspection target is melanoma and at least one of the type of melanoma comprises the following steps: Extracting a cluster of spectra of the current spectral image; detecting from the reference model a cluster of latent representations that is most similar to a cluster corresponding to the current spectral image; and Based on the clusters detected from the reference model, it is determined whether the cluster corresponding to the current spectral image is a melanoma and the type of the melanoma.

23. A server device supporting melanoma detection based on spectral images, the device comprising: a server communication circuit that establishes a communication channel with a melanoma inspection device; as well as A server processor functionally connected to the server communication circuit and configured to: receiving a current spectral image of an inspection target including skin lesions from the melanoma inspection device; detecting a latent vector by applying the current spectral image to a pre-stored reference model; determining whether the inspection target is at least one of a melanoma and a type of melanoma based on a directionality and a form of a latent representation corresponding to the latent vector; and The determined information is sent to the melanoma inspection device.

24. The device according to claim 23, wherein: The server processor is configured to: Extracting a cluster of spectra of the current spectral image; detecting from the reference model a cluster of latent representations that is most similar to a cluster corresponding to the current spectral image; and Based on the clusters detected from the reference model, it is determined whether the cluster corresponding to the current spectral image is a melanoma and the type of the melanoma.

Images