Apparatus and method for generating infection status information based on image information

Abstract

According to various embodiments of the present invention, by applying an apparatus and method for generating infection state information based on image information, an input of an image of a diagnostic kit used by a user is received, and an image input to a deep learning model learned using a dataset obtained by expanding a diagnostic kit image in various ways is applied. From the input image, infection state information regarding a disease of the user can be generated and provided.

Description

Technical Field

[0001] The present invention relates to an infection status information generation device and method. More specifically, the present invention relates to a device and method for generating infection status information based on input image information. The research of the present invention was conducted as a source technology development project "High-sensitivity home kit for COVID-19 on-site screening with enhanced preprocessing (No. 2021M3E5E3080741)" which is funded by the Korea Research Foundation sourced from the Ministry of Science and ICT of Korea.

Background Art

[0002] The content described in this section is merely for providing background information of the present embodiment and does not constitute prior art.

[0003] A diagnostic kit is a test device made using a chemical reaction for quickly and easily diagnosing a specific disease. A diagnostic kit is an accurate test device designed to quickly diagnose the presence or absence of infection without the need for a special testing facility or an expert, unlike other highly reliable diagnostic methods. Diagnostic kits have advantages such as being inexpensive, usable anywhere, easy to use, and providing a diagnostic result within 15 to 30 minutes.

[0004] After a user uses a diagnostic kit, the result lines that appear in the test line area and the control line area of the diagnostic kit window are often not clearly distinguishable by the naked eye, and there is a problem that it is difficult for non-expert users to determine whether they are positive.

[0005] Even when the result line that appears in the test line area is difficult to distinguish by the naked eye, it is necessary to research and develop a technology that can generate and provide infection status information so that even general non-expert users can clearly recognize whether they are positive.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] The present invention receives an input of an image of a diagnostic kit used by a user, applies the image input to a deep learning model learned using a dataset obtained by expanding the diagnostic kit image in various ways, and from the input image, generates infection status information for the user's disease, and provides an apparatus and method for generating infection status information based on the provided image information.

[0008] Other unspecified objects of the present invention can be further considered within the range that can be easily inferred from the following detailed description of the invention and its effects.

Means for Solving the Problems

[0009] To achieve the above problems, a computing device for generating infection status information based on image information according to an embodiment of the present invention includes a memory storing at least one program for generating infection status information, and at least one processor for executing an operation for generating the infection status information according to the at least one program, and the operation executed by the processor includes receiving an input image of a diagnostic kit to which a sample extracted from an inspection target is applied, determining a main region related to the inspection target in the input image, and determining a sub-region surrounding the main region so as to include the main region, and applying the sub-region to a machine-learned model to generate and output the infection status information for the inspection target.

[0010] The main region is a region including a test line whose color changes according to the infection state of the object to be inspected.

[0011] The infection state information includes infection information indicating whether the object to be inspected is infected or the concentration of the detection target contained in the object to be inspected predicted based on the color change of the test line.

[0012] The step of receiving the input image for the diagnostic kit is a step of receiving a plurality of input images including the input image taken at the current time and the input image taken at the previous time, and the step of generating and outputting the infection state information is to generate and output information regarding the transition of the infection state using the concentrations of a plurality of detection targets generated based on the plurality of input images.

[0013] The step of generating and outputting the infection state information provides a first message regarding the current infection state when the concentration of the detection target at the current time has increased compared to the concentration at the previous time, and provides a second message regarding the current infection state when the concentration of the detection target at the current time has decreased compared to the concentration at the previous time.

[0014] The machine learning model is trained using an extended dataset including a plurality of training data and transformed training data transformed based on the plurality of training data.

[0015] The conversion training data includes at least one selected from the group consisting of first conversion training data obtained by blurring the training data, second conversion training data obtained by changing the size of the training data, third conversion training data generated by distorting the training data, fourth conversion training data obtained by rotating the training data, fifth conversion training data obtained by adjusting the brightness of the training data, sixth conversion training data obtained by changing the surrounding environment of the training data, and seventh conversion training data obtained by adjusting the color temperature of the training data.

[0016] The step of determining the sub-region is characterized in that it includes the region corresponding to the test line and excludes the region corresponding to the control line.

[0017] The step of generating and outputting the infection status information is characterized in that it determines whether the diagnostic kit is normal based on the region corresponding to the control line, and generates and outputs information regarding whether it is normal.

[0018] The processor starts a mobile application and executes the operation by the started mobile application.

[0019] To achieve the above object, an infection status information generation method executed by a computing device that generates infection status information based on image information according to an embodiment of the present invention may include receiving an input image of a diagnostic kit to which a sample extracted from an inspection target is applied, determining a main region related to the inspection target in the input image, determining a sub-region that surrounds the main region and includes the main region, and applying the sub-region to a machine-learned model to generate and output the infection status information for the inspection target.

[0020] The main area is an area including a test line whose color changes according to the infection status of the object to be inspected.

[0021] The infection status information includes infection information indicating whether the object to be inspected is infected or the concentration of the detection target included in the object to be inspected predicted based on the color change of the test line.

[0022] The step of determining the sub-area includes an area corresponding to the test line and is determined to exclude an area corresponding to a control line.

[0023] In order to achieve the above object, a computer program according to an embodiment of the present invention is stored in a computer-readable recording medium and executes one of the infection status information generation methods on a computer.

Advantages of the Invention

[0024] As described above, according to an embodiment of the present invention, by applying an apparatus and a method for generating infection status information based on image information, an input of an image of a diagnostic kit used by a user is received, and an image input to a deep learning model learned using a data set obtained by expanding a diagnostic kit image in various ways is applied, and infection status information of the user with respect to a disease can be generated and provided from the input image.

[0025] Even when the effects are not explicitly described herein, the following effects described in the specification expected from the technical features of the present invention and their provisional effects are treated in the same manner as those described in the specification of the present invention.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 7G

Figure 7H

Figure 7I

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Mode for Carrying Out the Invention

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The advantages, features, and methods for achieving them of the present invention will become clear by referring to the embodiments described in detail later together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and can be realized in various forms. These embodiments are provided only to enable those with ordinary knowledge in the technical field to which the present invention belongs to fully understand the scope of the invention so that the disclosure of the present invention is complete. Therefore, the present invention is defined only by the scope of the claims. Unless otherwise specified, all terms (including technical and scientific terms) used in this specification are used in a meaning commonly understood by those with ordinary knowledge in the technical field to which the present invention belongs. Also, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless otherwise clearly stated.

[0028] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as "have", "may have", "include", and "may include" in this application indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and it should be understood that they do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms including ordinal numbers such as the second and the first may be used to describe various components, but the components are not limited by such terms.

[0029] The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the rights of the present invention, the second component can be named the first component, and similarly, the first component can be named the second component. The term "and / or" includes any combination or any one of a plurality of related listed items.

[0030] Hereinafter, with reference to the accompanying drawings, various embodiments of an apparatus and a method for generating infection status information based on image information according to the present invention will be described in detail.

[0031] FIG. 1 is an explanatory diagram regarding the configuration of an apparatus for generating infection status information based on image information according to an embodiment of the present invention and an infection status information generation system including the apparatus for generating infection status information based on image information.

[0032] The infection status information generation system 10 may include a computing device 100 and a server 200.

[0033] The apparatus for generating infection status information based on image information may be a computing device configured to generate infection status information based on image information.

[0034] Referring to FIG. 1, a computing device 100 (hereinafter referred to as "computing device") configured to generate infection status information based on image information may include a processor 110, a memory 120, a network interface 130, and a machine learning model 140.

[0035] The processor 110 may control the computing device to operate as a device that generates infection status information based on image information. For example, the processor 110 executes at least one program stored in the memory 120. The at least one program may include at least one computer-executable instruction statement.

[0036] The memory 120 is configured to store computer-executable instruction statements or program codes, program data, and / or other appropriate forms of information. The computer-executable instruction statements or program codes, program data, and / or other appropriate forms of information may be provided by the network interface 130. The program stored in the memory 120 includes a set of instruction statements 122 executable by the processor 110. In one embodiment, the memory 120 may be a memory (such as a volatile memory like a random access memory, a non-volatile memory, or a suitable combination thereof), at least one magnetic disk storage device, an optical disk storage device, a flash memory device, or other forms of storage media that can be accessed by the computing device 100 and store the desired information, or a suitable combination thereof. The memory 120 may store at least one program for generating infection status information.

[0037] The memory 120 may store data 121 that can be accessed by at least one processor. For example, the memory 120 can acquire, receive, access, write, generate, or store data that can be operated on by the processor 110.

[0038] Memory 120 may store instructions 122 provided in the form of a set of computer-executable instruction words. The instructions 122 may be stored in the form of a program and, when executed by the processor 110, may be configured to cause the computing device 100 to perform operations according to an exemplary embodiment.

[0039] The processor 110 may be interconnected with various other components of the computing device 100 via a communication bus including the memory 120.

[0040] The computing device 100 may also include at least one network interface 130 that provides an interface for at least one input / output device. The network interface 130 may operate as an input / output interface and a communication interface. The network interface 130 may be connected to the communication bus. Input / output devices (not shown) may be connected to other components of the computing device 100 via the network interface 130.

[0041] The computing device 100 may store or include at least one machine learning model 140. For example, the machine learning model 140 may include or store various machine learning models such as neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, linear models, and / or other types of models including non-linear models.

[0042] The neural network may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks (CNNs), and / or other forms of neural networks, or combinations thereof.

[0043] Referring to FIG. 1, the server 200 may include a processor 210, a memory 220, a network interface 230, a machine learning model 240, a model learning unit 250, and learning data 260.

[0044] The computing device 100 and the server 200 may follow a client - server relationship.

[0045] The processor 210 may be controlled to operate as a server of a device that generates infection status information based on image information. For example, the processor 210 executes at least one program stored in the memory 220. The at least one program may include at least one computer - executable instruction statement.

[0046] The memory 220 is configured to store computer - executable instruction statements or program codes, program data, and / or other appropriate forms of information. The computer - executable instruction statements or program codes, program data, and / or other appropriate forms of information may be provided by the network interface 230. The program stored in the memory 220 includes a set of executable instruction statements 222 by the processor 210. In one embodiment, the memory 220 may be a memory (such as a volatile memory like a random - access memory, a non - volatile memory, or a suitable combination thereof), at least one magnetic disk storage device, an optical disk storage device, a flash memory device, or other forms of storage media that can be accessed by the server 200 and store the desired information, or a suitable combination thereof.

[0047] The memory 220 may store data 221 that can be accessed by at least one processor. For example, the memory 220 can acquire, receive, access, write, generate, or store data that can be operated on by the processor 210.

[0048] Memory 220 may store instructions 222 provided in the form of a set of computer-executable instruction words. The instructions 222 may be stored in the form of a program and, when executed by the processor 210, may be configured to cause the server 200 to perform operations according to an exemplary embodiment.

[0049] The processor 210 may be interconnected with various other components of the server 200 via a communication bus including the memory 220.

[0050] The server 200 may also include at least one network interface 230 that provides an interface for at least one input / output device. The network interface 230 may operate as an input / output interface and a communication interface. The network interface 230 may be connected to the communication bus. Input / output devices (not shown) may be connected to other components of the server 200 via the network interface 230.

[0051] The server 200 may store or include at least one machine learning model 240. For example, the machine learning model 240 may include or store various machine learning models such as neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, linear models, and / or other types of models including non-linear models.

[0052] The server 200 may include at least one server computing device. Such server computing devices may, for example, be parallel computing architectures or combinations thereof, and may operate according to various computing architectures including sequential computing architectures.

[0053] The model learning unit 250 may train the machine learning models 140 and 240. The model learning unit 250 may use at least one training or learning algorithm. The model learning unit 250 may perform training using the labeled learning dataset 260, but is not necessarily limited thereto, and may perform training using the unlabeled learning dataset 260.

[0054] The learning data 260 may include a learning dataset necessary for the model learning unit 250 to learn the machine learning models 140 and 240.

[0055] The server 200 may communicate with the computing device 100 via the network 300.

[0056] The network 300 may be any type of network or a combination of networks that enable communication between devices. The network 300 may include at least one of a local area network, a wide area network, the Internet, a security network, a cellular network, a mesh network, and a peer-to-peer communication link.

[0057] The processor 110 may receive an input image of a diagnostic kit to which a sample extracted from the inspection target is applied, determine a main region related to the inspection target in the input image, determine a sub-region that surrounds and includes the main region, and generate and output infection status information for the inspection target by applying the sub-region to the machine learning model.

[0058] Here, the inspection target may include all humans, animals, and plants.

[0059] The sample is collected from the inspection target and may be saliva, sputum, nasal discharge, blood, urine, etc. of the inspection target, or may be something that can be applied to the diagnostic kit after being mixed with other substances, or may be something that can be directly applied to the diagnostic kit as the sample itself.

[0060] The server 200 may collect and manage the infection status information from the computing device 100. Based on the infection status information collected from the computing device 100, the symptoms of the user can be accurately analyzed and managed in a hospital or a specialized institution.

[0061] The server 200 may use a database (not shown) that stores and manages not only the infection status information collected from the computing device 100, but also the personal information of the user and the information analyzed by experts.

[0062] Not all of the blocks shown in FIG. 1 are essential components. In other embodiments, some of the blocks connected to the system 10 including the device 100 that generates the infection status information based on the image information and the device that generates the infection status information based on the image information may be added, changed, or deleted.

[0063] FIG. 2 is an explanatory diagram regarding the determination of the main area and the sub - area by the device that generates the infection status information based on the image information according to an embodiment of the present invention.

[0064] Reference numeral 400 represents a diagnostic kit. The diagnostic kit may be a device used to determine whether a user is infected with a disease by a sample collected from the user or a sample mixed with the test subject. The diagnostic kit may be, for example, a diagnostic kit for COVID - 19.

[0065] Reference numeral 410 indicates a window where an indication line (for example, a test line, a control line) showing the test result appears after the user uses the diagnostic kit.

[0066] The processor 110 may determine the main area related to the test subject by detecting the area corresponding to the test line 411 of the window 410 where the test result of the diagnostic kit 400 appears.

[0067] The main area may be an area including a test line whose color changes according to the infection status of the object to be inspected.

[0068] The processor 110 may determine a sub-region in such a manner that it includes the region corresponding to the test line 411 and excludes the region corresponding to the control line 413.

[0069] The processor 110 may determine a sub-region to include a part of the region corresponding to the window 410 excluding the region corresponding to the control line 413 and the region corresponding to the outside of the region corresponding to the window 410, and at least a part of the region corresponding to the diagnostic kit 400, and include the region corresponding to the test line 413.

[0070] For example, the processor 110 may determine, as the sub-region, the region where the test line 413 is located among the regions corresponding to half of the region corresponding to the window 410, and may further determine the sub-region to include the region corresponding to the diagnostic kit 400.

[0071] The processor 110 may determine whether the object to be inspected is positive based on the degree of concentration of the test line 411. The processor 110 may generate and output infection status information including information regarding whether the object to be inspected is positive.

[0072] The infection status information may include infection information indicating whether the object to be inspected is infected, or the concentration of the detection target included in the object to be inspected predicted based on the color change of the test line 411. The color change may be determined based on the degree of concentration of the test line.

[0073] The processor 110 may generate concentration information of the detection target included in the object to be inspected predicted based on the degree of concentration of the test line 411.

[0074] The degree of concentration can be divided into predetermined reference levels according to a preset reference. For example, the degree of concentration can be divided into a total of 11 levels from 0 to 10 levels, from low to high degrees of concentration. The processor 110 may determine that it is positive if the degree of concentration of the test line 411 is equal to or higher than a predetermined threshold level (for example, 3 levels). The degree of concentration may be chroma.

[0075] The detection target may be various types of viruses or bacteria such as HIV (Human Immunodeficiency Virus), COVID-19 virus, and influenza virus.

[0076] The processor 110 may generate and output infection status information including the generated concentration information of the detection target.

[0077] The processor 110 may predict the concentration of the detection target in a form corresponding to the degree of concentration of the detection target, and generate concentration information of the detection target.

[0078] For example, if the processor 110 determines that the degree of concentration of the test line 411 is 1 level, it predicts that the concentration of the detection target is 10 ng / mL, and if it determines that the degree of concentration of the test line 411 is 10 levels, it predicts that the concentration of the detection target is 100 ng / mL. That is, the concentration interval may be 10 ng / mL for each unit level interval.

[0079] FIG. 3 is a diagram showing a machine learning model used in an apparatus for generating infection status information based on image information according to an embodiment of the present invention.

[0080] Referring to FIG. 3, the infection status information generation method according to the present invention may include an object detection process of detecting a diagnostic kit from an input image and a process of classifying the presence or absence of a detection target (that is, positive or negative), or the degree of concentration of a test line, from a main region or a sub-region portion determined from the input image.

[0081] FIG. 4 is an explanatory diagram regarding a diagnostic kit image used as a dataset of an apparatus for generating infection state information based on image information according to an embodiment of the present invention.

[0082] FIG. 4 may be a concept corresponding to reference numeral 501 in FIG. 3.

[0083] Reference numeral 601 in FIG. 4 shows a case where the entire diagnostic kit is cropped from the input image.

[0084] Reference numeral 602 in FIG. 4 shows that both the test line and the control line are cropped from the input image.

[0085] Reference numeral 603 in FIG. 4 shows a case where all the test lines are cropped from the input image and the control line is not included.

[0086] FIGS. 4(a) and (b) are graph data showing that when corresponding to reference numeral 603, the computing device 100 can more accurately determine the presence or absence of infection than when corresponding to reference numeral 602, and when corresponding to reference numeral 602, the computing device 100 can more accurately determine the presence or absence of infection than when corresponding to reference numeral 601.

[0087] FIG. 5 is an explanatory diagram regarding a deep learning model and dataset expansion used in an apparatus for generating infection state information based on image information according to an embodiment of the present invention.

[0088] FIG. 5(a) is a diagram showing RMSD (Root Mean Square Deviation) by a CNN model.

[0089] FIG. 5(b) is a diagram showing the diagnostic accuracy of the inspection target of the computing device 100 according to the type of dataset.

[0090] Dataset #1 is the case where diagnostic kit images taken under a single environment and single lighting are used as the dataset. Dataset #2 is the case where Dataset #1 is extended as shown in Fig. 5(c). Dataset #3 is the case where Dataset #2 further includes diagnostic kit images taken under various lighting conditions. Dataset #4 is the case where Dataset #3 further includes diagnostic kit images taken under various lighting conditions and various environments.

[0091] That is, it can be confirmed that the performance of the computing device 100 improves as the variety of images included in the dataset increases.

[0092] In the present invention, the machine learning model used by the computing device 100 may be trained using an extended dataset including a plurality of training data and transformed training data transformed based on the plurality of training data. In the present invention, the machine learning model used by the computing device 100 may be a deep learning model.

[0093] Referring to Fig. 5(c), the training data 260 may include an extended dataset obtained by using at least one method among a method of blurring, a method of converting the size, a method of distorting, a method of rotating, a method of adjusting the brightness, a method of changing the surrounding environment, and a method of adjusting the color temperature, of the diagnostic kit images before and after use, which are training images, in order to increase the number of datasets.

[0094] That is, the transformed training data may include at least one selected from the group consisting of first transformed training data obtained by blurring the training data, second transformed training data obtained by changing the size of the training data, third transformed training data generated by distorting the training data, fourth transformed training data obtained by rotating the training data, fifth transformed training data obtained by adjusting the brightness of the training data, sixth transformed training data obtained by changing the surrounding environment of the training data, and seventh transformed training data obtained by adjusting the color temperature of the training data.

[0095] FIG. 6 is a diagram showing data comparing the performance of an apparatus for generating infection state information based on image information according to an embodiment of the present invention with the discrimination capabilities of experts and ordinary people in using a diagnostic kit.

[0096] FIG. 6(a) is a diagram showing a receiver operating characteristic (ROC) curve indicating the prediction performance of a non-expert, an expert, and the computing device 100 with respect to the test result determination performance after a user uses a diagnostic kit.

[0097] FIG. 6(b) is a diagram showing experimental data for confirming cross-reactivity.

[0098] FIG. 6(c) is data showing the concentration prediction performance of a detection target. It can be confirmed that the concentration prediction performance is excellent because the color corresponding to the actual concentration is similar to the predicted color.

[0099] FIG. 7 is a diagram showing data comparing the performance of an apparatus for generating infection state information based on image information according to an embodiment of the present invention with the discrimination capabilities of experts and ordinary people.

[0100] Figures 7(b) to 7(e) are diagrams showing the results of a blind test on the ability of the computing device 100, experts, and non-experts to recognize the positivity of a diagnostic kit used in the nasal method.

[0101] In the case of non-experts, the accuracy was 46.4% and the error was 7.1. In the case of experts, the accuracy was 63.2% and the error was 13.7. In the case of the present invention, the accuracy was 97.5%.

[0102] Figure 7(d) is a diagram showing the images correctly answered by the general public in the blind test, the images correctly answered only by the experts and the computing device 100, and the images correctly answered only by the computing device 100.

[0103] Figure 7(e) is a diagram showing the results of image processing of the second image that non-expert members of the general public could not distinguish among the images in Figure 7(d) and the third image that neither non-experts nor experts could distinguish.

[0104] In the case of the second image that non-expert members of the general public could not distinguish but that experts and the computing device 100 could distinguish, although it was not easy to distinguish visually, a faint coloration that could be recognized by experts appeared. When image processing was performed by adjusting the brightness and contrast, a slight test line was confirmed.

[0105] In the case of the third image that neither non-experts nor experts could distinguish but that the computing device 100 could distinguish, since it was difficult to detect visually, it was confirmed that there was no difference even after image processing. That is, even when a test line of a density level that cannot be detected visually appears, the computing device 100 can recognize this and determine that the test subject is positive.

[0106] Figures 7(f) to 7(i) are diagrams showing the results of a blind test on the ability of the computing device 100, experts, and non-experts to recognize the positivity of a diagnostic kit used in the saliva method.

[0107] In the case of non-experts, the accuracy was 85% and the error was 7. In the case of experts, the accuracy was 92.5% and the error was 5.6. In the case of the present invention, the accuracy was 100%.

[0108] FIG. 7(h) is a diagram showing an image correctly answered by the general public in a blind test, an image correctly answered only by the expert and the computing device 100, and an image correctly answered only by the computing device 100.

[0109] FIG. 7(i) is a diagram showing the results of image processing of a second image that a non-expert who is a member of the general public could not distinguish and a third image that neither non-experts nor experts could distinguish among the images of FIG. 7(h).

[0110] In the case of the second image that a non-expert who is a member of the general public could not distinguish but the expert and the computing device 100 could distinguish, although it is not easy to distinguish visually, a weak color development that can be recognized by the expert appears. When image processing is performed in such a way as to adjust the brightness and contrast, a test line can be confirmed slightly.

[0111] In the case of the third image that neither non-experts nor experts could distinguish but the computing device 100 could distinguish, since it is difficult to detect visually, it is confirmed that there is no difference even if image processing is performed. That is, even if a test line of a density level that cannot be detected visually appears, the computing device 100 can recognize this and determine that the inspection target is positive.

[0112] FIG. 8 is an explanatory diagram regarding the performance difference due to the dataset used to train the machine learning model used in the apparatus for generating infection state information based on the image information according to an embodiment of the present invention, and the performance difference when the diagnostic kit is applied in different diagnostic methods.

[0113] Figure 8(a) is a graph showing that the performance improves when a clinical sample is added to an image (DL..v2) of a kit in which a target substance is diluted in a standard sample (phosphate-buffered saline) (DL..v3). As shown in the figure, it can be seen that the more data there is, the more the performance of the computing device 100 improves.

[0114] Figures 8(b) to 8(c) are graphs comparing the performance of the present invention, which diagnoses whether it is positive or not from a diagnostic kit image using the nasal method and the saliva method, respectively, with the performance of the prior art.

[0115] Figure 9 is a diagram showing the positive determination performance of an apparatus for generating infection state information based on image information according to an embodiment of the present invention.

[0116] Figure 9 is a confusion matrix showing to what extent the computing device 100 according to the present invention accurately determines positive or negative based on the sample used for the cell membrane.

[0117] Figure 10 is a diagram showing the detection target concentration prediction performance according to the concentration of a sample and the detection target concentration prediction result obtained from an input image acquired over time in an apparatus for generating infection state information based on image information according to an embodiment of the present invention.

[0118] The step in which the processor 110 receives an input image for the diagnostic kit may be a step of receiving a plurality of input images including the input image taken at the current time and the input image taken at the previous time.

[0119] It is common for a user to perform self-diagnosis multiple times using a diagnostic kit in order to confirm their own diagnosis result. For example, the user can perform self-diagnosis using the diagnostic kit twice a day for 5 to 9 consecutive days. Correspondingly, the computing device 100 provides information regarding the transition of the infection state.

[0120] The infection state information may further include information regarding the transition of the infection state.

[0121] The processor 110 may generate concentration information of a detection target included in an inspection target predicted based on the degree of concentration of the test line 411.

[0122] The processor 110 may predict the concentration of the detection target in a form corresponding to the degree of concentration of the detection target, and generate concentration information of the detection target.

[0123] For example, if the processor 110 determines that the degree of concentration of the test line 411 is one level, it predicts that the concentration of the detection target is 10 ng / mL, and if it determines that the degree of concentration of the test line 411 is ten levels, it predicts that the concentration of the detection target is 100 ng / mL. That is, the concentration interval may be 10 ng / mL for each unit level interval. Also, when the processor 110 determines that the degree of concentration of the test line 411 corresponds to a level (for example, 5.6 levels) between the fifth and sixth levels, it is natural that the concentration of the detection target can be predicted to be a concentration (for example, 56 ng / mL) corresponding to between 50 and 60 ng / mL.

[0124] The processor 110 may receive a plurality of input images taken at different times. The processor 110 may generate and output information regarding the transition of the infection state using the concentration information of a plurality of detection targets generated based on the plurality of input images.

[0125] When the concentration of the detection target at the current time has increased compared to the concentration at the previous time, the processor 110 may generate and output infection state information in a form of providing a first message regarding the infection state at the current time, and when the concentration of the detection target at the current time has decreased compared to the concentration at the previous time, provide a second message regarding the infection state at the current time.

[0126] The first message may include a message indicating that the inspection target is in the infection period. The first message may further include a message recommending a PCR test since the inspection target is in the infection period.

[0127] The second message may include a message indicating that the subject of inspection is in the recovery period.

[0128] The processor 110 may set the first day based on the date when the first input image was received, predict the density of the detection target based on the input images input thereafter, and generate and output infection state transition information corresponding to the input time or date of the input image.

[0129] FIG. 10(a) is a diagram showing that the predicted density gradually decreases when the computing device 100 dilutes a sample for a certain patient with phosphate buffered saline (PBS).

[0130] FIG. 10(b) is a graph showing the infection state transition information generated by the computing device 100 and the transition information of the PCR test results. Referring to FIG. 10(b), it can be confirmed that the infection state transition information generated by the computing device 100 and the PCR test results show substantially the same transition.

[0131] Referring to FIG. 10(b), it can be confirmed that the density of the detection target contained in the body gradually decreases over time after the user is infected.

[0132] FIG. 11 is a diagram comparing the density prediction performance of the detection target of the device for generating infection state information based on the image information according to an embodiment of the present invention with the detection target density prediction capabilities of the general public and experts.

[0133] FIG. 11 is a graph comparing the density of the test target sample applied to the diagnostic kit with the densities predicted by the non-expert, expert, and computing device 100 from the image of the diagnostic kit. The closer it is to the Y = X graph, the better the prediction can be judged.

[0134] Referring to FIG. 11, the lowest concentration (LOD) at which the general public can visually determine a positive result is 1.25 ng / mL, and the lowest concentration at which an expert can visually determine a positive result is 0.62 ng / mL. The computing device 100 according to the present invention using dip running was able to determine a positive result up to 0.15 ng / mL.

[0135] FIG. 12 is an exemplary diagram regarding the case where an apparatus for generating infection status information based on image information according to an embodiment of the present invention is applied to a mobile device.

[0136] FIG. 13 is an exemplary diagram regarding a result screen presented to a user in an apparatus for generating infection status information based on image information according to an embodiment of the present invention.

[0137] The computing device 100 according to the present invention is an electronic device for analyzing an input image to determine whether a test subject applied to a diagnostic kit is positive, and includes various types of portable terminal devices such as smartphones, tablet PCs, desktop PCs, and the like. Therefore, the processor 110 may start a mobile application and execute operations by the started mobile application.

[0138] Referring to FIG. 12(a), the user takes a picture of an image of a diagnostic kit to which a test subject is applied using the computing device 100. As shown in FIG. 12(a), the user can directly obtain a diagnostic kit image using the computing device 100 equipped with a camera, but the present invention is not limited thereto, and the computing device 100 may receive a diagnostic kit image obtained using an external device.

[0139] Referring to FIG. 12(b), the user activates, by means of the computing device 100, a mobile application for executing the operations according to the present invention. The mobile application can be activated to operate in response to the user's operation on an input image stored in the computing device 100, and the user may photograph the diagnostic kit via the activated mobile application.

[0140] The user inputs information such as his / her symptoms (e.g., fever, cough, nasal discharge, phlegm, etc.), gender, age, height, weight, and the time of using the diagnostic kit into the activated mobile application, and the computing device 100 may generate infection status information including various types of information input by the user.

[0141] Referring to FIG. 12(c), the computing device 100 generates infection status information by operating based on the input image activated via the mobile application, and the generated infection status information is displayed to the user via the display unit 150 of the computing device 100. The computing device 100 may display the infection status information to the user via the display unit 150 included in the computing device 100 and transmit the infection status information to an external device, so that the external device may display the infection status information to the user.

[0142] The processor 110 may determine whether the diagnostic kit is normal based on the area corresponding to the control line in the input image corresponding to the control line, and generate and output information regarding whether it is normal. The information regarding whether it is normal may be displayed together with the infection status information.

[0143] When the processor 110 cannot recognize both the control line and the test line from the diagnostic kit included in the input image, or when the test line appears but the control line cannot be recognized, the processor 110 may determine that the diagnostic kit is malfunctioning.

[0144] Referring to FIG. 13, as described with reference to FIG. 10, the computing device 100 generates information regarding the transition of the infection state and displays to the user infection state information including the generated information regarding the transition of the infection state. As shown in FIG. 13, for the information regarding the transition of the infection state, the X-axis represents information regarding the date and the Y-axis represents information regarding the concentration of the detection target.

[0145] In this case, the processor 110 determines that it is the infection period from 22 / 09 / 01 to 22 / 09 / 04, determines that it is the recovery period from after 22 / 09 / 04 to 22 / 09 / 09, and includes the determined information in the infection state information and displays it to the user.

[0146] As shown in FIG. 13, when the user determines that the computing device 100 is benign by the diagnostic kit, the computing device 100 displays a message recommending a PCR (Polymerase Chain Reaction) test.

[0147] The computing device 100 may transmit the generated infection state information to a specialized medical institution.

[0148] FIG. 14 is a flowchart for explaining a method for generating infection state information according to an embodiment of the present invention.

[0149] The method for generating infection state information may be executed by a computing device that generates infection state information based on image information.

[0150] In step S100, the processor may execute a step of receiving an input image for a diagnostic kit to which a sample (specimen) extracted from the test subject is applied.

[0151] In step S200, the processor may execute a step of determining a main region related to the test subject in the input image and determining a sub-region surrounding the main region so as to include the main region.

[0152] In step S300, the processor may execute a step of applying the sub-region to the machine learning model and generating and outputting infection state information for the inspection target.

[0153] In FIG. 14, each process is described as being executed sequentially, but this is merely an illustrative explanation. Those skilled in the art can change the order described in FIG. 14, execute at least one process in parallel, or add other processes without departing from the essential characteristics of the embodiments of the present invention, and various modifications and changes can be applied.

[0154] FIGS. 15 and 16 are block diagrams for showing the process of the infection state information generation method according to an embodiment of the present invention.

[0155] FIG. 17 is an explanatory diagram regarding the infection state transition information provided to the user by the apparatus for generating infection state information based on the image information according to an embodiment of the present invention.

[0156] As described with reference to FIG. 10, the computing device 100 generates information regarding the transition of the infection state and displays the infection state information including the generated information regarding the transition of the infection state to the user.

[0157] When the concentration of the detection target at the current time is increased compared to the concentration at the previous time, the processor 110 may generate and output the infection state information in a form of providing a first message regarding the infection state at the current time, and when the concentration of the detection target at the current time is decreased compared to the concentration at the previous time, provide a second message regarding the infection state at the current time.

[0158] The information regarding the transition of the infection state may include the first message or the second message.

[0159] FIG. 17 is reference graph data showing the concentration of the virus detected over time after being infected with a disease (e.g., coronavirus).

[0160] The reference graph data of the virus concentration collected from the infected person over time may be statistical data, data generated based on known data, or data pre-stored in the computing device 100.

[0161] The processor uses the current concentration and the previous concentration to determine which infection interval the test subject corresponds to by comparing with the reference graph data in order to provide the user with information on the transition of the infection status of the test subject.

[0162] The infection intervals of the test subject may include a preliminary interval 1701, a first infection interval 1702, a second infection interval 1703, a third infection interval 1704, a fourth infection interval 1705, and a status improvement interval 1706.

[0163] For example, if the current concentration is higher than the previous concentration and the current concentration is less than or equal to the first critical concentration 1707, the processor 110 determines that the test subject corresponds to the preliminary interval 1701. In this case, the first message may include a message indicating that the test subject has not been infected yet but is expected to be infected.

[0164] If the current concentration is higher than the previous concentration, the current concentration is greater than or equal to the first critical concentration 1707 and less than or equal to the second critical concentration 1708, the processor 110 determines that the test subject corresponds to the first infection interval 1702. In this case, the first message may include a message indicating that the test subject is in an infected state and is expected to be in the initial stage of infection.

[0165] If the current concentration is higher than the previous concentration and the current concentration is greater than or equal to the second critical concentration 1708, the processor 110 determines that the test subject corresponds to the second infection interval 1703. In this case, the first message may include a message indicating that the possibility of the test subject being infected is very high and specialized medical treatment may be required.

[0166] When the current concentration is lower than the previous concentration and the current concentration is 1708 or higher, the second critical concentration, the processor 110 determines that the test subject corresponds to the third infection interval 1704. In this case, the second message may include a message indicating that the test subject is very likely to be infected but the condition is improving.

[0167] When the current concentration is lower than the previous concentration, the current concentration is 1707 or higher, the first critical concentration, and 1708 or lower, the second critical concentration, the processor 110 determines that the test subject corresponds to the fourth infection interval 1705. In this case, the second message may include a message indicating that the test subject is expected to be in the late stage of infection.

[0168] When the current concentration is lower than the previous concentration and the current concentration is 1707 or lower, the first critical concentration, the processor 110 determines that the test subject corresponds to the condition improvement interval 1706. In this case, the second message may include a message indicating that the test subject is not in an infected state, the infected state has improved, and the test subject can return to normal life.

[0169] Here, the first critical concentration 1707 may be a reference concentration for determining whether the test subject has a confirmed diagnosis. That is, when the concentration of the detection target collected from the test subject is 1707 or higher, the first critical concentration, the processor 110 determines that the test subject has a confirmed diagnosis.

[0170] The second critical concentration 1708 may be a reference concentration for determining whether the test subject is in a serious state beyond the confirmed diagnosis state. That is, when the concentration of the detection target collected from the test subject is 1708 or higher, the second critical concentration, the processor 110 determines that the test subject is in a serious state.

[0171] The first threshold concentration 1707 and the second threshold concentration 1708 may be reference values predetermined based on statistical data.

[0172] The processor 110 may determine the infection period of the inspection target using the current concentration and the previous concentration, and further consider the time interval between the current time and the previous time and the slope value calculated based on the difference between the current concentration and the previous concentration value to determine the infection period of the inspection target.

[0173] The processor 110 may predict the infection time of the user using at least one of the determined infection period of the inspection target, the reference graph data, the current concentration, and the slope value based on the previous concentration. The infection status information may include the predicted infection time.

[0174] This application also provides a computer storage medium. Program instructions are stored in the computer storage medium, and when the program instructions are executed by a processor, the infection status information generation method described above is realized.

[0175] Examples of the computer storage medium according to an embodiment of the present invention include, but are not necessarily limited to, a USB disk, an SD card, a PD optical drive, a mobile hard disk, a large-capacity floppy drive, a flash memory, a multimedia memory card, a server, and the like.

[0176] Even if all the components constituting the embodiments of the present invention described above are described as being combined or operating in combination, the present invention is not necessarily limited to those embodiments. That is, within the scope of the object of the present invention, all the components may operate by being selectively combined with at least one. Further, although all of those components may be realized as one independent hardware, a computer program having a program module that selectively combines some or all of the components and executes some or all of the functions combined with at least one piece of hardware may be used. Such a computer program may be stored in a computer-readable medium such as a USB memory, a CD disk, or a flash memory, and may realize an embodiment of the present invention when read and executed by a computer. Examples of the recording medium for the computer program include a magnetic recording medium and an optical recording medium.

[0177] The above description is merely an illustrative explanation of the technical idea of the present invention, and those having ordinary knowledge in the technical field to which the present invention pertains can make various modifications, changes, and substitutions without departing from the essential characteristics of the present invention. Therefore, the embodiments and the accompanying drawings disclosed in the present invention are not for limiting the technical idea of the present invention, but for explaining it, and the scope of the technical idea of the present invention is not limited by such embodiments and the accompanying drawings. The protection scope of the present invention should be interpreted according to the scope of the appended utility model claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of the present invention.

Description of Reference Numerals

[0178] 100 Computing device 110 Processor 120 Memory 130 Network interface 140 Machine learning model 200 Server 300 Network

Claims

1. In a computing device that generates infection status information based on image information, the computing device includes a memory that stores at least one program for generating the infection status information, and at least one processor that executes an operation for generating the infection status information according to the at least one program, and the operation executed by the processor is receiving an input image of a diagnostic kit to which a sample extracted from an inspection target is applied; determining a main region related to the inspection target in the input image, and determining a sub-region that surrounds the main region; applying the sub-region to a machine-learned model, and generating and outputting the infection status information for the inspection target.

2. The main region is a region including a test line whose color changes according to the infection status of the inspection target, according to the computing device of claim 1.

3. The infection status information is infection information indicating whether the inspection target is infected, or the concentration of a detection target included in the inspection target predicted based on the color change of the test line, according to the computing device of claim 2.

4. The step of receiving the input image of the diagnostic kit is receiving a plurality of input images including the input image taken at the current time and the input image taken at the previous time, and the step of generating and outputting the infection status information generates and outputs information regarding the transition of the infection status using the concentrations of a plurality of detection targets generated based on the plurality of input images, according to the computing device of claim 3.

5. The step of generating and outputting the infection status information provides a first message regarding the current infection status when the concentration of the detection target at the current time is higher than the concentration at the previous time, and provides a second message regarding the current infection status when the concentration of the detection target at the current time is lower than the concentration at the previous time, according to the computing device of claim 4.

6. The machine learning model is The computing device according to claim 1, characterized in that it is learned using an extended dataset including a plurality of training data and transformed training data transformed based on the plurality of training data.

7. The transformed training data is first transformed training data obtained by blurring the training data, second transformed training data with the size of the training data changed, third transformed training data generated by distorting the training data, fourth transformed training data obtained by rotating the training data, fifth transformed training data with the brightness of the training data adjusted, sixth transformed training data with the surrounding environment of the training data changed, The computing device according to claim 6, characterized in that it includes at least one selected from the group consisting of seventh transformed training data with the color temperature of the training data adjusted.

8. The step of determining the sub-region is determined to include the region corresponding to the test line and exclude the region corresponding to the control line (Control Line). The computing device according to claim 2.

9. The step of generating and outputting the infection status information is judging whether the diagnostic kit is normal based on the region corresponding to the control line, and generating and outputting information regarding whether it is normal. The computing device according to claim 8.

10. The processor starts a mobile application and executes the operation by the started mobile application. The computing device according to claim 1.

11. In an infection status information generation method executed by a computing device that generates infection status information based on image information, receiving an input image of a diagnostic kit to which a sample (specimen) extracted from an inspection target is applied; determining a main region related to the inspection target in the input image, and determining a sub-region that surrounds the main region so as to include the main region; applying the sub-region to a machine-learned model to generate and output the infection status information for the object to be inspected; an infection status information generation method comprising the step of

12. wherein the main region is a region including a test line whose color changes according to the infection status of the object to be inspected, the infection status information generation method according to claim 11.

13. wherein the infection status information includes infection information indicating whether the object to be inspected is infected or the concentration of the detection target included in the object to be inspected predicted based on the color change of the test line, the infection status information generation method according to claim 12.

14. wherein the step of determining the sub-region includes a region corresponding to the test line and is determined to exclude a region corresponding to a control line, the infection status information generation method according to claim 12.

15. A computer program stored in a computer-readable recording medium for causing a computer to execute the infection status information generation method according to any one of claims 11 to 14.

Images