Bacteria identification support method, multi-colony learning model generation method, bacteria identification support device, and computer program

Abstract

To provide a bacterial species identification support method capable of specifying a plurality of bacterial species included in a multi-colony.SOLUTION: A bacterial species identification support method acquires an image of a medium containing a multi-colony, which is a collection of a plurality of colonies formed by a plurality of types of cultured bacteria; when an image containing the multi-colony is input, performs image recognition; and inputs the acquired image into a multi-colony learning model that outputs information indicating the types of bacteria forming the multi-colony, to specify a plurality of bacterial species contained in the multi-colony.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a bacterial species identification support method, a multi-colony learning model generating method, a single-colony learning model generating method, a bacterial species identification support device, and a computer program. [Background technology]

[0002] Infectious diseases are caused by bacteria or other microorganisms entering the body. By identifying the infecting bacteria, the infection can be identified and measures such as treatment or prevention of the spread of infection can be taken. In medical facilities such as hospitals, patients are examined, and if an infection is suspected, a specimen is taken from the patient and a request is made for testing to a testing institution. The testing institution performs a test to identify the bacteria contained in the specimen. Many identification kits have been developed for bacterial identification tests, but only pure isolates can be used for identification kits. Therefore, in bacterial identification tests, it is necessary to first isolate and culture the bacteria. After isolation and culture, it is also possible to roughly estimate which identification kit is appropriate to use. Specifically, the laboratory technician can roughly estimate the bacterial species by observing the growth and culture conditions of the isolated strain during culture, the characteristics of the colony (color, odor, hemolysis, autolysis, penetration into the medium, migration, sliding, etc.), and Gram staining findings, etc. Bacterial identification tests require accuracy, simplicity, speed, and economic efficiency, so an identification kit suitable for determining the strain is selected based on the results of the species estimation of the isolated strain, and the bacteria can only be properly identified if the estimation is correct. However, if the estimation is incorrect, the bacteria cannot be identified and a second estimation is required. Estimation of bacterial species is called primary reading (primary image reading), secondary reading (secondary image reading), etc., depending on the number of estimations. This type of testing procedure is a common bacterial identification method. Meanwhile, in recent years, sequencers and mass spectrometers that identify genome base sequences have made it possible to identify bacteria, but isolation and culture remain necessary for accurate species identification. Bacteria grow by twofold division of a single cell, and although a single cell cannot be seen with the naked eye, if they grow to, say, 10 million or more, they form colonies on a plate medium (agar medium) and can be seen with the naked eye. Since a single colony originates from a single cell, the amount of bacteria (number of bacteria) in a sample before cultivation can be determined by counting the number of colonies.

[0003] Patent Document 1 discloses a colony identification system including a visible light irradiation image acquisition unit that acquires a visible light irradiation image, which is an image of a bacterial colony formed in a culture medium, photographed under the condition that the colony is irradiated with visible light, a non-visible light irradiation image acquisition unit that acquires a non-visible light irradiation image, which is an image of the colony photographed under the condition that the colony is irradiated with non-visible light, and a colony identification unit that identifies the bacteria forming the colony based on the visible light irradiation image and the non-visible light irradiation image.

[0004] Patent Document 2 discloses a bacterial colony identification device including: an illumination means for illuminating a bacterial colony cultured on a culture medium contained in an optically transparent container; a transmitted illumination means for transmitting illumination light through the optically transparent container and the culture medium to illuminate the bacterial colony; an imaging means for imaging the bacterial colony illuminated by each illumination means; a database storing images and spectral data of bacterial colonies at a plurality of culture times for a plurality of bacterial species; and a signal processing unit for comparing the image of the bacterial colony imaged by the imaging means with information stored in the database to identify the type of the bacterial colony. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-135240 A [Patent Document 2] JP 2020-18249 A Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to provide a bacterial species identification support method capable of identifying multiple bacterial species contained in a multi-colony, a method for generating a multi-colony learning model, a method for generating a single-colony learning model, a bacterial species identification support device, and a computer program. [Means for solving the problem]

[0007] The bacterial species identification support method according to the present embodiment acquires an image of a culture medium containing a multi-colony, which is a collection of multiple colonies formed by multiple types of cultured bacteria, and, when an image containing a multi-colony is input, identifies the multiple bacterial species contained in the multi-colony by inputting the acquired image into a multi-colony learning model which performs image recognition and outputs information indicating the types of bacteria that form the multi-colony.

[0008] The method for generating a multi-colony learning model according to the present embodiment is a method for generating a multi-colony learning model that, when an image of a culture medium containing a multi-colony consisting of a collection of multiple colonies formed by multiple types of cultured bacteria is input, outputs information indicating the types of bacteria that form the multi-colony. The method obtains an image including a multi-colony formed by multiple cultured bacteria, creates learning data by adding teacher data indicating the species of the multiple bacteria to the obtained image, and trains the learning model by machine learning so that, when an image of a culture medium containing a multi-colony is input, the multiple species of bacteria indicated by the teacher data of the image are output.

[0009] The method for generating a single colony learning model according to the present embodiment is a method for generating a single colony learning model that, when an image of a culture medium containing a single colony formed by cultured bacteria is input, outputs information indicating the position of the single colony and the bacterial species that formed the single colony, and obtains a first image containing a single colony formed by cultured bacteria of a first species and a second image containing a single colony formed by cultured bacteria of a second species, divides the obtained first and second images into small images each containing a single colony, and creates learning data by assigning teacher data indicating the first bacterial species and teacher data indicating the second bacterial species to each of the small images of the first and second images, respectively, and trains the learning model by machine learning so that, when a small image of the first image is input, information indicating the first bacterial species indicated by the teacher data is output, and when a small image of the second image is input, information indicating the second bacterial species indicated by the teacher data is output.

[0010] The bacterial species identification support device of this embodiment includes an acquisition unit that acquires an image of a culture medium including a multi-colony, which is a collection of multiple colonies formed by multiple types of cultured bacteria, and a multi-colony learning model that performs image recognition when an image including a multi-colony is input, and outputs information indicating the types of bacteria that form the multi-colony, and identifies the multiple bacterial species contained in the multi-colony by inputting the image acquired by the acquisition unit.

[0011] The computer program according to the present embodiment acquires an image of a culture medium containing a multi-colony, which is a collection of multiple colonies formed by multiple types of cultured bacteria, and when an image containing a multi-colony is input, performs image recognition and inputs the acquired image into a multi-colony learning model that outputs information indicating the types of bacteria that form the multi-colony, thereby causing the computer to execute a process of identifying the multiple bacterial species contained in the multi-colony. Effect of the Invention

[0012] According to the above, it is possible to identify a plurality of bacterial species contained in a multi-colony. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of the configuration of a bacterial species identification support device according to a first embodiment. [Diagram 2] FIG. 1 is a conceptual diagram showing a bacterial species identification support method according to a first embodiment. [Diagram 3] FIG. 1 is a conceptual diagram illustrating an example of the configuration of a multi-colony learning model. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of a single colony learning model. [Diagram 5] 13 is a flowchart showing a procedure for generating a single colony learning model. [Figure 6] FIG. 13 is a conceptual diagram showing a method for creating learning data for a single colony learning model. [Figure 7] 13 is a flowchart showing a procedure for generating a multi-colony learning model. [Figure 8] FIG. 13 is a conceptual diagram showing a method for creating learning data for a multi-colony learning model. [Figure 9] 1 is a flowchart showing an outline of the process of identifying a bacterial species, a certainty factor including the bacterial species, and a bacterial amount thereof. [Figure 10] 4 is a flowchart showing the processing procedure of image diagnosis in the bacterial species identification support method according to the first embodiment. [Figure 11] 4 is a flowchart showing a display processing procedure according to the first embodiment. [Figure 12] FIG. 13 is a schematic diagram showing an example of a result display screen. [Figure 13] FIG. 13 is a schematic diagram showing an example of a screen displaying the affiliation probability, etc. [Figure 14] FIG. 2 is a schematic diagram showing an example of a marker and a mask. [Figure 15] 10 is a flowchart showing a method for creating learning data according to the second embodiment. [Figure 16] FIG. 11 is a block diagram showing an example of the configuration of a bacterial species identification support device according to a third embodiment. [Figure 17]13 is a flowchart showing the processing steps of image diagnosis in the bacterial species identification support method according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Specific examples of a bacterial species identification support method, a multi-colony learning model generation method, a single-colony learning model generation method, a bacterial species identification support device, and a computer program according to the embodiments of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is indicated by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. In addition, at least some of the embodiments described below may be arbitrarily combined.

[0015] (Embodiment 1) FIG. 1 is a block diagram showing an example of the configuration of a bacterial species identification support device 1 according to the first embodiment. The bacterial species identification support device 1 is a device that supports the identification process of a bacterial species contained in a specimen collected from a patient at a medical institution or the like. In a testing institution, the bacteria contained in the specimen is isolated and cultured. In the cultured medium, single colonies and multi-colonies are formed by one or multiple bacteria. A single colony is a substantially circular or rod-shaped colony formed by a single cultured bacterium. A multi-colony is a colony consisting of a collection of multiple colonies formed by one or multiple types of cultured bacteria. The bacterial species identification support device 1 according to the first embodiment accurately identifies bacteria on a culture medium from an image of the culture medium (hereinafter referred to as a "culture medium image") that includes at least multiple colonies.

[0016] <Configuration of the bacterial species identification support device 1> The bacterial species identification support device 1 is a computer such as a personal computer or a server device. The bacterial species identification support device 1 includes a calculation unit 11, a memory 12, a storage unit 13, an operation unit 14, and a display unit 15. The bacterial species identification support device 1 may be a multi-computer including multiple computers. It may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the bacterial species identification support device will be described as being one computer.

[0017] The calculation unit 11 is an arithmetic processing device such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), or a tensor processing unit (TPU). The calculation unit 11 may be configured using a quantum computer. The calculation unit 11 reads out and executes a computer program 131 stored in the storage unit 13, thereby implementing the bacterial species identification method according to the first embodiment, such as identifying the bacterial species contained in the multi-colony, the confidence level of including the bacterial species, the probability of belonging to the colony, and the bacterial amount.

[0018] The memory 12 is a volatile memory such as a DRAM (Dynamic RAM) or an SRAM (Static RAM), and temporarily stores a computer program 131 read from the storage unit 13 when the calculation unit 11 executes calculation processing, or various data generated by the calculation processing of the calculation unit 11.

[0019] The storage unit 13 is, for example, a storage device such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), a flash memory, etc. The storage unit 13 stores various programs executed by the calculation unit 11 and various data required for processing by the calculation unit 11. In the present embodiment 1, the storage unit 13 stores a computer program 131 executed by the calculation unit 11, a single-colony learning model 2 for supporting identification of bacteria contained in a specimen from a culture image, and a multi-colony learning model 3.

[0020] The computer program 131 is recorded in a computer-readable manner, for example, on the recording medium 10. The storage unit 13 stores the computer program 131 read from the recording medium 10 by a reading device (not shown). The recording medium 10 is a semiconductor memory such as a flash memory, an optical disk, a magnetic disk, a magneto-optical disk, or the like. The computer program 131 may be stored in the storage unit 13 during the manufacturing stage of the bacterial species identification support device 1. Furthermore, the computer program 131 according to this embodiment 1 may be downloaded from an external server (not shown) connected to a communication network and stored in the storage unit 13.

[0021] The single colony learning model 2 is an object detection model, and is a trained model that, when a medium image including a single colony is input, outputs information indicating the position of the single colony and the bacterial species group that forms the single colony. The multi-colony learning model 3 is an image recognition model, and is a trained model that outputs information indicating one or more bacterial groups that form the multi-colony when a medium image including a multi-colony is input. The multi-colony learning model 3 only recognizes the type of bacterial species group, and does not detect the position of each colony that constitutes the multi-colony. The single-colony learning model 2 and the multi-colony learning model 3, as well as the bacterial species group, will be described in detail later.

[0022] The operation unit 14 is an input device that receives operations from an operator such as an engineer, etc. The input device is, for example, a keyboard or a pointing device.

[0023] The display unit 15 is an output device that outputs information such as a culture image, a bacterial identification result, etc. The output device is, for example, a liquid crystal display or an EL display.

[0024] <Outline of bacterial species identification support method and learning model> Fig. 2 is a conceptual diagram showing the bacterial species identification support method according to embodiment 1. In Fig. 2, a division processing unit 1a, a morphological classification unit 1b, a single-colony learning model 2, a multi-colony learning model 3, a first bacterial quantity calculation unit 2a, a second bacterial quantity calculation unit 3a, a display processing unit 4, and a correction processing unit 5 are realized by a calculation unit 11 executing a computer program 131.

[0025] The medium image to be processed will be described as an image of a medium on which a single colony and multiple colonies are formed.

[0026] The division processing unit 1a divides the medium image into a plurality of small images. For example, the division processing unit 1a divides the medium image into 4×4=16 small images in the vertical and horizontal directions. The size of the small images is not particularly limited as long as it is within the range of the processing capacity of the computer, such as the memory size of the GPU, but it is desirable that the size is such that one to three single colonies can be included. Each small image is, for example, a rectangular medium image divided into a lattice shape. The multiple small images are images of a size that can be input to the subsequent single colony learning model 2 and multi-colony learning model 3. In other words, the size of the small images is a size that can be input to each learning model without further reduction processing of the small images. The sizes of the small images may be different, but analysis is easier when they are the same size.

[0027] The morphological classification unit 1b is a functional unit that classifies a plurality of small images into small images including a single colony and small images including multiple colonies. For example, the morphological classification unit 1b extracts colony parts by performing image processing such as binarization processing, edge detection, morphological processing, and circle detection by Hough transform, etc., on the small images, and classifies the small images into those including multiple colonies when a colony having a pixel count of a region corresponding to a colony of a predetermined size or more is extracted, and classifies the small images into those including a single colony when only a colony of a smaller size is extracted. Note that the rule-based morphological classification by image processing is an example, and colonies may be detected and the small images may be classified using a learning model such as U-Net, Faster RCNN (Regions with Convolutional Neural Network), SSD (Single Shot Multibox Detector), or YOLO (You Only Look Once), which is trained to extract colony parts. The morphological classification unit 1b does not need to identify the species of bacteria, and the detection accuracy of the colony does not need to be high. The small image containing the classified single colony is input to the single-colony learning model 2. The small image containing the classified multiple colonies is input to the multi-colony learning model 3.

[0028] The single colony learning model 2 outputs the position, species group, and confidence level of a single colony. The first bacterial mass calculation unit 2a calculates the bacterial mass of a single colony. The multi-colony learning model 3 outputs the species group and confidence level included in a multi-colony. The second bacterial mass calculation unit 3a calculates the bacterial mass of a multi-colony. Details of each learning model and the method of calculating the bacterial mass will be described later.

[0029] The display processing unit 4 executes a process of displaying the identified bacterial species, the confidence level, the amount of bacteria, etc. together with the medium image on the display unit 15. The correction processing unit 5 executes a process of accepting corrections to the identified bacterial species.

[0030] 3 is a conceptual diagram showing an example of the configuration of the multi-colony learning model 3. The multi-colony learning model 3 is an image recognition model that recognizes one or more bacterial species groups forming multiple colonies, but does not recognize the positions of the colonies.

[0031] In the present embodiment 1, the multi-colony learning model 3 is a neural network having, for example, an input layer 31 that receives an input of an image, an intermediate layer 32 that extracts image features, and an output layer 33 that outputs a species group to which a species that forms a colony included in a small image belongs. The multi-colony learning model 3 in the present embodiment 1 is a CNN (Convolution Neural Network) such as ResNet or DenseNet. The species identification support device generates the multi-colony learning model 3 by performing deep learning on the CNN model to learn the relationship between the multi-colony image and the species group. The bacterial species group is a classification of the same bacterial group name as the identification target by physiological and biochemical tests, etc. performed after the bacterial species identification support process according to the first embodiment, or a classification of bacterial species in the phylogenetic classification of bacterial species. Examples of the former (bacterial group by physiological and biochemical tests, etc.) are classifications such as gram-negative bacteria, gram-positive cocci, gram-positive bacilli, anaerobic bacteria, yeast-like fungi, Enterobacteriaceae, Staphylococcus genus, Haemophilus genus, Neisseria genus, and related bacteria, which are identification targets by commercially available bacterial identification kits. Examples of the latter (bacterial group by phylogenetic classification) are classifications of bacterial group names that group together multiple bacterial species, such as genus (genus) and species (species). For example, the genus of bacteria generally called Escherichia coli is Escherichia, the species is Escherichia coli, and the smallest unit is a strain, such as Escherichia coli O157 strain. The former includes more bacterial species than the latter, and therefore it is generally difficult for the bacterial species identification support device 1 to learn image features (i.e., many learning images are required). On the other hand, the latter limits image features, particularly by limiting to species rather than genus, and strains rather than species, and it is easy for the bacterial species identification support device 1 to learn image features (i.e., there tends to be fewer learning images than the former). Therefore, if multiple bacterial species cannot be separated, learning with the former learning method is more efficient, whereas if they can be classified, learning with the latter method is more efficient.

[0032] The input layer 31 of the neural network has a plurality of nodes that accept input of pixel values ​​of each pixel related to the small image, and passes the input pixel values ​​to the intermediate layer 32. The intermediate layer 32 has a plurality of nodes that extract image features, and passes the extracted features to the output layer 33. For example, when the multi-colony learning model 3 is a CNN, the intermediate layer 32 has a configuration in which a convolution layer that convolutes the pixel values ​​of each pixel input from the input layer 31 and a pooling layer that maps the pixel values ​​convolved in the convolution layer are connected in a plurality of layers, and finally extracts image features while compressing the pixel information of the image. The output layer 33 has one or more nodes that output one or more species groups to which the species included in the multi-colony belongs. For example, the output layer 33 has nodes corresponding to each of the multiple species groups, and each node outputs a value (hereinafter referred to as a confidence level) indicating the possibility that the species forming the multi-colony belongs to each species group.

[0033] In the present embodiment 1, the multi-colony learning model 3 is a CNN such as DenseNet, but the configuration of the model is not limited to CNN. The multi-colony learning model 3 may be a multi-colony learning model 3 configured, for example, as a neural network other than CNN, a Support Vector Machine (SVM), a Bayesian network, or a decision tree such as XGBoost.

[0034] The certainty output from the multi-colony learning model 3 is a numerical value that serves as a classification index in the classification task performed by the multi-colony learning model 3, and is a real number ranging from 0 to 1, but does not necessarily match the actual probability that a species belonging to the corresponding bacterial species group is included in the multi-colony. Therefore, the bacterial species identification support device 1 may convert the certainty into the belonging probability using a conversion function stored in advance. In other words, the certainty is calibrated to the belonging probability. The conversion function is created as follows. First, for multiple multi-colonies, a frequency distribution of certainty is created from a set of certainty that the species contained in the multiple colonies belong to a specific species group. For example, a frequency distribution of certainty is created in intervals of 0.1. Meanwhile, for the multiple multi-colonies, the actual probability that the species contained in the multiple colonies belong to a specific species group, that is, the probability of belonging based on the judgment of a laboratory technician or the result of a confirmed test, is calculated. Then, a conversion function that associates the certainty and the probability of belonging is determined. The conversion function uses a logistic regression function, an isotonic regression function, or the like. By using the conversion function created in this way, the probability of belonging can be calculated from the value of the certainty. In the single colony learning model 2 described later, the certainty can be converted into the membership probability in a similar manner. The probability described in the claims is a concept that includes both the confidence level and the membership probability.

[0035] 4 is a block diagram showing an example of the configuration of the single colony learning model 2. The single colony learning model 2 includes a position detection unit 21, a colony area cutout unit 22, and a bacterial species classification unit . The position detection unit 21 is an object detection model such as YOLOv3, U-Net, Faster RCNN, SSD, etc., which detects colony images included in a small image. The object detection model recognizes the positions and ranges of all single colony images included in the small image, and the object detection model outputs information indicating the positions and ranges of at least a single colony image. The object detection model does not need to recognize the species of the colony. Even if the model recognizes the species, the discrimination accuracy is not particularly required. The position and range of the colony image detected by the object detection model is expressed as an image portion surrounded by, for example, a circular bounding circle or a rectangular bounding box (hereinafter referred to as a bound). Since the colony shape is usually circular, the detection of the colony area is more likely to be accurate when a circular bounding circle is used than when a rectangular bounding box is used. That is, the IOU (Intersection of Union), which is an index of the overlap ratio between the colony area and the bounding area, becomes a more accurate value in the bounding circle, so the detection accuracy of the colony can be improved by including the IOU in the loss function in the object detection model. Incidentally, when IOU is included in the loss function, the convergence of the loss function in the learning process often becomes poor, but by adding an additional term such as Distance IOU to the loss function, it becomes possible to ensure and efficiently converge. The position and range of the colony image detected by the object detection model are input to the colony area cutout unit 22.

[0036] The colony region cutout unit 22 cuts out from the small image an image portion surrounded by a bounding circle or bounding box centered on the central coordinates of the colony detected by the position detection unit 21. When cutting out, the radius of the bounding circle or the width and height of the bounding box may be cut out to be several percent larger than the size of the detected colony. Since the cut out image is generally saved in a rectangular image format, if it is cut out in a circular shape, it is saved as a square image inscribed in the circle. In this case, the pixel values ​​outside the circle are set to zero (zero padding) or the average value of the surrounding pixel values. While the size of the colony varies from colony to colony, it is preferable to make them all the same image size in order to make the subsequent processing in the bacterial species classification unit 23 more efficient. Therefore, the image size is enlarged to the largest image size among the cut out colony images or even larger, and all the images are unified to the same image size to prevent the loss of image features and to enable efficient subsequent analysis. The colony image thus obtained is input to the bacterial species classification unit 23.

[0037] When a colony image containing one single colony and with a uniform image size is input, the species classification unit 23 performs image recognition processing on the colony image and identifies the species that form the colony image using an image recognition model such as VGG, ResNet, or DenseNet. The species classification unit 23 recognizes a species group related to the species that form the colony, and outputs information indicating at least the species group that forms the colony and the degree of certainty that this species group forms the colony. The species classification model recognizes the species group to which the species of the colony belongs, but does not detect the position.

[0038] The single colony learning model 2 configured in this manner outputs the position of a single colony detected by the position detection unit 21 and the bacterial species group recognized by the bacterial species classification unit. The position of a single colony is expressed, for example, as a circular or rectangular bound that surrounds the image portion of the colony. By separately configuring the position detection unit 21, which is an object detection model, and the bacterial species classification unit 23, which is an image recognition model, it becomes possible to accurately recognize the position of a colony and the bacterial species group.

[0039] In the above example, the medium image is divided into small images of a predetermined size and input to the single colony learning model 2. However, a single colony included in the medium image may be detected using an object detection model similar to the position detection unit 21, divided into small images including the detected single colony, and the divided small images may be input to the single colony learning model 2. In this case, since the position of the single colony is specified by the object detection model, the single colony learning model 2 executes a process of recognizing the bacterial species forming the single colony. The single colony learning model 2 does not need to include the position detection unit 21 and the colony area cutout unit 22. The small image may be an image including a circular image portion surrounded by a bounding circle, or an image including a rectangular image portion surrounded by a bounding box.

[0040] <How to generate single colony learning model 2> Fig. 5 is a flowchart showing the procedure for generating the single colony learning model 2, and Fig. 6 is a conceptual diagram showing a method for creating learning data for the single colony learning model 2. In the following explanation, an example will be described in which the bacterial species identification support device 1 performs machine learning to generate the single colony learning model 2, but the single colony learning model 2 may be generated by another external computer or server.

[0041] First, a bacterial species belonging to a first bacterial species group is cultured, and a medium image (first image) including a single colony formed by the bacterial species is obtained (step S11). The first bacterial species group is, for example, gram-positive cocci. Similarly, a bacterial species belonging to a second bacterial species group is cultured, and a medium image (second image) including a single colony formed by the bacterial species is obtained (step S12). The second bacterial species group is, for example, gram-negative cocci.

[0042] Next, as shown in FIG. 6, each of the medium images obtained in steps S11 and S12 is divided into small images (step S13). Then, training data is created by adding the position of the colony and teacher data indicating that it is the first bacterial species group to the small images including a single colony of a bacterial species belonging to the first bacterial species group, and adding the position of the colony and teacher data indicating that it is the second bacterial species group to the small images including a single colony of a bacterial species belonging to the second bacterial species group (step S14). There are two main methods for adding teacher data: a mechanical method in which the small images divided in step S13 are binarized by Otsu's method, and circular areas of a certain size are mechanically recognized as colonies, and the name of each bacterial species group is assigned to each colony; and a manual method in which an inspection technician visually checks the colonies in the small images divided in step S13, and records the colony coordinates and colony diameter on the image together with the name of each bacterial species group. The procedure of correcting oversights and errors by performing the mechanical method and then performing the manual method is generally efficient. In addition, in the manual method, software is created that displays the small images divided in step S13 on a screen, and then the position and diameter of the colony on the screen is marked by mouse operation, and the software can be used to efficiently provide training data. Once training data is created and single-colony training model 2 is generated, it can be applied to other bacterial species (in this case, many mistakes and oversights are included), and training data can be efficiently created by correcting it using the manual method described above.

[0043] Next, the learning data created in step S14 is used to perform machine learning on the object detection model (single-colony learning model 2 before learning), thereby generating a single-colony learning model 2 (step S15).

[0044] Specifically, when a small image of learning data is input, the colony position and bacterial species group output from the output layer 33 of the object detection model are compared with the colony position and bacterial species group indicated by the training data attached to the small image, and the parameters of the object detection model are optimized so that the output result approaches the correct value. The parameters are, for example, weights (coupling coefficients) between each node. The method of optimizing the parameters is not particularly limited, but various parameters are optimized using, for example, the steepest descent method.

[0045] By the above process, when a small image including a single colony is input, a single colony learning model 2 can be generated that can identify the position and species group of a single colony included in the small image and the confidence level including the species group. Although the single colony learning model 2 cannot detect the position and species group of colonies formed by one or more species that constitute a multi-colony, it can accurately identify the position and species group of a single colony. Usually, to build a machine learning model, it is necessary to prepare training data similar to the test data. In other words, since the patient samples that serve as test data contain multiple bacterial species, the training data must be the results of a laboratory technician identifying the bacterial species on a medium in which a sample containing a mixture of multiple bacterial species is cultured, or the results of identifying the bacterial species through other tests. The problem with preparing these is that they are expensive and time-consuming. Therefore, by isolating bacteria of one species or one species group in advance and culturing the bacteria, multiple single colonies consisting of one species or one species group can be generated, making it easy to generate a large amount of training data with clear species or species groups. In other words, by using learning data of small images including a single colony of one bacterial species group, rather than learning data of small images including multiple single colonies composed of multiple bacterial species groups, the single colony can be efficiently machine-learned to learn a single colony. In addition, since the single colony learning model 2 is configured to receive small images obtained by dividing a medium image into colony units, the above learning method is suitable, and the detection accuracy of the bacterial species group by the single colony learning model 2 can be improved.

[0046] The bacterial species identification support device 1 according to the present embodiment 1 identifies a specific bacterial species group, not an accurate bacterial name classification. As described later, after the bacterial species group is identified by image processing of the bacterial species identification support device 1, the identified bacterial species group is verified and definitively diagnosed by physiological and biochemical tests, genetic tests, MS analysis, etc. For example, in the case of bacterial species A and bacterial species B that are the subject of the same physiological and biochemical tests, even if the bacterial species A and bacterial species B are not distinguished by image diagnosis, the bacterial species A and bacterial species B are definitively diagnosed by the subsequent physiological and biochemical tests, etc., so that the test result is not affected at all. Therefore, the classification content required of the bacterial species identification support device 1 is not an accurate bacterial name classification, but a classification of bacterial species group names that will be the same in the subsequent physiological and biochemical tests, etc. By using the bacterial species group names instead of accurate bacterial names as the learning data, the classification accuracy of the bacterial species can be gradually improved according to the classification granularity of the bacterial species group, and the single colony learning model 2 can be efficiently constructed.

[0047] In the above example, for simplicity, an example of machine learning of two bacterial species groups, gram-positive cocci and gram-negative cocci, has been described. However, it goes without saying that the single colony learning model 2 can be configured to recognize other bacterial species groups by machine learning using culture medium images of gram-positive bacilli, gram-negative bacilli, and other bacterial species groups.

[0048] In the above description, an example has been described in which the bacterial species identification support device 1 generates the single-colony learning model 2, but the single-colony learning model 2 may be generated by an external server, and various parameters required to configure the single-colony learning model 2 may be distributed to the bacterial species identification support device 1. The bacterial species identification support device 1 downloads the various parameters distributed from the external server, and configures the single-colony learning model 2 based on the parameters.

[0049] <How to generate multi-colony learning model 3> Fig. 7 is a flowchart showing the procedure for generating the multi-colony learning model 3, and Fig. 8 is a conceptual diagram showing a method for creating learning data for the multi-colony learning model 3. In the following explanation, an example will be described in which learning data for the multi-colony learning model 3 is created using a trained single-colony learning model 2, and machine learning is performed.

[0050] First, the bacterial species identification support device 1 acquires a medium image including single colonies and multiple colonies (step S31), and divides the acquired medium image into small images (step S32).

[0051] Next, the bacterial species identification support device 1 classifies the multiple small images into small images including single colonies and small images including multiple colonies by the morphological classification unit 1b (step S33). Then, as shown in Fig. 8, the bacterial species identification support device 1 inputs the small images including single colonies into the trained single colony learning model 2 to identify the bacterial species group included in the culture medium image and the confidence level of including the bacterial species group (step S34). Since it is presumed that the bacterial species forming a single colony also forms multiple colonies, the bacteria of the bacterial species group identified by the single colony learning model 2 are considered to be the bacterial species group of the bacterial species that constitute multiple colonies.

[0052] The bacterial species identification support device 1 creates learning data by adding teacher data indicating one or more bacterial species groups identified in step S34 to the small image including the multi-colony (step S35).

[0053] Next, the learning data created in step S35 is used to perform machine learning on the image recognition model (the multi-colony learning model 3 before learning), thereby generating the multi-colony learning model 3 (step S36). Specifically, when a small image of learning data is input, one or more bacterial species groups output from the output layer 33 of the image recognition model are compared with one or more bacterial species groups indicated by the teaching data attached to the small image, and the parameters of the image recognition model are optimized so that the output result approaches the correct answer. The parameters are, for example, weights (coupling coefficients) between nodes. There are no particular limitations on the method of optimizing the parameters, but various parameters are optimized using, for example, the steepest descent method.

[0054] By the above process, when a small image including a multi-colony is input, a multi-colony learning model 3 can be generated that can identify the bacterial species group of the multi-colony included in the small image and the confidence level of including the bacterial species group. The multi-colony learning model 3 is a model that cannot detect the position of the colony formed by each of one or more bacterial species that constitute a multi-colony, but can accurately recognize one or more bacterial species groups that form a multi-colony.

[0055] In the above description, an example has been described in which the bacterial species identification support device 1 generates the multi-colony learning model 3, but the multi-colony learning model 3 may be generated by an external server, and various parameters required to configure the multi-colony learning model 3 may be distributed to the bacterial species identification support device 1. The bacterial species identification support device 1 downloads the various parameters distributed from the external server, and configures the multi-colony learning model 3 based on the parameters.

[0056] <Bacteria species identification support method> FIG. 9 is a flow chart showing an outline of the process of identifying a bacterial species, a certainty factor including the bacterial species, and a bacterial amount. A laboratory technician cultures a bacterial species in order to identify a bacterial species contained in a specimen collected from a patient at a medical institution or the like (step S51). Then, the laboratory technician takes an image of a medium on which a colony is formed by culturing the bacterial species (step S52), and inputs the captured medium image into the bacterial species identification support device 1 to execute an image diagnostic process for identifying a bacterial species group (step S53). The medium image includes a single colony and a multi-colony. The image diagnostic process of the bacterial species identification support device 1 identifies the position of a single colony, the bacterial species group that forms the single colony, the certainty factor and the probability of belonging to the bacterial species group, and the bacterial amount. Also, one or more bacterial species groups that form a multi-colony, the certainty factor and the probability of belonging to the bacterial species group, and the bacterial amount are identified.

[0057] The laboratory technician determines whether or not isolation culture is required based on the results of the image diagnosis processing (step S54). For example, if there is a bacterial species group that is determined not to form a single colony but to be included in a multi-colony, the laboratory technician determines that isolation culture is required.

[0058] If it is determined that isolation and culture are necessary (step S54: YES), the laboratory technician picks up bacteria from the multi-colony and performs isolation and culture (step S55), and the process returns to the imaging step of step S52.

[0059] If it is determined that isolation culture is not necessary (step S54: NO), the laboratory technician picks up a single colony and then verifies the image diagnostic processing results and performs a definitive diagnosis using physiological and biochemical tests, genetic tests, MS analysis, etc., to identify the bacterial species (step S57).

[0060] 10 is a flowchart showing the processing procedure of image diagnosis in the bacterial species identification support method according to embodiment 1. In step S53, the bacterial species identification support device 1 executes the following processing: The bacterial species identification support device 1 acquires the medium image captured in step S52 (step S71), and executes pre-processing such as noise removal and trimming on the acquired medium image (step S72).

[0061] Next, the bacterial species identification support device 1 divides the medium image into small images (step S73), and classifies the divided small images into small images including single colonies and small images including multiple colonies (step S74).

[0062] The bacterial species identification support device 1 inputs a small image including a single colony to the single colony learning model 2, thereby identifying the position of the single colony, the bacterial species group, and the confidence level including the bacterial species group (step S75).

[0063] Next, the bacterial species identification support device 1 uses the above conversion function to calculate the belonging probability from the confidence calculated in step S75 (step S76).

[0064] Next, the bacterial species identification support device 1 calculates the bacterial amount of the bacterial species that form one or more single colonies contained in the culture medium (step S77). For example, the bacterial species identification support device can calculate the bacterial amount by counting the number of single colonies for each of multiple bacterial species groups based on information output from the single colony learning model for multiple small images obtained by dividing the culture medium image. For one bacterial species group, the number of single colonies contained in each small image is counted. Then, by adding up the number of single colonies contained in each small image, the number of single colonies formed by the bacterial species that belong to one bacterial species group contained in the culture medium image can be obtained. The number of single colonies calculated in this manner corresponds to the bacterial amount of the bacterial species that belong to one bacterial species group. The bacterial amount can be obtained in the same manner for one or more other bacterial species groups.

[0065] In addition, the bacterial species identification support device 1 inputs a small image including a multi-colony into the multi-colony learning model 3 to identify one or more bacterial species groups forming a multi-colony and the confidence level of including the bacterial species group (step S78).

[0066] Next, the bacterial species identification support device 1 uses the above conversion function to calculate the belonging probability from the confidence calculated in step S78 (step S79).

[0067] Next, the bacterial species identification support device 1 calculates the bacterial mass of the bacterial species that forms one or more multi-colonies contained in the medium (step S80).

[0068] For example, the bacterial species identification support device 1 estimates the bacterial mass of the bacterial species forming the multi-colony based on the area of ​​the multi-colony included in the medium image. Specifically, the bacterial species identification support device 1 stores a correspondence table or function that associates the area of ​​the multi-colony with the bacterial mass, and the bacterial species identification support device 1 calculates the bacterial mass of the bacterial species contained in the multi-colony from the area of ​​the multi-colony using the correspondence table or function. When a multi-colony contains multiple bacterial species, an approximate bacterial mass can be calculated by dividing the total number of bacterial masses of each bacterial species forming the multi-colony by the proportion of single colonies formed by the multiple bacterial species according to that proportion. In the method using the area of ​​multi-colonies, a bacterial solution with a previously adjusted bacterial solution concentration is applied to a medium, and the area of ​​the medium surface covered by colonies is measured after a certain incubation time, and then a correspondence table between the bacterial solution concentration and the area is created, or a linear regression equation is calculated with the area as the explanatory variable and the bacterial solution concentration as the objective variable, thereby calculating the bacterial solution concentration from the area. The amount of bacteria can be calculated by multiplying the bacterial solution concentration by the amount of sample.

[0069] As another method, the bacterial species identification support device 1 estimates the bacterial mass of a bacterial species that forms multiple colonies based on the distribution density of single colonies in a medium image. Specifically, the bacterial species identification support device 1 stores a correspondence table or function that associates the distribution density of single colonies formed by a specific bacterial species with the bacterial mass of the bacterial species that forms multiple colonies, and the bacterial species identification support device 1 calculates the bacterial mass of a bacterial species contained in the multiple colonies from the distribution density of single colonies using the correspondence table or function. The method using distribution density involves applying a bacterial solution with a previously adjusted bacterial concentration onto a medium using a platinum loop, magnetic beads, etc., setting the application path to be the same each time, and then counting the number of single colonies detectable after a certain incubation time to determine the correspondence relationship (correspondence table or linear regression equation) between the number of single colonies and the bacterial solution concentration. The amount of bacteria can be calculated by multiplying the bacterial solution concentration by the sample volume.

[0070] Next, the bacterial species identification support device 1 calculates the bacterial mass of each bacterial species and species group by adding the bacterial mass calculated in step S77 and the bacterial mass calculated in step S80 for each bacterial species or species group (step S81).

[0071] Next, the bacterial species identification support device 1 executes post-processing such as identifying resident bacteria (step S82). The bacterial species identification support device 1 accepts the type of specimen to be analyzed. The bacterial species identification support device 1 stores a correspondence table that associates the type of specimen with the type and concentration of resident bacteria. The bacterial species identification support device 1 identifies the resident bacteria from the correspondence table using the accepted type of specimen as a key.

[0072] Next, the bacterial species identification support device 1 executes a display process for displaying the image processing results on the display unit 15 (step S83), and ends the process.

[0073] 11 is a flowchart showing the procedure of the display process according to embodiment 1. The bacterial species identification support device 1 displays the result display screen 6 on the display unit 15 (step S91).

[0074] Fig. 12 is a schematic diagram showing an example of the result display screen 6, and Fig. 13 is a schematic diagram showing an example of the species belonging probability, etc. display section 62. The result display screen 6 includes a medium image display section 61 that displays a medium image, a species belonging probability, etc. display section 62 that displays the identified species group, the belonging probability, the amount of bacteria, etc., and a correction operation section 63 for correcting the identified species.

[0075] The bacterial species belonging probability display section 62 displays a list of the names of a plurality of bacterial species groups that form single colonies or multiple colonies, as shown in Fig. 13. Each bacterial species name is displayed in association with the bacterial species group belonging probability display section, the bacterial amount, and the bacterial amount ratio. The belonging probability display section includes a scale bar showing values ​​from 0 to 1. The scale bar displays an image showing the maximum value of the belonging probability of the bacterial species group, for example, an up arrow image. The scale bar also displays a down arrow image showing a designated probability for designating the target of a single colony to be marked or masked. The user can change the designated probability by sliding the down arrow image left or right. Furthermore, the scale bar displays a frequency distribution of the belonging probability of each bacterial species group that forms a colony included in the medium image.

[0076] Furthermore, the bacterial species affiliation probability display unit 62 displays a check box indicating whether or not the bacterial species is indigenous, in association with each bacterial species name. For bacterial species identified as indigenous in step S82, a check image is displayed in the check box. The user can manually check or uncheck the check box indicating whether or not the bacterial species is indigenous, and record the check content.

[0077] Furthermore, the species affiliation probability display unit 62 displays a first check box indicated by "display (low probability)" for displaying a mark image 61a or mask images 61b, 61c for a specific single colony in association with each species name, and a second check box indicated by "display (high probability)." When the first check button is checked, the mark image 61a, etc. is displayed for a single colony formed of a species in a species group having an affiliation probability equal to or lower than the designated probability. When the second check button is checked, the mark image 61a, etc. is displayed for a single colony formed of a species in a species group having an affiliation probability equal to or higher than the designated probability. When the first and second check boxes are unchecked, the mark image 61a, etc. is no longer displayed.

[0078] Furthermore, the bacterial species affiliation probability display unit 62 displays multiple check boxes represented as "Highlight," "Mask (other)," and "Mask (target)" in association with each bacterial species name to specify the display mode of a mark or mask, etc., to be displayed when the first check box or the second check box is checked.

[0079] FIG. 14 is a schematic diagram showing an example of a marker and a mask. When the "Highlight" check button is checked, the bacterial species identification support device 1 displays a mark image 61a surrounding a single colony having an affiliation probability equal to or greater than a specified probability or less than a specified probability on the medium image, as shown in FIG. 14A. When the "Mask (other)" check button is checked, the bacterial species identification support device 1 displays a mask image 61b that masks image portions other than single colonies having an affiliation probability equal to or greater than a specified probability or less than a specified probability, as shown in FIG. 14B. When the "Mask (target)" check button is checked, the bacterial species identification support device 1 displays a mask image 61c that masks single colonies having an affiliation probability equal to or greater than a specified probability or less than a specified probability, as shown in FIG. 14C.

[0080] Returning to FIG. 11, after completing the processing of step S91, the bacterial species identification support device 1 displays, as described above, the species name, affiliation probability, bacterial quantity, etc. of the bacterial species group to which the bacterial species that form single colonies and multi-colonies belong in the species affiliation probability etc. display unit 62 (step S92).

[0081] The bacterial species identification support device 1 allows the laboratory technician to easily grasp the proportion of the bacterial species group contained in the specimen by displaying a list of the bacterial species name of the bacterial species group displayed in the frame image in ascending or descending order of the bacterial amount or the proportion of the bacterial amount in the total bacterial amount. In particular, by grasping the bacterial species groups that form single colonies and multiple colonies, it becomes easier to determine the necessity of isolation and culture in step S54, and it also helps prevent mistakes in bacterial species determination.

[0082] Since the presence of resident bacteria can be an obstacle in the analysis to identify the bacteria with which a patient is infected, resident bacteria with a certain amount of bacteria or less are set not to be displayed as necessary. There are several known resident bacteria, such as α-Streptococcus in sputum samples and coagulase-negative Staphylococcus in nasal mucus. Since a large amount of bacteria may not be determined to be a resident bacteria, it is determined whether or not it is a resident bacteria based on a combination of the sample material, amount of bacteria, and type of bacteria (specified in step S82), and the color of the frame image for resident bacteria is set to a different color from other bacterial species groups, so that the display setting does not interfere with the analysis.

[0083] After completing the process of step S92, the bacterial species identification support device 1 accepts the designated probability with a down arrow image displayed on the scale bar of the bacterial species belonging probability etc. display unit 62 (step S93). The bacterial species identification support device 1 also accepts the display mode with check boxes for "highlight", "mask (other)", and "mask (target)" (step S94). Then, the bacterial species identification support device 1 displays the mark image 61a or mask images 61b, 61c according to the accepted designated probability and display correspondence by superimposing them on the medium image (step S95).

[0084] The bacterial species identification support device 1 executes a process of converting the position of the single colony identified by the process of step S75 into a position in the medium image before division into small images.

[0085] Then, the bacterial species identification support device 1 displays an image showing the processing result of step S75 etc. superimposed or arranged on the medium image. For example, the bacterial species identification support device 1 displays a circular or rectangular bound indicating the position and size of a single colony and the bacterial species group superimposed on the medium image. The bound includes a circular or rectangular frame image surrounding the image portion of the single colony, and characters or symbols indicating the bacterial species group, or the color of the frame image, or a transparent color indicating the bacterial species group for the area inside the frame image. Furthermore, the bacterial species identification support device 1 displays characters or symbols indicating one or more bacterial species groups forming a multi-colony, or the color of the frame image, or a transparent color indicating the bacterial species group for the area inside the frame image, in or near the image portion of a multi-colony. The color or transparent color of the frame image indicating the bacterial species group is set to a shade of color or a gradation from blue to red, etc., depending on the certainty and probability of belonging to the bacterial species group of each colony output in the processing of steps S75 and S76, and a toggle button can be used to switch whether or not to display the color or transparent color of the frame image for certainty and probability of belonging that are greater or smaller than a set threshold value.

[0086] Furthermore, the bacterial species identification support device 1 accepts corrections to the identified bacterial species in the correction operation unit 63 (step S96).

[0087] For example, the bacterial species identification support device 1 accepts the selection of a specific single colony image on a culture medium image. For example, the user can select a specific single colony image by double-clicking the mouse pointer. When a specific single colony image is selected, the bacterial species identification support device 1 displays a correction operation section 63 as shown in FIG.

[0088] The correction operation unit 63 displays a confirmation check box, the species names of the selected multiple species groups that can form single colonies, and the confidence or probability that each species group forms a single colony. The correction operation unit 63 also displays a species name input section for inputting the name of a species or species group other than the species names displayed in the list.

[0089] When the estimation of the species by the species identification support device 1 is regarded as the first reading, the laboratory technician who checks the result corresponds to the second reading, but the laboratory technician who performs the second reading can check the candidate species name, the certainty, or the probability of belonging on the correction operation unit 63. The laboratory technician can store the confirmation history by checking that the correspondence between the colony image and the species name is correct and operating the confirmation check button. The laboratory technician can instruct the correction of the species name by checking the confirmation check button of the species name different from the species name with the highest certainty identified by the single colony learning model 2. In addition, the laboratory technician can input the species name that is not listed in the correction operation unit 63 into the species name input unit. The species name input unit is, for example, a pull-down list, and the laboratory technician can select an appropriate species from the species listed in the pull-down list.

[0090] A confirmation check button may be displayed in association with the bacteria species name input section. When another medical technician performs the tertiary interpretation, he / she can check the check button if he / she judges that the bacteria species name input section is correct.

[0091] The bacterial species identification support device 1 stores the confirmation result of the secondary interpretation received in step S96, and the bacterial species identification support device stores the change content of the identified bacterial species (step S97).

[0092] The thus configured bacterial species identification support device 1 etc. can identify multiple bacterial species groups contained in a multi-colony, the certainty factor and the probability of belonging to the bacterial species group, and the bacterial amount. It can also identify a bacterial species group that forms a single colony, the certainty factor and the probability of belonging to the bacterial species group, and the bacterial amount.

[0093] Generally, high-resolution images can be obtained by imaging devices, but the processing power of the computer memory and GPU required for image detection and classification is relatively low, which tends to become a bottleneck in computer processing. Therefore, computer processing is generally made possible by reducing the size of the medium image, but in this case, image features important for bacterial species identification may be lost. Therefore, in this embodiment 1, the medium image is divided into multiple small images without being reduced, and the small images are input to the single colony learning model 2 and the multi-colony learning model 3, thereby improving the detection accuracy.

[0094] The single colony learning model 2 includes a position detection unit 21 that mainly detects the position of a colony, and a species classification unit 23 that identifies the species group that forms a colony and the certainty that includes the species group through image recognition. By separating the detection process from the classification process, the accuracy of identifying the species group can be improved.

[0095] Furthermore, the single-colony learning model 2 and the multi-colony learning model 3 can be configured to specify bacterial species groups rather than accurately classifying bacterial species by name, thereby improving the accuracy of bacterial species classification.

[0096] Furthermore, in this embodiment 1, as shown in FIG. 6, a plurality of bacterial species groups are individually cultured to create learning data, and the single colony learning model 2 is trained. Therefore, learning data for the single colony learning model 2 can be easily created. In addition, by configuring the single colony learning model 2 to input a small image including a single colony, the single colony learning model 2 generated using the above learning data can specify the position and species group of a single colony with high accuracy, and the confidence level including the species group. The belonging probability can be calculated with high accuracy based on the confidence level specified with high accuracy. In addition, the amount of bacteria can be calculated with high accuracy based on the position and species group of a single colony specified with high accuracy, and the confidence level including the species group.

[0097] Furthermore, in the first embodiment, as shown in FIG. 8, by using the single-colony learning model 2, learning data for the multi-colony learning model 3 can be easily created. In the first embodiment, an example in which a bacterial species group is identified has been described, but the bacterial species may be specified.

[0098] (Embodiment 2) The bacterial species identification support device 1 according to the second embodiment differs from that according to the first embodiment in the method of creating learning data. Since the other configurations of the bacterial species identification support device 1 are similar to those of the bacterial species identification support device 1 according to the first embodiment, the same reference numerals are used for similar parts and detailed descriptions are omitted.

[0099] 15 is a flowchart showing a method for creating learning data according to the second embodiment. In the second embodiment, in order to create learning data, a laboratory technician starts the monoculture of a group of multiple types of bacteria (step S211) and captures a medium image at a first time point (step S212). The first time point corresponds to the initial stage in which a single colony is formed. At this time point, the center position of the single colony in the medium image can be detected with high accuracy, but it is difficult to identify the bacterial group.

[0100] The bacterial species identification support device 1 acquires a medium image captured at a first time point (step S213), and detects the center position of a single colony based on the acquired medium image (step S214).

[0101] Next, the laboratory technician captures a medium image at a second time point (step S215). The second time point is when the single colonies have grown large enough to identify the bacterial species group. At the second time point, no multiple colonies have formed.

[0102] The bacterial species identification support device 1 acquires a medium image captured at a second time point (step S216), and identifies the bacterial species group of the single colony and the probability of it belonging to the group (step S217).

[0103] Next, the laboratory technician captures an image of the medium at a third time point (step S218). At the third time point, multiple colonies are formed.

[0104] The bacterial species identification support device 1 acquires the culture medium image captured at the third point in time (step S219), identifies the image portion of the multi-colony, and creates small images of the multi-colony by dividing the image portion into small images (step S220).

[0105] The bacterial species identification support device 1 creates learning data by adding teacher data indicating one or more bacterial species groups identified in step S217 to a small image including multiple colonies (step S221).

[0106] It is possible that the colony may grow to a size that allows a bacterial species group to be identified at the third time point. Therefore, the bacterial species group identified in step S217 plus the bacterial species group identified based on the medium image at the third time point may be added as training data to the small image of the multi-colony. Here, there are three types of medium images that can be used to identify the bacterial species group of a single colony: images from the first, second, and third time points. In each case, the position and bacterial species group of the single colony can be estimated, and the probability of the estimated bacterial species group affiliation can be compared at multiple time points. The bacterial species group name with the highest probability of affiliation can then be adopted as the bacterial species group name of the single colony at that position.

[0107] Next, the learning data created in step S221 is used to perform machine learning on the image recognition model (the multi-colony learning model 3 before learning), thereby generating the multi-colony learning model 3 (step S222).

[0108] According to the bacterial species identification support device 1 of the second embodiment, the accuracy of the teacher data for the multi-colony learning model 3, that is, the accuracy of the bacterial species group, can be improved. Therefore, the recognition accuracy of the multi-colony learning model 3 can be improved, and the bacterial species identification support device 1 can more accurately identify one or more bacterial species groups included in the multi-colony. In addition, since the number of single colonies (first time point) included in the multi-colony at the third time point is counted and the bacterial species forming the multi-colony at the third time point is identified (second time point), the single colonies forming the multi-colony at the third time point can be counted for each bacterial species. Furthermore, by adding the single colonies not forming a multi-colony at the third time point counted for each bacterial species to this number, the bacterial amount (number of bacteria) for each bacterial species in the sample at the time of application to the medium can be identified. Note that, if the sample is diluted before application to the medium, the concentration of the bacterial amount for each bacterial species in the sample can be identified by multiplying the reciprocal of the dilution rate.

[0109] Although a method for generating the multi-colony learning model 3 has been described here, the positions and species groups of single colonies detected and identified in steps S214 and S217 may be used as training data to machine-train the single-colony learning model 2.

[0110] (Embodiment 3) A bacterial species identification support device 301 according to the third embodiment differs from that according to the first embodiment in the configuration of a multi-colony learning model 3. Other configurations of the bacterial species identification support device 301 are similar to those of the bacterial species identification support device 1 according to the first embodiment, so similar parts are given the same reference numerals and detailed descriptions are omitted.

[0111] 16 is a block diagram showing an example of the configuration of a bacterial species identification support device 301 according to embodiment 3. The storage unit 13 of the bacterial species identification support device 301 according to embodiment 3 includes a first multi-colony learning model 303a and a second multi-colony learning model 303b.

[0112] The first multi-colony learning model 303a is an image recognition model similar to the multi-colony learning model 3 of the first embodiment, and is configured to accurately identify one or more bacterial species groups contained in the multi-colony and their certainty factors.

[0113] The second multi-colony learning model 303b is an object detection model or image segmentation model such as YOLOv3, U-Net, Faster RCNN, SSD, etc., which detects multiple colony regions forming a multi-colony and calculates the bacterial species group forming each colony and its certainty. The position and area of ​​the colony image detected by the second multi-colony learning model 303b are expressed as an image portion surrounded by, for example, a circular or rectangular bound, or an irregular boundary line. However, the area detection accuracy and the identification accuracy of the bacterial species group of the second multi-colony learning model 303b tend to be lower than those of the single colony learning model 2 or the first multi-colony learning model 303a because the image features to be learned tend to be complex.

[0114] The method of generating the second multi-colony learning model 303b is as follows. A medium image including multiple colonies is divided into small images. Then, training data indicating the colony areas and bacterial species groups included in the small images is added to the small images to generate training data. The training data is manually generated by checking the medium image and the small images. Next, the second multi-colony learning model 303b is generated by machine learning an object detection model (the second multi-colony learning model 303b before learning) using the created training data.

[0115] Fig. 17 is a flowchart showing the processing procedure of image diagnosis in the bacterial species identification support method according to embodiment 3. The bacterial species identification support device 301 according to embodiment 3 executes the same processes as steps S71 to S77 in embodiment 1 in steps S371 to S375. However, in Fig. 17, the calculation processes of the belonging probability and the bacterial amount are omitted.

[0116] Next, the bacterial species identification support device 301 inputs the small image including the multi-colony to the first multi-colony learning model 303a to specify one or more bacterial species groups forming the multi-colony and the confidence that the bacterial species group is included (step S376). Although omitted in Fig. 17, the belonging probability and the bacterial amount are calculated as in the first embodiment.

[0117] Then, the bacterial species identification support device 301 judges whether or not there is a bacterial species group that is not included in a single colony but is included in a multi-colony based on the identification results of steps S375 and S376 (step S377). Next, the bacterial species identification support device 301 inputs a small image including a multi-colony into the second multi-colony learning model 303b to identify the position of each colony forming a multi-colony, the bacterial species group, and its certainty or belonging probability (step S378). The process of step S378 may be executed only when there is a bacterial species group that is not included in a single colony but is included in a multi-colony. Whether or not there is truly a bacterial species group that is not included in a single colony but is included in a multi-colony obtained in step S377 can be confirmed by separately isolating and culturing the candidate bacterial species group, isolating it as a single colony, and then identifying the bacterial species group of the single colony by the method of embodiment 1. By replacing the output data in step S377 with this identified bacterial species group, the discrimination accuracy of the second multi-colony learning model 303b can be improved.

[0118] The bacterial species identification support device 301 executes post-processing such as identifying resident bacteria (step S379), displays the image processing results on the display unit 15 (step S380), and ends the process.

[0119] For example, the bacterial species identification support device 301 may display on the display unit 15 the position of the colonies forming the multi-colony and the information on the bacterial species group specified in step S378 together with the judgment result in step S377. Specifically, the bacterial species identification support device 301 may display, in the image portion of the multi-colony or its vicinity, characters, symbols, and the color of a frame image indicating one or more bacterial species groups forming the multi-colony, or a transparent color indicating the bacterial species group for the area inside the frame image, as the recognition result by the first multi-colony learning model 303a, and may display an indefinite area surrounding the multiple colonies forming the multi-colony (hereinafter, referred to as an indefinite bound) as the recognition result by the second multi-colony learning model 303b. The area includes a frame image surrounding the image portion of each colony forming the multi-colony, and characters or symbols or the color of a frame image indicating the bacterial species group, or a transparent color indicating the bacterial species group for the area inside the frame image. It is not necessary to surround all colonies with an indefinite bound, and it is sufficient to display an indefinite bound for some colonies. Furthermore, the bacterial species identification support device 301 may selectively superimpose and display an amorphous bound indicating a bacterial species group that is not included in a single colony but is included in a multi-colony on the medium image.

[0120] The laboratory technician can refer to the output result of the bacterial species identification support device 301 and pick up bacteria from the multi-colony and perform isolation and culture for the bacterial species group that is buried in the multi-colony and has not been sufficiently separated and identified. Specifically, the judgment result of step S377 allows the laboratory technician to confirm the possibility that there is a bacterial species group that is not included in a single colony and is buried in the multi-colony. In addition, the location of such a bacterial species group can be estimated based on the identification result of step S378.

[0121] In addition to the above explanation, the following items are added. (Appendix 1) Acquire an image of a culture medium containing a multi-colony formed by a group of multiple colonies formed by the cultured multiple types of bacteria; When an image including a multi-colony is input, image recognition is performed and the acquired image is input to a multi-colony learning model that outputs information indicating the type of bacteria that forms the multi-colony, thereby identifying the multiple bacterial species contained in the multi-colony. A method for assisting in the identification of bacterial species.

[0122] (Appendix 2) Acquire an image of a culture medium including a single colony formed by the cultured bacteria and a multi-colony formed by a group of multiple colonies formed by the cultured multiple types of bacteria; Identifying the location and species of a single colony by inputting the acquired image into a single colony learning model that outputs information indicating the location of the single colony and the species of bacteria that form the single colony when an image including a single colony is input; The acquired image is input to the multi-colony learning model to identify the multiple bacterial species contained in the multi-colony. A method for supporting bacterial species identification as described in Appendix 1.

[0123] (Appendix 3) The multi-colony learning model does not recognize the positions of the colonies of the multiple bacterial species that constitute the multi-colony, and outputs information indicating the multiple bacterial species contained in the multi-colony. A method for supporting bacterial species identification as described in Appendix 2.

[0124] (Appendix 4) Divide the acquired image of the medium into multiple small images; The divided small images are input to the single colony learning model to identify the location and species of the single colony. A bacterial species identification support method according to claim 2 or 3.

[0125] (Appendix 5) The single colony learning model is Detect the location and size of a single colony using an object detection model; Cut out the image containing a single colony. The extracted image is input into an image recognition model to identify the species of the single colony. A bacterial species identification support method according to any one of claims 2 to 4.

[0126] (Appendix 6) Divide the acquired image of the medium into multiple small images each containing a single colony; The divided small images are input to the single colony learning model to identify the location and species of the single colony. A bacterial species identification support method according to claim 2 or 3.

[0127] (Appendix 7) Colonies larger than the identified single colony are identified as multi-colonies; The image of the identified multi-colony is input to the multi-colony learning model to identify the multiple bacterial species contained in the multi-colony. A bacterial species identification support method as described in Appendix 6.

[0128] (Appendix 8) The information indicating the species output by the single colony learning model is information indicating a group of a specific species. A bacterial species identification support method according to any one of claims 2 to 7.

[0129] (Appendix 9) Identifying the probability that a bacterial species identified as forming a single colony forms said single colony A bacterial species identification support method according to any one of appendices 2 to 8.

[0130] (Appendix 10) Display the identified species and the probability A bacterial species identification support method as described in Appendix 9.

[0131] (Appendix 11) Accept the specified probability, Displaying an image of the medium together with an image of the medium, an image marking a single colony having a probability equal to or greater than the specified probability or equal to or less than the specified probability, an image masking images other than a single colony having a probability equal to or greater than the specified probability or equal to or less than the specified probability, or an image masking a single colony having a probability equal to or greater than the specified probability or equal to or less than the specified probability. A bacterial species identification support method as described in appendix 10.

[0132] (Appendix 12) Calculate the amount of bacteria that form one or more single colonies in the medium. A bacterial species identification support method according to any one of claims 2 to 11.

[0133] (Appendix 13) Calculate the probability that the multiple bacterial species identified as forming multiple colonies form the multiple colonies. A bacterial species identification support method according to any one of claims 1 to 12.

[0134] (Appendix 14) Calculate the amount of bacteria of one or more multi-colony forming bacteria species contained in the medium A bacterial species identification support method according to any one of claims 1 to 8.

[0135] (Appendix 15) Accepting amendments to the types of identified bacterial species; If the correction is accepted, change the type of the identified species. A bacterial species identification support method according to any one of claims 1 to 14.

[0136] (Appendix 9) A method for generating a single colony learning model, which outputs information indicating a position of the single colony and a bacterial species forming the single colony when an image of a medium containing a single colony formed by a cultured bacterium is input, comprising: obtaining a first image including a single colony formed by a cultured first species of bacteria and a second image including a single colony formed by a cultured second species of bacteria; Dividing the acquired first and second images into small images each including one single colony; creating learning data by providing training data indicating the first bacterial species and training data indicating the second bacterial species to the small images of the first image and the second image, respectively; The learning model is trained by machine learning so that, when a small image of the first image is input, information indicating the first bacterial species indicated by the training data is output, and, when a small image of the second image is input, information indicating the second bacterial species indicated by the training data is output. How to generate single-colony learning models. [Explanation of symbols]

[0137] 1. Bacteria identification support device 1a Division processing section 1b Morphological classification section 2. Single-colony learning model 2a 1st bacteria amount calculation section 3. Multi-colony learning model 3a 2nd bacteria amount calculation section 4. Display Processing Section 5. Correction Processing Section 6 Results display screen 10 Recording media 11 Arithmetic section 12. Memory 13 Storage section 14 Control section 15 Display section 21 Position detection unit 22 Bacterial species classification department 61 Culture medium image display section 62 Bacteria species affiliation probability display section 63 Modification operation section 131 Computer Programs 303a First Multi-Colony Learning Model 303b Second Multi-Colony Learning Model

Claims

1. A method for detecting a bacterial infection by culture using a culture medium including a single colony formed by cultured bacteria and a multi-colony formed by a group of multiple colonies formed by cultured multiple types of bacteria, The acquired image of the medium is divided into multiple small images each containing a single colony. When an image including a single colony is input, the position and the species of the single colony are specified by inputting the divided small images into a single colony learning model that outputs information indicating the position of the single colony and the species of the bacteria that forms the single colony; Colonies larger than the identified single colony are identified as multi-colonies; When an image including a multi-colony is input, image recognition is performed, and the multiple species of bacteria contained in the multi-colony are identified by inputting the identified image of the multi-colony into a multi-colony learning model that outputs information indicating the multiple species of bacteria contained in the multi-colony without recognizing the positions of the colonies of the multiple species of bacteria that constitute the multi-colony. A method for assisting in the identification of bacterial species.

2. Divide the acquired image of the medium into multiple small images; The divided small images are input to the single colony learning model to identify the location and species of the single colony. The method for supporting bacterial species identification according to claim 1 .

3. The single colony learning model is Detect the location and size of a single colony using an object detection model; Cut out the image containing a single colony. The extracted image is input into an image recognition model to identify the species of the single colony. The method for supporting bacterial species identification according to claim 1 or 2.

4. The information indicating the species output by the single colony learning model is information indicating a group of a specific species. The method for supporting bacterial species identification according to any one of claims 1 to 3.

5. Identifying the probability that a bacterial species identified as forming a single colony forms said single colony The method for supporting bacterial species identification according to any one of claims 1 to 4.

6. Display the identified species and the probability The method for supporting bacterial species identification according to claim 5.

7. Accept the specified probability, Displaying an image of the medium together with an image of the medium, an image marking a single colony having a probability equal to or greater than the specified probability or equal to or less than the specified probability, an image masking images other than a single colony having a probability equal to or greater than the specified probability or equal to or less than the specified probability, or an image masking a single colony having a probability equal to or greater than the specified probability or equal to or less than the specified probability. The method for supporting bacterial species identification according to claim 6.

8. Calculate the probability that the multiple bacterial species identified as forming multiple colonies form the multiple colonies. The method for supporting bacterial species identification according to any one of claims 1 to 7.

9. Accepting amendments to the types of identified bacterial species; If the correction is accepted, change the type of the identified species. The method for supporting bacterial species identification according to any one of claims 1 to 8.

10. An acquisition unit for acquiring images of a culture medium including a single colony formed by a cultured bacterium and a multi-colony formed by a collection of multiple colonies formed by a plurality of cultured types of bacteria; a single colony learning model that outputs, when an image including a single colony is input, information indicating the position of the single colony and the species of the bacteria that forms the single colony; a multi-colony learning model that, when an image including a multi-colony is input, performs image recognition and outputs information indicating the multiple bacterial species contained in the multi-colony without recognizing the positions of the colonies of the multiple bacterial species that constitute the multi-colony; Calculation section and Equipped with The calculation unit is Dividing the image of the medium acquired by the acquisition unit into a plurality of small images including a single colony; The position and the species of a single colony are identified by inputting the divided small images into the single colony learning model; Colonies larger than the identified single colony are identified as multi-colonies; An image of the identified multi-colony is input to the multi-colony learning model, thereby identifying the multiple bacterial species contained in the multi-colony. Bacteria identification support device.

11. Acquiring an image of a culture medium including a single colony formed by a cultured bacterium and a multi-colony formed by a collection of multiple colonies formed by a plurality of cultured types of bacteria, The acquired image of the medium is divided into multiple small images each containing a single colony. When an image including a single colony is input, the position and the species of the single colony are specified by inputting the divided small images into a single colony learning model that outputs information indicating the position of the single colony and the species of the bacteria that forms the single colony; Colonies larger than the identified single colony are identified as multi-colonies; When an image including a multi-colony is input, image recognition is performed, and the multiple species of bacteria contained in the multi-colony are identified by inputting the identified image of the multi-colony into a multi-colony learning model that outputs information indicating the multiple species of bacteria contained in the multi-colony without recognizing the positions of the colonies of the multiple species of bacteria that constitute the multi-colony. A computer program for causing a computer to execute a process.

Images