Program, display method, and display system

Abstract

To provide a program or the like that allows a computer to execute processing of outputting a photographed cultured bacterial colony image and a current image of a cultured bacterial colony which is obtained by photographing a petri dish, in a comparable manner.SOLUTION: A program acquires a first image obtained by photographing a bacterial colony on a petri dish, acquires a second image obtained by newly photographing a bacterial colony on the petri dish, and allows a computer to execute a processing of outputting the first image and the second image which have been acquired on a display part in a comparable manner.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a program or the like that causes a computer to execute processing for outputting an image.

Background Art

[0002] In bacterial culture identification tests, techniques for automatically determining bacterial species from images of cultured bacterial colonies have been proposed. However, it is realistically difficult to identify bacterial species with a 100% accuracy rate using such techniques. For accurate bacterial species determination, visual confirmation of cultured bacterial colonies by an inspector such as a clinical laboratory technologist is essential. In addition, by using the bacterial cells constituting colonies isolated to a single bacterial species after bacterial culture for identification tests such as MALDI-TOF-MS (Matrix Assisted Laser Desorption / Ionization - Time of Flight - Mass Spectrometry), it is possible to estimate the bacterial species name of the bacterial cells with higher accuracy. However, performing identification tests on all of a large number of colonies is not realistic because it increases test costs and test time. For this reason, visual confirmation by inspectors is mandatory. In addition, when confirming the presence or absence of an infectious disease caused by a specific microorganism in an infectious disease patient, overlooking the presence (colony) of a specific type of microorganism (particularly one that causes severe infectious diseases), for example overlooking even a single type of colony, will lead to serious diagnostic errors. Therefore, in general, in clinical testing sites, measures to prevent errors have been devised, for example, by having multiple inspectors perform duplicate checks.

[0003] For inspectors to visually confirm cultured bacterial colonies, it is necessary to display the image analysis results on a monitor screen and then observe the petri dish containing the cultured bacteria. The cultured bacterial colonies to be observed are small, each only a few millimeters in diameter. Furthermore, since multiple cultured bacterial colonies are present on the petri dish, identifying the target colony is not easy. After identifying (estimating) the bacterial species of the cultured bacterial colonies on the petri dish, if subsequent tests are to be performed, it is necessary to collect isolated bacterial cells from the agar plate to separate them into a single species. The bacterial suspension obtained by collecting the cultured bacterial colonies from the petri dish is then suspended in physiological saline or culture medium prepared in a well plate or test tube to make the bacterial suspension easier to handle and use in subsequent testing steps. In the above steps, if the inspector collects cultured bacterial colonies from the petri dish again, it is necessary to select the corresponding well on a 96-well plate or the corresponding test tube on a test tube rack with multiple test tubes as the suspension destination. Performing this location confirmation multiple times is time-consuming and carries a high risk of error.

[0004] Therefore, there is a need for technology to support the operation of work and the equipment used to perform that work. Patent Document 1 discloses an operation support system in which the position of the operating means to be operated by the user is detected from image information captured by a camera, and a projector projects given information (for example, a light spot) to the position of the detected operating means. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2005-300971 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, while the above-mentioned conventional techniques can assist in operations, they have difficulty assisting in the process of comparing images of cultured bacterial colonies with actual cultured bacterial colonies on a petri dish. The present invention has been made in view of this situation. Its purpose is to provide a program or the like that performs a process to output images of cultured bacterial colonies that have been captured and images of the cultured bacterial colonies obtained by photographing a petri dish in a comparable manner. [Means for solving the problem]

[0007] A program according to one aspect of the present invention causes a computer to perform the following processes: acquire a first image of bacterial colonies on a petri dish that has already been photographed; acquire a second image of bacterial colonies on the petri dish that has been newly photographed; and output the acquired first and second images to a display unit in a manner that allows for comparison. [Effects of the Invention]

[0008] In one aspect of this invention, it is possible to output a first image and a second image in a comparable manner. The first image is a previously taken image of a cultured bacterial colony, and the second image is a current image of the colony obtained by photographing a petri dish. [Brief explanation of the drawing]

[0009] [Figure 1] This is an explanatory diagram showing an example configuration of the support system. [Figure 2] This is an explanatory diagram illustrating an example of a workflow for determining bacterial species. [Figure 3] This is a block diagram showing an example of the hardware configuration of an information processing device. [Figure 4] This is an explanatory diagram showing an example of an image database. [Figure 5] This flowchart shows an example of the image display processing procedure. [Figure 6] This is an explanatory diagram showing an example of a display screen. [Figure 7] This flowchart shows an example of the lighting control process procedure. [Figure 8]This is a flowchart showing an example of the procedure of lighting control processing. [Figure 9] This is an explanatory diagram relating to selection of bacterial colonies. [Figure 10] This is an explanatory diagram showing another configuration example of the support system. [Figure 11] This is a flowchart showing another example of the procedure of illumination control processing. [Figure 12] This is an explanatory diagram showing a state in which bacterial colonies are illuminated by lighting. [Figure 13] This is an explanatory diagram showing another configuration example of the support system. [Figure 14] This is a block diagram showing an example of the hardware configuration of a tablet terminal. [Figure 15] This is a flowchart showing another example of the procedure of image display processing. [Figure 16] This is an explanatory diagram showing another example of a display screen. [Mode for Carrying Out the Invention]

[0010] (Embodiment 1) Hereinafter, embodiments will be described with reference to the drawings. Figure 1 is an explanatory diagram showing a configuration example of a support system. The support system 100 includes an information processing device 1, an installation stand 2, a camera 3, a lighting device 4, a code reader 5, and an image DB 6. Further, Figure 1 shows a petri dish 7 and an agar medium 8 formed in the petri dish 7.

[0011] The information processing device 1 is configured by a computer such as a desktop PC (Personal Computer), a notebook PC, or a tablet computer. The information processing device 1 may be configured by a multi-computer composed of a plurality of computers, a virtual machine virtually constructed by software, or a quantum computer. Furthermore, the functions of the information processing device 1 may be realized by a cloud service.

[0012] The mounting table 2 is a table on which the petri dish 7 is placed, for an inspector to perform visual confirmation of cultured bacterial colonies (hereinafter referred to as "bacterial colonies"). The mounting table 2 is provided with a reference line CL that serves as a reference for the placement position of the petri dish 7.

[0013] The camera 3 includes an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. The camera 3 is installed above the mounting table 2, and photographs the agar medium 8 formed in the petri dish 7 placed on the mounting table 2. The camera 3 transmits the image obtained by photographing to the information processing device 1.

[0014] The lighting device 4 is a lighting device capable of surface light emission. In addition, the lighting device 4 has a function of illuminating a part of the light emitting surface and displaying the other part as dark. A tablet computer can be used as the lighting device 4. In the following description, it is assumed that the lighting device 4 is configured by a tablet computer. In FIG. 1, the portion of the mounting table 2 where the petri dish 7 is placed is made of transparent glass or an acrylic plate, so that the illumination from the lighting device 4 passes through and illuminates the petri dish 7. The lighting device 4 is an example of a light emitting device.

[0015] The code reader 5 reads a one-dimensional code, a two-dimensional code, or an RFID (radio-frequency identification device) code. The code reader 5 transmits the read code to the information processing device 1. The code reader 5 is an example of a reader.

[0016] The image DB 6 stores captured images obtained by the camera 3 photographing the agar medium 8 formed in the petri dish 7. The image DB 6 stores the images and a petri dish number that can uniquely identify the petri dish 7.

[0017] Petri dish 7 is marked with a barcode 71. The barcode 71 contains a petri dish number for individual identification of petri dish 7. Instead of the barcode 71, a two-dimensional code or an RFID tag may be attached to petri dish 7.

[0018] Bacterial colonies have formed on the agar medium 8 as a result of culturing bacteria. In the example in Figure 1, three types of bacteria have formed a total of four bacterial colonies. Bacterial colony 81 shows two bacterial colonies of the same bacterium. Bacterial colonies 82 and 83 show bacterial colonies of two different bacteria, respectively.

[0019] Next, we will explain the workflow for visual identification of bacterial species in bacterial testing. Figure 2 is an explanatory diagram showing an example of the workflow for bacterial species identification. The agar medium 8 on which bacterial colonies are cultured is photographed with camera 3. The barcode on the petri dish 7 containing the agar medium 8 is read with code reader 5. The captured images are recorded in image DB 6, corresponding to the petri dish number. From the captured images, the AI ​​(machine learning model) detects the location of the bacterial colonies and then estimates the bacterial species forming those colonies. Alternatively, the first inspector visually estimates the bacterial species. The estimated bacterial species result is stored in image DB 6. At this time, the position coordinates and diameter of the colonies are also stored in image DB 6. The second inspector confirms the estimation result made by the AI ​​(machine learning model) or the first inspector. The support system 100 assists the second inspector in the confirmation process. After confirmation by the second inspector, subsequent tests are performed according to the bacterial species.

[0020] Figure 3 is a block diagram showing an example of the hardware configuration of an information processing device. The information processing device 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a display panel 14, an operation unit 15, a serial communication unit 16, an image output unit 17, a communication unit 18, and a reading unit 19. Each component is connected by bus B.

[0021] The control unit 11 has one or more arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The control unit 11 reads and executes a control program 1P (program, program product) stored in the auxiliary storage unit 13, thereby performing various information processing, control processing, etc., and realizing functional units such as a code acquisition unit, a first image acquisition unit, a second image acquisition unit, and an output unit.

[0022] The main memory unit 12 consists of SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc. The main memory unit 12 primarily temporarily stores data necessary for the control unit 11 to perform arithmetic processing.

[0023] The auxiliary storage unit 13 is a hard disk or SSD (Solid State Drive), and stores the control program 1P and various databases (DBs) necessary for the control unit 11 to execute processing. The various databases stored in the auxiliary storage unit 13 may be stored in a database server or cloud storage different from that of the information processing device 1. The image DB 6 may also be stored in the auxiliary storage unit 13.

[0024] The display panel 14 can be made of a liquid crystal panel or an organic EL (Electro-Luminescence) display, etc. The operation unit 15 can be made of, for example, a touch panel incorporated into the display panel 14, and can perform predetermined operations on the display panel 14. The operation unit 15 can also perform operations on a software keyboard displayed on the display panel 14. The operation unit 15 may also be a hardware keyboard, mouse, etc.

[0025] The serial communication unit 16 communicates serially with the camera 3 and receives captured images from the camera 3. The serial communication unit 16 also communicates serially with the code reader 5 and receives codes read from barcodes, 2D barcodes, or RFID tags. For example, the serial communication conforms to the USB (Universal Serial Bus) standard.

[0026] The image output unit 17 outputs an image to be displayed on the lighting device 4. The communication unit 18 communicates with the image database 6 via the network.

[0027] The reading unit 19 reads a portable storage medium 1a, including CD (Compact Disc)-ROM and DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 19 and store it in the auxiliary storage unit 13. Alternatively, the control unit 11 may download the control program 1P from another computer via a network N or the like and store it in the auxiliary storage unit 13. Furthermore, the control unit 11 may also read the control program 1P from the semiconductor memory 1b.

[0028] Figure 4 is an explanatory diagram showing an example of an image database. Image DB6 includes a Petri dish number column, an image column, a position coordinate column, a diameter column, and a bacterial species column. In this embodiment, the Petri dish number column, image column, and position coordinate column are essential. The diameter column is optional. The bacterial species column is not required in this embodiment. The Petri dish number column stores the Petri dish number. The image column stores the file ID associated with the image of Petri dish 7 taken by camera 3. The image is a photograph of the agar medium 8 formed in Petri dish 7. The image column may store the file name of the image file instead of the image itself. In this case, the image file is stored separately in Image DB6. The position coordinate column stores the coordinates of the center position of the bacterial colony. These coordinates are coordinate values ​​in the image coordinate system. The diameter column stores the diameter of the circle because the morphology of the bacterial colony is close to a perfect circle. This diameter is an estimated value of the actual length calculated from the length on the image. The bacterial species column stores the bacterial species of the bacteria constituting the bacterial colony. The multiple images stored in image DB6 are an example of the first image group.

[0029] Figure 5 is a flowchart showing an example of the image display processing procedure. The second inspector places the petri dish 7 to be used to confirm the bacterial species estimation result on the mounting stand 2, following the guideline CL, so that the barcode 71 faces the reading section of the code reader 5. The second inspector starts the image display processing. The control unit 11 of the information processing device 1 obtains the petri dish number from the code reader 5 (step S1). The control unit 11 obtains the image associated with the petri dish number from the image DB 6 (step S2). The control unit 11 performs image processing on the first image obtained from the image DB 6 (step S3). This image processing involves scaling, resizing, or rotating the first image so that when the first image and the second image obtained from the camera 3 are displayed side by side, the correspondence between the colonies in both images can be visually confirmed. The control unit 11 outputs the first and second images to the display panel 14 (step S4) and terminates the processing. Instead of a barcode 71, identification information consisting of alphanumeric characters or symbols may be attached to Petri dish 7. The inspector visually reads the identification information and inputs it into the information processing device 1.

[0030] Let me add some details about the image processing in step S4. In the image processing in step S3, feature matching is performed first. Features extracted from the first image and the second image are matched. Feature matching can be achieved using known techniques such as AKAZE (Accelerated KAZE), SURF (Speeded-Up Robust Features), SIFT (Scale Invariant Feature Transform), kNN (k-Nearest Neighbor), or Brute-Force Matcher.

[0031] Based on the results of feature matching, processing such as alignment of the second image is performed. Several methods for transforming the coordinates of two corresponding images are known, but a linear transformation (affine transformation) can be used here. The affine transformation can be calculated from the coordinate values ​​of four corresponding feature points in the first and second images. In the affine transformation, the coordinates before the transformation (x n ,y n ) and the transformed coordinates (X n , Y n The relationship between ) can be expressed by the following equation (1).

[0032]

number

[0033] λ is the scaling factor, θ is the rotation angle, (T x , T y ) indicates the translation distance. λ, θ, and (Tx, Ty) can be determined by the coordinates of the four feature points before and after the affine transformation. As a result, it is possible to display the first and second images with their positions, sizes, and orientations aligned, that is, to output the first and second images to the display panel 14 in a way that allows for comparison.

[0034] Although the image processing was performed on the first image, it may also be performed on the second image. In this case, the image processing may be performed on predetermined frames rather than on all frames constituting the video output from camera 3. Alternatively, a still image may be captured by camera 3, and the image processing may be performed on that still image before displaying it.

[0035] Figure 6 is an explanatory diagram showing an example of a display screen. Display screen d01 includes a camera image area d011, an estimated image area d012, an attribute table d013, and a sequence number d014. The camera image area d011 displays an image of the agar medium 8 formed in petri dish 7, captured by camera 3. The estimated image area d012 displays an image obtained from a previous capture of the agar medium 8 by camera 3. The attribute table d013 displays the diameter of the bacterial colonies. The sequence number d014 displays a number to allow the attribute table to be associated with the bacterial colonies.

[0036] Using the image displayed in the estimated image area d012, the AI ​​or the first inspector performs an estimation of the bacterial species. The second inspector can determine whether the bacterial species estimated by the AI ​​or determined by the first inspector is valid by referring to the estimation or determination result and the image displayed in the camera image area d011.

[0037] (Control of lighting equipment) This section describes how to control the lighting device 4 to facilitate observation of bacterial colonies. For example, if a bacterial colony is selected on the display screen d01, the lighting will be directed only at the selected bacterial colony. Further explanation follows.

[0038] Figure 7 is a flowchart showing an example of the lighting control process procedure. The second inspector performs an operation using the operation unit 15 in the estimated image area d012 of the display screen d01. The control unit 11 of the information processing device 1 accepts the selection (step S21). The control unit 11 determines whether the accepted operation is a bacterial colony selection operation (step S22). A selection operation is, for example, specifying the bacterial colony to be illuminated using a pointing device such as a mouse. If the control unit 11 determines that the accepted operation is a bacterial colony selection operation (YES in step S22), it performs lighting control processing (step S23) and returns the process to step S21. The lighting control processing will be described later with reference to Figure 8. If the control unit 11 determines that the accepted operation is not a bacterial colony selection operation (NO in step S22), it determines whether it is a light-off operation (step S24). A light-off operation is, for example, an operation to deselect all selected bacterial colonies. If the control unit 11 determines that the received operation is a light-off operation (YES in step S24), it performs a light-off control (step S25) and returns the process to step S21. For example, light-off control involves outputting a completely black image to the tablet computer used as a lighting device, making the screen used as lighting a dark screen. If the control unit 11 determines that the received operation is not a light-off operation (NO in step S24), it determines whether it is a termination operation (step S26). If the control unit 11 determines that it is not a termination operation (NO in step S26), it returns the process to step S21. If the control unit 11 determines that it is a termination operation (YES in step S26), it terminates the process.

[0039] Figure 8 is a flowchart showing an example of the lighting control process. The control unit 11 obtains the position coordinates of the selected bacterial colony (step S41). The position coordinates are the coordinates on the display screen d01. The control unit 11 identifies the selected bacterial colony (step S42). The control unit 11 converts the position coordinates obtained in step S41 into coordinates in the image displayed in the estimated image area d012. By referring to the converted coordinates and the values ​​in the position column and diameter column of the image DB6, the control unit 11 can identify the selected bacterial colony. The control unit 11 generates an image for illumination (step S43). For example, the control unit 11 generates an image in which the area of ​​the circle identified from the values ​​in the position column and diameter column of the image DB6 is white, and the other area is black. The control unit 11 may enlarge or reduce the generated image to match the specifications of the screen of the tablet computer used as an illumination device. The control unit 11 outputs the image (step S44) and returns the process to the caller. The above image, which enables the tablet computer to function as an illumination device, is an example of an illumination command.

[0040] Figure 9 is an explanatory diagram regarding the selection of bacterial colonies. Figure 9A shows an example of the display screen. In Figure 9A, the same reference numerals are used for display elements similar to those in Figure 6. In Figure 9A, bacterial colony number 4 is selected. For example, if you move the mouse pointer MP to the display area of ​​the bacterial colony you want to select and click the mouse, that bacterial colony will be selected. If you move the mouse pointer MP to an area other than the bacterial colony and click the mouse, the selection will be deselected. Alternatively, the bacterial colony may be selected when the mouse pointer MP hovers over the display area of ​​the bacterial colony you want to select. As shown in attribute table d013 in Figure 9A, the selected bacterial colony may be indicated by hatching the background of the corresponding cell. Alternatively, bacterial colony selection may be made possible by selecting a cell in attribute table d013. It is not limited to one bacterial colony that can be selected at a time; multiple bacterial colonies may be selected.

[0041] Figure 9B is an illustrative diagram illustrating how selected bacterial colonies are illuminated by lighting. Bacterial colony 83, number 4, is being illuminated by light from lighting device 4.

[0042] This embodiment provides the following advantages. The second image captured by camera 3 is displayed side-by-side with the first image used by the AI ​​to estimate the bacterial species or by the first inspector to determine the bacterial species, so that their size and orientation match. This makes it easy to visually grasp the correspondence between bacterial colonies in the first image and those in the second image. Furthermore, when a bacterial colony is selected, the selected bacterial colony is illuminated, allowing the second inspector to observe it thoroughly under favorable conditions. As a result, the second inspector can not only efficiently visually confirm the bacterial species of the colony, but also efficiently and easily perform the subsequent inspection step of collecting (picking) bacterial colonies on the selected colony. The petri dish 7 is placed on the stand 2 so that the barcode 71 attached to it faces the code reader 5, reducing the difference between the orientation of the second image and the orientation of the first image. This reduces the image processing required to align the orientations of the first and second images. If an RFID tag is attached to Petri dish 7 instead of a barcode 71, the code reader 5 can still read the code from the RFID tag even if Petri dish 7 is not positioned so that the RFID tag is facing the code reader 5. However, it is desirable to attach some kind of mark to Petri dish 7 for alignment purposes in order to minimize the difference between the orientation of the second image and the orientation of the first image.

[0043] (Embodiment 2) This embodiment relates to a configuration in which the lighting device 4 is a projector. A projector is an optical projection device that enlarges and projects / displays video images, such as the operation screen of a personal computer, onto a screen. Figure 10 is an explanatory diagram showing another example of the configuration of the support system. The support system 100 includes an information processing device 1, a mounting base 2, a camera 3, a lighting device 4, a code reader 5, and an image DB 6. The information processing device 1, mounting base 2, camera 3, code reader 5, and image DB 6 are the same as in Embodiment 1, so their description is omitted.

[0044] The lighting device 4 is installed above the mounting base 2, similar to the camera 3. The installation position and display magnification of the lighting device 4 are adjusted so that the projection range of the lighting device 4 overlaps with the petri dish 7 placed on the mounting base 2.

[0045] When a bacterial colony is selected on the display screen, the control unit 11 of the information processing device 1 generates an image in which only that bacterial colony is illuminated by white light, and outputs the generated image to the illumination device 4. In Figure 10, bacterial colony 83, number 4, is selected. Light 41 from the illumination device 4 is then irradiated onto the bacterial colony 83.

[0046] This embodiment provides the same effects as Embodiment 1, plus the following: Since a projector is used as the lighting device 4, compared to Embodiment 1, which illuminates the petri dish containing the culture medium from below, it is possible to increase the amount of light irradiated onto the bacterial colonies because there is no attenuation effect of the culture medium on the illumination light.

[0047] (Embodiment 3) This embodiment relates to a configuration in which bacterial colonies are selected based on their species to be irradiated with light from the lighting device 4. The hardware configuration of this embodiment is the same as that of the embodiment described above, so its description will be omitted. The following description will mainly focus on the differences from the embodiment described above.

[0048] In this embodiment, the bacterial species column in image DB6 is essential. The bacterial species column stores the bacterial species name of each bacterial colony. The bacterial species stored is either the result estimated by the AI ​​or the result determined by the first inspector. In image DB6, a bacterial group column may be provided instead of the bacterial species column, and the bacterial group name may be stored for each bacterial colony. In this case, in the following description, the bacterial species name will be replaced with the bacterial group name.

[0049] In this embodiment, the image display processing is the same as in the embodiment described above, but the bacterial colony selection operation is slightly different. In this embodiment, bacterial colonies are selected using the bacterial species name. The lighting control processing will now be described. Figure 11 is a flowchart showing another example of the lighting control processing procedure. The control unit 11 of the information processing device 1 acquires the bacterial species name selected by the second inspector (step S51). When the second inspector selects a bacterial species name from the bacterial species menu on the display screen, the control unit 11 acquires the selected bacterial species name. The control unit 11 refers to the image DB 6 and identifies a bacterial colony with the same bacterial species name as the acquired bacterial species name (step S52). The control unit 11 generates an image in which the identified bacterial colony is illuminated by white light, etc. (step S53). The control unit 11 outputs the generated image to the lighting device 4 (tablet computer or projector) (step S54) and returns the process to the caller.

[0050] The bacterial species selection operation may be performed by selecting a bacterial colony, similar to the embodiment described above. In this case, two selection modes are provided. One is a mode for selecting a bacterial colony. In this mode, the operation is the same as in the embodiment described above. The other is a mode for selecting a bacterial species. In this case, the control unit 11 obtains the position coordinates of the selected bacterial colony. The control unit 11 converts the obtained position coordinates into coordinates in the image. The control unit 11 identifies the selected bacterial colony by referring to the converted coordinates and the values ​​in the position column and diameter column of the image DB6, and obtains the bacterial species name of the bacterial colony from the bacterial species column of the image DB6.

[0051] Figure 12 is an explanatory diagram showing how bacterial colonies are illuminated. In Figure 12, the screen of the information processing device 1 displays the bacterial species menu d015. The bacterial species menu d015 displays Escherichia coli and Pseudomonas aeruginosa as options. Here, Escherichia coli is selected. The four colonies labeled with the code 81 correspond to the E. coli colonies. Therefore, the projector 4 is illuminating the four bacterial colonies 81.

[0052] In addition to the effects of the above-described embodiment, this embodiment provides the following effects. Even if a bacterial colony contains a novel bacterial species that is not targeted for detection, it is not easy for the AI ​​to identify it. In this case, the AI ​​estimates that the novel bacterial species is a known species. As a result, among the bacterial colonies of the selected species name, there will be some that have different morphological characteristics. The second inspector can identify the novel bacterial species by visual inspection based on the differences in morphological characteristics. Therefore, this method allows the second inspector to efficiently correct errors in the AI's estimation results.

[0053] By combining Embodiment 1 and Embodiment 2 described above, the lighting device 4 may be configured to include a tablet computer and a projector. Depending on the bacterial species and the state of the bacterial colony, switching between illuminating from above or below allows for observation of the bacterial colony under appropriate lighting conditions. Furthermore, the lighting may be applied simultaneously from both above and below.

[0054] (Embodiment 4) This embodiment is a configuration that narrows down the required hardware. Figure 13 is an explanatory diagram showing another example of the support system configuration. The support system 100 includes a tablet terminal 20, a code reader 5, and an image database 6. The code reader 5 and the image database 6 are the same as in the embodiment described above, so their description is omitted.

[0055] The touch panel display of the tablet terminal 20 is divided into two parts for use. One part is used as a display unit 241 that displays images read from the image DB 6. The other part is used as an illumination unit that places a petri dish 7 and illuminates the agar medium 8 formed in the petri dish 7. Guide lines are displayed on the illumination unit to indicate the placement of the petri dish 7. A transparent film sheet with the guide lines drawn on it may be attached. Alternatively, two rod-shaped members 9 may be attached to the illumination unit for positioning the petri dish 7. The two rod-shaped members 9 are placed on the same circumference at a predetermined distance apart. This predetermined distance should match the horizontal length of the barcode 71 attached to the petri dish 7. By doing so, the petri dish 7 can be positioned so that the barcode 71 fits between the two rod-shaped members 9, thereby aligning the orientation of the petri dish 7. The rod-shaped members 9 may be cylindrical as shown in Figure 13, but their horizontal cross-section may be crescent-shaped to match the outer surface of the petri dish 7. Instead of the two rod-shaped members 9, a wall member with a circular arc shape in plan view may be used. The horizontal length of the wall member is configured to match the horizontal length of the barcode 71. When setting up the petri dish 7, the barcode is positioned to overlap with the wall member. This makes it possible to set up the petri dishes 7 in the same orientation. In addition, the height of the wall member is set so that part of the barcode 71 is not hidden. This makes it possible to read the barcode 71 with the code reader 5. As shown in Figure 13, since the touch panel display of the tablet terminal 20 is provided with a display unit 241 and an illumination unit, the display area of ​​the touch panel display must be at least twice the bottom area of ​​the petri dish 7.

[0056] Figure 14 is a block diagram illustrating an example of the hardware configuration of a tablet device. The tablet device 20 consists of a tablet computer or a notebook PC that can be used as a tablet. The tablet device 20 includes a control unit 21, a main memory unit 22, an auxiliary memory unit 23, a display panel 24, a touch panel 25, a serial communication unit 26, and a communication unit 27. Each component is connected by bus B.

[0057] The control unit 21 has one or more arithmetic processing units such as CPUs, MPUs, and GPUs. The control unit 21 reads and executes control programs 20P (programs, program products) stored in the auxiliary storage unit 23, thereby performing various information processing, control processing, etc., and realizing various functional units.

[0058] The main memory unit 22 is an SRAM, DRAM, flash memory, etc. The main memory unit 22 primarily temporarily stores data necessary for the control unit 21 to perform calculations.

[0059] The auxiliary storage unit 23 is a hard disk or SSD, and stores the control program 20P and various databases necessary for the control unit 21 to execute processing. The various databases stored in the auxiliary storage unit 23 may be stored on a database server or cloud storage separate from the tablet terminal 20.

[0060] The display panel 24 can be made of a liquid crystal panel or an organic EL display, etc. The touch panel 25 is integrated into the display panel 24. The display panel 24 and the touch panel 25 constitute a touch panel display. The tablet terminal 20 may also be operated using a hardware keyboard, mouse, etc.

[0061] As described above, a petri dish 7 is placed on a portion of the touch panel display. It is desirable that the touch panel 25 be a projected capacitive type so that the touch panel 25 can operate even when the petri dish 7 is placed. Since the projected capacitive type supports multi-touch, even if the position of the petri dish 7 is detected, operations on the display unit 241 can also be detected. However, if operations on the display unit 241 can be detected even when the petri dish 7 is placed, other types of touch panels may be used as the touch panel 25 of the tablet terminal 20.

[0062] The serial communication unit 26 communicates with the code reader 5 via serial communication and receives codes read from barcodes or 2D barcodes. For example, the serial communication unit 26 communicates in accordance with USB standards and Bluetooth® standards. The communication unit 27 communicates with the image DB 6 via the network.

[0063] Figure 15 is a flowchart showing another example of the image display processing procedure. The second inspector places the petri dish 7 to be examined for bacterial species estimation in the illumination unit, following the guideline, so that the barcode 71 is positioned between the two rod-shaped members 9. The second inspector has the code reader 5 read the barcode 71. The control unit 21 of the tablet terminal 20 obtains the petri dish number from the code reader 5 (step S71). The control unit 21 obtains the image associated with the petri dish number from the image DB 6 (step S72). The control unit 21 enlarges or reduces the first image obtained from the image DB 6 (step S73). The enlargement or reduction is to ensure that when the first image is displayed on the display panel 24, the size of the petri dish 7 in the first image and the size of the actual petri dish 7 viewed from above are visually similar. The magnification or reduction ratio can be determined from the size of the image and the size of the display unit 241 provided on the touch panel display. The control unit 11 outputs the first image and the second image to the display panel 24 (step S74), and then terminates the process.

[0064] The lighting control process and the ignition control process are the same as in the embodiment described above, so a detailed explanation will be omitted. In this embodiment, the output destination of the image generated by the lighting control process is the lighting section of the touch panel display. Furthermore, the output image is not an image that illuminates only the selected bacterial colonies, but an image in which the center position of the bacterial colonies intersects with cross-shaped illumination. By illuminating the petri dish 7 with cross-shaped light 242 as shown in Figure 13, the second inspector can easily grasp the position of the selected bacterial colonies on the petri dish 7.

[0065] This embodiment provides the following advantages. When observing Petri dish 7, the first image used by the AI ​​to estimate the bacterial species, or when the first inspector to determine the bacterial species, is displayed on the display unit 241 in a manner that matches how the Petri dish 7 actually appears to the second inspector. This makes it easy to grasp the correspondence between the bacterial colonies shown in the first image and the bacterial colonies actually being observed. Furthermore, when a bacterial colony selection operation is performed, the selected bacterial colony is illuminated with cross-shaped lighting, allowing the second inspector to observe the bacterial colony sufficiently under favorable conditions. In addition, the hardware configuration eliminates the need for the information processing device 1, the mounting stand 2, the camera 3, and the projector as the lighting device 4, thereby reducing hardware costs and the man-hours required to assemble the support system 100.

[0066] Furthermore, although the input process becomes more complicated, if the second inspector inputs the petri dish number, which is printed alongside the barcode 71, into the tablet terminal 20, the code reader 5 will also be unnecessary. Additionally, although the process becomes more complicated, if a tablet terminal 20 with a camera is used, and the barcode 71 on petri dish 7 is photographed with the camera before placing petri dish 7 on the tablet terminal 20, the code reader 5 can also be eliminated.

[0067] (modified version) While the illumination light for Petri dish 7 is assumed to be white light, it is not limited to this. The color of the illumination light may be changed depending on the bacterial species. Furthermore, the intensity of the illumination light may also be changed depending on the bacterial species and the reliability (output value) of the bacterial species estimation by AI. The intensity and color of the light may also be changed manually.

[0068] (Correction of bacterial species, instructions for follow-up testing) The second inspector will correct the bacterial species if they determine that the AI ​​or the first inspector's estimation is inappropriate. The second inspector will also give instructions for subsequent testing. Bacterial colonies to be tested for subsequent testing will be picked from Petri dish 7. The picked bacteria will be suspended in the corresponding wells on a 96-well plate or in the corresponding test tubes on a test tube rack.

[0069] Figure 16 is an explanatory diagram showing another example of a display screen. Display screen d02 includes a camera image d021, an estimated image d022, a marker d023, a colony number d024, a display icon d025, a hide icon d026, and a list table d027.

[0070] Camera image d021 is an image of petri dish 7 taken by camera 3. Estimated image d022 is the image input to the AI ​​for estimating bacterial species, or the image when the first inspector determined the bacterial species. Marker d023 is an overlay display of the outline of the bacterial colony. Colony number d024 is an overlay display of the sequential number of the bacterial colony. The sequential number is used to establish a correspondence with the record shown in list d027. Display icon d025 and hide icon d026 are icons that switch the display state of marker d023 and colony number d024, which are overlaid on estimated image d022, between display and hide states. Display icon d025 indicates that marker d023 and colony number d024 are displayed. Hide icon d026 indicates that marker d023 and colony number d024 are hidden. Clicking or performing other actions on the display icon d025 will switch it to the hidden icon d026, and the marker d023 and colony number d024 will be hidden. Clicking or performing other actions on the hidden icon d026 will switch it back to the display icon d025, and the marker d023 and colony number d024 will be displayed. The list table d027 displays information for each bacterial colony, such as the estimated bacterial species.

[0071] Table d027 includes columns for number, estimated bacterial species, estimated bacterial group, MALDI, susceptibility testing, simple identification, bacterial quantity, and remarks. The number column displays the sequential number of the colony. The estimated bacterial species column displays the bacterial species estimated by the AI ​​or determined by the first inspector. The estimated bacterial group column displays the bacterial group corresponding to the bacterial species. The MALDI column indicates whether or not a subsequent test, such as MALDI-TOF-MS testing (matrix-assisted laser desorption / ionization mass spectrometry), should be performed. The susceptibility testing column indicates whether or not a subsequent test, such as drug susceptibility testing, should be performed. The simple identification column indicates whether or not a subsequent test, such as simple identification of the bacterial species, should be performed. A check mark in the MALDI, susceptibility testing, and simple identification columns indicates that the test should be performed, while a minus sign indicates that the test does not need to be performed. The bacterial quantity column displays the bacterial quantity in the bacterial colony. The remarks column displays comments entered by the first inspector. In Figure 16, the percentages displayed in the estimated bacterial column indicate the confidence level of the AI-generated estimation. The data displayed in list d027 is stored in image DB6, etc.

[0072] The second inspector can change the bacterial species displayed in the estimated bacterial column if they determine that it differs from the species visually confirmed and is incorrect. Furthermore, they can toggle the display of whether or not to perform tests on the MALDI column, susceptibility test column, and simple identification column by performing operations such as mouse clicks. The results of the second inspector's operations are stored in the image DB6 and the database that stores test results.

[0073] The technical features (constituent elements) described in each embodiment are combinable with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and is intended to include all modifications in the sense and scope equivalent to the claims. Furthermore, the claims use, but are not limited to, a multi-claim format in which claims refer to two or more other claims. They may also be written in a multi-claim format in which multi-claims refer to at least one multi-claim (multi-multi-claim). [Explanation of symbols]

[0074] 100: Support System 1: Information Processing Device 11: Control Unit 12: Main memory 13: Auxiliary storage section 14: Display Panel 15:Operation unit 16: Serial communication unit 17: Image output unit 18: Communications Department 19: Reading section 1P: Control Program 1a: Portable storage medium 1b: Semiconductor memory 2: Installation stand 3: Camera 4: Lighting equipment 5: Code reader 6: Image Database 7: Petri dish 20: Tablet devices 21: Control Unit 22: Main memory 23: Auxiliary storage section 24: Display Panel 25: Touch panel 26: Serial communication unit 27: Communications Department 20P: Control Program B: Bus N: Network

Claims

1. A first image, which is stored in the memory unit after photographing a bacterial colony on a petri dish, is retrieved from the memory unit. The light-emitting device that irradiates the petri dish with light outputs an irradiation command to the position on the petri dish corresponding to the position identified in the acquired first image. With the light-emitting device irradiating the aforementioned position on the petri dish with light in accordance with the irradiation command, a second image of the bacterial colony on the petri dish is newly captured. The acquired first image and second image are output to the display unit in a comparable manner. A program that instructs a computer to perform a process.

2. The code attached to the petri dish is read, and the first image corresponding to the read code is obtained from the first image group previously taken and stored in the memory unit. The program according to claim 1.

3. The code is obtained from the petri dish, which is arranged so that the reader and the code face each other, via the reader. The program according to claim 2.

4. Rotate, enlarge, reduce, or perform a coordinate transformation on either the first or second image so that the first and second images can be compared. The program according to any one of claims 1 to 3.

5. The selection of the bacterial colonies in the first image is accepted. The light-emitting device located on the underside of the petri dish outputs an irradiation command to the position of the petri dish corresponding to the selected bacterial colony. The program according to any one of claims 1 to 3.

6. The selection of the bacterial colonies in the first image is accepted. The projector, located above or below the petri dish, outputs an irradiation command to the position on the petri dish corresponding to the received position. The program according to any one of claims 1 to 3.

7. The bacterial colonies in the first image are classified into several types depending on the bacterial species. The selection of the aforementioned bacterial species is accepted. Identify the location of bacterial colonies corresponding to the accepted bacterial species, The light-emitting device located below the petri dish outputs an irradiation command to a position on the petri dish corresponding to the specified position. The program according to any one of claims 1 to 3.

8. The bacterial colonies in the first image are classified into several types depending on the bacterial species. The selection of the aforementioned bacterial species is accepted. Identify the location of bacterial colonies corresponding to the accepted bacterial species, The projector, located above or below the petri dish, outputs an irradiation command to the position of the petri dish corresponding to the specified location. The program according to any one of claims 1 to 3.

9. Computers A first image, which is stored in the memory unit after photographing a bacterial colony on a petri dish, is retrieved from the memory unit. The light-emitting device that irradiates the petri dish with light outputs an irradiation command to the position on the petri dish corresponding to the position identified in the acquired first image. With the light-emitting device irradiating the aforementioned position on the petri dish with light in accordance with the irradiation command, a second image of the bacterial colony on the petri dish is newly captured. The acquired first and second images are output to the display unit in a way that allows for comparison. A method for displaying the execution of a process.

10. A camera to photograph bacterial colonies on a petri dish, A reader for reading the code attached to the petri dish, A code acquisition unit that acquires the code from a petri dish arranged so that the reader and the code face each other, through the reader. A first image acquisition unit acquires a first image corresponding to an acquired code from a first group of images previously captured by the aforementioned camera. A light-emitting device that irradiates light onto the petri dish is provided with an irradiation command output unit that outputs an irradiation command to a position on the petri dish corresponding to a position identified in the acquired first image. A second image acquisition unit acquires a second image of the bacterial colonies on the petri dish using the camera while the light-emitting device irradiates light onto the position on the petri dish in accordance with the irradiation command, and Output unit that displays the acquired first and second images on the display unit in a comparable manner. Information processing device having A display system equipped with the following features.

Images