Inspection system, inspection method, and inspection program

Abstract

An inspection system and inspection method are provided that reduce the time required to inspect environmental microorganisms. [Solution] An inspection system for inspecting microorganisms or mold that occur in indoor environments or equipment, which has an imaging unit that directly photographs samples collected from the indoor environment or equipment, and an output unit that inspects microorganisms or mold in image data photographed by the imaging unit and outputs the inspection results.

Description

[Technical field]

[0001] The present disclosure relates to an inspection system 、 Testing method and inspection programs Regarding. [Background technology]

[0002] Conventionally, there have been services that test the presence and type of environmental microorganisms (e.g., mold) contained in dust inside air conditioners and provide the test results. By providing such services, service providers can make suggestions (e.g., cleaning the air conditioner, replacing the filters, etc.) to realize an optimal air environment according to the test results. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-195454 A [Patent Document 2] Special Publication No. 2020-529869 [Patent Document 3] International Publication No. 2019 / 074926 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the past, to test for the presence and type of environmental microorganisms, a culture method was used in which collected samples (such as dust containing environmental microorganisms) were cultured in a culture medium, and it took a long time to provide the test results.

[0005] The present disclosure provides a testing system and a testing method that reduce the time required to test for environmental microorganisms. [Means for solving the problem]

[0006] A first aspect of the present disclosure is a method for manufacturing a semiconductor device comprising: An inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: an imaging unit that directly images a sample collected from an indoor environment or a device; and an output section for inspecting image data captured by the imaging section for microorganisms or fungi and outputting the inspection results.

[0007] According to the first aspect of the present disclosure, it is possible to provide a testing system that shortens the time required to test for environmental microorganisms.

[0008] In addition, the second aspect of the present disclosure is An inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: an imaging unit that captures an image of the appearance of a microorganism or mold individual sampled from an indoor environment or a device; and an output section for inspecting image data captured by the imaging section for microorganisms or fungi and outputting the inspection results.

[0009] According to the second aspect of the present disclosure, it is possible to provide a testing system that shortens the time required for testing environmental microorganisms.

[0010] A third aspect of the present disclosure is an inspection system according to the first or second aspect, comprising: The imaging target captured by the imaging unit is a solution collected from an indoor environment or a device, and the imaging unit captures an image of the imaging target through a lens under visible light or ultraviolet light.

[0011] A fourth aspect of the present disclosure is the inspection system according to the third aspect, The solution is obtained by dissolving a sample taken from the room environment or from an instrument.

[0012] A fifth aspect of the present disclosure is the inspection system according to the third aspect, The samples were collected from air conditioners, air purifiers, humidifiers, ventilation equipment, fans, and indoor environment surfaces.

[0013] A sixth aspect of the present disclosure is the inspection system according to the third aspect, The image data includes a first trained model that determines attributes of each region in the image data.

[0014] A seventh aspect of the present disclosure is the inspection system according to the sixth aspect, The system has a second trained model that determines the type of microorganism or mold for an area determined to contain a microorganism or mold by the first trained model.

[0015] An eighth aspect of the present disclosure is an inspection system according to any one of the first to seventh aspects, The output section counts and outputs the number of microorganisms or fungi by type in the image data captured by the imaging section.

[0016] In addition, a ninth aspect of the present disclosure is An inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: an imaging unit that directly images a sample collected from an indoor environment or a device; an output unit that inspects the image data captured by the imaging unit for microorganisms or fungi and outputs the inspection results; The output unit displays the type of microorganism or mold, the number or ratio of microorganisms or mold, and image data captured by the imaging unit as the inspection results, as well as one of information indicating the indoor environment or equipment, a description of the microorganism or mold, the level of contamination, and comparison results with other inspection results.

[0017] In addition, a tenth aspect of the present disclosure is An inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: an imaging unit that captures an image of the appearance of a microorganism or mold individual sampled from an indoor environment or a device; an output unit that inspects the image data captured by the imaging unit for microorganisms or fungi and outputs the inspection results; The output unit displays the type of microorganism or mold, the number or ratio of microorganisms or mold, and image data captured by the imaging unit as the inspection results, as well as one of information indicating the indoor environment or equipment, a description of the microorganism or mold, the level of contamination, and comparison results with other inspection results.

[0018] An eleventh aspect of the present disclosure is an inspection system according to any one of the third to seventh aspects, The imaging target is obtained by dispersing a collected sample into a solution.

[0019] A twelfth aspect of the present disclosure is the inspection system according to the eleventh aspect, The imaging target is obtained by dispersing a sample taken from an air conditioner into a solution.

[0020] A thirteenth aspect of the present disclosure is an inspection system according to the eleventh or twelfth aspect, The imaging target is obtained by dispersing a collected sample in physiological saline in which a surfactant has been dissolved, to form a solution.

[0021] A fourteenth aspect of the present disclosure is an inspection system according to any one of the eleventh to thirteenth aspects, The imaging subject includes a first solution obtained by dispersing a collected sample in physiological saline containing a surfactant, and a second solution obtained by further diluting the first solution with physiological saline containing a surfactant.

[0022] A fifteenth aspect of the present disclosure is an inspection system according to the seventh aspect, Each region whose attributes are determined by the first trained model is 32×32 pixels or larger.

[0023] A sixteenth aspect of the present disclosure is the inspection system according to the seventh aspect, The first trained model is generated by performing a learning process using learning data in which image data containing the same type of microorganism or mold is associated with location information in which the location of each area in the image data that is determined to contain a microorganism or mold is identified by multiple coordinates.

[0024] A seventeenth aspect of the present disclosure is the inspection system according to the seventh aspect, When the second trained model determines the type of the microorganism or fungus, it counts the microorganism or fungus for each determined type.

[0025] An eighteenth aspect of the present disclosure is an inspection system according to any one of the first to seventeenth aspects, the inspection system is a mobile terminal, The imaging unit is built into the mobile terminal.

[0026] A nineteenth aspect of the present disclosure is the inspection system according to the seventh aspect, the inspection system includes a portable imaging stand on which the imaging target is placed, and a mobile terminal connected to the imaging stand; The imaging stand has the imaging unit, The mobile terminal has the first trained model, the second trained model, and the output unit.

[0027] A twentieth aspect of the present disclosure is an inspection system according to any one of the seventh, fifteenth, or sixteenth aspects, comprising: The first trained model is trained YOLO.

[0028] A twenty-first aspect of the present disclosure is an inspection system according to the seventh or seventeenth aspect, The second trained model is generated by performing a learning process using training data in which partial image data of each area determined to contain a microorganism or mold in image data containing the same type of microorganism or mold is associated with the type of the microorganism or mold.

[0029] A twenty-second aspect of the present disclosure is an inspection system according to any one of the seventh, seventeenth, or twenty-first aspects, comprising: The second trained model is a trained DML.

[0030] In addition, a twenty-third aspect of the present disclosure is An inspection method for an inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: taking a direct image of the sample taken from the room environment or device; and a step of inspecting the photographed image data for microorganisms or fungi and outputting the inspection results.

[0031] In addition, a twenty-fourth aspect of the present disclosure is An inspection method for an inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: A step of photographing the appearance of a microorganism or mold individual collected from an indoor environment or equipment; and a step of inspecting the photographed image data for microorganisms or mold and outputting the inspection results. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a first diagram showing an application example of the testing service providing system in the learning phase. [Diagram 2] FIG. 1 is a first diagram showing an application example of the inspection service providing system in the inspection phase. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile terminal and an image processing device. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of functions related to learning data generation processing of the image processing device. [Diagram 5] FIG. 2 is a diagram illustrating an example of a functional configuration of functions related to a learning process of the image processing device. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of a function related to an inspection process of a mobile terminal. [Figure 7]FIG. 13 is a diagram showing an example of providing test results by a mobile terminal. [Figure 8] 13 is a flowchart showing the flow of a learning process. [Figure 9] 13 is a flowchart showing a flow of an inspection process. [Figure 10] FIG. 2 is a second diagram showing an application example of the testing service providing system in the learning phase. [Figure 11] FIG. 2 is a second diagram showing an application example of the inspection service providing system in the inspection phase. [Figure 12] FIG. 13 is a diagram showing an example of a report including inspection results. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0034] [First embodiment] <Testing service provision system during the learning phase> First, an application example of the testing service providing system in the learning phase, among the testing service providing systems including the testing system according to the first embodiment, will be described. Fig. 1 is a first diagram showing an application example of the testing service providing system in the learning phase. As shown in Fig. 1, in the learning phase, the testing service providing system 100 has a mobile terminal 120 (an example of the testing system according to the first embodiment), an imaging stand 130, and an image processing device 140.

[0035] In the learning phase, the imaging target to be photographed using the mobile terminal 120 is acquired in the following procedure, and is directly photographed under visible light (for example, under fluorescent light) or under ultraviolet light. (1) An experimenter collects a sample (dust 160) from an air conditioner (indoor unit) 150 for testing (note that the sample may be collected from inside the air conditioner (indoor unit) 150 or from the outer surface of the air conditioner (indoor unit) 150). (2) The collected sample is dispersed and dissolved in a physiological saline solution 170 containing a surfactant. Note that "dispersing the sample and dissolving it" may also include "drying the dispersed and dissolved sample." (3) The soluble sample is collected from the solution 170 (for example, 10 μl) and dropped onto a slide glass to generate a preparation. At this time, samples 181_1 to 181_6 are collected so that only the same type of environmental microorganism is included, and each type is dropped onto a different slide glass to generate preparations 180_1 to 180_6. Note that the example in FIG. 1 shows a case where the sample (dust 160) collected from the air conditioner (indoor unit) 150 contains six types of environmental microorganisms. (4) The generated slides 180_1 to 180_6 are sequentially placed on the imaging stand 130 as imaging targets, and are photographed under visible light (eg, fluorescent light) or ultraviolet light using the mobile terminal 120.

[0036] In other words, "taking pictures directly" in the learning phase means that the steps from collecting the sample (1) to photographing (4) do not include "cultivating environmental microorganisms" (photographing is done without multiplying the environmental microorganisms from the state in which the sample is collected). Also, "photographing" refers to obtaining information on the appearance (color and shape) of individual environmental microorganisms (in the case of mold, an individual mold, specifically one unit of spores and hyphae; in the case of bacteria, one individual bacterial organism).

[0037] The imaging stand 130 is configured to be portable, and has a lens 131, a mounting portion 132, and a lens support portion 133, as shown in FIG.

[0038] The lens 131 magnifies the image pickup target (in the example of FIG. 1, the slide 180_1) to a predetermined magnification.

[0039] The placement section 132 is a member on which an imaging target (in the example of FIG. 1, a slide 180_1) is placed.

[0040] Lens support section 133 is a member that supports lens 131 at a position a predetermined distance away from the imaging target placed on mounting section 132, and is a member on whose upper surface mobile terminal 120 is placed. Lens support section 133 may be provided with a lifting mechanism for varying the distance between lens 131 and the imaging target placed on mounting section 132.

[0041] The mobile terminal 120 has an internal imaging device 121 (an example of an imaging section) and is placed on the lens support section 133 so that the position of the lens 131 coincides with the position of the imaging device 121, thereby capturing an image of an object to be imaged (in the example of FIG. 1, slide 180_1) under visible light.

[0042] In addition, in the learning phase, the mobile terminal 120 transmits the captured image data 122_1 to the image processing device 140.

[0043] The image processing device 140 generates learning data based on the image data 122_1 transmitted from the mobile terminal 120. The image processing device 140 also performs a learning process on the learning model using the generated learning data to generate a trained model. The trained model generated in the learning phase is installed in the mobile terminal 120 in the inspection phase.

[0044] <Testing service provision system in the testing phase> Next, an application example of the inspection service providing system in the inspection phase, among the inspection service providing systems including the inspection system according to the first embodiment, will be described. Fig. 2 is a first diagram showing an application example of the inspection service providing system in the inspection phase. As shown in Fig. 2, in the inspection phase, the inspection service providing system 100 has a mobile terminal 120 (an example of the inspection system according to the first embodiment) and an imaging stand 130.

[0045] In the inspection phase, the image of the object to be imaged using the mobile terminal 120 is acquired by the following procedure, and is directly imaged under visible light (eg, under fluorescent light) or under ultraviolet light. (1) The service provider collects a sample (dust 220) from the air conditioner (indoor unit) 210 of the user to whom the test results will be provided (note that the sample may be collected from inside the air conditioner (indoor unit) 210 or from the outer surface of the air conditioner (indoor unit) 210). (2) The collected sample is dispersed and dissolved in a physiological saline solution 230 containing a surfactant. Note that "dispersing the sample and dissolving it" may also include "drying the dispersed and dissolved sample." (3) The solution 241 is taken from the solution 230 (for example, 10 μl) and dropped onto a slide glass to produce a preparation 240. (4) The prepared slide 240 is placed on the imaging stand 130 as an imaging target, and is photographed under visible light (eg, fluorescent light) or ultraviolet light using the mobile terminal 120.

[0046] In other words, "taking pictures directly" in the inspection phase means that the steps from sample collection (1) to photographing (4) do not include "cultivating environmental microorganisms" (photographing is done without multiplying the environmental microorganisms from the state in which the sample is collected). Also, "photographing" refers to obtaining information on the appearance (color and shape) of individual environmental microorganisms (in the case of mold, individual molds, specifically, one unit of spores and hyphae; in the case of bacteria, one individual bacterial organism).

[0047] 2, the imaging stand 130 has a lens 131, a mounting portion 132, and a lens support portion 133. Note that the imaging stand 130 used in the inspection phase is similar to the imaging stand 130 used in the learning phase (FIG. 1), and therefore detailed description of each portion will be omitted here.

[0048] The mobile terminal 120 has an internal imaging device 121 and is placed on the lens support section 133 so that the position of the lens 131 coincides with the position of the imaging device 121, thereby capturing an image of the subject (in the example of FIG. 2, slide 240) under visible light (e.g., under fluorescent light) or ultraviolet light.

[0049] In addition, in the inspection phase, a trained model is installed in the mobile terminal 120, and the captured image data 250 is processed using the trained model to provide inspection results 260.

[0050] In this way, by configuring the testing service providing system 100 to use a trained model to test for the presence and type of environmental microorganisms, the following effects can be enjoyed. Environmental microorganisms in captured image data can be accurately identified, even if they are similar in shape regardless of species, such as mold (which belongs to the fungus kingdom, particularly yeasts and ascomycetes). - Because there is no need to use a culture method, it is possible to shorten the time required to provide test results. For example, when a service provider visits a user to whom the test results will be provided, they can take a sample from the user's air conditioner and provide the test results on the spot. As a result, they can immediately make suggestions to realize the optimal air environment based on the test results (cleaning the air conditioner, replacing the filters, etc.). Since there is no need to use culture methods, it is possible to reduce the costs required to provide test results (there is no need to request testing from a specialized company). - Since there is no need to use culture methods, it is possible to identify dead environmental microorganisms (i.e., not only live but also dead bacteria) (eliminating the need to collect live environmental microorganisms).

[0051] <Hardware configuration of mobile terminal and image processing device> Next, a description will be given of the hardware configuration of the mobile terminal 120 and the image processing device 140. Fig. 3 is a diagram showing an example of the hardware configuration of the mobile terminal and the image processing device.

[0052] (1) Hardware configuration of mobile terminal Fig. 3(a) is an example of a hardware configuration of the mobile terminal 120. As shown in Fig. 3(a), the mobile terminal 120 has a processor 301, a memory 302, an auxiliary storage device 303, a display device 304, an operation device 305, a communication device 306, and an imaging device 121. The hardware components of the mobile terminal 120 are connected to each other via a bus 307.

[0053] The processor 301 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 301 reads various programs (for example, an inspection program in the inspection phase) onto the memory 302 and executes them.

[0054] The memory 302 has a main storage device such as a read only memory (ROM) and a random access memory (RAM). The processor 301 and the memory 302 form a so-called computer, and the processor 301 executes various programs read onto the memory 302, causing the computer to realize various functions.

[0055] The auxiliary storage device 303 stores various programs and various data used when the processor 301 executes the various programs.

[0056] The display device 304 is a display device that displays the captured image data 122_1, 250, the examination results 260, etc. The operation device 305 is an input device that is used when inputting various instructions to the mobile terminal 120.

[0057] The communication unit 306 is, for example, a communication device for communicating with the image processing device 140. The imaging device 121 captures images of the slides 180_1 to 180_6 and 240 that are the imaging targets.

[0058] It should be noted that the various programs installed in the auxiliary storage device 303 are assumed to be installed by being downloaded from a network via the communication device 306, for example.

[0059] (2) Hardware configuration of image processing device 3(b) shows an example of a hardware configuration of the image processing device 140. Note that the hardware configuration of the image processing device 140 is generally the same as the hardware configuration of the mobile terminal 120, so the following description will focus on the differences from the mobile terminal 120.

[0060] The processor 321, for example, reads out a learning program or the like onto the memory 322 and executes it. The auxiliary storage device 323, for example, realizes a learning data storage unit (described later).

[0061] The drive device 327 is a device for setting the recording medium 329. The recording medium 329 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 329 may also include semiconductor memories that record information electrically, such as ROM, flash memory, etc.

[0062] The various programs to be installed in the auxiliary storage device 323 are installed, for example, by setting the distributed recording medium 329 in the drive device 327 and reading out the various programs recorded in the recording medium 329 by the drive device 327. Alternatively, the various programs to be installed in the auxiliary storage device 323 may be installed by being downloaded from a network via the communication device 326.

[0063] <Functional configuration of image processing device> Next, a functional configuration of the image processing device 140 will be described. As described above, a learning program is installed in the image processing device 140, and the image processing device 140 performs the following operations by executing the learning program: Functions related to the learning data generation process for generating learning data; - Functions related to learning processing that performs learning processing on a learning model using learning data; Therefore, in the following, these functions realized by the image processing device 140 will be explained separately.

[0064] (1) Functions related to the generation and processing of learning data First, functions related to the learning data generation process will be described. Fig. 4 is a diagram showing an example of a functional configuration of functions related to the learning data generation process of an image processing device. As shown in Fig. 4, the image processing device 140 has an image data acquisition unit 410, a correct label acquisition unit 420, and a learning data generation unit 430 as functions related to the learning data generation process.

[0065] The image data acquisition unit 410 acquires image data 122_1 to 122_6 captured under visible light (e.g., fluorescent light) or ultraviolet light by the mobile terminal 120 in the learning phase. The image data 122_1 to 122_6 are image data captured of the slides 180_1 to 180_6 under visible light (e.g., fluorescent light) or ultraviolet light, respectively.

[0066] The correct label acquisition unit 420 acquires the correct label input by the experimenter 450. The experimenter 450 inputs the types of environmental microorganisms contained in the samples 181_1 to 181_6 of the slides 180_1 to 180_6, respectively, as correct data to the image processing device 140, and the correct label acquisition unit 420 acquires this.

[0067] In this embodiment, the experimenter 450 inputs "mold A" as the type of environmental microorganisms contained in the sample 181_1, and inputs "mold B" as the type of environmental microorganisms contained in the sample 181_2. The experimenter 450 also inputs "mold C" as the type of environmental microorganisms contained in the sample 181_3, and "mold D" as the type of environmental microorganisms contained in the sample 181_4. Furthermore, the experimenter 450 inputs "mold E" as the type of environmental microorganisms contained in the sample 181_5, and "mold F" as the type of environmental microorganisms contained in the sample 181_6.

[0068] The learning data generating unit 430 generates learning data by associating the image data 122_1 to 122_6 acquired by the image data acquiring unit 410 with the corresponding types of environmental microorganisms ("Mold A" to "Mold F") acquired by the correct label acquiring unit 420. The learning data generating unit 430 also stores the generated learning data in the learning data storage unit 440.

[0069] 4, learning data 441 to 446 indicate learning data generated by learning data generation unit 430 and stored in learning data storage unit 440. As shown in Fig. 4, learning data 441 to 446 have information items such as "image data", "image area", "partial image", and "correct label".

[0070] Among these, "image data" stores the file names of the image data acquired by the image data acquisition unit 410. In the example of Fig. 4, "image data 1" to "image data 6" are the file names of the image data 122_1 to 122_6, respectively.

[0071] The "image area" stores the position information (multiple (x coordinates, y coordinates), for example, the (x coordinates, y coordinates) of the two diagonal vertices of the area) of each area determined to contain one environmental microorganism in the image data. In the case of FIG. 4, for example, "area 1-1" (=coordinates (x 11a ,y 11a ), (x 11b ,y11b )) is stored.

[0072] "Partial image" stores the file name of the image data (partial image data) of each area determined to contain one environmental microorganism in the image data, as specified by "image area." Incidentally, containing one environmental microorganism means, for example, containing one spore unit when the environmental microorganism is mold.

[0073] The "correct label" stores the type of environmental microorganism in each area specified by the "image area." In the example of Fig. 4, "mold A" to "mold F" are stored in the "correct label" of learning data 441 to the "correct label" of learning data 446, respectively.

[0074] (2) Learning process functions Next, functions related to the learning process will be described. Fig. 5 is a diagram showing an example of a functional configuration of functions related to the learning process of an image processing device. As shown in Fig. 5, the image processing device 140 has a first learning unit 510 and a second learning unit 520 as functions related to the learning process.

[0075] The first learning unit 510 learns a process for determining an area containing environmental microorganisms in image data. Specifically, the first learning unit 510 includes a YOLO 511 and a comparison / change unit 512.

[0076] YOLO511 is a learning model that uses image data with a file name specified by "image data" as input in the learning data, and determines the location information and attribute information of each area within the image data (whether it is an environmental microorganism, etc.).

[0077] The comparison / change unit 512 compares the location information and attribute information of each area determined by YOLO 511 with the location information of each area determined to contain environmental microorganisms specified by the "image area" of the learning data, and calculates an error. The comparison / change unit 512 also backpropagates the calculated error and updates the model parameters of YOLO 511.

[0078] For example, in the case of the learning data 441, the comparison / change unit 512 By inputting image data 122_1 with file name = "Image data 1" into YOLO511, the location information and attribute information of each area output by YOLO511, Location information of each area determined to contain environmental microorganisms, identified by image area="area 1-1", "area 1-2", "area 1-3", ...; and calculate the error.

[0079] The second learning unit 520 learns a process for determining the type of environmental microorganism from the image data (partial image data) of each region determined to contain the environmental microorganism in the image data. Specifically, the second learning unit 520 includes a DML (Deep Metric Learning) 521 and a comparison / change unit 522.

[0080] DML 521 is a learning model that receives partial image data specified by "partial image" in the learning data as input and outputs types of environmental microorganisms (six types in this embodiment) (outputs six types of classification probabilities).

[0081] The comparison / change unit 522 compares the type of environmental microorganisms determined by the DML 521 with the type of environmental microorganisms (classification probability (e.g., 1.0)) identified by the "correct label" of the learning data, and calculates an error. The comparison / change unit 522 also back-propagates the calculated error, and updates the model parameters of the DML 521.

[0082] For example, in the case of the learning data 441, the comparison / change unit 522 By inputting the partial image data of the file name = "Image 1-1" into the DML521, the type of environmental microorganisms determined by the DML521 and · Correct answer label = "Mold A" and calculate the error.

[0083] <Functional configuration of mobile terminal> Next, a functional configuration of the mobile terminal 120 will be described. As described above, an inspection program is installed in the mobile terminal 120, and the mobile terminal 120 realizes a function related to the inspection process by executing the program. Therefore, the following describes the function related to the inspection process realized by the mobile terminal 120.

[0084] Fig. 6 is a diagram showing an example of a functional configuration of functions related to inspection processing of a mobile terminal. As shown in Fig. 6, the mobile terminal 120 has an image data acquisition unit 610, a first inference unit 620, a partial image extraction unit 630, a second inference unit 640, and an output unit 650 as functions related to inspection processing. The first inference unit 620, the partial image extraction unit 630, and the second inference unit 640 function as an identification unit that identifies environmental microorganisms in the image data.

[0085] The image data acquisition unit 610 acquires image data 660 captured under visible light (e.g., fluorescent light) or ultraviolet light in the inspection phase. The image data 660 is image data of the slide 240 captured under visible light (e.g., fluorescent light) or ultraviolet light.

[0086] The first inference unit 620 has a trained YOLO 621 (an example of a first trained model) generated by the first learning unit 510 performing a learning process on the YOLO 511 in the learning phase. The first inference unit 620 executes the trained YOLO 621 by inputting image data 660 to the trained YOLO 621. As a result, the trained YOLO 621 determines the position information of each area in the image data 660 and the attribute information of each area (whether or not it contains environmental microorganisms, etc.).

[0087] In the example of FIG. 6, in the image data 660, The coordinates (x 1a ,y 1a ), (x 1b ,y 1b ) and determining that the region contains environmental microorganisms; The coordinates (x 2a ,y2a ), (x 2b ,y 2b ) and determining that the region contains environmental microorganisms; The coordinates (x 3a ,y 3a ), (x 3b ,y 3b ) and determining that the region contains environmental microorganisms; The coordinates (x 4a ,y 4a ), (x 4b ,y 4b ) and determining that the region contains environmental microorganisms; The coordinates (x 5a ,y 5a ), (x 5b ,y 5b ) and determining that the region contains environmental microorganisms; Areas other than areas 1 to 5 are determined to be areas that do not contain environmental microorganisms. This shows how it was done.

[0088] The first inference unit 620 notifies the partial image extraction unit 630 and the output unit 650 of the position information of each area and the attribute information of each area in the image data 660 determined by the trained YOLO 621.

[0089] The partial image extraction unit 630 extracts image data (partial image data) of each region including environmental microorganisms in the image data 660 based on the position information of each region and the attribute information of each region in the image data 660 notified by the first inference unit 620. The example in Fig. 6 shows a state in which the partial image extraction unit 630 extracts partial image data 631 to 635 of regions 1 to 5.

[0090] The partial image extraction unit 630 notifies the second inference unit 640 of the extracted partial image data 631 to 635.

[0091] The second inference unit 640 has a trained DML 641 (second trained model) that the second learning unit 520 generates by performing a learning process on the DML 521 in the learning phase. The second inference unit 640 executes the trained DML 641 by inputting the partial image data 631-635 to the trained DML 641. As a result, the trained DML 641 determines the types of environmental microorganisms contained in each of the partial image data 631-635.

[0092] The example of Figure 6 shows how partial image data 631 is input and the trained DML 641 determines it to be "mold C," and how partial image data 632 is input and the trained DML 641 determines it to be "mold A." The example of Figure 6 also shows how partial image data 633 is input and the trained DML 641 determines it to be "mold A," and how partial image data 634 is input and the trained DML 641 determines it to be "mold B." The example of Figure 6 also shows how partial image data 635 is input and the trained DML 641 determines it to be "mold C."

[0093] The second inference unit 640 notifies the output unit 650 of the type of the environmental microorganism determined by the learned DML 641.

[0094] The output unit 650 generates test results by performing visualization processing based on the position information and attribute information of each area in the image data 660 notified by the first inference unit 620 and the type of environmental microorganisms notified by the second inference unit 640, and displays the test results on the display device 304.

[0095] The test results generated by the output unit 650 may include, for example, Image data in which each area determined to contain environmental microorganisms in the image data 660 is colored according to the type of environmental microorganism; - Aggregated data of the number (or percentage) of each type of environmental microorganism in the image data 660; etc. are included.

[0096] Thus, the "test" in the testing phase includes at least quantifying the types and numbers of environmental microorganisms.

[0097] <Examples of test results provided> Next, a description will be given of an example of providing test results by the mobile terminal 120. Fig. 7 is a diagram showing an example of providing test results by the mobile terminal.

[0098] Of these, Fig. 7(a) is an example of image data 710 in which each area determined to contain environmental microorganisms in image data 660 is colored according to the type of environmental microorganism. The example of Fig. 7(a) shows "mold C" colored blue, "mold A" colored red, and "mold C" colored yellow (for convenience, Fig. 7(a) shows different types of hatching).

[0099] Moreover, Fig. 7(b) is an example of aggregated data 720 that has been obtained by aggregating the number of each type of environmental microorganism in image data 660. As shown in Fig. 7(b), aggregated data 720 includes information items such as "mold type" and "number." Of these, "mold type" stores the names of six types of mold and names other than mold ("other"), and "number" stores the number of the corresponding type of mold or other numbers.

[0100] <Learning process flow> Next, a description will be given of the flow of the learning process by the testing service providing system 100. Fig. 8 is a flowchart showing the flow of the learning process.

[0101] In step S801, an experimenter 450 collects dust 160 from an air conditioner (indoor unit) 150 for testing.

[0102] In step S802, the experimenter 450 disperses the collected sample to turn it into a solution.

[0103] In step S803, the experimenter 450 classifies the solutions by type of environmental microorganisms.

[0104] In step S804, the experimenter 450 collects samples from each of the classified solutions and drops them onto slide glasses to generate slides 180_1 to 180_6. Furthermore, the experimenter 450 photographs the generated slides 180_1 to 180_6 (imaging targets) under visible light (e.g., fluorescent light) or ultraviolet light using the mobile terminal 120.

[0105] In step S805, the image processing device 140 generates learning data 441 to 446 for each type of environmental microorganism based on the captured image data.

[0106] In step S806, the image processing device 140 performs a learning process on YOLO using the generated learning data 441 to 446, and learns the position information and attribute information of each area in the image data.

[0107] In step S807, the image processing device 140 performs a learning process on the DML using the generated learning data 441 to 446, and learns the types of environmental microorganisms for partial image data of each region in the image data.

[0108] <Inspection process flow> Next, a description will be given of the flow of inspection processing by the inspection service providing system 100. Fig. 9 is a flowchart showing the flow of the inspection processing.

[0109] In step S901, the service provider collects dust 220 from the air conditioner (indoor unit) 210 of the user to whom the inspection results are to be provided.

[0110] In step S902, the service provider disperses the collected sample into a solution.

[0111] In step S903, the service provider collects a sample from the solution and drops it onto a slide glass to generate a slide 240. The service provider also photographs the generated slide 240 (imaging target) using the mobile terminal 120 under visible light or ultraviolet light.

[0112] In step S904, the mobile terminal 120 determines the position information and attribute information of each area in the captured image data, and extracts partial image data of each area determined to contain environmental microorganisms.

[0113] In step S905, the mobile terminal 120 determines the type of environmental microorganisms for each of the extracted partial image data.

[0114] In step S906, the mobile terminal 120 performs visualization processing based on the position information and attribute information of each area in the image data and the type information of the environmental microorganisms, and provides the test results to the user.

[0115] <Summary> As is clear from the above description, the mobile terminal 120, which is an example of the inspection system according to the first embodiment, The collected dust is dissolved and then dropped onto a glass slide to create a slide, which is then photographed through a lens under visible or ultraviolet light. In other words, the dust is photographed directly, or the external appearance of individual environmental microorganisms is photographed. Environmental microorganisms in the captured image data are identified using a trained YOLO that determines the attributes of each area in the image data, and a trained DML that determines the type of environmental microorganism for areas determined to contain environmental microorganisms. -Visualize the identified environmental microorganisms in the image data and provide the test results.

[0116] In this way, by configuring the device to identify environmental microorganisms in image data using a trained model, a mobile terminal, which is an example of an inspection system, can accurately identify environmental microorganisms in the captured image data, even if the environmental microorganisms have similar shapes.

[0117] This eliminates the need to use a culture method and shortens the time required to provide test results. As a result, for example, when a service provider visits a user to whom the test results are to be provided, the service provider can collect a sample from the user's air conditioner and provide the test results on the spot.

[0118] In other words, according to the first embodiment, it is possible to provide an inspection system and an inspection method that reduce the time required to inspect environmental microorganisms.

[0119] [Second embodiment] In the above first embodiment, a case has been described in which the mobile terminal 120 functions as an inspection system having the imaging device 121, an identification unit (second inference unit 640, etc.), and an output unit 650. However, some of the functions of the mobile terminal 120 may be arranged on an imaging stand, and the imaging stand and the mobile terminal 120 may form an inspection system.

[0120] Fig. 10 is a second diagram showing an application example of the inspection service providing system in the learning phase. The difference from Fig. 1 is that in the second embodiment, the inspection service providing system 100 in the learning phase has an inspection system 1010 and an image processing device 140, and the inspection system 1010 includes an imaging stand 1020 and a mobile terminal 120. Also, the difference from Fig. 1 is that in the second embodiment, the imaging stand 1020 has an imaging device 1021 and is connected to the mobile terminal 120.

[0121] Similarly, Fig. 11 is a second diagram showing an application example of the inspection service providing system in the inspection phase. The difference from Fig. 2 is that in the second embodiment, the inspection service providing system 100 in the inspection phase has an inspection system 1010, which includes an imaging stand 1020 and a mobile terminal 120. Also, the difference from Fig. 2 is that in the second embodiment, the imaging stand 1020 has an imaging device 1021 and is connected to the mobile terminal 120.

[0122] In this way, even when the imaging device 1021 is placed on the imaging stand 1020 and the mobile terminal 120 functions as an identification section and an output section, the same effects as those of the first embodiment can be obtained.

[0123] In this embodiment, the imaging device 1021 is arranged on the imaging stand 1020, but other functions realized in the mobile terminal 120 may be arranged on the imaging stand 1020. That is, in the inspection system 1010 including the imaging stand 1020 and the mobile terminal 120, the combination of functions shared between the imaging stand 1020 and the mobile terminal 120 is arbitrary.

[0124] [Third embodiment] In the above first and second embodiments, an example of providing the test results has been described as outputting at least the type and number of environmental microorganisms and image data, but the method of providing the test results is not limited to this, and the results may be summarized, for example, as a report in a form that is easy for the user to understand.

[0125] 12 is a diagram showing an example of a report including the test results. As shown in FIG. 12, a report 1200 includes the following information items: "User information" (code 1210), "Photo showing collection location" (code 1220), "Level" (code 1230), "Test results" (code 1240), "Description of mold" (Code 1250), etc. are included.

[0126] Of these, "user information" describes the circumstances under which the sample was collected, such as the date it was collected and the name of the location where it was collected. Furthermore, a photograph showing the collection location is inserted into "photo showing collection location" as information showing the environment in which the sample was collected. Furthermore, "level" describes the level of contamination calculated based on the inspection results. Furthermore, "inspection results" includes the captured image data and a graph showing the type and number (or ratio) of mold detected. Furthermore, "mold description" describes a detailed description of each type of mold (mold A to mold F in the example of FIG. 12).

[0127] Although not shown in report 1200 of Fig. 12, report 1200 may further include a comparison result with past test results. The past test results here include either or both of past test results from the same location and past test results from different collection locations.

[0128] [Fourth embodiment] In the above first embodiment, no specific mention was made of the size of the image data (partial image data) of each area determined to contain environmental microorganisms, but the size of the image data of each area is, for example, 32 x 32 pixels or more, or 50 x 50 pixels or more.

[0129] In addition, in the above first embodiment, a case has been described in which dust is collected from an air conditioner (indoor unit), but the collection destination from which dust is collected is not limited to an air conditioner (indoor unit) and may be other equipment. Note that other equipment includes, for example, air purifiers, humidifiers, ventilation equipment, blowers, etc. In addition, the collection destination from which dust is collected is not limited to equipment and may be, for example, the surface of an object other than equipment (wall, desk, indoor equipment, etc.) as long as it is an indoor environment. In other words, the inspection system according to the above first and second embodiments is an inspection system that inspects microorganisms or mold that occur in an indoor environment or equipment.

[0130] In the above first embodiment, the case where the imaging target is obtained by collecting dust and dissolving it has been described, but the method of obtaining the imaging target is not limited to this. For example, the imaging target may be obtained by collecting dirt and dissolving it. Alternatively, the imaging target may be obtained by obtaining the solution as it is. In other words, the imaging target may be a solution obtained by collecting it from an indoor environment or a device.

[0131] In the above first embodiment, the learning process is described as being performed using the learning data 441 to 446, but the learning data used in the learning process is not limited to the learning data 441 to 446. Additional learning data may be generated and a re-learning process may be performed to further improve the accuracy of identifying environmental microorganisms.

[0132] In addition, in the above first embodiment, the types of environmental microorganisms are described as six types, but the types of environmental microorganisms are not limited to six types. Image data may be added to generate learning data corresponding to seven or more types of environmental microorganisms. In the above first embodiment, six types of learning data are generated based on the number of types of environmental microorganisms. However, since DML is used when determining the types of environmental microorganisms, even if environmental microorganisms other than the six types are included, the environmental microorganisms other than the six types can be accurately classified. Alternatively, even if a substance other than environmental microorganisms is included, the substance can be accurately classified. In other words, in the case of DML, even if an unlearned substance is included, it can be accurately classified.

[0133] In the first embodiment, mold is given as an example of environmental microorganisms, but the environmental microorganisms are not limited to mold. For example, they may be microorganisms such as bacteria. The bacteria referred to here include, for example, Legionella, Bacillus, Micrococcus cocci, and the like.

[0134] In the above first embodiment, a physiological saline solution containing a surfactant is used to dissolve the collected sample. However, the solution used to dissolve the collected sample is not limited to this, and a solution diluted with physiological saline solution containing a surfactant may be used. In other words, the solution used to dissolve the collected sample may include: A first solution obtained by dispersing the collected sample in physiological saline containing a surfactant; A second solution obtained by diluting the first solution with saline containing a surfactant, Includes:

[0135] The reason for diluting the solution with physiological saline containing a surfactant is that the detection accuracy can be improved by dilution. The dilution ratio is, for example, about 1 to 1000 times, and preferably about 5 to 20 times.

[0136] In the above first embodiment, the output unit 650 is described as tallying up the number (or ratio) of each type of environmental microorganism in the image data 660. However, the number of each type of environmental microorganism may be counted, for example, by a trained DML.

[0137] Although the embodiments have been described above, it will be understood that various changes in form and details are possible without departing from the spirit and scope of the claims. [Explanation of symbols]

[0138] 100: Inspection service provider system 120: Mobile terminal 121: Imaging device 130: Photo stand 132: Placement section 133: Lens support part 140: Image processing device 150: Air conditioner (indoor unit) 160: Dust 180_1~180_6: Preparation 210: Air conditioner (indoor unit) 220: Dust 240: Preparation 410: Image data acquisition unit 420: Correct label acquisition section 430: Learning data generation unit 441~446: Learning data 510: First Study Section 520: Second Learning Section 610: Image data acquisition unit 620: 1st inference part 630: Partial image extraction section 640:Second reasoning part 650: Output section 730: Aggregated data 1010: Inspection systems 1020: Photo stand 1021: Imaging device 1200 : Report

Claims

1. An inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: an imaging unit that images an imaging target obtained by dispersing and dissolving a sample collected from an indoor environment or a device without staining the imaging target; A processing unit for processing captured image data, The processing unit includes: Inspecting the type of microorganism or mold individual in the image data by processing the captured image data using a trained model trained with training data including image data capable of identifying the color and shape of the appearance of the microorganism or mold individual and the type of the microorganism or mold individual; Output the inspection results, Inspection system.

2. An inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: an imaging unit that captures an image of the appearance of a microorganism or mold individual collected from an indoor environment or a device when the microorganism or mold individual is dispersed and dissolved; A processing unit for processing captured image data, The processing unit includes: Inspecting the type of microorganism or mold individual in the image data by processing the captured image data using a trained model trained with training data including image data capable of identifying the color and shape of the appearance of the microorganism or mold individual and the type of the microorganism or mold individual; Output the inspection results, Inspection system.

3. The inspection system according to claim 1 , wherein the imaging unit captures images through a lens under visible light or ultraviolet light.

4. The inspection system according to claim 3 , wherein the sampling target is any one of an air conditioner, an air purifier, a humidifier, a ventilation system, a blower, and a surface in an indoor environment.

5. The inspection system according to claim 1 , wherein the processing unit counts and outputs the number of microorganisms or fungi individuals in the image data captured by the imaging unit by type.

6. 5. The inspection system according to claim 1, wherein the processing unit displays the type of microorganism or mold individual, the number or ratio of microorganism or mold individual, and image data captured by the imaging unit as inspection results, and also displays one of information indicating the indoor environment or equipment, a description of the microorganism or mold, a level of contamination, and a comparison result with other inspection results.

7. The inspection system according to claim 1 , wherein the imaging target is obtained by dispersing a sample collected from an air conditioner into a solution.

8. The inspection system according to claim 7 , wherein the imaging target is obtained by dispersing a collected sample in physiological saline in which a surfactant is dissolved, to form a solution.

9. 9. The inspection system according to claim 7 or 8, wherein the imaging target includes a first solution obtained by dispersing a collected sample in physiological saline having a surfactant dissolved therein to form a solution, and a second solution obtained by further diluting the first solution with physiological saline having a surfactant dissolved therein.

10. the inspection system is a mobile terminal, The inspection system according to claim 1 , wherein the imaging unit is built into the mobile terminal.

11. the inspection system includes a portable imaging stand on which the imaging target is placed, and a mobile terminal connected to the imaging stand; The imaging stand has the imaging unit, The inspection system according to claim 1 , wherein the mobile terminal comprises the processing unit.

12. An inspection method for an inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: an imaging step of imaging an imaging target obtained by dispersing and dissolving a sample collected from an indoor environment or an apparatus without staining the imaging target; A processing step for processing the captured image data, The processing step comprises: Inspecting the type of microorganism or mold individual in the image data by processing the captured image data using a trained model trained with training data including image data capable of identifying the color and shape of the appearance of the microorganism or mold individual and the type of the microorganism or mold individual; Output the inspection results, Testing method.

13. An inspection method for an inspection system for inspecting microorganisms or mold occurring in an indoor environment or equipment, comprising: a photographing step of photographing the appearance of the microorganism or mold individual collected from the indoor environment or the equipment when the microorganism or mold individual is dispersed and dissolved; A processing step for processing the captured image data, The processing step comprises: Inspecting the type of microorganism or mold individual in the image data by processing the captured image data using a trained model trained with training data including image data capable of identifying the color and shape of the appearance of the microorganism or mold individual and the type of the microorganism or mold individual; Output the inspection results, Testing method.

14. A step of inputting image data of an object to be imaged, which is obtained by dispersing and dissolving a sample collected from an indoor environment or a device, without staining the object; A step of inspecting the type of microorganism or mold individual in the image data by processing the input image data using a trained model trained with training data including image data capable of identifying the color and shape of the appearance of the microorganism or mold individual and the type of the microorganism or mold individual; A step of outputting the inspection results; A test program to be executed on a computer.

15. A step of inputting image data of the appearance of a microorganism or mold individual collected from an indoor environment or a device when the microorganism or mold individual is dispersed and dissolved; A step of inspecting the type of microorganism or mold individual in the image data by processing the input image data using a trained model trained with training data including image data capable of identifying the color and shape of the appearance of the microorganism or mold individual and the type of the microorganism or mold individual; A step of outputting the inspection results; An inspection program that causes a computer to execute the above.

Images