Program, program generation method, and image diagnostic method

Abstract

To provide a method for performing abnormality diagnosis on captured image data obtained by imaging an object without requiring time and trouble for the generation of an abnormal object.SOLUTION: A control unit acquires local normal data obtained by imaging a normal state of an object, acquires experimental normal data obtained by imaging a normal state at a trial location, which is a location where a normal state can be imaged outside a predetermined location, generates an abnormal state of the object at the trial location, acquires experimental abnormal data obtained by imaging an abnormal state, generates an image generation artificial intelligence model from an image generation program capable of generating a predetermined image based on the experimental normal data and the experimental abnormal data, inputs the local normal data, generates an image of the object in an abnormal state in a pseudo manner and outputs the same, generates a learning model using the local normal data and the pseudo abnormal data as teacher data, inputs the captured image data obtained by imaging the object to the learning model, and determines at least one of presence / absence and type of abnormality of an object in the captured image data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a program, a method for generating a program, and an image diagnosis method.

Background Art

[0002] Conventionally, for example, in Patent Document 1, as a machine learning method for detecting an object with high accuracy, the distance values of at least some of the pixels included in the detection target portion of the original distance image composed of a plurality of pixels indicating the distance values to the object are changed to abnormal values. A method of constructing a learned model has been proposed by performing supervised learning using the processed distance image subjected to the processing as an input parameter and the label given to the detection target portion and the original distance image as output parameters.

[0003] Further, in Patent Document 2, first image information, which is normal image information of a portion to be detected of a building, is generated from BIM information, which is three-dimensional information of the building, and second image information obtained by actually imaging the detection target location of the building is obtained. Third image information with newly added abnormal locations is generated by performing image processing on at least one of the first image information and the second image information, and the third image information is learned by a generation model using a GAN. A technique has been proposed in which fourth image information as image information for magnification is generated from the first image information by the learned generation model, and an anomaly detection model is learned using the second image information and the fourth image information as teacher data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the above-mentioned prior art, for example, by imaging an object such as a plant with an imaging device in a test field or the like, after collecting a normal image in which the object is in a normal state and an abnormal image in which the object is in an abnormal state, machine learning is performed to generate a learning model. Then, a program for performing an abnormality diagnosis as to whether an image of an arbitrary object is normal or abnormal can be generated using the generated learning model.

[0006] However, for example, in the case of plants, if the cultivation environment is different, not only the way the captured image looks, but also differences occur in the form and color of the plants. Therefore, when diagnosing image data (on-site image data) captured at the location (on-site) where an object such as a plant exists, the accuracy of the abnormality diagnosis may decrease. Thus, a method of additionally performing machine learning using normal data and abnormal data obtained by imaging an object such as a plant on-site can be considered. However, there is a problem that creating an abnormal object on-site requires time and effort. Furthermore, it is difficult to generate an abnormal object itself, and there is also a problem that generating an abnormal object causes a significant loss. These problems are the same even if the object is something other than a plant, and there has been a demand for the development of a technology that can diagnose the abnormality of a captured object without the time and effort required for generating an abnormal object.

[0007] The present invention has been made in view of such circumstances, and its object is to provide a program, a method for generating the program, and an image diagnosis method capable of performing an abnormality diagnosis on captured image data obtained by imaging an object without the time and effort required for generating an abnormal object.

Means for Solving the Problems

[0008] In order to solve the above-described problems and achieve the object, a program according to an aspect of the present invention causes a control unit that diagnoses the presence or absence of an abnormality of an object based on captured image data of the object to acquire on-site normal data obtained by capturing a normal state of the object at a predetermined location where the object exists, acquire experimental normal data obtained by capturing a normal state of the object at a trial location that is a location other than the predetermined location and where the normal state of the object can be imaged, after generating an abnormal state of the object at the trial location, acquire experimental abnormal data obtained by capturing the abnormal state of the object, generate an image generation artificial intelligence model from an image generation program capable of generating a predetermined image based on the experimental normal data and the experimental abnormal data, input the on-site normal data as input parameters into the image generation artificial intelligence model, output pseudo-abnormal data obtained by pseudo-generating an image of the object in an abnormal state at the predetermined location as output parameters, generate a learning model generated by machine learning using the on-site normal data and the pseudo-abnormal data as teacher data, and input the captured image data of the object into the learning model to execute determination of at least one of the presence or absence and type of an abnormality in the object.

[0009] In a program according to an aspect of the present invention, in the above invention, by inputting a predetermined prompt together with the on-site normal data into an image generation program including the image generation artificial intelligence model, the pseudo-abnormal data of the object is generated and output. In a program according to an aspect of the present invention, in this configuration, the image generation program is an image generation artificial intelligence that combines a diffusion model and an image classification model that converts the prompt into image features by an algorithm that converts language into image features.

[0010] In a program according to an aspect of the present invention, in the above invention, information regarding the abnormality of the object in the pseudo-abnormal data is included as the teacher data.

[0011] A method for generating a program according to an aspect of the present invention is a method for generating a program that generates a program according to the above invention. The method includes collecting on-site normal data obtained by imaging the normal state of the object at a predetermined location where the object exists, experimental normal data obtained by imaging the normal state of the object at a trial location that is a location outside the predetermined location and where the normal state of the object can be imaged, and experimental abnormal data obtained by imaging the abnormal state of the object after generating the abnormal state of the object at the trial location. Then, the collected experimental normal data and experimental abnormal data are input into an image generation means capable of generating a predetermined image to generate an image generation artificial intelligence model. The on-site normal data is input into the image generation artificial intelligence model to pseudo-generate an image of the object in an abnormal state at the predetermined location and output it as pseudo-abnormal data. A learning model generated by machine learning is generated using the on-site normal data and the pseudo-abnormal data as teacher data.

[0012] An image diagnosis method according to an aspect of the present invention is an image diagnosis method executed by a control unit that diagnoses the presence or absence of an abnormality in an object based on imaging image data obtained by imaging the object. The method includes acquiring on-site normal data obtained by imaging the normal state of the object at a predetermined location where the object exists, acquiring experimental normal data obtained by imaging the normal state of the object at a trial location that is a location other than the predetermined location and where the normal state of the object can be imaged, after generating an abnormal state of the object at the trial location, acquiring experimental abnormal data obtained by imaging the abnormal state of the object, generating an image generation artificial intelligence model from an image generation program capable of generating a predetermined image based on the experimental normal data and the experimental abnormal data, inputting the on-site normal data as an input parameter into the image generation artificial intelligence model, and outputting pseudo-abnormal data obtained by pseudo-generating an image of the object in an abnormal state at the predetermined location as an output parameter, generating a learning model generated by machine learning using the on-site normal data and the pseudo-abnormal data as teacher data, and inputting the imaging image data obtained by imaging the object into the learning model to determine at least one of the presence or absence and type of an abnormality in the object.

Advantages of the Invention

[0013] According to the program, program generation method, and image diagnosis method according to the present invention, it is possible to perform an abnormality diagnosis on imaging image data obtained by imaging an object without requiring time and effort for generating an abnormal object.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings of the following embodiment, the same or corresponding parts are denoted by the same reference numerals. Further, the present invention is not limited to the embodiment described below.

[0016] First, when explaining an information processing apparatus according to an embodiment of the present invention, the intensive studies conducted by the present inventor will be described. That is, according to the findings of the present inventor, as an abnormality diagnosis program, which is a program for determining whether a predetermined object is normal or abnormal, it is conceivable to adopt an abnormality classification model or an abnormality determination model generated by machine learning such as a support vector machine (SVM) or LightGBM (Light Gradient Boosting Machine). The abnormality classification model can be generated by collecting abnormal images (abnormal data) of objects such as experimentally obtained crops in an abnormal state and normal images (normal data) of objects such as crops obtained in actual cultivation in a normal state. That is, a method of generating an abnormality classification model by performing machine learning using normal data and abnormal data, which are obtained by imaging objects in a normal state and objects in an abnormal state respectively, as teacher data, is conceivable. Since the abnormality classification model can determine the presence or absence of an abnormality based on the classified abnormality, it is possible to make the abnormality classification model function as an abnormality determination model.

[0017] However, in the prior art, while it is possible to generate so-called normal objects such as normal crops and organisms, it has been extremely difficult to generate so-called abnormal objects such as abnormal crops and organisms. Also, when the cultivation environment differs for crops or the growth environment differs for organisms, not only the appearance of the images obtained by imaging but also the form and color of the plants or organisms will vary. Therefore, when diagnosing image data (hereinafter referred to as on-site data) captured at a location where there are objects to be diagnosed (hereinafter referred to as the on-site), such as a farm, greenhouse, or plant factory where plants are actually cultivated or organisms are growing, using an abnormality diagnosis model based on image data captured at other fields, the accuracy of the abnormality diagnosis decreases.

[0018] Based on the above points, the inventor also considered a method of performing additional machine learning using normal data and abnormal data obtained by imaging objects on-site. However, creating abnormal objects on-site has problems such as requiring time and effort. Furthermore, it is itself difficult to generate abnormal objects, and problems such as causing significant losses by generating abnormal objects also occur.

[0019] Therefore, the inventor conducted further intensive studies on the generation of normal data and abnormal data. That is, the inventor cultivated and generated objects such as plants experimentally at a trial location different from the on-site, and came up with a method of collecting a normal image (hereinafter referred to as an experimental normal image) captured when the object was in a normal state and an abnormal image (hereinafter referred to as an experimental abnormal image) captured when the object was in an abnormal state at the trial location and using them for machine learning. Specifically, the inventor thought of a method of generating pseudo-abnormal images by using the experimental normal images as experimental normal data, the experimental abnormal images as experimental abnormal data, and using the on-site normal images (hereinafter referred to as on-site normal images) as on-site normal data and inputting them into an image generation artificial intelligence (hereinafter referred to as generative AI: Generative Artificial Intelligence) that can generate a predetermined image.

[0020] Specifically, first, an image generation AI model is generated from the above-described generative AI using an experimental normal image, which is experimental normal data, and an experimental abnormal image, which is experimental abnormal data. Next, for the generated image generation AI model, by inputting on-site normal data as input parameters and performing prompt specification, an abnormal image is generated and output as output parameters. The generated abnormal image is a so-called pseudo-abnormal image (hereinafter referred to as a pseudo-abnormal image), which is not necessarily the same as an on-site abnormal image obtained by actually imaging an object in an abnormal state on-site, and is output as pseudo-abnormal data. Also, since information about the on-site environment is input when on-site normal data is input to the generative AI, the pseudo-abnormal image can be treated as a high-precision abnormal image reflecting information such as the on-site environment. Furthermore, by performing prompt specification on the generative AI, it becomes possible to efficiently generate pseudo-abnormal images compared to the case of randomly generating abnormalities.

[0021] Then, machine learning is performed using, as teacher data, abnormal data (pseudo-on-site abnormal data) consisting of pseudo-abnormal images obtained as described above and on-site normal data consisting of on-site normal images captured on-site by a normal imaging device. As a result, an abnormal classification model capable of diagnosing whether an object in a captured image of an object on which actual cultivation, growth, etc. have been performed is normal or abnormal can be generated. Here, when generating the above-described pseudo-abnormal images (pseudo-on-site abnormal data), since prompt specification is performed, for each of the pseudo-on-site abnormal data, labeling can be performed regarding various abnormalities created according to the prompt (instruction). Therefore, the abnormal classification model becomes a learning model that has learned not only the presence or absence of abnormalities but also the correspondence between the information and images of the labeled types of abnormalities. Note that as the type information, in the case of a complex abnormality, etc., information on the degree of abnormality of each type may be adopted after quantification. Also, the abnormal classification model may be generated by re-learning or transfer learning of an abnormal classification model created from experimental normal images and experimental abnormal images. By generating an abnormal classification model, it becomes possible to construct an abnormal diagnosis program including the abnormal classification model.

[0022] In addition, as an abnormality diagnosis program, in addition to diagnosing normal and abnormal conditions using an abnormality classification model, when outputting a diagnosis result of being abnormal, the inventor may also construct it to be able to output a diagnosis name capable of recognizing the abnormal state, additional information, and supplementary information which are additional information. Further, according to the inventor's findings, by using various image generation AIs such as Stable Diffusion, Mid Journey, DALL E3, or SeaArt as the above-described generative AI, it becomes possible to specify what kind of abnormal object image to generate. Here, an image generation AI is a technology that can approximate a target image by a prompt by combining an image classification model that converts a prompt into image features by an algorithm that converts language (text) into image features, such as CLIP, in addition to a diffusion model.

[0023] As a result, it becomes possible to efficiently create an image that causes an abnormality having a form and color matching the situation at the site where the object is imaged. Furthermore, by inputting a diagnosis name or symptom of the abnormality, it is also possible to adopt functions such as outputting an assumed abnormal image and outputting a normal image after appropriate countermeasures (normal image after countermeasures) for the abnormal image after appropriate countermeasures have been taken.

[0024] In addition, there are problems where various conditions differ between an experimental environment (hereinafter referred to as the experimental environment) where a large amount of data can be acquired and the local environment (hereinafter referred to as the local environment) where actual application is carried out. Such problems include not only those caused by the shooting environment such as brightness and background in the local environment and the experimental environment, but also changes that occur in the object itself, such as the color and shape of plants and organisms, due to differences in the cultivation and growth environment. When there is not enough local abnormal data, the above-mentioned differences cannot be learned by machine learning, which will reduce the image diagnosis accuracy of the image diagnosis program including the image diagnosis learning model. On the contrary, by using image generation AI such as Stable Diffusion, it becomes possible to generate a pseudo-abnormal image assuming a situation where an abnormality occurs in the object in the local environment based on the information of the local environment as pseudo-abnormal data.

[0025] When generating a large number of pseudo-abnormal images as described above, it becomes possible to cover the variation range of the local abnormal images by generating pseudo-abnormal images by noise generation or the like. In this case, compared with the conventional image generation technology GAN (Generative Adversarial Networks), according to the technology of generative AI such as Stable Diffusion, it becomes possible to input or limit instruction information regarding what kind of abnormality is to be generated for a given local image. Therefore, compared with conventional technologies such as GAN, it becomes possible to efficiently generate abnormal images adapted to the local area.

[0026] Also, when using pseudo-abnormal images in machine learning for generating an abnormal classification model, it is also possible for an operator to determine whether the abnormal state of the object is the desired abnormal state for the abnormal images adopted as abnormal data and make a selection. Thereby, the normal images captured locally are used as normal data, and the pseudo-abnormal images that are actually impossible to capture generated by the generative AI are used as abnormal data, and the judgment accuracy of the abnormal classification model generated by machine learning can be improved.

[0027] As described above, the inventor has conceived a method of using, as teaching data, a normal image actually captured on-site and a pseudo-abnormal image that cannot actually be captured on-site but can be generated by a generative AI, in machine learning. Note that the object of the abnormality diagnosis is applicable not only to plant cultivation and biological growth but also to objects where the environment for obtaining abnormal data is different from the environment where it is actually desired to be applied. Specifically, in the case of agricultural crops, it is applicable to a discrimination program for diseases, physiological disorders, and pest damage from on-site images. In addition, it is also applicable to flames in incinerators, plant facilities in water treatment plants, bridges, fish farming, and the like. The following embodiment is devised based on the above earnest studies by the inventor.

[0028] (Information processing device) FIG. 1 shows an information processing device according to an embodiment of the present invention. As shown in FIG. 1, the information processing device 10 is configured to be able to input and output data with the imaging device 20. The imaging device 20 is composed of, for example, an imaging camera. The imaging device 20 captures an object, a background, etc. at a site or a test site, and transmits the captured image data generated by the capture to the information processing device 10 via the communication unit 13.

[0029] Here, the input / output of data between the imaging device 20 and the information processing device 10 can be executed by network communication via a network or the cloud, or by contactless communication such as Bluetooth (registered trademark). Note that it is also possible to execute data transfer via a disk recording medium such as a USB (Universal Serial Bus) memory, a CD (Compact Disc), a DVD (Digital Versatile Disc), or a BD (Blu-ray (registered trademark) Disc). The network is configured by appropriately combining wired communication and wireless communication, and is composed of a communication network such as an Internet line network or a mobile phone line network. The network consists of, for example, a dedicated line, a public communication network such as the Internet, a telephone communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a mobile phone, a public line, one or a combination of a plurality of VPNs (Virtual Private Networks), and the like.

[0030] As shown in FIG. 1, the information processing device 10 according to the present embodiment includes a control unit 11, a storage unit 12, a communication unit 13, and an input / output unit 14. Specifically, the control unit 11 includes a processor having hardware such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and an FPGA (Field-Programmable Gate Array), and a main storage unit (both not shown) such as a RAM (Random Access Memory) and a ROM (Read Only Memory).

[0031] The storage unit 12 is composed of a storage medium selected from a volatile memory such as a RAM, a non-volatile memory such as a ROM, an EPROM (Erasable Programmable ROM), a hard disk drive (HDD), and a removable medium. Note that as the removable medium, for example, a USB memory or a disk recording medium such as a CD, a DVD, or a BD can be adopted. Also, the storage unit 12 may be configured using a computer-readable recording medium such as a memory card that can be externally attached.

[0032] The storage unit 12 can store an operating system (OS), various programs such as an image processing application, various tables, various databases, etc., for executing the operations of the information processing apparatus 10. These various programs can also be recorded on a computer-readable recording medium such as a hard disk, a flash memory, a CD-ROM, a DVD-ROM, a flexible disk, etc. and widely distributed.

[0033] The communication unit 13 is, for example, a LAN interface board, a wired communication circuit for wired communication, or a wireless communication circuit for wireless communication. The LAN interface board, the wired communication circuit, and the wireless communication circuit are connected to a network. The communication unit 13 as a transmission unit and a reception unit connects to the network and communicates with the information processing apparatus 10.

[0034] The input / output unit 14 can be composed of, for example, a touch panel display, a speaker microphone, etc. The input / output unit 14 as input means may include an interface that inputs various information transmitted from the imaging device 20 through the communication unit 13 and outputs it to the control unit 11. Further, the input / output unit 14 includes a user interface such as a keyboard, input buttons, a lever, a touch panel for manual input provided superimposed on a display such as a liquid crystal, or a microphone for voice recognition. By an operator or the like operating the input / output unit 14, it is configured to be able to input predetermined information to the control unit 11. The input / output unit 14 as output means displays a predetermined image or the like on a display monitor, displays characters, figures, etc. on the screen of the touch panel display, or outputs voice from a speaker according to the control by the control unit 11. That is, the input / output unit 14 is configured to be able to notify external parties of predetermined information. Note that the input unit and the output unit in the input / output unit 14 may be configured separately.

[0035] The control unit 11 loads the program stored in the storage unit 12 into the working area of the main memory and executes it, and controls each component through the execution of the program, thereby realizing a function that matches a predetermined purpose. That is, the control unit 11 can execute the functions of the image acquisition unit 111, the abnormal image generation unit 112, the abnormal determination unit 113, and the learning unit 114 by executing various programs loaded from the storage unit 12. Further, the various programs also include a program for realizing an artificial intelligence and a learning model (also referred to as a learned model) capable of realizing the processing according to the present embodiment.

[0036] The storage unit 12 stores a local normal image database 121, a pseudo-abnormal image database 122, an experimental normal image database 123, an experimental abnormal image database 124, an image diagnosis program 125 including an abnormal classification model 125a, and a generation program 127. The local normal image database 121, the pseudo-abnormal image database 122, the experimental normal image database 123, and the experimental abnormal image database 124 are all stored in the storage unit 12 so as to be searchable as databases.

[0037] The on-site normal image database 121 stores on-site normal data, which are on-site normal images among a plurality of captured image data obtained by capturing objects such as plants and organisms to be subjected to abnormality diagnosis with the imaging device 20 on-site. The pseudo-abnormal image database 122 stores pseudo-on-site abnormal data, which are pseudo-abnormal images generated by the abnormal image generation unit 112 of the control unit 11. The experimental normal image database 123 stores experimental normal data, which are experimental normal images in which the object is imaged in a normal state at a trial location different from the on-site location. The experimental abnormal image database 124 stores experimental abnormal data, which are experimental abnormal images in which the object is imaged in an abnormal state at a trial location different from the on-site location.

[0038] The image diagnosis program 125 is an abnormality diagnosis program that can diagnose whether the captured object is normal or abnormal, and is executed by the abnormality determination unit 113. The image diagnosis program 125 is constructed to include an abnormality classification model 125a. The abnormality classification model 125a is a learning model that can be generated by deep learning (deep learning) using, for example, a neural network by the learning unit 114. In the present embodiment, machine learning by the learning unit 114 is executed using the pseudo-on-site abnormal data and the on-site normal data as teacher data.

[0039] The generation program 127 as an image generation means is an image generation program configured to include a so-called generation AI that can generate a pseudo-abnormal image and is executed by the abnormal image generation unit 112. The generation program 127 is a program that can generate a pseudo-abnormal image by the generation AI when predetermined information is input. The generation program 127 includes an image generation AI model 127a. The image generation AI model 127a is generated by inputting experimental normal data and experimental abnormal data into a generation AI such as the above-described Stable Diffusion. In the present embodiment, as the predetermined information, experimental normal data, experimental abnormal data, and on-site normal data regarding a predetermined object are adopted. When on-site normal data is input as an input parameter and prompt designation is performed, the image generation AI model 127a of the generation program 127 generates a pseudo-abnormal image and outputs it as an output parameter. The generation program 127 may be stored in an external server that can communicate through a network, that is, in the cloud, without being stored in the storage unit 12.

[0040] (Method for Generating Diagnostic Learning Model and Image Diagnosis Method) Next, a method for generating a diagnostic learning model executed by the information processing apparatus 10 configured as described above and an image diagnosis method using an image diagnosis program including the generated diagnostic learning model will be described. FIG. 2 is a flowchart for explaining the method for generating a diagnostic learning model and the image diagnosis method according to the present embodiment.

[0041] As shown in FIG. 2, first, in step ST1, the imaging device 20 images a state where an object such as a plant or a living thing is normal at the site and outputs on-site normal data. The image acquisition unit 111 in the control unit 11 of the information processing apparatus 10 acquires the on-site normal data output by the imaging device 20 via the communication unit 13. The image acquisition unit 111 of the control unit 11 stores the acquired on-site normal data in the on-site normal image database 121.

[0042] Next, in step ST2, the imaging device 20 images an object such as a plant or a living thing in a test site different from the actual site in a normal state and outputs experimental normal data. The image acquisition unit 111 of the information processing device 10 acquires the experimental normal data output by the imaging device 20 via the communication unit 13. The image acquisition unit 111 stores the acquired experimental normal data in the experimental normal image database 123.

[0043] Next, in step ST3, at a test location different from the actual site, an operator creates an abnormal state of the object, for example. Next, the imaging device 20 images the object in the created abnormal state and outputs experimental abnormal data. The image acquisition unit 111 of the information processing device 10 acquires the experimental abnormal data output by the imaging device 20 via the communication unit 13. The image acquisition unit 111 of the control unit 11 stores the acquired on-site abnormal data in the experimental abnormal image database 124. Regarding the above steps ST1 to ST3, they may be executed in reverse order or in parallel, and the execution order of steps ST1 to ST3 can be set arbitrarily.

[0044] After the execution of steps ST1 to ST3, when moving to step ST4, the abnormal image generation unit 112 of the control unit 11 reads the experimental normal data from the experimental normal image database 123 and reads the experimental abnormal data from the experimental abnormal image database 124. The abnormal image generation unit 112 uses the read experimental normal data and experimental abnormal data as input parameters, and further inputs the specification by the prompt into the generation program 127. The generation program 127 constructs an image generation AI model 127a based on the input experimental normal data and experimental abnormal data, and stores it as the image generation AI model 127a in the generation program 127 of the storage unit 12.

[0045] Next, the abnormal image generation unit 112 reads local normal data from the local normal image database 121, and reads the image generation AI model 127a from the generation program 127. The abnormal image generation unit 112 inputs the local normal data and the prompt into the image generation AI model 127a to generate a pseudo-abnormal image at the local site, and outputs it as pseudo-local abnormal data. Since information about the local environment is also input when the local normal data is input into a generation AI such as Stable Diffusion, the pseudo-abnormal image can be treated as a high-precision abnormal image reflecting information such as the local environment. Furthermore, by specifying a prompt for the generation AI, it becomes possible to efficiently generate pseudo-abnormal images as compared with the case of randomly generating abnormalities. The abnormal image generation unit 112 stores the pseudo-local abnormal data in the pseudo-abnormal image database 122.

[0046] Next, in step ST5, the learning unit 114 of the control unit 11 reads local normal data (local normal images) from the local normal image database 121 and reads pseudo-local abnormal data from the pseudo-abnormal image database 122. The learning unit 114 as a learning means performs machine learning using the local normal data and the pseudo-abnormal data as teacher data. That is, the learning unit 114 acquires the local normal data and the pseudo-abnormal data from the storage unit 12 as learning input parameters, and generates a learning model by machine learning such as supervised learning with the presence or absence of an abnormality of the object in the local normal data and the pseudo-abnormal data as learning output parameters. Note that, for example, information about abnormalities such as a diagnosis name, a name related to an abnormal state, and a location where an abnormality has occurred may be adopted as output parameters together. Thereby, the learning unit 114 generates the abnormality classification model 125a. The learning unit 114 stores the generated abnormality classification model 125a in the storage unit 12 to construct the image diagnosis program 125. Note that various machine learning such as deep learning (deep neural network) using a neural network can be adopted for the machine learning, and it can be performed by mini-batch learning or the like.

[0047] Next, in step ST6, the imaging device 20 images an arbitrary object to be diagnosed to generate an on-site captured image and outputs it as on-site image data to the information processing device 10. The image acquisition unit 111 in the control unit 11 of the information processing device 10 acquires the on-site image data output by the imaging device 20 via the communication unit 13. The image acquisition unit 111 outputs the acquired on-site image data to the abnormality determination unit 113. The abnormality determination unit 113 reads the image diagnosis program 125 from the storage unit 12 and inputs the on-site image data. The abnormality determination unit 113 diagnoses the state of the object included in the on-site image data input to the image diagnosis program 125, outputs the data of the diagnosis result, and supplies it to the input / output unit 14. The input / output unit 14 displays the diagnosis result on a display unit such as a display monitor. Here, regarding the diagnosis result, it is possible to output information regarding abnormalities based on various results. For example, it is possible to output together information such as the diagnosis name indicating what kind of abnormality it is and information about the location where the abnormality appears.

[0048] As described above, the abnormality classification model 125a is generated in the information processing device 10 and the image diagnosis program 125 is constructed. As a result, it becomes possible to diagnose whether an abnormality has occurred in the object of the on-site image data obtained by imaging an arbitrary object on-site, and information regarding the abnormality occurring in the object.

[0049] According to the above-described embodiment, it is possible to accurately diagnose the presence or absence of an abnormality in the on-site image data obtained by imaging an arbitrary object without requiring time and effort to generate an object in an abnormal state.

[0050] (Recording medium) In the above-described embodiment, a program for causing the information processing apparatus 10 to execute the processing method can be recorded on a recording medium readable by a device such as a computer, other machines, or wearable devices (hereinafter referred to as a computer or the like). By causing a computer or the like to read and execute the program of this recording medium, the computer or the like functions as a movement control device. Here, a recording medium readable by a computer or the like refers to a non-transitory recording medium that accumulates information such as data and programs by an electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer or the like. Examples of removable recording media among such recording media include flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, BDs, DATs, magnetic tapes, memory cards such as flash memories, and the like. Also, recording media fixed to a computer or the like include hard disks, ROMs, and the like. Furthermore, an SSD can be used as both a removable recording medium from a computer or the like and a recording medium fixed to a computer or the like.

[0051] As described above, one embodiment of the present invention has been specifically described. However, the present invention is not limited to the above-described embodiment, and various modifications based on the technical idea of the present invention are possible. For example, the numerical values given in the above-described embodiment are merely examples, and different numerical values may be used as necessary. The present invention is not limited by the description and drawings that form a part of the disclosure of the present invention according to this embodiment.

[0052] For example, in the generation of the anomaly classification model according to the above-described embodiment, it is also possible to perform machine learning using experimental normal data, experimental anomaly data, on-site normal data, and pseudo-anomaly data. An image diagnosis program 125 can be created using the experimental normal data and the experimental anomaly data, and an anomaly classification model can also be generated by performing additional learning or transfer learning using on-site normal images (on-site normal data) and pseudo-anomaly images (pseudo-anomaly data).

[0053] In addition, in the above-described embodiment, deep learning using a neural network as an example of machine learning is used, but machine learning based on other methods may also be performed. For example, other supervised learning methods such as support vector machines, decision trees, naive Bayes, and k-nearest neighbor methods may be used. Further, semi-supervised learning may be used instead of supervised learning.

[0054] In addition, in one embodiment, the "unit" described above can be replaced with a "circuit" or the like. For example, the control unit can be replaced with a control circuit.

[0055] Further effects and modifications can be easily derived by those skilled in the art. The broader aspects of the present disclosure are not limited to the specific details and representative embodiments presented and described as above. Therefore, various changes can be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and their equivalents.

Explanation of Reference Numerals

[0056] 10 Information processing apparatus 11 Control unit 12 Storage unit 13 Communication unit 14 Input / output unit 20 Imaging device 111 Image acquisition unit 112 Abnormal image generation unit 113 Abnormal determination unit 114 Learning unit 121 Local normal image database 122 Suspected abnormal image database 123 Experimental normal image database 124 Experimental abnormal image database 125 Image diagnosis program 125a Abnormal classification model 127 Generation program 127a Image generation AI model

Claims

1. A control unit for diagnosing the presence or absence of an abnormality in the object based on the captured image data of the object, acquires on-site normal data obtained by imaging the normal state of the object at a predetermined location where the object exists, acquires experimental normal data obtained by imaging the normal state of the object at a trial location which is a location other than the predetermined location and where the normal state of the object can be imaged, after generating an abnormal state of the object at the trial location, acquires experimental abnormal data obtained by imaging the abnormal state of the object, generates an image generation artificial intelligence model from an image generation program capable of generating a predetermined image based on the experimental normal data and the experimental abnormal data, inputs the on-site normal data as input parameters into the image generation artificial intelligence model, and outputs, as output parameters, pseudo-abnormal data obtained by pseudo-generating an image of the object in an abnormal state at the predetermined location, generates a learning model generated by machine learning using the on-site normal data and the pseudo-abnormal data as teacher data, inputs the captured image data of the object into the learning model, and determines at least one of the presence or absence and type of abnormality in the object A program for causing execution of this.

2. In an image generation program including the image generation artificial intelligence model, by inputting a predetermined prompt together with the on-site normal data, the pseudo-abnormal data of the object is generated and output The program according to claim 1.

3. The image generation program is an image generation artificial intelligence that combines a diffusion model and an image classification model that converts the prompt into image features by an algorithm that converts language into image features The program according to claim 2.

4. Including information regarding the abnormality of the object in the pseudo-abnormal data as the teacher data The program according to claim 1.

5. A program generation method for generating the program according to any one of claims 1 to 4, Collect local normal data obtained by imaging the normal state of the object at a predetermined location where the object exists, experimental normal data obtained by imaging the normal state of the object at a trial location which is a location other than the predetermined location and where the normal state of the object can be imaged, and experimental abnormal data obtained by imaging the abnormal state of the object after generating the abnormal state of the object at the trial location. Input the collected experimental normal data and experimental abnormal data into an image generation means capable of generating a predetermined image to generate an image generation artificial intelligence model. Input the local normal data into the image generation artificial intelligence model to pseudo-generate an image of the object in an abnormal state at the predetermined location and output it as pseudo-abnormal data. Generate a learning model generated by machine learning using the local normal data and the pseudo-abnormal data as teacher data. A method for generating a program.

6. An image diagnosis method executed by a control unit for diagnosing the presence or absence of an abnormality of an object based on imaging image data of the object, comprising: Acquire local normal data obtained by imaging the normal state of the object at a predetermined location where the object exists. Acquire experimental normal data obtained by imaging the normal state of the object at a trial location which is a location other than the predetermined location and where the normal state of the object can be imaged. After generating the abnormal state of the object at the trial location, acquire experimental abnormal data obtained by imaging the abnormal state of the object. Based on the experimental normal data and the experimental abnormal data, generate an image generation artificial intelligence model from an image generation program capable of generating a predetermined image. Input the local normal data as an input parameter into the image generation artificial intelligence model, and output pseudo-abnormal data obtained by pseudo-generating an image of the object in an abnormal state at the predetermined location as an output parameter. Generate a learning model generated by machine learning using the local normal data and the pseudo-abnormal data as teacher data. Input the imaging image data of the object into the learning model to determine at least one of the presence or absence and type of abnormality in the object. An image diagnosis method.

Images