Learning device, learning method, and storage medium

Abstract

A learning device 1X mainly comprises a pseudo normal data generation means 35X and an abnormality learning means 36X. The pseudo normal data generation means 35X generates, from abnormality data representing an abnormal state of an object including a first region in which an abnormality of the object appeared, pseudo normal data by converting the first region into a pseudo normal region on the basis of a second region of the abnormality data excluding the first region. The abnormality learning means 36X performs, on the basis of a pair of the pseudo normal data and the abnormality data, machine learning of an inferencer that performs, from data representing a normal state of the object, inference pertaining to pseudo abnormality data in which pseudo abnormality of the object appeared.

Description

Learning device, learning method, and storage medium

[0001] The present disclosure relates to the technical fields of a learning device, a learning method, and a storage medium related to machine learning of a model for generating training data.

[0002] Conventionally, techniques for generating training data to be used in machine learning of a model that detects abnormalities such as lesions have been known. For example, Patent Literature 1 discloses a technique in which abnormal images are generated by inserting abnormality simulation data into normal images, which are training images prepared in advance for training, and the generated abnormal images are used to train a model that outputs normal images.

[0003] International Publication WO2020 / 184069

[0004] Training a model to detect anomalies requires abnormal data, such as abnormal images that represent abnormal conditions, but in the case of medical images, it is difficult to obtain abnormal data that represents anomalies. Therefore, it is possible to use a generative model based on deep learning to increase the amount of abnormal data. When training a generative model, it is desirable to use normal data that is as similar as possible to the abnormal data, except for the abnormal data and the parts that are deformed by the anomaly, in order to train a highly accurate generative model.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide a learning device, a learning method, and a storage medium that are capable of training an inference device that performs inference on abnormal data with high accuracy.

[0006] One aspect of the learning device is a learning device having: a pseudo-normal data generation means for generating pseudo-normal data from abnormal data representing an abnormal state of an object, including a first region where an abnormality of the object is manifested, by converting the first region into a pseudo-normal region based on a second region of the abnormal data other than the first region; and an abnormality learning means for executing machine learning of an abnormal data inferencer that makes inferences regarding pseudo-abnormal data in which a pseudo abnormality of the object is manifested from data representing the normal state of the object, based on a pair of the pseudo-normal data and the abnormal data.

[0007] One aspect of the learning method is a learning method in which a computer generates pseudo-normal data from abnormal data representing an abnormal state of an object, including a first region where an abnormality of the object is manifested, by converting the first region into a pseudo-normal region based on a second region of the abnormal data other than the first region, and performs machine learning of an abnormal data inference device that makes inferences regarding the pseudo-abnormal data in which a pseudo-abnormality of the object is manifested from data representing the normal state of the object, based on a pair of the pseudo-normal data and the abnormal data.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute a process of performing machine learning of an abnormal data inference device that generates pseudo-normal data by converting abnormal data representing an abnormal state of an object, including a first region where an abnormality of the object is manifested, into a pseudo-normal region based on a second region of the abnormal data other than the first region, and that makes inferences regarding pseudo-abnormal data in which a pseudo-abnormality of the object is manifested from data representing the normal state of the object, based on a pair of the pseudo-normal data and the abnormal data.

[0009] As an example of an effect of the present disclosure, it is possible to train an inference device that can make inferences regarding abnormal data with high accuracy.

[0010] 6A shows a schematic configuration of a learning system; FIG. 6B shows a hardware configuration of a learning device; FIG. 6C shows an overview of the processing performed by the learning device; FIG. 6D shows an example of functional blocks of a learning device; (A) an example of a normal image; (B) an example of a normal image in which a mask region is set; (C) a normal image showing a loss calculation region used for calculating losses in the learning of a normal image inference device; (A) an example of an abnormal image; (B) an example of an abnormal image in which a mask region based on abnormal region information is set; (C) an example of a pseudo-normal image generated from the image shown in FIG. 6B; (A) an example of a pseudo-normal image; (B) an example of a pseudo-normal image showing an inference target region where inference should be performed by the abnormal image inference device; (C) an abnormal image showing a loss calculation region used for calculating losses in the learning of the abnormal image inference device; (A) a first example of a pseudo-abnormal image generated using the abnormal image inference device; (B) a second example of a pseudo-abnormal image generated using the abnormal image inference device; FIG. 6E shows an example of a flowchart showing processing related to the learning of a normal image inference device. 1 is an example of a flowchart showing processing related to learning of an abnormal image inference device; FIG. 2 is an example of a functional block of a learning device that performs processing based on moving images; FIG. 3 is an example of a flowchart showing processing related to learning of a normal structure data inference device; FIG. 4 is an example of a flowchart showing processing related to learning of an abnormal structure data inference device; FIG. 5 is a block diagram of a learning device; and FIG. 6 is an example of a flowchart executed by a learning device.

[0011] Hereinafter, embodiments of a learning device, a learning method, and a storage medium will be described with reference to the drawings.

[0012] <First embodiment> (1) System configuration Fig. 1 shows a schematic configuration of a learning system 100. As shown in Fig. 1, the learning system 100 generates learning data (especially, abnormal images showing abnormalities in the object) necessary for training a model that detects anomalies from images of the object in which anomalies are to be detected. The learning system 100 mainly includes a learning device 1 and a storage device 2.

[0013] Hereinafter, a description will be given of the generation of training data used to train a lesion detection model that detects image regions (also referred to as "abnormal regions") representing suspected abnormalities such as lesions from endoscopic images of a human subject taken during an endoscopic examination. As will be described later, endoscopic images are suitable examples in that it is generally difficult to obtain images containing abnormal regions and that the abnormal regions resemble other regions (i.e., the foreground and background are similar). Examples of endoscopes that are subject to the present disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, angioscopes, and epidural endoscopes.

[0014] Furthermore, the training data generated in the learning system 100 is not limited to training data used for training the lesion detection model described above. For example, the training data generated in the learning system 100 may be training data used for training a model that detects lesion areas from any medical images other than endoscopic images (e.g., images obtained in ultrasound examinations, CT examinations, or MRI examinations). In another example, the training data generated in the learning system 100 may be training data used for training a model that detects object areas in which abnormalities (abnormalities other than lesions) appear in images of any object. Examples of models that detect object areas in which abnormalities other than lesions appear include models that detect abnormal areas that are suspected of being abnormal from images (appearance inspection images) taken in appearance inspections of objects other than humans.

[0015] The learning device 1 performs learning of inference devices (normal image inference device and abnormal image inference device) required for generating learning data based on information stored in the storage device 2. Hereinafter, an image representing a normal state of an object (here, an endoscopic image) will be referred to as a "normal image," and an image representing an abnormal state of an object (here, an endoscopic image) will be referred to as an "abnormal image." A normal image is an example of "normal data based on an image captured of a normal state of an object." Furthermore, the normal image inference device is an example of a "normal data inference device."

[0016] The normal image inferer is a machine learning model that, when a normal image in which a mask area is set is input, performs machine learning to infer an image of the mask area representing a normal object based on the input image. The mask area refers to an image area set to a specific pixel value. The normal image inferer is a model (engine) generated by self-supervised learning to infer the mask area set in the normal image from an area other than the mask area. In other words, the normal image inferer is a model that learns the relationship between a normal image of an object in which a mask area is set and an image representing the normal state of the object in the mask area. Hereinafter, a pseudo-normal image generated based on the inference result of the normal image inferer will also be referred to as a "pseudo-normal image."

[0017] The abnormal image inference device is a machine learning model that performs machine learning when a normal image of an object (i.e., an endoscopic image showing a normal state) is input, to infer an image showing an abnormality in the object based on the input image. The abnormal image inference device performs machine learning using an abnormal image and a pseudo-normal image generated from the abnormal image using the normal image inference device. The abnormal image inference device is a model that learns the relationship between an image showing the normal state of the object and an image showing an abnormal state of the object in the captured range of the image. Hereinafter, an abnormal image pseudo-generated based on the inference result of the abnormal image inference device will also be referred to as a "pseudo-abnormal image."

[0018] The storage device 2 is a memory that stores various information necessary for the processing of the learning device 1. The storage device 2 has a normal image DB 21, an abnormal image DB 22, normal image inference device information 23, and abnormal image inference device information 24.

[0019] The normal image DB 21 is a database that stores normal images of objects and is used for training the normal image inference device.

[0020] The abnormal image DB 22 is a database that stores abnormal images of objects. The abnormal image DB 22 is used for training the abnormal image inference device. Here, each abnormal image stored in the abnormal image DB 22 is associated with abnormal area information (label information) that indicates an abnormal area in the abnormal image. The abnormal area information may be, for example, information indicating the size (vertical width and horizontal width) and position of a bounding box, or information indicating an arbitrary shape and position that forms the outer frame of the abnormal area. The abnormal area is an example of a "first area," and an area of ​​the abnormal image other than the abnormal area is an example of a "second area." Here, it is assumed that no abnormal part of the object exists in the area of ​​the abnormal image other than the abnormal area indicated by the abnormal area information, and that the area of ​​the abnormal image other than the abnormal area represents the normal state of the object.

[0021] Examples of pathologies detected as abnormal regions in endoscopic images include the following (a) to (f): (a) head and neck: pharyngeal cancer, malignant lymphoma, papilloma; (b) esophagus: esophageal cancer, esophagitis, hiatal hernia, Barrett's esophagus, esophageal varices, esophageal achalasia, esophageal submucosal tumor, benign esophageal tumor; (c) stomach: gastric cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor; (d) duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma; (e) small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease; (f) large intestine: large intestine cancer, large intestine neoplastic disease, large intestine inflammatory disease, large intestine polyp, large intestine polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids.

[0022] The normal image inferencing device information 23 stores parameters of the normal image inferencing device required to configure the normal image inferencing device. The parameters of the normal image inferencing device stored in the normal image inferencing device information 23 are updated by learning performed by the learning device 1. The normal image inferencing device is a machine learning model (including statistical models; the same applies hereinafter) having any architecture, such as a neural network or a support vector machine. When the normal image inferencing device is configured using a neural network, the normal image inferencing device information 23 includes various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number of filters and filter size in each layer, and the weight of each element of each filter. Note that the normal image inferencing device information 23 stores initial values ​​of the parameters of the normal image inferencing device before learning.

[0023] The abnormal image inference device information 24 stores parameters of the abnormal image inference device necessary to configure the abnormal image inference device. The parameters of the abnormal image inference device stored in the abnormal image inference device information 24 are updated by learning performed by the learning device 1. The abnormal image inference device is a machine learning model having any architecture, such as a neural network or a support vector machine. The abnormal image inference device may be, for example, any model used in image generation AI that generates images from images (e.g., Leonardo.ai's Image2Image). When the abnormal image inference device is configured using a neural network, the abnormal image inference device information 24 includes various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weight of each element of each filter. Note that the abnormal image inference device information 24 stores initial values ​​of the parameters of the abnormal image inference device before learning.

[0024] The storage device 2 may be an external storage device such as a hard disk connected to or built into the learning device 1, a storage medium such as a flash memory, or a server device that communicates data with the learning device 1. The storage device 2 may also be composed of multiple storage devices, and may have the above-mentioned storage units distributed among them.

[0025] The configuration of learning system 100 shown in Figure 1 is an example, and various modifications may be made. For example, learning device 1 and storage device 2 may be implemented as a single device. In another example, learning device 1 may be implemented as a plurality of devices. In this case, the plurality of devices that make up learning device 1 exchange information necessary to execute pre-assigned processing between them via direct wired or wireless communication or via communication via a network.

[0026] (2) Hardware Configuration Fig. 2 shows an example of the hardware configuration of the learning device 1. The learning device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 19.

[0027] The processor 11 executes a program stored in the memory 12 to function as a controller (arithmetic unit) that performs overall control of the learning device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0028] Memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Memory 12 also stores programs executed by learning device 1. Some of the information stored in memory 12 may be stored in an external storage device, such as storage device 2, that can communicate with learning device 1, or in a storage medium that is detachable from learning device 1. Memory 12 may also store information stored in storage device 2 instead.

[0029] The interface 13 is an interface for electrically connecting the learning device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0030] The hardware configuration of the learning device 1 is not limited to the configuration shown in Fig. 2. For example, the learning device 1 may further include a display unit such as a display, an input unit such as a keyboard or a mouse, and an audio output unit such as a speaker.

[0031] (3) Overview Figure 3 shows an overview of the processing executed by the learning device 1. As will be described later, the learning device 1 sequentially executes four steps: learning of the normal image inferencing device, inference by the normal image inferencing device, learning of the abnormal image inferencing device, and inference by the abnormal image inferencing device. Note that the inference by the abnormal image inferencing device may be executed by any device other than the learning device 1 that references abnormal image inferencing device information 24, which stores the parameters of the abnormal image inferencing device after learning.

[0032] First, in the first step, the learning device 1 uses normal images stored in the normal image DB 21 to train the normal image inferential device. In this case, the learning device 1 sets a mask area in the normal image and updates the parameters of the normal image inferential device so as to minimize the loss (error) between the normal image and the inference result output by the normal image inferential device when the normal image with the mask area set is input to the normal image inferential device. Then, the learning device 1 stores the parameters of the normal image inferential device updated by learning in the normal image inferential device information 23.

[0033] Next, in the second step, the learning device 1 performs inference using the normal image inference device obtained by learning. In this case, the learning device 1 uses the normal image inference device obtained by learning to generate pseudo-normal images by converting the abnormal regions of each of the abnormal images stored in the abnormal image DB 22 into normal regions. Here, the abnormal image and the pseudo-normal image match in areas other than the area indicated by the abnormal region information associated with the abnormal image (i.e., background), and also match in patterns, etc., in the abnormal region (i.e., foreground). This allows for the acquisition of multiple pairs of abnormal images and pseudo-normal images suitable for learning by the abnormal image inference device.

[0034] It is desirable that the normal image paired with the abnormal image in the learning of the abnormal image inference device be a normal image that is as similar as possible in elements other than the abnormal area (lesion). Specifically, when the abnormality is a lesion, it is desirable that the normal image paired with the abnormal image not only matches the appearance of the area surrounding the lesion in the image, but also reproduces the color and structure of the tissue that would originally have been present at the location of the lesion. For example, it is desirable that the pseudo-normal image paired with the abnormal image also shares the structure of the lesion portion shown in the abnormal image (e.g., whether the lesion is formed on a "fold" in the intestinal wall or on a flat surface) with the abnormal image. In other words, lesions generally form when cells and their aggregate tissue, which normally form a uniform pattern, break down. Therefore, in order to learn the deformation that occurs during the process of developing a lesion, it is desirable that the pseudo-normal image paired with the abnormal image be a pair that does not exhibit any differences in such patterns. Taking the above into consideration, in the second step, the learning device 1 prepares a pseudo-normal image in which the abnormal image and the background other than the abnormal area match, and the abnormal area is inferred to be in a normal state from the background.

[0035] Next, in the third step, the learning device 1 trains the abnormal image inferencing device based on multiple pairs of abnormal images and pseudo-normal images. In this case, for each pair of an abnormal image and a pseudo-normal image, the learning device 1 updates the parameters of the abnormal image inferencing device so as to minimize the loss between the abnormal image and the inference result output by the abnormal image inferencing device when the pseudo-normal image is input to the abnormal image inferencing device. The learning device 1 then stores the parameters of the abnormal image inferencing device updated by learning in abnormal image inferencing device information 24.

[0036] Then, in the fourth step, the learning device 1 generates pseudo-abnormal images from normal images (or pseudo-normal images) using the trained abnormal image inferencing device. This increases the number of abnormal images that are generally difficult to obtain, and makes it possible to acquire a sufficient amount of learning data necessary for training a lesion detection model (anomaly detection model) with high accuracy.

[0037] The effect of applying the above process to medical images is further explained below. Medical images generally do not change much in appearance near the boundary of an arbitrarily set mask region. For example, even if a portion of the intestinal wall is cut out, the appearance of the surrounding area is very similar. On the other hand, in general object images, the area near the mask may be a different object, making it difficult to predict the appearance of an object next to an unrelated object. Therefore, when using such images, the normal image inference machine is likely to fail to infer the mask region. Furthermore, in medical images, more normal image data is obtained than abnormal images (images containing lesion areas). For example, the majority of videos captured during a single endoscopic examination are normal images, and there are relatively few abnormal images, so there is a strong need to increase the number of abnormal images.

[0038] (4) Functional Blocks Figure 4 shows an example of functional blocks of the learning device 1. Functionally, the processor 11 of the learning device 1 has a random mask generation unit 31, a mask area setting unit 32, a normal learning unit 33, a mask area setting unit 34, a pseudo-normal image generation unit 35, an abnormal learning unit 36, and a pseudo-abnormal image generation unit 37. Note that in Figure 4, blocks that exchange data are connected by solid lines, but the combination of blocks that exchange data is not limited to this. The same applies to other functional block diagrams described below.

[0039] The random mask generation unit 31 sequentially acquires normal images from the normal image DB 21 and determines the position and shape of a mask area to be set for each acquired normal image. In this case, the random mask generation unit 31, for example, randomly acquires normal images from the normal image DB 21 and sets parameters that determine the position and shape of a mask area to be set for the acquired normal images to random values. For example, if the mask area is specified by a rectangle (bounding box), the random mask generation unit 31 sets the width and height of the rectangle to randomly determined values ​​(i.e., values ​​determined probabilistically). Note that instead of randomly determining the position and shape of the mask area, the random mask generation unit 31 may determine at least one of the position and / or shape of the mask area based on a predetermined rule. The random mask generation unit 31 supplies information specifying the acquired normal image and the mask area to be set to the mask area setting unit 32.

[0040] Based on the information received from the random mask generation unit 31, the mask area setting unit 32 sets, in the normal image, a mask area having the position and shape determined by the random mask generation unit 31. In this case, the mask area setting unit 32 sets, in the normal image, for example, a mask area whose pixel value is 0. Then, the mask area setting unit 32 supplies, to the normal learning unit 33, a pair of the normal image and the normal image in which the mask area has been set.

[0041] The normal learning unit 33 trains the normal image inferencing device based on pairs of normal images before and after the mask area is set. In this case, the normal learning unit 33 updates the parameters of the normal image inferencing device so as to minimize the loss (error) between the inference result output by the normal image inferencing device when the normal image with the mask area set is input to the normal image inferencing device and the normal image before and after the mask area is set. The algorithm for determining the parameters to minimize the loss may be any learning algorithm used in machine learning, such as gradient descent or backpropagation. The loss function may be, for example, the L1 distance. The normal learning unit 33 then updates the parameters of the normal image inferencing device for pairs generated from each normal image acquired by the random mask generation unit 31 and stores the updated parameters in the normal image inferencing device information 23.

[0042] After the normal image inference device has completed learning, the mask area setting unit 34 sequentially acquires, from the abnormal image DB 22, abnormal images used for learning the abnormal image inference device and abnormal area information associated with the abnormal images, and sets a mask area in the acquired abnormal image based on the abnormal area information. The mask area setting unit 34 processes the abnormal image, for example, so that the pixel values ​​in the area indicated by the abnormal area information become 0. The mask area setting unit 34 then supplies the abnormal image in which the mask area has been set to the pseudo-normal image generation unit 35, and supplies the abnormal image before the mask area setting to the abnormal learning unit 36. Note that, since the image area in the abnormal image in which the abnormal state of the object appears is limited to the abnormal area that becomes the mask area, the image area of ​​the abnormal image other than the mask area represents the normal state of the object.

[0043] The pseudo-normal image generation unit 35 generates a pseudo-normal image based on a normal image inference device configured with reference to the normal image inference device information 23 and an abnormal image in which a mask area is set. In this case, the pseudo-normal image generation unit 35 generates a pseudo-normal image based on the inference result output by the normal image inference device when an abnormal image in which a mask area is set is input to the normal image inference device. Here, the pseudo-normal image may be an image in which the mask area and the captured area are the same (i.e., an image in which the mask area is replaced with a normal area representing the normal state of the object), or it may be an abnormal image in which the mask area is converted based on the inference result of the normal image inference device. In this case, the normal image inference device may output an image in which the mask area is converted to the normal area of ​​the object as the inference result, or an image in which the captured area is the same as the abnormal image as the inference result. The pseudo-normal image generation unit 35 supplies the generated pseudo-normal image to the abnormality learning unit 36.

[0044] The abnormality learning unit 36 ​​trains the abnormal image inferencing device based on a pair of an abnormal image supplied from the mask area setting unit 34 and a pseudo-normal image generated by the pseudo-normal image generation unit 35 based on the abnormal image. In this case, the abnormality learning unit 36 ​​updates the parameters of the abnormal image inferencing device so as to minimize the loss (error) between the abnormal image and the inference result output by the abnormal image inferencing device when the pseudo-normal image is input to the abnormal image inferencing device. The algorithm for determining the parameters to minimize the loss may be any learning algorithm used in machine learning, such as gradient descent or backpropagation. Note that if the pseudo-normal image input to the normal image inferencing device corresponds to a mask area, the abnormality learning unit 36 ​​limits the range of the abnormal image to be compared with the inference result output by the abnormal image inferencing device to the same area as the mask area. Furthermore, if the pseudo-normal image input to the abnormal image inferencing device is in the same captured area as the abnormal image, the abnormality learning unit 36 ​​may input information specifying the area to be inferred (i.e., the same area as the mask area) to the abnormal image inferencing device along with the pseudo-normal image.

[0045] Then, the abnormality learning unit 36 ​​updates the parameters of the abnormal image inference device for each pair of an abnormal image and a pseudo-normal image generated from the abnormal image acquired by the mask area setting unit 34, and stores the updated parameters in the abnormal image inference device information 24.

[0046] After the abnormal image inferencing device has learned, the pseudo-abnormal image generating unit 37 generates a pseudo-abnormal image based on the abnormal image inferencing device configured with reference to the abnormal image inferencing device information 24. In this case, the pseudo-abnormal image generating unit 37 acquires a pseudo-abnormal image based on the inference result output by the abnormal image inferencing device by inputting a pseudo-normal image generated from a normal image or an abnormal image using the normal image inferencing device to the abnormal image inferencing device. This allows the pseudo-abnormal image generating unit 37 to substantially increase the number of abnormal images used for model learning.

[0047] Each of the components, including the random mask generation unit 31, the mask area setting unit 32, the normal learning unit 33, the mask area setting unit 34, the pseudo-normal image generation unit 35, the abnormal learning unit 36, and the pseudo-abnormal image generation unit 37, can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. At least some of the components may be configured using an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0048] (5) Specific Examples Next, specific examples of the processing executed by each block shown in FIG. 4 will be described.

[0049] Fig. 5(A) is an example of a normal image. Fig. 5(B) is an example of a normal image with a masked region set. Fig. 5(C) shows a normal image in which a loss calculation region, which is an image region used to calculate the loss in training the normal image inference device, is clearly indicated by a dashed frame.

[0050] In this example, the random mask generator 31 first acquires the normal image shown in Fig. 5(A) from the normal image DB 21 and determines the shape and position of the mask area. The mask area setting unit 32 then sets a mask area based on the shape and position determined by the random mask generator 31 in the normal image (see Fig. 5(B)). The normal learning unit 33 then updates the parameters of the normal image inferencing device so as to minimize the loss (error) between the image area (i.e., the image area obtained by converting the mask area) that is the inference result output by the normal image inferencing device when the normal image after the mask area setting shown in Fig. 5(B) is input to the normal image inferencing device, and the loss calculation area shown in Fig. 5(C).

[0051] Fig. 6(A) is an example of an abnormal image. Fig. 6(B) is an example of an abnormal image in which a mask region based on abnormal region information is set. Fig. 6(C) is an example of a pseudo-normal image generated from the image shown in Fig. 6(B).

[0052] In Figure 6 (A), the abnormal area indicated by the abnormal area information is clearly indicated by a dashed frame, and in Figure 6 (C), the image area where inference was performed by the normal image inference device (also called the "inferred area") is clearly indicated by a two-dot chain frame.

[0053] In this example, the mask area setting unit 34 first acquires the abnormal image shown in FIG. 6A from the abnormal image DB 22 and sets a mask area in the abnormal image based on abnormal area information associated with the acquired abnormal image (see FIG. 6B). The pseudo-normal image generation unit 35 then inputs the image shown in FIG. 6B into the normal image inference device, and acquires the pseudo-normal image shown in FIG. 6C based on the inference result output by the normal image inference device. Here, the normal image inference device converts the mask area of ​​the abnormal image into an inferred area representing a normal state based on information about its surrounding area. This generates a pseudo-normal image in which the abnormal image shown in FIG. 6A and the rest of the area are identical, and the inferred area replacing the abnormal area has a pattern similar to that of the abnormal area.

[0054] Fig. 7(A) is an example of a pseudo-normal image. Fig. 7(B) is an example of a pseudo-normal image that clearly indicates the inference target region, which is the image region where inference should be performed by the abnormal image inference device. Fig. 7(C) shows an abnormal image that clearly indicates the loss calculation region used to calculate the loss in learning by the abnormal image inference device.

[0055] In this example, the abnormality learning unit 36 ​​acquires the pseudo-normal image shown in FIG. 7A from the pseudo-normal image generation unit 35 and sets the area identified by the abnormal area information as the inference target area (see FIG. 7B). The abnormality learning unit 36 ​​then inputs an image obtained by cutting out the inference target area from the pseudo-normal image, or inputs the pseudo-normal image and information specifying the inference target area to the abnormal image inference device. The abnormality learning unit 36 ​​then updates the parameters of the abnormal image inference device to minimize the loss between the inferred area indicated by the inference result of the abnormal image inference device (i.e., an image obtained by converting the inference target area to reveal an abnormality) and the loss calculation area of ​​the abnormal image shown in FIG. 7C (i.e., the area identified by the abnormal area information). In this way, the abnormality learning unit 36 ​​can use a pair of an abnormal image and a pseudo-normal image that have common features except for the deformation due to the abnormality to train the abnormal image inference device to output highly accurate inference results regarding the pseudo-abnormal image.

[0056] The abnormality learning unit 36 ​​may use a randomly generated noise vector in training the abnormal image inferencing device. In this case, when a pseudo-normal image and a random noise vector are input to the abnormal image inferencing device, the abnormality learning unit 36 ​​updates the parameters of the abnormal image inferencing device so as to minimize the loss (e.g., L1 distance) between the pseudo-abnormal image output by the abnormal image inferencing device and the abnormal image used to generate the pseudo-normal image. The abnormality learning unit 36 ​​may also provide a classifier that distinguishes between the pseudo-abnormal image output by the abnormal image inferencing device and the abnormal image, and update the parameters of the abnormal image inferencing device so as to minimize an objective function including the loss term and an adversarial loss term based on the classifier. Training of an inferencing device (an engine that infers images from images) using such vector noise and a classifier is disclosed, for example, in the following literature: Isola Phillip, Zhu Jun-Yan, Zhou Tinghui, and Efros Alexei A. 2017. Image-to-image translation with conditional adversarial networks. arXiv:1611.07004v3 [cs.CV] 26 Nov 2018.

[0057] FIG. 8(A) shows a first example of a pseudo-abnormal image generated using the abnormal image inference device, and FIG. 8(B) shows a second example of a pseudo-abnormal image generated using the abnormal image inference device.

[0058] In a first example of generating a pseudo-abnormal image, the pseudo-abnormal image generation unit 37 generates a pseudo-abnormal image from a normal image (which may be a normal image used to train the normal image inference unit) stored in the normal image DB 21 or another storage device using a trained abnormal image inference unit. In this case, the pseudo-abnormal image generation unit 37 designates an arbitrary region on the normal image as an inference target region and inputs an image obtained by cutting out the inference target region from the normal image, or information specifying the normal image and the inference target region, to the abnormal image inference unit. The pseudo-abnormal image generation unit 37 then acquires a pseudo-abnormal image based on the inference result output by the abnormal image inference unit. Here, the pseudo-abnormal image generation unit 37 acquires a pseudo-abnormal image by converting the inference target region of the normal image into an inferred region based on the inference result output by the abnormal image inference unit. According to this example, the pseudo-abnormal image generation unit 37 can generate a pseudo-abnormal image from a normal image.

[0059] In a second example of generating a pseudo-abnormal image, the pseudo-abnormal image generation unit 37 generates a pseudo-normal image using a trained normal image inference device from an abnormal image stored in the abnormal image DB 22 or another storage device (which may be an abnormal image used to train the normal image inference device). In this case, the pseudo-abnormal image generation unit 37 sets the abnormal region indicated by the abnormal region information associated with the abnormal image as a mask region, and generates a pseudo-normal image based on the inference result output by the normal image inference device when the abnormal image with the mask region is input to the normal image inference device. The pseudo-abnormal image generation unit 37 then designates the image region that was the mask region in the pseudo-normal image as the inference target region, and inputs an image obtained by cutting out the inference target region from the pseudo-normal image, or inputs the pseudo-normal image and information specifying the inference target region to the abnormal image inference device. The pseudo-abnormal image generation unit 37 then generates a pseudo-abnormal image based on the inference result output by the abnormal image inference device. According to this example, the pseudo-abnormal image generation unit 37 can generate another pseudo-abnormal image from an abnormal image.

[0060] (6) Processing Flow Fig. 9 is an example of a flowchart showing the processing related to learning of the normal image inference device. The learning device 1 repeatedly executes the processing of the flowchart shown in Fig. 9.

[0061] First, the learning device 1 acquires normal images from the normal image DB 21 (step S11). In this case, for example, the learning device 1 randomly samples normal images from the normal image DB 21.

[0062] Next, the learning device 1 sets a mask area in the normal image (step S12). In this case, the learning device 1 sets a mask area with a randomly determined position and shape in the normal image acquired in step S11, for example.

[0063] Next, the learning device 1 inputs the normal image in which the mask region has been set to the normal image inference device and obtains the inference result output by the normal image inference device (step S13).The learning device 1 then updates the parameters of the normal image inference device based on the inference result of the normal image inference device and the loss between the normal image and the normal image (step S14).The learning device 1 stores the updated parameters in normal image inference device information 23.The learning device 1 then determines whether a predetermined learning termination condition is met, and if the learning termination condition is not met, executes the processing of the flowchart again.

[0064] 10 is an example of a flowchart showing the process related to learning of the abnormal image inference device. The learning device 1 repeatedly executes the process of the flowchart shown in FIG.

[0065] First, the learning device 1 acquires abnormal images and abnormal area information associated with the abnormal images from the abnormal image DB 22 (step S21). In this case, for example, the learning device 1 randomly samples pairs of abnormal images and associated abnormal area information from the abnormal image DB 22.

[0066] Next, the learning device 1 sets a mask area in the abnormal image based on the abnormal area information (step S22). In this case, the learning device 1 processes the abnormal image so that the abnormal area specified by the abnormal area information becomes a mask area with a predetermined pixel value.

[0067] Next, the learning device 1 inputs the abnormal image with the masked region to the normal image inference device, and obtains a pseudo-normal image output by the normal image inference device (step S23). In this case, the pseudo-normal image may be an image in which the image region corresponding to the masked region is extracted, or may be the entire abnormal image in which the masked region has been converted into an inferred region.

[0068] Next, the learning device 1 acquires the inference result output by the abnormal image inference device when the pseudo-normal image is input to the abnormal image inference device (step S24).The learning device 1 then updates the parameters of the abnormal image inference device based on the inference result of the abnormal image inference device and the loss of the abnormal image (step S25).The learning device 1 stores the updated parameters in abnormal image inference device information 24.The learning device 1 then determines whether a predetermined learning termination condition is met, and if the learning termination condition is not met, executes the processing of the flowchart again.

[0069] (7) Modification Example The normal image inferencing device and the abnormal image inferencing device may be trained by different devices. In this case, for example, a device other than the learning device 1 trains the normal image inferencing device and updates the parameters of the normal image inferencing device. Then, the learning device 1 refers to normal image inferencing device information 23, which stores the updated parameters of the normal image inferencing device, the abnormal image DB 22, and the abnormal image inferencing device information 24, and performs the processes executed by the mask region setting unit 34, pseudo-normal image generation unit 35, and abnormal learning unit 36 ​​shown in FIG. 4 to train the abnormal image inferencing device.

[0070] Second Embodiment The learning device 1 in the second embodiment differs from the learning device 1 in the first embodiment in that it represents an object as stereoscopic three-dimensional data based on moving images and generates learning data necessary for learning a model that detects abnormal regions in the three-dimensional data. Hereinafter, the same components as those in the first embodiment will be appropriately designated by the same reference numerals, and their description will be omitted. The configuration of the learning system 100 is the same as the configuration shown in FIG. 1, and the hardware configuration of the learning device 1 is the same as the configuration shown in FIG. 2.

[0071] 11 shows an example of functional blocks of a learning device 1 that performs processing based on moving images. The processor 11 of the learning device 1 functionally includes a structure data generation unit 30A, a random mask generation unit 31A, a mask area setting unit 32A, a normal learning unit 33A, a mask area setting unit 34A, a pseudo-normal structure data generation unit 35A, and an abnormal learning unit 36A. The storage device 2 also stores a normal moving image DB 21A, an abnormal moving image DB 22A, normal structure data inference device information 23A, and abnormal structure data inference device information 24A.

[0072] The normal video DB21A is a database that stores normal videos, which are videos captured of the normal state of an object. The normal video DB21A is used for training the normal structure data inference device. The normal videos stored in the normal video DB21A are converted into three-dimensional data (three-dimensional structure data) of the object. The three-dimensional structure data is, for example, voxel data that represents the three-dimensional structure using data in voxel units. In the second embodiment, the three-dimensional structure data of the object generated from the normal video is also simply referred to as "normal structure data." The normal structure data is an example of "normal data based on an image captured of the normal state of an object."

[0073] The abnormal video DB 22A is a database that stores abnormal videos, which are videos captured of objects in which an abnormality appears. The abnormal video DB 22A is used for training the abnormal structure data inference device. The abnormal videos stored in the abnormal video DB 22A are converted into three-dimensional data (three-dimensional structure data) of the objects. In the second embodiment, the three-dimensional structure data of the objects generated from the abnormal videos is also simply referred to as "abnormal structure data." Here, each abnormal video stored in the abnormal video DB 22A is associated with abnormal area information that indicates an abnormal area in each image of the abnormal video.

[0074] The normal structure data inferor information 23A stores parameters of the normal structure data inferor required to configure the normal structure data inferor. The parameters of the normal structure data inferor stored in the normal structure data inferor information 23A are updated by learning performed by the learning device 1. The normal structure data inferor is a machine learning model having any architecture, such as a neural network or a support vector machine. The normal structure data inferor is an example of a "normal data inferor". The abnormal structure data inferor information 24A stores parameters of the abnormal structure data inferor required to configure the abnormal structure data inferor. The parameters of the abnormal structure data inferor stored in the abnormal structure data inferor information 24A are updated by learning performed by the learning device 1. The abnormal structure data inferor is a machine learning model having any architecture, such as a neural network or a support vector machine.

[0075] The normal structure data inference device is a machine learning model that performs machine learning to infer structural data of a mask region representing a normal object based on input normal structure data in which a mask region is set. The mask region in the second embodiment is a three-dimensional region having a three-dimensional bounding box or other arbitrary shape, and values ​​within the region are set to predetermined values. The normal structure data inference device is a model that undergoes self-supervised learning to infer a mask region using normal structure data. In other words, the normal structure data inference device is a model that learns the relationship between normal structure data of an object in which a mask region is set and structural data representing the normal state of the object in the mask region. Hereinafter, normal structure data that is pseudo-generated (with the mask region replaced) based on the inference result of the normal structure data inference device will also be referred to as "pseudo-normal structure data."

[0076] The abnormal structure data inference device is a machine learning model that performs machine learning when normal structure data of an object is input, to infer structure data that indicates an abnormality of the object based on the input structure data. The abnormal structure data inference device is a model that learns the relationship between structure data that indicates the normal state of the object and structure data that indicates the abnormal state of the object. Hereinafter, the pseudo-anomalous structure data generated based on the inference result of the abnormal structure data inference device will also be referred to as "pseudo-anomalous structure data."

[0077] In the learning stage of the normal structure data inference device, the structure data generation unit 30A samples normal videos from the normal video DB 21A and generates normal structure data from the sampled normal videos. In this case, the structure data generation unit 30A may generate normal structure data of the object from the normal videos using any three-dimensional structure data generation technology such as SfM (Structure from Motion) or Neural Radiance Fields (NeRF). Similarly, in the learning stage of the abnormal structure data inference device, the structure data generation unit 30A samples abnormal videos from the abnormal video DB 22A and generates abnormal structure data from the sampled abnormal videos.

[0078] The random mask generation unit 31A determines the position and shape of a mask area to be set in the normal structure data supplied from the structure data generation unit 30A. In this case, the random mask generation unit 31A randomly (i.e., stochastically) determines, for example, each parameter of the position and shape of the mask area. The random mask generation unit 31A supplies the normal structure data and information about the position and shape of the set mask area to the mask area setting unit 32A.

[0079] The mask area setting unit 32A sets, in the normal structure data, a mask area having the position and shape determined by the random mask generation unit 31 A. For example, the mask area setting unit 32A processes the normal structure data so that the values ​​of voxels in the mask area become 0.

[0080] The normal learning unit 33A learns the normal structure data inferor based on a pair of normal structure data and normal structure data in which a masked region is set. In this case, the normal learning unit 33 updates the parameters of the normal structure data inferor so as to minimize the loss (error) between the normal structure data and the inference result output by the normal structure data inferor when normal structure data in which a masked region is set is input to the normal structure data inferor. Note that, for example, Swin UNETR (Swin UNET Transformers) exists as a configuration for predicting masked portions in three-dimensional structure data and updating the parameters of a model so as to minimize the loss.

[0081] The mask area setting unit 34A sets a mask area in the acquired abnormal structure data based on the abnormal structure data supplied from the structure data generation unit 30A and the abnormal area information associated with the abnormal structure data. The mask area setting unit 34A processes the abnormal structure data, for example, so that the values ​​of voxels in the abnormal area specified by the abnormal area information become 0. The mask area setting unit 34A then supplies the abnormal structure data in which the mask area has been set to the pseudo-normal structure data generation unit 35A, and supplies the unprocessed abnormal structure data to the abnormality learning unit 36A.

[0082] The pseudo-normal structure data generation unit 35A generates pseudo-normal structure data based on a normal structure data inference unit configured by referring to the normal structure data inference unit information 23A and abnormal structure data in which a mask region is set. In this case, the pseudo-normal structure data generation unit 35A generates pseudo-normal structure data based on the inference result output by the normal structure data inference unit when abnormal structure data in which a mask region is set is input to the normal structure data inference unit.

[0083] The abnormality learning unit 36A learns the abnormal structure data inference unit based on a pair of the abnormal structure data supplied from the mask region setting unit 34A and the pseudo-normal structure data generated by the pseudo-normal structure data generating unit 35A based on the abnormal structure data. In this case, the abnormality learning unit 36A updates the parameters of the abnormal structure data inference unit so as to minimize the loss (error) between the inference result output by the abnormal structure data inference unit and the abnormal structure data when the pseudo-normal structure data is input to the abnormal structure data inference unit.

[0084] Furthermore, in the second embodiment, the learning device 1 may also perform a process of generating pseudo-abnormal structural data from normal structural data or pseudo-normal structural data using the abnormal structural data inference device after training, thereby suitably increasing the amount of training data required for training a model that detects abnormal regions from structural data.

[0085] 12 is an example of a flowchart showing the process related to learning of the normal structure data inference device 1. The learning device 1 repeatedly executes the process of the flowchart shown in FIG.

[0086] First, the learning device 1 acquires a normal video from the normal video DB 21A (step S31). In this case, for example, the learning device 1 randomly samples normal videos from the normal video DB 21A. Then, the learning device 1 generates normal structure data of the object from the normal video using any 3D structure data generation technology such as SfM.

[0087] Next, the learning device 1 sets a mask region in the normal structure data (step S33). Next, the learning device 1 inputs the normal structure data with the mask region set to the normal structure data inference device and obtains the inference result output by the normal structure data inference device (step S34). Then, the learning device 1 updates the parameters of the normal structure data inference device based on the inference result of the normal structure data inference device and the loss between the normal structure data (step S35). The learning device 1 stores the updated parameters in normal structure data inference device information 23A. Then, the learning device 1 determines whether a predetermined learning termination condition is met, and if the learning termination condition is not met, executes the processing of the flowchart again.

[0088] 13 is a flowchart showing an example of processing related to learning by the abnormal structure data inference device 1. The learning device 1 repeatedly executes the processing of the flowchart shown in FIG.

[0089] First, the learning device 1 acquires an abnormal video and abnormal area information associated with the abnormal video from the abnormal video DB 22A, and generates abnormal structure data of the object from the acquired abnormal video (step S41). In this case, for example, the learning device 1 randomly samples an abnormal video and associated abnormal area information from the abnormal video DB 22A, and generates abnormal structure data of the object from the abnormal video using any three-dimensional structure data generation technology such as SfM.

[0090] Next, the learning device 1 sets a mask region in the abnormal structure data based on the abnormal region information (step S42). In this case, the learning device 1 processes the abnormal structure data so that the values ​​of voxels in the three-dimensional abnormal region specified by the abnormal region information become predetermined values.

[0091] Next, the learning device 1 inputs the abnormal structure data in which the mask region is set to the normal structure data inference device, and obtains pseudo-normal structure data output by the normal structure data inference device (step S43). In this case, the pseudo-normal structure data may be structure data representing the portion corresponding to the mask region, or may be the entire abnormal structure data in which the mask region has been converted based on the inference result.

[0092] Next, the learning device 1 acquires the inference result output by the abnormal structure data inference device when the pseudo-normal structure data is input to the abnormal structure data inference device (step S44).The learning device 1 then updates the parameters of the abnormal structure data inference device based on the inference result of the abnormal structure data inference device and the loss of the abnormal structure data (step S45).The learning device 1 stores the updated parameters in the abnormal structure data inference device information 24A.The learning device 1 then determines whether a predetermined learning termination condition is met, and if the learning termination condition is not met, executes the processing of the flowchart again.

[0093] 14 is a block diagram of a learning device 1X according to a third embodiment. The learning device 1X mainly includes a pseudo-normal data generating means 35X and an abnormality learning means 36X. The learning device 1X may be composed of multiple devices.

[0094] The pseudo-normal data generating means 35X generates pseudo-normal data by converting abnormal data representing an abnormal state of an object, including a first region where an abnormality of the object is present, into a pseudo-normal region based on a second region of the abnormal data other than the first region. The "first region" is, for example, the "abnormal region" in the first and second embodiments. The "second region" is, for example, an image region of the abnormal image other than the abnormal region in the first embodiment, and a three-dimensional region of abnormal structure data other than the abnormal region in the second embodiment. The "pseudo-normal data" is, for example, the "pseudo-normal image" in the first embodiment and the "pseudo-normal structure data" in the second embodiment. Examples of the pseudo-normal data generating means 35X include the pseudo-normal image generating unit 35 in the first embodiment and the pseudo-normal structure data generating unit 35A in the second embodiment.

[0095] The abnormality learning means 36X executes machine learning of an abnormal data inference device that performs inference regarding pseudo-abnormal data that represents a pseudo-abnormality of an object from data that represents a normal state of the object, based on a pair of pseudo-normal data and abnormal data. The "pseudo-abnormal data" is, for example, the "pseudo-abnormal image" in the first embodiment and the "pseudo-abnormal structure data" in the second embodiment. The "abnormal data inference device" is, for example, the "abnormal image inference device" in the first embodiment and the "abnormal structure data inference device" in the second embodiment. Examples of the abnormality learning means 36X include the abnormality learning unit 36 ​​in the first embodiment and the abnormality learning unit 36A in the second embodiment.

[0096] 15 is an example of a flowchart showing a processing procedure in the third embodiment. The pseudo-normal data generating means 35X generates pseudo-normal data by converting the first region, which includes a first region where an abnormality of the object appears, into a pseudo-normal region based on a second region of the abnormal data other than the first region, from abnormal data representing the abnormal state of the object (step S51). The abnormality learning means 36X executes machine learning of an abnormal data inferring device that performs inference on the pseudo-abnormal data that shows a pseudo-abnormality of the object from data representing the normal state of the object based on a pair of the pseudo-normal data and the abnormal data (step S52).

[0097] According to the second embodiment, the learning device 1X can learn an abnormal data inference device that performs highly accurate inference when learning an inference device necessary for data augmentation of abnormal data.

[0098] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0099] In addition, part or all of the above-described embodiments (including variations, the same applies below) may also be described as, but are not limited to, the following supplementary notes. Furthermore, not only the devices, methods, and storage media described in the supplementary notes, but also various hardware, software, various recording means (including storage media) for recording software, or systems may be made to depend on part or all of the configurations described in the supplementary notes, as long as they do not deviate from the above-described embodiments.

[0100] [Supplementary Note 1] A learning device comprising: pseudo-normal data generation means for generating pseudo-normal data from abnormal data representing an abnormal state of an object, including a first region where the abnormality of the object is expressed, by converting the first region into a pseudo-normal region based on a second region of the abnormal data other than the first region; and abnormal learning means for executing machine learning of an abnormal data inferencer that infers pseudo-abnormal data in which the pseudo-abnormality of the object is expressed from data representing the normal state of the object, based on a set of the pseudo-normal data and the abnormal data. [Supplementary Note 2] The learning device according to Supplementary Note 1, wherein the pseudo-normal data generation means generates the pseudo-normal data from the abnormal data, in which the first region is set as a mask region, based on a normal data inferencer that infers data representing the normal state of the object in the mask region from data in which the mask region is set. [Supplementary Note 3] The learning device according to Supplementary Note 2, further comprising: a mask area setting means for acquiring normal data based on an image of the object in a normal state and setting the mask area on the normal data; and a normal learning means for executing machine learning of the normal data inferor based on a pair of the normal data in which the mask area has been set and the normal data before the mask area was set. [Supplementary Note 4] The learning device according to Supplementary Note 3, wherein the abnormal data is associated with first area information indicating the first area in the abnormal data, and the mask area setting means sets the mask area based on the first area information. [Supplementary Note 5] The learning device according to Supplementary Note 3, wherein the mask area setting means randomly determines at least one of the position and shape of the mask area. [Supplementary Note 6] The learning device according to Supplementary Note 1, further comprising pseudo-abnormal data generation means for generating the pseudo-abnormal data based on the abnormal data inferor on which the machine learning has been performed and data representing the normal state of the object. [Supplementary Note 7] The learning device according to Supplementary Note 6, wherein the data representing the normal state of the object is normal data based on an image of the object in a normal state, or the pseudo-normal data converted from the abnormal data. [Supplementary Note 8] The learning device according to Supplementary Note 1, wherein the second region is the entire region of the abnormal data other than the first region, and the second region does not include a region where an abnormality appears in the object.[Supplementary Note 9] The abnormal data is an abnormal image capturing an abnormal state of the object, the pseudo-normal data generation means generates a pseudo-normal image from the abnormal image by converting the first region into the pseudo-normal region as the pseudo-normal data, the abnormality learning means executes machine learning in the abnormal data inference device based on a pair of the pseudo-normal image and the abnormal image, and the abnormal data inference device makes inferences regarding a pseudo-abnormal image that shows a pseudo-abnormality of the object from an image that represents a normal state of the object. [Supplementary Note 10] The learning device according to Supplementary Note 1, wherein the abnormal data is abnormal structure data that represents an abnormal state of the object in three dimensions, the pseudo-normal data generation means generates pseudo-normal structure data from the abnormal structure data by converting the first region into the pseudo-normal region as the pseudo-normal data, the abnormality learning means executes machine learning in the abnormal data inference device based on a pair of the pseudo-normal structure data and the abnormal structure data, and the abnormal data inference device makes an inference regarding pseudo-abnormal structure data that represents a pseudo-abnormality of the object from structure data that represents the normal state of the object. [Supplementary Note 11] The learning device according to Supplementary Note 10, further comprising structure data generation means that generates the abnormal structure data based on a plurality of abnormal images obtained by capturing the abnormal state of the object. [Supplementary Note 12] A learning method in which a computer generates pseudo-normal data from abnormal data representing an abnormal state of an object, including a first region where an abnormality of the object is manifested, by converting the first region into a pseudo-normal region based on a second region of the abnormal data other than the first region, and performs machine learning of an abnormal data inference device that makes inferences regarding the pseudo-abnormal data in which a pseudo-abnormality of the object is manifested from data representing the normal state of the object, based on a pair of the pseudo-normal data and the abnormal data.[Supplementary Note 13] A storage medium storing a program for causing a computer to execute a process of executing machine learning of an abnormal data inferencing device that generates pseudo-normal data by converting a first region, including a first region where an abnormality of the object, into a pseudo-normal region based on a second region of the abnormal data other than the first region, from abnormal data representing an abnormal state of the object, based on a pair of the pseudo-normal data and the abnormal data, and performs inference regarding the pseudo-abnormal data in which the pseudo-abnormality of the object appears from data representing a normal state of the object, based on the pair of the pseudo-normal data and the abnormal data. [Supplementary Note 14] The learning method according to Supplementary Note 12, wherein the pseudo-normal data is generated from the abnormal data in which the first region is set as a mask region based on a normal data inferencing device that infers data representing a normal state of the object in the mask region from the data in which the mask region is set. [Supplementary Note 15] The learning method according to Supplementary Note 14, wherein normal data is obtained based on an image capturing a normal state of the object, the mask region is set for the normal data, and the machine learning of the normal data inferencing device is performed based on a pair of the normal data in which the mask region is set and the normal data before the mask region was set. [Supplementary Note 16] The learning method of Supplementary Note 15, wherein first region information indicating the first region in the abnormal data is associated with the abnormal data, and the mask region is set based on the first region information. [Supplementary Note 17] The learning method of Supplementary Note 15, wherein at least one of the position and shape of the mask region is randomly determined. [Supplementary Note 18] The learning method of Supplementary Note 12, wherein the pseudo-abnormal data is generated based on the abnormal data inference device on which the machine learning was performed and data representing a normal state of the object. [Supplementary Note 19] The learning method of Supplementary Note 18, wherein the data representing the normal state of the object is normal data based on an image of the normal state of the object, or the pseudo-normal data converted from the abnormal data. [Supplementary Note 20] The learning method of Supplementary Note 12, wherein the second region is the entire region of the abnormal data excluding the first region, and the second region does not include a region where an abnormality appears in the object.[Supplementary Note 21] The abnormal data is an abnormal image capturing an abnormal state of the object, the pseudo-normal data generation means generates a pseudo-normal image from the abnormal image by converting the first region into the pseudo-normal region as the pseudo-normal data, the abnormality learning means executes machine learning in the abnormal data inference device based on a pair of the pseudo-normal image and the abnormal image, and the abnormal data inference device makes an inference regarding a pseudo-abnormal image that shows a pseudo-abnormality of the object from an image that shows a normal state of the object. [Supplementary Note 22] The learning method according to Supplementary Note 12, wherein the abnormal data is abnormal structure data that represents an abnormal state of the object in three dimensions, the pseudo-normal data generation means generates pseudo-normal structure data from the abnormal structure data by converting the first region into the pseudo-normal region as the pseudo-normal data, the abnormality learning means executes machine learning in the abnormal data inference device based on a pair of the pseudo-normal structure data and the abnormal structure data, and the abnormal data inference device makes an inference regarding pseudo-abnormal structure data that represents a pseudo-abnormality of the object from structure data that represents the normal state of the object. [Supplementary Note 23] The learning method according to Supplementary Note 22, further comprising structure data generation means that generates the abnormal structure data based on a plurality of abnormal images obtained by capturing the abnormal state of the object.

[0101] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference.

[0102] 1, 1X Learning device 2 Storage device 11 Processor 12 Memory 13 Interface 21 Normal image DB 21A Normal video DB 22 Abnormal image DB 22A Abnormal video DB 23 Normal image inference device information 23A Normal structure data inference device information 24 Abnormal image inference device information 24A Abnormal structure data inference device information 100 Learning system

Claims

1. A learning device comprising: a pseudo-normal data generation means for generating pseudo-normal data from abnormal data representing an abnormal state of an object, including a first region where an abnormality of the object is manifested, by converting the first region into a pseudo-normal region based on a second region of the abnormal data other than the first region; and an abnormality learning means for executing machine learning of an abnormal data inferencer that makes inferences regarding pseudo-abnormal data in which a pseudo abnormality of the object is manifested from data representing the normal state of the object, based on a pair of the pseudo-normal data and the abnormal data.

2. The learning device of claim 1, wherein the pseudo-normal data generating means generates the pseudo-normal data from the abnormal data in which the first region is set in the mask region based on a normal data inference device that infers data representing the normal state of the object in the mask region from data in which the mask region is set.

3. The learning device according to claim 2, further comprising: a mask area setting means for acquiring normal data based on an image of the object in a normal state and setting the mask area on the normal data; and a normal learning means for executing machine learning of the normal data inference device based on a pair of the normal data in which the mask area has been set and the normal data before the mask area was set.

4. The learning device described in claim 3, wherein the abnormal data is associated with first region information indicating the first region in the abnormal data, and the mask region setting means sets the mask region based on the first region information.

5. The learning device according to claim 3, wherein said mask area setting means randomly determines at least one of the position and shape of said mask area.

6. The learning device according to claim 1, further comprising a pseudo-abnormal data generation means for generating the pseudo-abnormal data based on the abnormal data inference device on which the machine learning has been performed and data representing the normal state of the object.

7. The learning device according to claim 6, wherein the data representing the normal state of the object is normal data based on an image of the object in its normal state, or pseudo-normal data converted from the abnormal data.

8. A learning device according to claim 1, wherein the second region is the entire region of the abnormal data other than the first region, and the second region does not include a region in which an abnormality appears in the object.

9. The learning device described in claim 1, wherein the abnormal data is an abnormal image captured of an abnormal state of the object, the pseudo-normal data generation means generates a pseudo-normal image from the abnormal image by converting the first region into the pseudo-normal region, as the pseudo-normal data, the abnormality learning means executes machine learning in the abnormal data inference device based on a pair of the pseudo-normal image and the abnormal image, and the abnormal data inference device makes inferences regarding a pseudo-abnormal image that shows a pseudo-abnormality of the object from an image that shows a normal state of the object.

10. The learning device described in claim 1, wherein the abnormal data is abnormal structure data that represents the abnormal state of the object in three dimensions, the pseudo-normal data generation means generates pseudo-normal structure data from the abnormal structure data by converting the first region into the pseudo-normal region, and the abnormality learning means performs machine learning on the abnormal data inference device based on a pair of the pseudo-normal structure data and the abnormal structure data, and the abnormal data inference device makes inferences regarding pseudo-abnormal structure data that represents a pseudo-abnormality of the object from structure data that represents the normal state of the object.

11. The learning device according to claim 10, further comprising a structure data generating means for generating the abnormal structure data based on a plurality of abnormal images obtained by photographing an abnormal state of the object.

12. A learning method in which a computer generates pseudo-normal data from abnormal data representing an abnormal state of an object, including a first region in which an abnormality of the object is observed, by converting the first region into a pseudo-normal region based on a second region of the abnormal data other than the first region, and performs machine learning of an abnormal data inference device that makes inferences regarding the pseudo-abnormal data in which a pseudo-abnormality of the object is observed from data representing the normal state of the object, based on a pair of the pseudo-normal data and the abnormal data.

13. A storage medium storing a program that causes a computer to execute a process of executing machine learning of an abnormal data inference device that generates pseudo-normal data by converting abnormal data representing an abnormal state of an object, including a first region where an abnormality of the object is manifested, into a pseudo-normal region based on a second region of the abnormal data other than the first region, and that makes inferences regarding pseudo-abnormal data in which a pseudo-abnormality of the object is manifested from data representing the normal state of the object based on a pair of the pseudo-normal data and the abnormal data.

14. The learning method described in claim 12, wherein the first region generates the pseudo-normal data from the abnormal data set in the mask region based on a normal data inference device that infers data representing the normal state of the object in the mask region from data in which the mask region is set.

15. The learning method described in claim 14, further comprising: acquiring normal data based on an image of the object in a normal state; setting the mask area on the normal data; and performing machine learning of the normal data inference device based on a pair of the normal data with the mask area set and the normal data before the mask area was set.

16. The learning method described in claim 15, wherein the abnormal data is associated with first region information indicating the first region in the abnormal data, and the mask region is set based on the first region information.

17. The learning method according to claim 15, wherein at least one of the position and the shape of the mask region is determined randomly.

18. The learning method according to claim 12, wherein the pseudo-abnormal data is generated based on the abnormal data inference device on which the machine learning has been performed and data representing the normal state of the object.

19. The learning method described in claim 18, wherein the data representing the normal state of the object is normal data based on an image of the object in its normal state, or pseudo-normal data converted from the abnormal data.

20. A learning method as described in claim 12, wherein the second region is the entire region of the abnormal data other than the first region, and the second region does not include the region where an abnormality appears in the object.

Images