Information processing apparatus, information processing method, and computer-readable recording medium

Abstract

Even when diverse variations in an image become explicit, the desired changes can be detected more appropriately. An information processing apparatus includes: a generation unit that generates a standardized image by performing standardization processing on an input image using a learned model, the learned model being a pre-built model based on machine learning with multiple normal images as input; a determination unit that determines whether the standardized image is substantially consistent with a predetermined reference image; and an output unit that outputs notification information to a predetermined output destination based on the determination result.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing methods, and computer-readable recording media. Background Technology

[0002] With the development of technologies related to image analysis and various recognition methods, various studies have been conducted on the application of so-called image recognition technologies, which enable the detection and recognition of various objects captured in images. For example, Patent Document 1 proposes a technology that applies image recognition technology to so-called remote monitoring of desired objects using a camera or other imaging device, thereby enabling the detection of anomalies even without the user visually confirming the image.

[0003] In addition, in recent years, techniques for applying learned models based on machine learning to image recognition have been studied. For example, Patent Document 2 discloses an example of a technique that utilizes a learned model based on machine learning in the monitoring of a monitored object using an image corresponding to the capture result of a surveillance camera.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2016-48910

[0007] Patent Document 2: Japanese Patent Application Publication No. 2019-219147 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] On the other hand, images obtained by capturing images of a desired object using imaging devices are not limited to changes in that object; for example, changes may occur due to other reasons such as changes in the lighting environment. Thus, the data used in various recognition and detection processes is not limited to changes in the object being recognized or detected; sometimes changes caused by various factors become apparent. In such cases, for example, when monitoring a desired object using the analysis results of the desired data (e.g., an image corresponding to the captured image), it can be difficult to identify whether the apparent changes in the data are caused by an anomaly occurring in the monitored object or could occur even under normal conditions.

[0010] The present invention addresses the aforementioned problems and aims to detect the desired changes in a more appropriate manner, even when diverse changes in an image can be made explicit.

[0011] Methods for solving problems

[0012] The information processing apparatus of the present invention comprises: a generation unit that generates a standardized image by performing standardization processing on an input image using a learned model, wherein the learned model is a model pre-constructed based on machine learning with multiple normal images as input; a determination unit that determines whether the standardized image is substantially consistent with a predetermined reference image; and an output unit that outputs notification information to a predetermined output destination based on the determination result.

[0013] Invention Effects

[0014] According to the present invention, even when diverse changes in an image that is the object can be made explicit, the desired changes can be detected in a more appropriate manner. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating an example of the system architecture of an information processing system.

[0016] Figure 2 This is a diagram illustrating an example of the hardware structure of an information processing device.

[0017] Figure 3 This is a block diagram illustrating an example of the functional structure of an information processing system.

[0018] Figure 4 It is a diagram used to illustrate the technical ideas related to the implementation of information processing systems.

[0019] Figure 5 This is a diagram representing an example of a normal image used in the construction of a learned model.

[0020] Figure 6 This is a diagram used to illustrate an example of the construction related to building a learned model.

[0021] Figure 7 This is a diagram illustrating the structures involved in detecting anomalies generated in a monitored object.

[0022] Figure 8 This is a diagram representing one example of an abnormality detected.

[0023] Figure 9 This is a graph representing an example of simulation results related to the accuracy of anomaly detection.

[0024] Figure 10 This is a diagram representing an example of a normal image used in the construction of a learned model.

[0025] Figure 11 This is a diagram representing one example of an abnormality detected. Detailed Implementation

[0026] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, in this specification and the accompanying drawings, constituent elements having substantially the same functional structure are given the same reference numerals, and repeated descriptions are omitted.

[0027] <System Architecture>

[0028] Reference Figure 1 This describes an example of the system architecture of the information processing system of this embodiment. The information processing system of this embodiment provides a structure capable of monitoring the status of a remotely located object (e.g., various devices) using a network-connected imaging device (e.g., a camera). For example, Figure 1 This embodiment illustrates an example of the structure of the information processing system 1, which monitors the status of each of the remotely located devices 800a-800d based on images corresponding to the capture results of each of the capturing devices 300a-300d. Specifically, the information processing system 1 includes capturing devices 300a-300d, information processing devices 100 and 200, and a terminal device 400. Hereinafter, without specifically distinguishing between capturing devices 300a-300d, it is sometimes referred to as "capturing device 300". Similarly, without specifically distinguishing between devices 800a-800d, it is sometimes referred to as "device 800". Furthermore, each of the capturing devices 300a-300d, each of the information processing devices 100 and 200, and the terminal device 400 are configured to transmit and receive various types of information and data via network N1.

[0029] The type of network N1 connecting the devices constituting the information processing system 1 is not particularly limited. As a specific example, network N1 can be composed of a LAN (Local Area Network), the Internet, a dedicated line, or a WAN (Wide Area Network). Furthermore, network N1 can be composed of either a wired or wireless network. Additionally, network N1 can include multiple networks, and may include networks of different types as part of the network. Moreover, as long as communication between the devices is logically established, the physical structure of network N1 is not particularly limited. As a specific example, communication between devices can be relayed through other communication devices. Furthermore, the series of devices constituting the information processing system 1 do not necessarily need to be connected to a common network. That is, as long as communication can be established between devices that transmit and receive information and data, some devices (two or more) can be connected to each other through different networks.

[0030] The imaging device 300 can be implemented, for example, by a device configured as a digital camera (e.g., a surveillance camera) capable of capturing images such as moving or still images. The imaging device 300 transmits the image data (hereinafter also referred to as "image data") corresponding to the imaging result to other devices (e.g., information processing devices 100 and 200, etc.) via network N1.

[0031] Information processing devices 100 and 200 provide a configuration that enables monitoring the state of device 800 based on image data sent from imaging device 300 corresponding to the imaging result of device 800 by imaging device 300.

[0032] Specifically, the information processing device 200 performs predetermined parsing processing on the image corresponding to the image data sent from the imaging device 300, and detects the occurrence of an anomaly when the state of the device 800, which is the subject of the photograph, malfunctions. At this time, the information processing device 200 outputs notification information corresponding to the detection result to a predetermined output destination (e.g., the terminal device 400 described later). Thus, in the event of an anomaly in the device 800, the occurrence of the anomaly can be notified to the user (e.g., an administrator).

[0033] Information processing device 100 constructs the learned model used by information processing device 200 in the above-described analytical processing based on machine learning. As a specific example, information processing device 100 may also use a neural network, known as a Generative Adversarial Network (GAN), to construct the aforementioned learned model.

[0034] Furthermore, the structure and processing of information processing devices 100 and 200 will be described in detail later.

[0035] Terminal device 400 schematically illustrates a terminal device used by a user in the management of each of the information processing devices 100 and 200 and each of the imaging devices 300. As a specific example, terminal device 400 serves as an input interface for receiving input from the user related to the management of each of the information processing devices 100 and 200 and each of the imaging devices 300, and also serves as an output interface for outputting various information to the user.

[0036] in addition, Figure 1The structure shown is merely an example. The system structure of the information processing system 1, as long as it can achieve the functions of each component of the information processing system 1 described later, is not necessarily limited. As a specific example, two or more of the information processing device 100, information processing device 200, and terminal device 400 can be integrated into one unit. As another example, the function of at least one of the information processing device 100, information processing device 200, and terminal device 400 can be achieved through the cooperation of multiple devices.

[0037] <Hardware Structure>

[0038] Reference Figure 2 An example of the hardware structure of the information processing device 900 of the terminal device 400, which can be used as each of the information processing devices 100 and 200 in the information processing system 1 of this embodiment, will be described. For example... Figure 2 As shown, the information processing apparatus 900 of this embodiment includes a CPU (Central Processing Unit) 910, a ROM (Read Only Memory) 920, and a RAM (Random Access Memory) 930. Additionally, the information processing apparatus 900 includes an auxiliary storage device 940 and a network I / F 970. Furthermore, the information processing apparatus 900 may include at least one of an output device 950 and an input device 960. The CPU 910, ROM 920, RAM 930, auxiliary storage device 940, output device 950, input device 960, and network I / F 970 are interconnected via a bus 980.

[0039] CPU 910 is the central processing unit that controls various operations of information processing device 900. For example, CPU 910 can control the overall operation of information processing device 900. ROM 920 stores control programs, boot programs, etc., that can be executed by CPU 910. RAM 930 is the main memory of CPU 910 and is used as a working area or a temporary storage area for running various programs.

[0040] The auxiliary storage device 940 stores various data and programs. The auxiliary storage device 940 is implemented by a storage device that can temporarily or continuously store various data, such as an HDD (Hard Disk Drive) or a non-volatile memory such as an SSD (Solid State Drive).

[0041] The output device 950 is a device for outputting various information to prompt the user. In this embodiment, the output device 950 is implemented using a display device such as a monitor. The output device 950 prompts the user by displaying various information. However, as another example, the output device 950 may also be implemented using an audio output device that outputs voice, electronic sounds, or other sounds. In this case, the output device 950 prompts the user by outputting voice, electronic sounds, or other sounds. Furthermore, the device used as the output device 950 may be appropriately changed depending on the medium used to prompt the user.

[0042] Input device 960 is used to receive various instructions from the user. In this embodiment, input device 960 includes input devices such as a mouse, keyboard, and touch panel. However, as another example, input device 960 may include a microphone sound collection device, and can collect speech emitted by the user. In this case, by performing various parsing processes such as sound analysis and natural language processing on the collected speech, the content shown by the speech is recognized as an instruction from the user. In addition, the device used as input device 960 can be appropriately changed according to the method of recognizing instructions from the user. Furthermore, various devices can be used as input device 960.

[0043] The Network I / F970 is used to communicate with external devices via a network. Furthermore, the devices used as the Network I / F970 can be appropriately changed depending on the type of communication path and the communication method applied.

[0044] CPU 910 expands the program stored in ROM 220 or auxiliary storage device 940 into RAM 930 and executes the program, thereby achieving... Figure 3 The functional structures and references of the information processing devices 100 and 200 shown are as follows. Figure 4 , Figure 6 as well as Figure 7 The information processing devices 100 and 200, as described above, each perform their respective processing.

[0045] <Functional Structure>

[0046] Reference Figure 3 With particular attention to the structures of the information processing devices 100 and 200, an example of the functional structure of the information processing system 1 of this embodiment will be described.

[0047] First, an example of the functional structure of the information processing apparatus 100 will be explained. As described above, the information processing apparatus 100 constructs a learned model based on machine learning. This learned model is used by the information processing apparatus 200 (described later) to perform predetermined analytical processing on an image corresponding to image data sent from the imaging device 300. The learning processing unit 110 schematically shows the constituent elements that perform various processes related to the construction of the learned model. That is, the learning processing unit 110 corresponds to an example of a "construction unit" involved in the construction of the learned model.

[0048] The learning processing unit 110 takes multiple images representing the correct state (hereinafter, these images will also be referred to as "normal images") as input and constructs the learned model based on machine learning. Alternatively, the learning processing unit 110 can use any one of the multiple normal images as a reference image and perform image processing on the other normal images to make the features of the other normal images closer to the features of the reference image. Furthermore, in the following description, this image processing will also be referred to as "normalization processing," and the image generated by performing this normalization processing on the image being processed will also be referred to as a "normalized image."

[0049] For example, the learning processing unit 110 may also construct a learned model based on machine learning utilizing a neural network called a GAN. In this case, the learning processing unit 110 may also include a generator 101 and a discriminator 103 constituting a GAN. Furthermore, the structure related to the construction of the learned model utilizing the generator 101 and the discriminator 103, and the standardization processing performed by the information processing apparatus 200 on the input image using the learned model, will be described in detail later.

[0050] The learned model constructed based on machine learning is output to the information processing device 200. Furthermore, the method is not particularly limited as long as the information processing device 200 can obtain the learned model. As a specific example, the information processing device 100 may also send the learned model to the information processing device 200 via network N1. Furthermore, as another example, the information processing device 100 may store the constructed learned model in a desired storage area. In this case, the information processing device 200 may also read the learned model from that storage area to obtain the learned model.

[0051] Alternatively, the reference image used in the construction of the learned model can be output to the information processing device 200 along with the learned model.

[0052] Next, an example of the functional structure of the information processing device 200 will be described. The information processing device 200 includes a generation unit 201, a determination unit 203, and an output control unit 205.

[0053] The generation unit 201 acquires image data from the imaging device 300 based on the imaging results captured by the device 800, which is the object of the imaging device 300's monitoring. The generation unit 201 then performs standardization processing on the image corresponding to the acquired image data using a learned model acquired from the information processing device 100, thereby generating a standardized image. Thus, by performing standardization processing on the image corresponding to the imaging results of the imaging device 300 to make the features of the image more closely resemble those of a reference image, a standardized image is generated.

[0054] The determination unit 203 determines whether the standardized image generated by the generation unit 201 is approximately consistent with the aforementioned reference image. In this case, the determination unit 203 can also determine whether the images are approximately consistent by comparing the standardized image and the reference image on a pixel-by-pixel basis. As a specific example, the determination unit 203 can also utilize the sum of absolute differences (SAD) between the pixel values ​​of each pixel in the comparison between the standardized image and the reference image.

[0055] Alternatively, the determination unit 203 can also determine whether the standardized image and the reference image are substantially similar by comparing the standardized image and the reference image pixel by pixel for each element (e.g., RGB). More specifically, it can also calculate the absolute value of the difference for each element value (e.g., RGB) of each pixel component between the standardized image and the reference image, and determine whether the pixels being considered are substantially similar based on whether the average of the absolute values ​​of the differences calculated for each element value exceeds a threshold. Furthermore, it can also determine whether the standardized image and the reference image are substantially similar based on the number of pixels that are substantially similar between the standardized image and the reference image.

[0056] Furthermore, as "component element values," various calculated values ​​of nearby pixels can be applied, such as the average value of nearby pixels, the difference with nearby pixels, etc. More specific examples of such calculated values ​​include edge-emphasizing features such as HOG features, SIFT features, and SURF features.

[0057] At this point, if the difference between the features of the image that serves as the source of the standardized image and the features of the reference image is within a range that can be reproduced based on the features of each of the multiple normal images used in the construction of the learned model, then the standardized image is approximately consistent with the reference image in the comparison. On the other hand, if the image that serves as the source of the standardized image has features that are difficult to be explicitly derived from the features of each of the multiple normal images, then the standardized image is not approximately consistent with the reference image in the comparison. Therefore, when a change in the state of the monitored device 800 occurs, resulting in a state change that is not conceivable under normal conditions, the standardized image is not approximately consistent with the reference image, and the occurrence of this change can be detected. Furthermore, the construction for detecting anomalies occurring in the monitored object (e.g., device 800) will be described in detail later with specific examples.

[0058] Then, the determination unit 203 outputs the information corresponding to the result of the above determination to the output control unit 205.

[0059] The output control unit 205 outputs information corresponding to the determination result of the determination unit 203 to a predetermined output destination. For example, the output control unit 205 may also output notification information corresponding to the determination result of the determination unit 203 to the terminal device 400. Thus, notification information corresponding to the determination result of the determination unit 203 can be sent to the user via the terminal device 400. Alternatively, as another example, the output control unit 205 may store information corresponding to the determination result of the determination unit 203 in a predetermined storage area.

[0060] Furthermore, the above is merely one example. As long as functions equivalent to those of the constituent elements of the information processing system 1 (especially the constituent elements of information processing devices 100 and 200) can be achieved, the functional structure of the information processing system 1 is not limited. For example, the functional structures of information processing devices 100 and 200 can be achieved through the cooperation of multiple devices.

[0061] As a specific example, some of the components of the information processing apparatus 100 may be located in other devices different from the information processing apparatus 100. Furthermore, as another example, the load related to the processing of at least some of the components of the information processing apparatus 100 may be distributed across multiple devices. The same applies to the information processing apparatus 200.

[0062] <Technical Features>

[0063] Next, as a technical feature of the information processing system 1 of this embodiment, the features involved in the information processing device 100 constructing the learned model and the features involved in the information processing device 200 detecting anomalies in the monitored object will be described in detail below.

[0064] First, refer to Figure 4 The technical concepts involved in implementing the information processing system 1 of this embodiment will be explained below. Furthermore, to facilitate a clearer understanding of the technical features of the information processing system 1 of this embodiment, the explanation will assume that GANs are used in the construction of the learned model.

[0065] exist Figure 4 In this model, the generator 101 and the discriminator 103 constitute a generative adversarial network (GAN). Furthermore, in constructing the learned model, multiple normal images D113 with different shooting conditions (e.g., lighting conditions and other shooting environment-related conditions) are used. Specifically, any one of the multiple normal images D113 is used as a reference image D111, and the other normal images D113 besides the reference image D111 are input into the generator 101.

[0066] The generation unit 101 performs a standardization process on the input normal image D113 using a learned model to make the features of the normal image D113 closer to the features of the reference image D111, thereby generating a standardized image D115.

[0067] The discrimination unit 103 is input with either the reference image D111 or the normalized image D115 generated by the generation unit 101, and determines which of the two images is the reference image D111 or the normalized image D115.

[0068] Based on this structure, the generation unit 101 uses a GAN based on the group of the discrimination unit 103 to perform a standardization process on the normal image D113, which makes it more difficult for the discrimination unit 103 to distinguish the standardized image D115 from the reference image D111, and constructs the above-mentioned learned model.

[0069] As described above, the generation unit 101 constructs the learned model separately for each monitored object and establishes a correspondence between the learned model and the reference image D111 (an image captured with the monitored object as the subject) used when constructing the learned model.

[0070] The learned model constructed by the generation unit 101 is applied to the generation unit 201 of the information processing device 200 for standardization processing of the input image. Specifically, the generation unit 201 uses the learned model to perform standardization processing on the captured image D123 corresponding to the capturing result of the capturing device 300, making the features of the captured image D123 closer to the features of the reference image D111, to generate a standardized image D125.

[0071] At this point, if the difference between the captured image D123 and the reference image D111 is a difference that can be inferred from the features of the multiple normal images D113 used in the construction of the learned model, then a standardized image D125 that eliminates this difference can be generated. That is, in this case, the determination unit 203 outputs a determination result that the standardized image D125 is approximately consistent with the reference image D111.

[0072] In contrast, if the difference between the captured image D123 and the reference image D111 is one that is difficult to infer from the features of the multiple normal images D113 used in the construction of the learned model, it is difficult to eliminate this difference when generating the standardized image D125. That is, in this case, the determination unit 203 outputs a determination result that the standardized image D125 is not approximately consistent with the reference image D111.

[0073] Based on this characteristic, for example, even if changes become visible in the image corresponding to the image captured by the monitored device 800, as long as the changes are visible under normal conditions, they are considered normal. Furthermore, if changes that are not visible under normal conditions become visible in the image corresponding to the image captured by the monitored device 800, the occurrence of such changes can be detected as abnormal. With this structure, even if a change that is conceivable under normal conditions occurs in the monitored object, the occurrence of such change can be prevented from being detected as abnormal, and if a change that is outside the conceivable range occurs, the occurrence of such change can be detected as abnormal. In other words, according to the information processing system of this embodiment, even when various changes in the image of the object can become visible, the desired change becoming visible can be detected in a more appropriate manner.

[0074] (Construction of the learned model)

[0075] Next, refer to Figure 5 as well as Figure 6 In particular, focusing on the use of GANs, this paper explains the construction involved in building a learned model to generate a normalized image by performing normalization processing on the input image.

[0076] One commonly used machine learning method is supervised learning, which involves building a model by assigning positive labels to the data used for learning, thereby making the predicted values ​​closer to the positive labels. Supervised learning, for example, uses images labeled with normal and abnormal states during learning, and allows the learned model to be classified as either a normal or abnormal state during discrimination.

[0077] However, when using a learned model built on general machine learning to detect anomalies in monitored objects, sometimes data representing anomalous states is needed in addition to the data representing normal states during learning. Conversely, sometimes it is difficult to prepare data representing anomalous states (in other words, incorrect data) during learning. For example, as... Figure 1 As illustrated in the example, in situations where a learned model is built for each device 800 that becomes the object of monitoring to implement the monitoring of that device 800, it is sometimes difficult to prepare an image representing the abnormal state of that device 800.

[0078] In view of this situation, in the information processing system of this embodiment, machine learning is applied using normal images representing the normal state of the monitored object when constructing the learned model.

[0079] On the other hand, for images corresponding to the captured images of the monitored object, it is also possible to imagine situations that change constantly with different lighting environments, etc. Under such circumstances, if only machine learning utilizing normal images is applied, each image is classified as representing a different state, and as a result, it is sometimes difficult to determine whether the monitored object is normal.

[0080] Therefore, in this embodiment, a technique is proposed that possesses robustness in determining a normal state as long as the explicit changes in the input data (e.g., images corresponding to the captured images of the monitored object) are within an allowable range, and in detecting an abnormal state when changes outside the allowable range become explicit. Specifically, in the information processing system of this embodiment, a learned model for detecting explicit anomalies in the input data is constructed using a neural network called GAN, even without using anomaly images representing the abnormal state of the monitored object, as described above.

[0081] By applying such a construction, in the information processing system of this embodiment, even without using abnormal images, a learned model that can be used to detect anomalies in monitored objects can be constructed using normal images.

[0082] Furthermore, in order to more easily understand the features of the information processing system of this embodiment, the following explanation will focus on the case where images are used as data for learning, and describe the construction involved in building the learned model.

[0083] First, refer to Figure 5 An example of an image (normal image) used in the construction of a learned model will be described. As described above, in the information processing system of this embodiment, multiple normal images D113 with different conditions are used when constructing a learned model.

[0084] For example, Figure 5 The normal images D113a to D113d shown represent examples of images captured by the monitored device 800 under conditions different from the lighting environment. Specifically, in Figure 5 In the example shown, although the monitored device 800 is in a normal state, the lighting conditions change over time due to changes in sunlight and other factors, resulting in constantly changing lighting environment conditions. As a result, images with different brightness and chromaticity are obtained.

[0085] Furthermore, any image corresponding to the capture result of the device 800 in normal condition is not limited to... Figure 5 The examples shown can also be used as normal images, corresponding to changes in various conditions. As a specific example, as time passes and weather (e.g., rain, wind) changes, the shooting environment and the state of the shooting device change, resulting in variations in brightness, color, the shape of shadows cast by sunlight, the presence or absence of water droplets adhering to the lens, and subtle shifts in the shooting device. These variations are sometimes visible in the image corresponding to the shooting result. Such images, as long as they correspond to the result of shooting the device as the subject under normal conditions, can also be used as normal images.

[0086] Next, refer to Figure 6 Focusing on utilization Figure 5 The example shown illustrates the construction involved in building a learned model using a series of normal images D113. Figure 6 In the example shown, for convenience, Figure 5 Of the normal images D113a to D113d shown, normal image D113a is used as the reference image D111. That is, Figure 6 The reference image D111 and normal image D113 (normal images D113b~D113d) shown are equivalent to Figure 4 The reference image D111 and the normal image D113 are shown.

[0087] Specifically, normal images D113b to D113d, other than the normal image D113a used as the reference image D111, are input to the generation unit 101. The generation unit 101 performs normalization processing on the input normal image D113 (each of normal images D113b to D113d) to make the features of the normal image D113 closer to the features of the reference image D111 (normal image D113a), thereby generating a normalized image D115. As a specific example, in... Figure 6 In the example shown, by adjusting the brightness and chroma of the normal image D113 input to the generation unit 101, a normalized image D115 with features similar to the reference image D111 is generated.

[0088] The discrimination unit 103 is input with a reference image D111 and a normalized image D115 generated by the generation unit 101. The discrimination unit 103 determines whether the input image is the reference image D111 or the normalized image D115.

[0089] As described above, the generation unit 101 and the discrimination unit 103 constitute a GAN. Based on this structure, the generation unit 101 constructs a learned model related to the application of the standardization process applied to the input normal image D113, in a manner that makes it more difficult for the discrimination unit 103 to distinguish between the reference image D111 and the standardized image D115. Furthermore, the discrimination unit 103 further improves the accuracy related to the discrimination of the reference image D111 and the standardized image D115, so as to be able to distinguish between the reference image D111 and the standardized image D115 with higher accuracy.

[0090] With the structure described above, if the difference between the input image and the reference image D111 is within the range that can be extrapolated from the differences in features between normal images D113a to D113d, then the learned model described above can be constructed that can generate a standardized image D115 with features more similar to the reference image D111 with high accuracy based on the input image.

[0091] Then, the constructed learned model and the reference image D111 used for constructing the learned model are used by the information processing device 200 to monitor the device 800 that is the object of monitoring (in other words, the detection of anomalies generated in the device 800).

[0092] Furthermore, the above example illustrates a case where image data is used in the construction of the learned model, but the category of data is not necessarily limited, and other categories of data can also be used. As a specific example, audio data such as speech, music, and ambient sounds can also be used in the construction of the learned model. In this case, the learned model is constructed by performing a standardization process on the input audio data to make the features of the audio data more closely resemble the features of the baseline audio data.

[0093] (Anomaly detection)

[0094] Next, refer to Figures 7-9 This section explains how an anomaly is detected when a monitored object exhibits an anomaly by utilizing a learned model built based on machine learning and related to the application of standardized processing of the input image. Furthermore, to facilitate a clearer understanding of the features of the information processing system in this embodiment, the following section focuses on utilizing reference... Figure 5 as well as Figure 6 This section describes an example of using a learned model to detect anomalies generated in a monitored object, illustrating the above construction.

[0095] First, refer to Figure 7 This section describes the construction involved in detecting anomalies generated within the monitored object. Figure 7 In this context, D111 is equivalent to a reference. Figure 6 The reference image D111 is explained. Additionally, D123 corresponds to the captured image obtained by the capturing device 300 from the monitored device 800. That is, Figure 7 The reference image D111 and the captured image D123 shown are equivalent to Figure 4 The reference image D111 and the captured image D123 are shown. Furthermore, the reference image D111 and the captured image D123 are images based on images captured using a common device 800 as the subject.

[0096] The generation unit 201 performs a standardization process on the input captured image D123 using a pre-built learned model based on machine learning, making the features of the captured image D123 closer to the features of the reference image D111, thereby generating a standardized image D125. Based on this, the determination unit 203 determines whether the generated standardized image D125 is approximately consistent with the reference image D111 (for example, whether the features of these images are approximately consistent).

[0097] At this point, if it is possible to use a series of normal images D113 (e.g., in the construction of the learned model described above) as examples, Figure 5By analogy with the characteristics of the normal images D113a to D113d shown, the characteristics of the captured image D123 and the reference image D111 are deduced. Then, the characteristics of the generated standardized image D125 are roughly consistent with the characteristics of the reference image D111.

[0098] On the other hand, if the captured image D123 contains differences from the reference image D111 that are difficult to infer from the characteristics of each of the aforementioned series of reference images D111, the characteristics of the generated normalized image D125 are not substantially consistent with the characteristics of the reference image D111. As a specific example, if a change in the monitored device 800 that is unthinkable under normal conditions becomes apparent in the captured image D123 due to some anomaly, the effect of this change also becomes apparent in the generated normalized image D125. Therefore, in this case, the generated normalized image D125 is not substantially consistent with the reference image D111.

[0099] Here, refer to Figure 8 A specific example is given to illustrate an anomaly detected by comparing a standardized image D125 with a reference image D111. D1111 represents an example of the reference image D111. Additionally, D1251 represents an example of a standardized image D125 generated by applying a standardized processing method to an captured image D123 using a machine learning-based learned model, making the features of the captured image D123 more similar to the features of the reference image D1111. Specifically, standardized image D1251 represents an example of a standardized image D125 generated when an anomaly occurs in the monitored device 800. Furthermore, regarding this learned model, compared with the reference... Figure 7 Similarly, the examples illustrate the application of reference. Figure 5 as well as Figure 6 The learned model is described.

[0100] In the standardized image D1251, the differences V1251, V1252, and V1253, which are not present in the reference image D1111, are made explicit. Specifically, difference V1251 schematically represents the difference that becomes explicit due to damage such as cracking in a part of the monitored device 800. Furthermore, difference V1252 schematically represents the difference that becomes explicit due to deformation in a part of the device 800. Additionally, difference V1253 schematically represents the difference that becomes explicit due to discoloration caused by oil leakage or other reasons resulting from damage to a part of the device 800.

[0101] The damage, deformation, and discoloration of the device 800, which are the main reasons for the manifestation of differences V1251, V1252, and V1253, are state changes that would be unimaginable under normal conditions. Therefore, it is difficult to extrapolate the changes corresponding to differences V1251, V1252, and V1253 based solely on the characteristics of the normal images D113a to D113d, which differ only under different lighting conditions. Under such circumstances, even if the captured image D123 is normalized to make its features closer to those of the reference image D1111, differences V1251, V1252, and V1253 will still be manifest in the generated normalized image D1251.

[0102] By utilizing this characteristic, the generated standardized image D1251 is compared with the reference image D1111 to detect differences (such as differences V1251, V1252, and V1253), thereby enabling the detection of anomalies in the monitored device 800.

[0103] In addition to the anomalies mentioned above, it can also detect situations that are unimaginable under normal conditions, such as the slope of the monitored object, the detachment of a part, the generation of smoke due to certain reasons, oil leaks, water leaks, the breakage of a part, and contact with animals or plants.

[0104] On the other hand, as described above, if the difference between the captured image D123 and the reference image D111 can be inferred by analogy based on the characteristics of each of a series of normal images D113 (e.g., normal images D113a to D113d), then the standardized image D125 is approximately the same as the reference image D111. Therefore, even if the conceivable changes of the device 800 under normal conditions become explicit relative to the captured image D123, it can be determined that the device 800 is in a normal state.

[0105] Based on the features described above, the information processing system according to this embodiment has the robustness to determine a normal state if the visible changes in the captured image D123 are within an allowable range, and to detect an abnormal state when changes outside the allowable range become visible.

[0106] For example, Figure 9 This is a graph illustrating an example of the simulation results related to the accuracy of detecting anomalies occurring in the monitored object of the information processing system of this embodiment. Specifically, Figure 9This example illustrates an ROC (Receiver Operating Characteristic) curve calculated by comparing images to determine abnormal threshold changes, with the vertical axis set to the true positive rate and the horizontal axis to the false positive rate. Figure 9 As shown, the information processing system according to this embodiment can suppress the occurrence of so-called over-detection, where a state that should be judged as normal is detected as abnormal, to a lower degree, and can detect the occurrence of abnormalities with higher accuracy.

[0107] (Replenish)

[0108] Furthermore, the above is merely one example and does not limit the application of the technology disclosed herein.

[0109] For example, refer to Figure 5 as well as Figure 6 The application of the learned model described is not necessarily limited to the reference model. Figures 7-9 This describes a system related to the monitoring of a predetermined object. As a specific example, the aforementioned learned model can be applied whenever data of a predetermined form is processed to make its characteristics more closely resemble those of data of the same form used as a benchmark (e.g., standardization). In this case, the type of data to which this processing is applied is not limited to image data, as described above.

[0110] In addition, in reference Figures 7-9 In systems related to the monitoring of a predetermined object, the learned model used in the standardization processing of captured images is not necessarily limited to a learned model constructed using GANs. That is, as long as the input captured image can be standardized to make its features closer to the features of a predetermined reference image, the construction method and type of the learned model used to implement the standardization processing are not particularly limited.

[0111] Thus, referring to Figure 5 as well as Figure 6 The techniques and references involved in constructing the learned model are explained. Figures 7-9 The technologies involved in monitoring the predetermined objects described can be implemented independently. Furthermore, there are no particular limitations regarding other technologies that can be used in combination with these technologies.

[0112] <Variation Example>

[0113] Next, refer to Figure 10 and Figure 11As a variation of the information processing system of this embodiment, an example of applying the information processing system of this embodiment to the inspection of so-called batch product replicas will be described. Specifically, the information processing system of this variation has the robustness to determine that if the deviation generated during the manufacture of the replica is within the allowable error range, it is considered to be in a normal state, and it detects the deviation that is outside the allowable range as an abnormal state.

[0114] For example, Figure 10 This diagram illustrates an example of a normal image used in the construction of the learned model in this variation. Specifically, normal image D113e is an image with a copy generated according to a standard as the subject. In contrast, normal images D113f to D113h are images with copies containing manufacturing deviations that are produced within permissible error ranges as the subject.

[0115] Specifically, normal image D113f schematically represents an image of a copy with dimensional deviations within the allowable error range as the subject. Similarly, normal image D113g schematically represents an image of a copy with at least a portion discolored within the allowable error range as the subject. Furthermore, normal image D113h schematically represents an image of a copy with a portion missing or deformed within the allowable error range as the subject.

[0116] In the information processing system of this variant example, when... Figure 10 Using the normal image D113e from the normal images D113e to D113h as the reference image D111, the other normal images D113f to D113h are input into the generation unit 101. Based on this, the discriminator 103, which forms a GAN with the generation unit 101, distinguishes between the normalized image D115 generated by the generation unit 101 based on the other normal images D113f to D113h and the aforementioned reference image D111. Based on this structure, the generation unit 101 constructs a learned model related to the application of this normalization process on the input image by performing normalization processing on the input normal image D113, making it more difficult for the discriminator 103 to distinguish between the reference image D111 and the normalized image D115.

[0117] Therefore, if the difference between the input image and the reference image D111 is within the range that can be extrapolated based on the differences in features between normal images D113e to D113h, then the above-mentioned learned model can be constructed to generate a standardized image D115 with features more similar to the reference image D111 with high accuracy based on the input image.

[0118] Furthermore, when inspecting the manufactured replicas, the generation unit 201 performs standardization processing on the captured image D123 corresponding to the captured image of the replica, based on the learned model, to make the features of the captured image closer to the features of the reference image D111 (normal image D113e), thereby generating a standardized image D125. Based on this, the determination unit 203 determines whether the replica under inspection has produced manufacturing deviations outside the permissible range, based on whether the generated standardized image D125 is approximately consistent with the reference image D111.

[0119] Furthermore, if the manufacturing-related deviations in the manufactured copy are within a range that can be extrapolated from the differences in features between the normal images D113e to D113h, then the features of the standardized image D125 generated from the captured image D123 of the copy are approximately consistent with the features of the reference image D111. That is, in this case, the determination unit 203 determines that the copy to be inspected has no problems (the manufacturing-related deviations are within the allowable error range).

[0120] In contrast, when the manufacturing-related deviations of the produced replica are difficult to extrapolate from the differences in features between normal images D113e to D113h, the features of the standardized image D125 generated based on the captured image D123 of the replica are not substantially consistent with the features of the reference image D111. That is, in this case, the determination unit 203 determines that the replica to be inspected is defective (the manufacturing-related deviations are outside the allowable error range).

[0121] Here, refer to Figure 11 A specific example is given to illustrate an anomaly detected by comparing the standardized image D125 with the reference image D111. D1113 represents an example of the reference image D111 in this variation. The reference image D1113 is equivalent to... Figure 10 The normal image D113e is shown. Additionally, D1253 represents an example of a normalized image D125 generated by normalizing the image D123 of the copy to be inspected using a learned model built on machine learning. Specifically, an example of a normalized image D125 generated when manufacturing-related deviations in the copy to be inspected are outside the allowable error range is shown.

[0122] In the standardized image D1253, differences V1254, V1255, and V1256, which are not present in the reference image D1113, are made explicit. Specifically, difference V1254 schematically represents a difference that becomes explicit due to defects or deformations outside the permissible error range in a copy that becomes part of the object being inspected. Furthermore, difference V1255 schematically represents a difference that becomes explicit due to the inclusion of foreign matter not present under normal conditions. Additionally, difference V1253 schematically represents a difference that becomes explicit due to discoloration outside the permissible error range in a copy that becomes part of the object being inspected.

[0123] The manufacturing deviations that occur in the reproduction, which are the main reason for the explicitness of differences V1254, V1255, and V1256, are, as described above, deviations outside the permissible error range. Therefore, even if a normalization process is performed on the captured image D123 to make its features closer to those of the reference image D1113, differences V1254, V1255, and V1256 become explicit in the generated normalized image D1253.

[0124] By utilizing this characteristic, the generated standardized image D1253 is compared with the reference image D1111 to detect differences (e.g., differences V1251, V1252, and V1253, etc.), thereby determining whether the manufactured copy has produced manufacturing deviations outside the allowable error range (i.e., inspecting the copy).

[0125] In addition to the deviations exemplified above, it can also detect deviations in manufacturing that are unthinkable when manufactured according to standards, such as cracks, dirt, deformation, and abnormal size.

[0126] <Summary>

[0127] As explained above, the information processing apparatus 200 of this embodiment performs standardization processing on the input image by using a pre-built learned model based on machine learning with multiple normal images as input to generate a standardized image. Based on determining whether the standardized image is approximately consistent with a predetermined reference image, notification information is output to a predetermined output destination according to the result of the determination.

[0128] This structure provides robustness, allowing normal states to be determined as long as the changes manifested in the input image are within acceptable limits, and detecting abnormal states when changes outside the acceptable limits become apparent. Therefore, for example, even when monitoring the state of a predetermined object using images captured by the imaging device, the occurrence of over-detection (mistaking conceivable state changes for abnormalities) can be suppressed to a lower level, and the occurrence of abnormalities can be detected with higher accuracy.

[0129] Furthermore, the information processing apparatus 100 of this embodiment constructs a learned model based on machine learning. This learned model generates a standardized image by performing a standardization process on an input image that makes the features of the input image closer to the features of a predetermined reference image. Specifically, the information processing apparatus 100 uses any one of a plurality of normal images as a reference image and constructs the learned model using a generative adversarial network composed of a learned model and a discriminator. This makes it more difficult to perform the standardization process on the input image to distinguish between the standardized image and the reference image by the discriminator. The discriminator is used to distinguish between the standardized image generated by performing standardization processing on other normal images besides the reference image among the plurality of normal images using the learned model and the reference image.

[0130] With this structure, the learned model described above can be constructed even when it is difficult to prepare images representing abnormal states. Furthermore, according to this structure, if the explicit changes in the input image are variations within a range that can be inferred from the features of the multiple normal images, a learned model can be constructed that can generate a standardized image with features approximately consistent with a reference image based on that input image. Additionally, in the information processing system of this embodiment, by limiting the reference image used as the comparison object with the standardized image, the processing load involved in constructing the learned model can be further reduced.

[0131] Furthermore, as mentioned above, the data used in constructing the learned model is not necessarily limited to image data; other types of data can also be used. In this case, the learned model is constructed by inputting multiple data points that are considered the correct solution (equivalent to data from a normal image). Specifically, any one of the multiple data points that are considered the correct solution is used as the baseline data (equivalent to data from a baseline image), and the other data is subjected to a standardization process to make the features of the other data more similar to the features of the baseline data.

[0132] As a more specific example, when audio data is used as input, the learned model is constructed to perform a standardization process on the input audio data to make the features of the audio data more similar to the features of the benchmark audio data.

[0133] Furthermore, when using the learned model for determination, the data input to the learned model (i.e., the data that has undergone standardization) is data of the same category as the data that primarily serves as the reference. Of course, the structure involved in acquiring this data can be appropriately modified (equivalent to the structure of the imaging device 300 in the case of image data) depending on the category of the data input to the learned model. Additionally, a desired determination (e.g., a determination of whether an anomaly is detected) is made based on a comparison between the data that has undergone standardization by the learned model (equivalent to standardized image data) and the data that serves as the reference.

[0134] The present invention has been described above together with the above-described embodiments. However, the present invention is not limited to the above-described embodiments. Various modifications can be made without departing from the technical concept of the present invention, and the above-described embodiments or variations can be combined as appropriate.

[0135] In addition, the present invention includes a method, a program for implementing the functions of the above embodiments, and a computer-readable recording medium storing the program.

[0136] Symbol Explanation

[0137] 1. Information processing system

[0138] 100 Information Processing Devices

[0139] 101 Production Department

[0140] 103 Discrimination Department

[0141] 110 Learning Processing Department

[0142] 200 information processing devices

[0143] Production Department 201

[0144] 203 Judgment Department

[0145] 205 Output Control Unit

[0146] 300 shooting device

[0147] 400 terminal device.

Claims

1. An information processing device, characterized in that, have: The generation unit generates a normalized image by performing normalization processing on the input image using a learned model, the learned model being a pre-built model based on machine learning with multiple normal images as input. The determination unit determines whether the standardized image is consistent with a predetermined reference image; as well as The output unit outputs the notification information to a predetermined output destination based on the result of the determination. In the machine learning, the learned model and the discriminator form a generative adversarial network. The discriminator distinguishes between the standardized image generated by applying the normalization process to other normal images besides the reference image among the plurality of normal images and the reference image. The learned model is constructed such that, taking any one of the plurality of normal images as the reference image, the input image is subjected to a standardization process that makes it more difficult for the discriminator to distinguish between the standardized image generated from the plurality of other normal images and the reference image.

2. The information processing device according to claim 1, characterized in that, In generating the standardized image, which serves as the discriminator's discrimination object, the learned model utilizes other normal images whose features are more similar to the reference image.

3. The information processing apparatus according to claim 1 or 2, characterized in that, The plurality of normal images comprises multiple images that differ from each other in terms of the conditions related to the shooting.

4. The information processing apparatus according to claim 3, characterized in that, The conditions related to shooting include at least one of the following: conditions related to the shooting environment, conditions related to the shooting parameters of the shooting device, and conditions related to the subject.

5. The information processing apparatus according to claim 4, characterized in that, Conditions related to the shooting environment include those related to the lighting environment.

6. The information processing apparatus according to claim 4, characterized in that, Conditions relating to the subject include conditions relating to the predetermined state of the subject.

7. The information processing apparatus according to claim 4, characterized in that, Conditions relating to the subject include conditions relating to deviations in the generation of a copy of the subject.

8. The information processing apparatus according to claim 1 or 2, characterized in that, The determination unit compares the standardized image with the reference image in pixels to determine whether the standardized image is consistent with the reference image.

9. The information processing apparatus according to claim 8, characterized in that, The determination unit compares the standardized image and the reference image pixel by pixel according to each element of the color that constitutes the pixel, thereby determining whether the standardized image is consistent with the reference image.

10. The information processing apparatus according to claim 1 or 2, characterized in that, If it is determined that the standardized image is not consistent with the reference image, the output unit will output the notification information indicating the anomaly to a predetermined output destination.

11. An information processing method executed by an information processing device, characterized in that, The information processing method includes the following steps: The generation step generates a standardized image by performing a standardization process on the input image using a learned model, wherein the learned model is a model pre-built based on machine learning with multiple normal images as input. The determination step involves determining whether the standardized image is consistent with a predetermined reference image; and The output step involves sending the notification information to the predetermined output destination based on the determination result. In the machine learning, the learned model and the discriminator form a generative adversarial network. The discriminator distinguishes between the standardized image generated by applying the normalization process to other normal images besides the reference image among the plurality of normal images and the reference image. The learned model is constructed such that, taking any one of the plurality of normal images as the reference image, the input image is subjected to a standardization process that makes it more difficult for the discriminator to distinguish between the standardized image generated from the plurality of other normal images and the reference image.

12. A computer-readable recording medium containing a program, characterized in that, This program causes the computer to perform the following steps: The generation step generates a standardized image by performing a standardization process on the input image using a learned model, wherein the learned model is a model pre-built based on machine learning with multiple normal images as input. The determination step involves determining whether the standardized image is consistent with a predetermined reference image. as well as The output step involves sending the notification information to the predetermined output destination based on the determination result. In the machine learning, the learned model and the discriminator form a generative adversarial network. The discriminator distinguishes between the standardized image generated by applying the normalization process to other normal images besides the reference image among the plurality of normal images and the reference image. The learned model is constructed such that, taking any one of the plurality of normal images as the reference image, the input image is subjected to a standardization process that makes it more difficult for the discriminator to distinguish between the standardized image generated from the plurality of other normal images and the reference image.

13. An information processing device, characterized in that, The information processing device includes: a construction unit that constructs a learned model based on machine learning, the learned model generating a standardized image by performing standardization processing on an input image to make the features of the input image more similar to the features of a predetermined reference image. The construction unit takes any one of the multiple normal images as the reference image and uses a generative adversarial network composed of the learned model and the discriminator to construct the learned model, so that the normalization process performed on the input image makes it more difficult for the discriminator to distinguish between the normalized image and the reference image. The discriminator is used to distinguish between the normalized image and the reference image generated by performing the normalization process on other normal images among the multiple normal images other than the reference image through the learned model.

14. An information processing method executed by an information processing device, characterized in that, The information processing method includes a construction step: constructing a learned model based on machine learning, wherein the learned model generates a standardized image by performing standardization processing on the input image to make the features of the input image closer to the features of a predetermined reference image. In the construction step, any one of the multiple normal images is taken as the reference image. The learned model is constructed using a generative adversarial network consisting of the learned model and the discriminator. This makes it more difficult for the discriminator to distinguish between the normalized image and the reference image when the input image is subjected to the normalization process. The discriminator is used to distinguish between the normalized image and the reference image generated by applying the normalization process to other normal images other than the reference image among the multiple normal images through the learned model.

15. A computer-readable recording medium containing a program, characterized in that, This program causes the computer to perform the build steps. In this construction step, a learned model is built based on machine learning. This learned model generates a standardized image by performing a standardization process on the input image to make its features more similar to the features of a predetermined reference image. In the construction step, any one of the multiple normal images is taken as the reference image. The learned model is constructed using a generative adversarial network consisting of the learned model and the discriminator. This makes it more difficult for the discriminator to distinguish between the normalized image and the reference image when the input image is subjected to the normalization process. The discriminator is used to distinguish between the normalized image and the reference image generated by applying the normalization process to other normal images other than the reference image among the multiple normal images through the learned model.

Images