Information processing device, information processing method, and program

Abstract

The objective is to provide an information processing device, an information processing method, and a program capable of identifying the nature of an anomaly. [Solution] The information processing device according to the embodiment comprises an acquisition means, a first generation means, a first extraction means, a second generation means, a identification means, and an output means. The acquisition means acquires a first image including the object to be inspected. The first generation means uses the first image and an anomaly detection model to generate an anomaly score map in which an anomaly score representing the degree of anomaly for each region included in the first image is assigned to that region. The first extraction means extracts a first region from the first image based on the anomaly score map. The second generation means generates a first text representing the area within the first region. The identification means identifies the content of the anomaly occurring in the object to be inspected based on the first text. The output means outputs the content of the anomaly.

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, for example, from the viewpoint of improving quality control, etc., technologies for automatically detecting abnormalities in inspection targets have been developed. Such abnormality detection is also useful for strengthening security and ensuring the reliability of automated systems.

[0003] In particular, since an image contains rich visual information, abnormality detection using the image is used for product quality inspection and assisting image diagnosis in the medical field.

[0004] Generally, for example, it is conceivable to detect an abnormality in an inspection target from an image using an abnormality detection model (trained model) generated by performing supervised learning. However, in order to perform such supervised learning, it is necessary to prepare an image (hereinafter referred to as an abnormal image) including an inspection target in an abnormal state and an annotation, and it is difficult to prepare the abnormality detection model.

[0005] Therefore, generating an abnormality detection model by performing unsupervised learning (that is, unsupervised abnormality detection) has attracted attention. Unsupervised learning is useful in that it does not require abnormal images because it uses only normal images. In addition, unsupervised learning also has an advantage in that it can detect unknown abnormalities.

[0006] However, in unsupervised abnormality detection, it is difficult to specify the content of the abnormality, and there is a possibility that sufficient abnormality detection cannot be realized in actual operation.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

[0008] Therefore, the problem that the present invention aims to solve is to provide an information processing device, an information processing method, and a program that can identify the nature of an anomaly. [Means for solving the problem]

[0009] The information processing device according to the embodiment comprises an acquisition means, a first generation means, a first extraction means, a second generation means, a identification means, and an output means. The acquisition means acquires a first image including the object to be inspected. The first generation means calculates an anomaly score representing the degree of anomaly for each region included in the first image using the acquired first image and an anomaly detection model generated by learning an image of the object to be inspected in a normal state, and generates an anomaly score map in which the calculated anomaly scores are assigned to the regions. The first extraction means extracts a first region from the first image based on the generated anomaly score map. The second generation means generates a first text representing the extracted first region. The identification means identifies the content of the anomaly occurring in the object to be inspected based on the generated first text. The output means outputs the content of the identified anomaly. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing an example of the functional configuration of the information processing device according to the first embodiment. [Figure 2] A diagram showing an example of the hardware configuration of an information processing device. [Figure 3] A flowchart illustrating an example of the processing procedure for an information processing device. [Figure 4] A diagram to specifically explain the operation of an information processing device. [Figure 5] A diagram illustrating the outline of the second embodiment. [Figure 6]A block diagram showing an example of the functional configuration of an information processing device. [Figure 7] A flowchart illustrating an example of the processing procedure for an information processing device. [Figure 8] A diagram to specifically explain the operation of an information processing device. [Modes for carrying out the invention]

[0011] The embodiments will be described below with reference to the drawings. (First Embodiment) First, the first embodiment will be described. The information processing device according to this embodiment operates as an anomaly detection device for detecting abnormalities in an object to be inspected using an image that includes the object to be inspected.

[0012] Figure 1 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. As shown in Figure 1, the information processing device 10 includes a first model storage unit 101, an image storage unit 102, a second model storage unit 103, an image acquisition unit 104, an anomaly detection unit 105, a region extraction unit 106, a text generation unit 107, a state determination unit 108, and an output unit 109.

[0013] The first model storage unit 101 stores pre-prepared anomaly detection models. These anomaly detection models are used, for example, to calculate an anomaly score based on an image containing the object to be inspected, as described later.

[0014] The image storage unit 102 corresponds to an image database that stores pre-prepared images. The images stored in the image storage unit 102 are, for example, images containing an inspection target in a normal state (hereinafter referred to as a normal image), but may also be images containing an inspection target in an abnormal state (hereinafter referred to as an abnormal image).

[0015] The second model storage unit 103 stores a pre-prepared generation model. The generation model stored in the second model storage unit 103 is used, for example, to generate text representing the inside of an image based on the image.

[0016] The image acquisition unit 104 acquires an image including the inspection target (hereinafter referred to as an inspection target image), for example, when inspecting a predetermined inspection target. The inspection target image is an image obtained, for example, by imaging the inspection target with a fixed-point camera, but may also be an image obtained by imaging the inspection target with a camera used by the user.

[0017] The abnormality detection unit 105 calculates an abnormality score corresponding to the inspection target image acquired by the image acquisition unit 104 using the abnormality detection model stored in the first model storage unit 101, and generates an abnormality score map based on the calculated abnormality score. The abnormality score map corresponds to map-form data in which an abnormality score representing the degree of abnormality is assigned to each region (pixel or batch) of the inspection target image. The abnormality score is an index regarding an abnormality such that, for example, the value increases as the degree of abnormality increases.

[0018] According to the above-described abnormality score map, the abnormality detection unit 105 can detect that an abnormality has occurred in the inspection target included in the inspection target image based on the abnormality score assigned in the abnormality score map.

[0019] However, even if it is detected that an abnormality has occurred in the inspection target as described above, it is difficult to specify the content (type) of the abnormality in the abnormality score map (abnormality detection model).

[0020] Furthermore, there are technologies that can use natural language processing to provide answers to questions (hereinafter referred to as question-answering technologies). When such question-answering technologies are applied to anomaly detection, it may be possible to identify the nature of the anomaly based on the answers (responses) to questions about anomalies occurring in the object being inspected, which are included in the image being inspected.

[0021] However, simply applying question-answering techniques to identify the nature of anomalies from images under examination requires precisely crafting questions to identify the location and nature of the anomaly. This results in a massive number of questions and a complex question design process (i.e., high cost for question design).

[0022] Therefore, in this embodiment, the above-described question design procedure is simplified, and a mechanism is provided that can identify the nature of the anomaly.

[0023] The region extraction unit 106 extracts a first region from the image under inspection based on the anomaly score map generated by the anomaly detection unit 105. The first region extracted by the region extraction unit 106 is a region of the image under inspection that has been assigned a high anomaly score in the anomaly score map (i.e., a region with a high anomaly score), and can be said to be a part of the image under inspection.

[0024] Furthermore, the region extraction unit 106 acquires a normal image from the image storage unit 102 and extracts a second region from the normal image. The normal image stored in the image storage unit 102 is, for example, an image obtained by imaging an inspection target in a normal state with a fixed-point camera. The second region extracted by the region extraction unit 106 is a region with the same range (position and size) as the first region described above.

[0025] The text generation unit 107 generates a first text representing the area within the first region using the first region extracted by the region extraction unit 106 and the generation model stored in the second model storage unit 103. The text generation unit 107 also generates a second text representing the area within the second region using the second region extracted by the region extraction unit 106 and the generation model stored in the second model storage unit 103.

[0026] The first text is text information describing the object itself or its state within the first region (i.e., the region with a high abnormality score in the image being examined), and the second text is text information describing the object itself or its state within the second region (the same region as the first region in the normal image). Furthermore, the generative model corresponds to the base model based on the question answering technology described above, and is assumed to be generated by self-supervised learning using, for example, a large amount of data (a set of images, questions, and answers). However, the generative model may be generated based on other learning methods.

[0027] The state determination unit 108 determines the state of the object to be inspected included in the image to be inspected based on the first and second texts generated by the text generation unit 107. In this embodiment, "determining the state of the object to be inspected" includes identifying the nature of the abnormality occurring in the object to be inspected.

[0028] The output unit 109 outputs the determination result from the status determination unit 108 (i.e., the details of the abnormality occurring in the object being inspected).

[0029] Figure 2 shows an example of the hardware configuration of the information processing device 10 shown in Figure 1. The information processing device 10 includes a CPU 10a, non-volatile memory 10b, main memory 10c, and communication device 10d, etc.

[0030] The CPU 10a is a processor for controlling the operation of various components within the information processing device 10. The CPU 10a may be a single processor or may consist of multiple processors. The CPU 10a executes various programs loaded from the non-volatile memory 10b into the main memory 10c. These programs executed by the CPU 10a include, for example, an operating system (OS) and application programs.

[0031] The non-volatile memory 10b is a storage medium used as an auxiliary storage device. The main memory 10c is a storage medium used as the main storage device. Although only the non-volatile memory 10b and the main memory 10c are shown in Figure 2, the information processing device 10 may also be equipped with other storage devices.

[0032] The communication device 10d is a device configured to communicate with an external device (for example, a server device).

[0033] In this embodiment, the first model storage unit 101, the image storage unit 102, and the second model storage unit 103 shown in Figure 1 are implemented by, for example, a non-volatile memory 10b or other storage device.

[0034] Furthermore, some or all of the image acquisition unit 104, anomaly detection unit 105, region extraction unit 106, text generation unit 107, state determination unit 108, and output unit 109 included in the information processing device 10 shown in Figure 1 are implemented by having the CPU 10a (i.e., the computer of the information processing device 10) execute a predetermined program, i.e., by software. This program may be stored and distributed on a computer-readable storage medium, or it may be downloaded to the information processing device 10 via a network. Note that some or all of these units 104 to 109 may be implemented by hardware such as ICs (Integrated Circuits), or by a combination of software and hardware.

[0035] Although not shown in Figure 2, the information processing device 10 may further include input devices such as a mouse or keyboard, and display devices such as a display.

[0036] Below, an example of the processing procedure of the information processing device 10 according to this embodiment will be described with reference to the flowchart in Figure 3.

[0037] First, the image acquisition unit 104 acquires an image to be inspected (step S1). In this embodiment, the object to be inspected included in the image to be inspected is assumed to be, for example, a product manufactured in a factory, but the object to be inspected can be any object in which an abnormality occurs in the appearance (surface) represented in the image.

[0038] When the process of step S1 is executed, the anomaly detection unit 105 generates an anomaly score map based on the anomaly score calculated using the inspection target image acquired in step S1 and the anomaly detection model stored in the first model storage unit 101 (step S2).

[0039] The following describes the process of step S2. Here, the anomaly detection model in this embodiment is assumed to be an autoencoder generated by, for example, learning using normal images (unsupervised learning). In this case, the anomaly detection model (autoencoder) is constructed to output an image similar to a normal image when a normal image is input, and when an image different from a normal image (for example, an abnormal image) is input, it is unable to reconstruct an image similar to that image and outputs an image different from that image. With such an anomaly detection model, the anomaly detection unit 105 can calculate an anomaly score based on the reconstruction error between the image to be inspected (input image) input to the anomaly detection model and the image output from the anomaly detection model (output image). The anomaly score is calculated for each region in the image to be inspected. Specifically, the anomaly score is calculated on a pixel or batch basis that constitutes the image to be inspected.

[0040] In step S2, the anomaly detection unit 105 can generate an anomaly score map by assigning the anomaly scores calculated for each region as described above to the respective regions.

[0041] In this embodiment, the anomaly detection model is described as an autoencoder, and the anomaly score is calculated using the autoencoder. However, the anomaly detection model in this embodiment only needs to contribute to the generation of an anomaly score map. For example, it may be a pre-trained model that has been trained to output an anomaly score for each region when an image to be inspected is input to the anomaly detection model, or it may be a pre-trained model that has been trained to output an anomaly score map when an image to be inspected is input to the anomaly detection model.

[0042] Next, the region extraction unit 106 extracts a first region from the image under examination based on the abnormal score assigned to each region in the abnormal score map generated in step S2 (step S3). In step S3, for example, regions to which an abnormal score greater than or equal to a predetermined value has been assigned are extracted as the first region.

[0043] Furthermore, the region extraction unit 106 acquires a normal image from the image storage unit 102 (step S4).

[0044] Here, assuming that the inspection target image acquired in step S1 is, for example, an image obtained by imaging the inspection target with a fixed-point camera, the normal image acquired in step S4 is an image obtained by imaging the inspection target (or a similar product) in a normal state with the same fixed-point camera.

[0045] In this case, the region extraction unit 106 extracts a second region with the same range as the first region from the normal image acquired in step S4 (step S5).

[0046] Next, the text generation unit 107 generates the first and second texts using the first region extracted in step S3, the second region extracted in step S5, and the generation model stored in the second model storage unit 103 (step S6).

[0047] Here, the generative model in this embodiment is assumed to be a trained model that has been trained to realize question answering technology. The method of training the generative model is not limited, but the generative model should be constructed to output, for example, an answer corresponding to a pre-prepared question sentence when an image is input.

[0048] According to such a generative model, the text generation unit 107 can generate first text representing the contents of a first region (a part of the image to be inspected) based on the answer to a question output from the generative model, by inputting the first region (a part of the image to be inspected) into the generative model. Similarly, the text generation unit 107 can generate second text representing the contents of a second region based on the answer to a question output from the generative model, by inputting the second region (a part of a normal image) into the generative model, by inputting the second region (a part of the normal image) into the generative model.

[0049] Note that the question used to generate the first text and the question used to generate the second text are the same. Furthermore, while the question may be created by the user, for example, to reduce the burden on the user in creating the question, the question may be generated using a generative model or similar method.

[0050] Furthermore, although this embodiment has been described as generating the first and second texts using a generative model, this embodiment only requires a configuration that can generate the first and second texts that represent the contents of the first and second regions, and the first and second texts may be generated by methods other than the generative model described above.

[0051] When the process in step S6 is executed, the state determination unit 108 determines (identifies) the state of the object to be inspected included in the image to be inspected based on the first and second text generated in step S6 (step S7).

[0052] As will be explained in more detail later, if, for example, the state of the subject cannot be determined without comparing the image to be examined (first region) with a normal image (second region), the process in step S7 is performed using, for example, the difference between the first and second texts (i.e., the difference in the state responses within the first and second regions).

[0053] On the other hand, if, for example, the state of the object to be inspected can be determined from the image to be inspected (first region), then the process in step S7 may be performed using, for example, only the first text (state within the first region).

[0054] When the processing in step S7 is executed, the output unit 109 outputs the judgment result in step S7 (step S8). The judgment result output in step S8 (i.e., the state of the object being inspected) corresponds to the content of the abnormality occurring in the object being inspected. In step S8, the judgment result may be output to the communication device 10d for transmission to, for example, an external server device of the information processing device 10, or it may be output to a display device (e.g., a display) for presentation to the user. In step S8, for example, the position of the first region in the image of the object being inspected (i.e., information about the location of the abnormality) may be output together with the judgment result (i.e., the content of the abnormality). If the position of the first region is output to a display device, the position of the first region may be displayed, for example, on the image of the object being inspected.

[0055] According to the process shown in Figure 3 above, anomaly detection is performed on the image to be inspected (an anomaly score map is generated) to identify the location and extent of the anomaly, and the content of the anomaly (the state of the object being inspected) can be identified from the answers to the questions in the question answering technology.

[0056] In step S4 described above, multiple normal images may be acquired. In this case, the nature of the abnormality occurring in the object under inspection can be identified based on, for example, a first text representing the first region extracted from the image under inspection and multiple second texts representing the second region extracted from each of the multiple normal images (i.e., multiple second texts generated for each normal image). Specifically, the state of the object under inspection can be determined multiple times based on the first text and each of the multiple second texts, and the nature of the abnormality occurring in the object under inspection can be identified by majority vote of the determination results. With such a configuration, even if, for example, an inappropriate normal image (a normal image in which the object under inspection was not properly captured) is stored in the image storage unit 102, it is possible to suppress a decrease in the accuracy of identifying the nature of the abnormality by considering other normal images.

[0057] Furthermore, although step S4 was described as acquiring a normal image, an abnormal image (an image containing the inspected object in an abnormal state) may be acquired instead. In this configuration, it is possible to identify the nature of the abnormality occurring in the inspected object by comparing the inspected object image with the abnormal image. Moreover, in step S4, both a normal image and an abnormal image may be acquired. In this case, it is possible to identify the nature of the abnormality occurring in the inspected object based on multiple second texts representing the second region extracted from both the normal image and the abnormal image.

[0058] Furthermore, in this embodiment, the anomaly detection unit 105 can determine whether or not an anomaly has occurred in the inspection target based on the anomaly score assigned to the anomaly score map generated in step S2. Specifically, the anomaly detection unit 105 compares, for example, the maximum value of the anomaly score assigned to the anomaly score map with a pre-prepared threshold (a threshold for distinguishing between normal and abnormal), and determines that an anomaly has occurred in the inspection target if the maximum value of the anomaly score is greater than or equal to the threshold. On the other hand, the anomaly detection unit 105 determines that no anomaly has occurred in the inspection target if the maximum value of the anomaly score is less than the threshold. In such a configuration, for example, if it is determined that an anomaly has occurred in the inspection target, the processing from step S3 onwards may be executed, and if it is determined that no anomaly has occurred in the inspection target, the processing from step S3 onwards may not be executed (i.e., the processing shown in Figure 3 is terminated).

[0059] Here, with reference to Figure 4, the operation of the information processing device 10 according to this embodiment will be described in detail. In this embodiment, we assume that the object to be inspected includes multiple objects (that is, multiple objects are arranged in the object to be inspected). Specifically, in the example shown in Figure 4, the object to be inspected is a food product, and the food product contains food or ingredients such as fish, kamaboko (fish cake), and sausages.

[0060] First, as shown in Figure 4, when an image of the food product to be inspected, obtained by imaging the food product with a fixed-point camera, is acquired, an anomaly score map 202 is generated using the image of the food product to be inspected 201 and the anomaly detection model.

[0061] Here, if a foreign object (e.g., a fly) is found in the head of a fish contained in the food product, an abnormality score map 202 is generated in which the region corresponding to the foreign object is assigned a higher abnormality score than other regions. According to such an abnormality score map 202, a first region 201a with a high abnormality score is extracted from the image under inspection 201 (i.e., a portion of the image corresponding to the first region 201a is cut out). Also, a second region 203a with the same range as the first region 201a is extracted from the normal image 203 (i.e., a portion of the image corresponding to the second region 203a is cut out). Note that the first region 201a and the second region 203a are, for example, rectangular regions.

[0062] Next, using the first domain 201a described above, a pre-prepared question, and a generative model, a first text representing the contents of the first domain 201a is generated. Similarly, using the second domain 203a described above, a pre-prepared question, and a generative model, a second text representing the contents of the second domain 203a is generated.

[0063] In the example shown in Figure 4, the question is "What is shown?", the first text (i.e., answer 1 to the question) is "A fly", and the second text (i.e., answer 2 to the question) is "A fish".

[0064] According to the first and second texts described above, it is possible to identify the abnormality in which a foreign object (in this case, a fly) is found in the first region 201a of the food product included in the image being inspected.

[0065] Here, we have described how to identify the nature of an anomaly based on the difference between the first and second texts. However, if the information processing device 10 (state determination unit 108) recognizes that the presence of a fly in the food product being inspected is an anomaly, it is also possible to identify the nature of the anomaly solely from the first text (i.e., that a fly is visible in the first region 201a). In other words, if the objective is to identify the presence of a foreign object in food products, it is sufficient to use the first text to identify the nature of the anomaly (determine the state of the product being inspected), and the process of extracting the second region 203a from the normal image 203 and generating the second text can be omitted.

[0066] Furthermore, although it differs from the example shown in Figure 4, if the content of the abnormality is to identify a defect in food or ingredients contained in a food product (for example, kamaboko), the first text can be generated using the first region containing the kamaboko, the question "Is there a defect in the object shown?", and a generative model. In this case, if the first text is, for example, "Yes (i.e., there is a defect)," then it can be determined that there is a defect in the kamaboko.

[0067] Furthermore, when identifying the nature of a wide range of anomalies, the anomaly can be identified based on multiple texts generated using several questions, such as "What is being shown?" or "Is there any damage to the object being shown?". Alternatively, for example, it may be possible to determine whether the anomaly can be identified based on the text generated using a predetermined question, and if the anomaly cannot be identified, further text may be generated using the next question.

[0068] Furthermore, in the example shown in Figure 4, it is assumed that the inspection target image 201 and the normal image 203 are images obtained by capturing the inspection target (food product) with a fixed-point camera (i.e., the inspection target is captured from the same position). However, if the positional relationship between the inspection target and the fixed-point camera is shifted (i.e., there is a positional shift of the inspection target between the inspection target image 201 and the normal image 203), different parts (positions) of the inspection target will be captured in the first region 201a and the second region 203a, respectively, which may reduce the accuracy of identifying the abnormality based on the difference between the first and second texts.

[0069] Therefore, in this embodiment, for example, before the process of extracting the second region 203a is executed, a process to correct the positional misalignment between the normal image 203 and the image to be inspected 201 (i.e., the positional misalignment of the object to be inspected in the image) may be executed. Such correction of positional misalignment can be achieved, for example, by extracting feature points representing the characteristics of the object to be inspected from both the image to be inspected 201 and the normal image 203, and then performing image processing to match the corresponding feature points between the images (i.e., matching the position of the object to be inspected in the normal image 203 to the position of the object to be inspected in the image to be inspected 201).

[0070] In this case, the second region 203a can be extracted from the normal image 203, which has been corrected for positional misalignment as described above.

[0071] As described above, the information processing device 10 according to this embodiment acquires an image of the object to be inspected (first image) including the object to be inspected, and uses an anomaly detection model generated by learning the acquired image of the object to be inspected and a normal image (an image including the object to be inspected in a normal state) to calculate an anomaly score representing the degree of anomaly for each region included in the image of the object to be inspected, and generates an anomaly score map in which the calculated anomaly scores are assigned to the regions. Furthermore, the information processing device 10 according to this embodiment extracts a first region from the image of the object to be inspected based on the generated anomaly score map and generates a first text representing the contents of the extracted first region. In addition, the information processing device 10 according to this embodiment identifies the content of the anomaly occurring in the object to be inspected based on the generated first text and outputs the content of the identified anomaly.

[0072] In this embodiment, the above-described configuration makes it possible to identify the nature of anomalies, which is generally difficult with anomaly detection methods that use anomaly detection models generated by unsupervised learning.

[0073] Specifically, in this embodiment, the scope of the anomaly is extracted (identified) using an anomaly score map, and then the content of the anomaly is identified based on the text generated by applying question answering technology. This eliminates the need to prepare questions to identify the anomaly location (i.e., the question design procedure can be omitted), thus enabling efficient anomaly detection.

[0074] In this embodiment, an abnormality score is calculated, with the value increasing as the degree of abnormality increases. A first region containing an abnormality score greater than or equal to a predetermined value is extracted from the image to be examined in the abnormality score map.

[0075] However, the first region in this embodiment may be a region extracted from a different perspective. Specifically, the first region may be a region that has been modified (e.g., enlarged or reduced) from a region to which an abnormality score greater than or equal to a predetermined value has been assigned. Furthermore, if, for example, the abnormality score decreases as the degree of abnormality increases, the first region including a region to which an abnormality score less than a predetermined value has been assigned in the abnormality score map may be extracted from the image under examination.

[0076] Furthermore, in this embodiment, it is assumed that the first region extracted from the image to be inspected is, for example, a rectangular region. However, the shape of the first region may be other than a rectangle, or it may be a shape determined according to, for example, the shape of the object to be inspected.

[0077] Furthermore, the first text representing the first domain in this embodiment can be generated by inputting the first domain into a generative model called a base model. This generative model (base model) is constructed, for example, to output a response corresponding to the first domain to a pre-prepared question when the first domain is input, and the first text can be generated based on the response output from the generative model. The question used to generate the first text may be selected from a large number of pre-prepared questions according to the subject to be examined, and the first text may be generated based on the response output from the generative model when the first domain and the selected question are input into the generative model.

[0078] Furthermore, this embodiment may be configured to generate the first text from the first region, and for example, the first text may be generated by performing image processing on the first region.

[0079] Furthermore, the information processing device 10 according to this embodiment may be configured to extract a second region corresponding to the first region from a pre-prepared normal image (second image including the object to be inspected), further generate a second text representing the extracted second region, and identify the content of the abnormality occurring in the object to be inspected based on the difference between the first text and the generated second text. With such a configuration, it may be possible to identify the content of the abnormality occurring in the object to be inspected with higher accuracy compared to simply using only the first text (first region). However, for example, in cases where the processing load of the information processing device 10 needs to be reduced, a configuration that identifies the content of the abnormality occurring in the object to be inspected using only the first text may be adopted. Furthermore, whether to use only the first text or both the first and second texts when identifying the content of the abnormality occurring in the object to be inspected may be appropriately selected depending on the object to be inspected, for example.

[0080] Furthermore, in this embodiment, the normal image from which the second region used to generate the second text described above is extracted may be corrected for positional misalignment with the image under inspection based on the image under inspection and feature points extracted from the normal image. In this case, the second region is extracted from the normal image whose positional misalignment has been corrected. In this embodiment, as described above, the same part of the object under inspection is included in the first and second regions, making it possible to improve the accuracy of identifying the nature of the abnormality occurring in the object under inspection.

[0081] In this embodiment, the second region was described as being extracted from a normal image, but the second region may be extracted from each of a plurality of normal images, or from at least one abnormal image (an image containing an object being examined in an abnormal state), or from both a normal image and an abnormal image.

[0082] In other words, in this embodiment, the system may be configured to identify the nature of the abnormality occurring in the object being inspected based on the difference (response difference) between each of the multiple second texts obtained using the first text and multiple images in the image storage unit 102 (image database).

[0083] Furthermore, although this embodiment has been described as outputting the content of an anomaly identified based on, for example, the first text (i.e., the content of the anomaly occurring in the object being inspected), other information such as the location of the first region in the image of the object being inspected (i.e., the location where the anomaly occurred in the object being inspected) may also be output in addition to the content of the anomaly.

[0084] In this embodiment, the information processing device 10 has been described as including the parts 101 to 109 shown in Figure 1, but the configuration of the information processing device 10 may differ from that shown in Figure 1. Specifically, the information processing device 10 according to this embodiment may have a configuration in which at least a part of the parts 101 to 109 shown in Figure 1 is located externally, or it may have a configuration that further includes functional parts other than the parts 101 to 109. Furthermore, the information processing device 10 according to this embodiment may be realized in the form of an information processing system or the like, comprising a first device including a part of the parts 101 to 109 shown in Figure 1, and a second device including the other parts of the parts 101 to 109.

[0085] (Second Embodiment) Next, a second embodiment will be described. In this embodiment, the description of parts that are the same as those of the first embodiment described above will be omitted, and the parts that differ from the first embodiment will be described mainly.

[0086] In the first embodiment described above, a first region containing an abnormality score greater than or equal to a predetermined value in the abnormality score map is extracted from the image to be inspected. However, in such a first region, the accuracy of identifying the nature of the abnormality occurring in the image to be inspected may decrease.

[0087] Specifically, as shown in Figure 5, if the food product being inspected has, for example, a blemish on a part of a kamaboko (fish cake), an abnormality score map 302 is generated from the inspection target image 301 that includes the food product in question.

[0088] In this case, the abnormal score map 302 assigns a high abnormal score to the damaged part of the kamaboko (fish cake), and the first region containing the area with a high abnormal score is extracted from the image 301 to be examined.

[0089] However, the first region extracted from the image 301 under examination based on such an anomaly score map 302 may be a smaller region compared to the kamaboko itself, and even if question answering techniques are applied to this first region, appropriate text may not be generated.

[0090] Therefore, unlike the first embodiment described above, this embodiment employs a configuration in which a region containing an object is extracted as the first region.

[0091] Figure 6 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. In Figure 6, the same reference numerals are used for parts similar to those in Figure 1 described above, and their detailed explanations are omitted.

[0092] As shown in Figure 1, the information processing device 10 includes a region estimation unit 110. Based on the anomaly score map generated by the anomaly detection unit 105, the region estimation unit 110 identifies objects in the image to be inspected that have been assigned a high anomaly score, and estimates the region encompassing the identified object (hereinafter referred to as the object region).

[0093] In this embodiment, the region extraction unit 106 extracts the object region estimated by the region estimation unit 110 as the first region from the image to be inspected.

[0094] The hardware configuration of the information processing device 10 is the same as that shown in Figure 2 above, so a detailed explanation is omitted here. In this embodiment, part or all of the region estimation unit 110 is implemented by having the CPU 10a execute a predetermined program (i.e., software), but it may also be implemented by hardware, or by a combination of software and hardware.

[0095] Below, an example of the processing procedure of the information processing device 10 according to this embodiment will be described with reference to the flowchart in Figure 7.

[0096] First, the processes of steps S11 and S12, which correspond to the processes of steps S1 and S2 shown in Figure 3 above, are executed.

[0097] Next, the region estimation unit 110 estimates the region of an object in the image under inspection that exhibits anomalies (i.e., the object region) based on the region with a high anomaly score in the anomaly score map generated in step S12 (step S13). In step S13, the unit identifies an object that overlaps with a region in the anomaly score map that has been assigned an anomaly score greater than or equal to a predetermined value, and estimates the object region based on the identified object. The object region may be a rectangular region or a region with a shape that follows the contour of the object. Furthermore, object identification may be performed using techniques such as GrabCut or SAM (Segment Anything Model).

[0098] The region extraction unit 106 extracts a first region from the image to be inspected based on the object region estimated in step S13, for example (step S14).

[0099] Once the process in step S14 is executed, the processes in steps S15 to S19, which correspond to the processes in steps S4 to S8 shown in Figure 3 above, are executed.

[0100] Although not shown in Figure 7, a process to correct the misalignment described in the first embodiment may be performed between steps S15 and S16.

[0101] Here, with reference to Figure 8, the operation of the information processing device 10 according to this embodiment will be specifically described. Note that the object of inspection in the example shown in Figure 8 is the food product described in Figure 4 above.

[0102] First, as shown in Figure 8, when an inspection target image 301 obtained by imaging food products with a fixed-point camera is acquired, an anomaly score map 302 is generated using the inspection target image 301 and the anomaly detection model.

[0103] If a portion of the kamaboko (fish cake) contained in the food product is damaged, an abnormality score map 302 is generated in which the region corresponding to the damaged area is assigned a higher abnormality score than other regions.

[0104] In this case, kamaboko (fish cake) is identified as an object that overlaps with the region assigned a high abnormal score in the abnormal score map 302, and the object region encompassing the kamaboko is estimated. In this embodiment, the estimated object region is designated as the first region 301a and extracted from the image to be inspected 301. In addition, a second region 303a, which has the same range as the first region 301a (object region), is extracted from the normal image 303. Note that the first region 301a and the second region 303a are, for example, rectangular regions.

[0105] Next, using the first domain 301a described above, a pre-prepared question, and a generative model, a first text representing the contents of the first domain 301a is generated. Similarly, using the second domain 303a described above, a pre-prepared question, and a generative model, a second text representing the contents of the second domain 303a is generated.

[0106] In the example shown in Figure 8, question 1 is "What is shown?", the first text (i.e., answer 1-1 to question 1) is "kamaboko", and the second text (i.e., answer 1-2 to question 1) is "kamaboko".

[0107] Furthermore, in the example shown in Figure 8, question 2 is "Is there a scratch on the object shown?", the first text (i.e., answer 2-1 to question 2) is "Yes", and the second text (i.e., answer 2-2 to question 2) is "No".

[0108] According to the first and second texts described above, it is possible to identify the nature of the abnormality as a scratch on the kamaboko (fish cake) located at the position of the first region 301a, rather than the presence of a foreign object.

[0109] In this embodiment, since the object regions are extracted from the inspection target image 301 and the normal image 303 as the first region 301a and the second region 303a, the answers 1-1 and 1-2 to question 1 are "kamaboko" (i.e., the object included in the first region 301a and the second region 303a can be recognized). However, if, for example, only the damaged part of the kamaboko (i.e., a part of the kamaboko) is extracted as the first and second regions, the answer corresponding to those first and second regions will not be "kamaboko," and there is a possibility that an inappropriate abnormality will be identified.

[0110] In contrast, this embodiment extracts not just an abnormal range (an area with a high abnormality score), but an object region (first region 301a and second region 303a), and applies question-answering technology to that object region. Therefore, it is possible to identify the nature of the abnormality occurring in the object being inspected with high accuracy.

[0111] According to at least one embodiment described above, it is possible to provide an information processing device, an information processing method, and a program capable of identifying the nature of an anomaly.

[0112] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0113] 10... Information processing device, 10a... CPU, 10b... Non-volatile memory, 10c... Main memory, 10d... Communication device, 101... First model storage unit, 102... Image storage unit, 103... Second model storage unit, 104... Image acquisition unit, 105... Anomaly detection unit, 106... Region extraction unit, 107... Text generation unit, 108... State determination unit, 109... Output unit, 110... Region estimation unit.

Claims

1. An acquisition means for acquiring a first image including the subject of inspection, A first generation means calculates an anomaly score representing the degree of anomaly for each region included in the first image using an anomaly detection model generated by learning the acquired first image and an image of the subject to be inspected in a normal state, and generates an anomaly score map in which the calculated anomaly scores are assigned to the region. A first extraction means for extracting a first region from the first image based on the generated anomaly score map, A second generation means for generating a first text representing the extracted first region, Based on the first text generated, a means for identifying the nature of the abnormality occurring in the object being inspected, Output means for outputting the details of the identified abnormality, An information processing device equipped with the following.

2. The information processing apparatus according to claim 1, wherein the first region includes a region in the generated abnormal score map to which an abnormal score greater than or equal to a predetermined value has been assigned.

3. The information processing apparatus according to claim 1, wherein the first region includes a region that contains objects to which an abnormal score greater than or equal to a predetermined value has been assigned in the generated abnormal score map.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the first region is a rectangular region.

5. The information processing apparatus according to claim 1, wherein the first text is generated by inputting the extracted first region into the base model.

6. The aforementioned base model is constructed to output a pre-prepared answer to a question sentence corresponding to the first domain when the first domain is input. The above-mentioned first text is generated based on the response output from the base model. The information processing apparatus according to claim 5.

7. The system further comprises a second extraction means for extracting a second region corresponding to the extracted first region from a second image containing the subject to be inspected, which is prepared in advance. The second generation means further generates a second text representing the extracted second region, The identification means identifies the nature of the abnormality occurring in the object under inspection based on the difference between the generated first text and the generated second text. The information processing apparatus according to claim 1.

8. The second image is corrected for positional misalignment with the first image based on the feature points extracted from the first image and the second image. The information processing apparatus according to claim 7, wherein the second extraction means extracts the second region from the second image in which the positional displacement has been corrected.

9. The information processing apparatus according to claim 7, wherein the second image includes at least one image of an object to be inspected in a normal state or an image of an object to be inspected in an abnormal state.

10. The information processing apparatus according to claim 1, wherein the output means further outputs the position of the first region in the first image.

11. An information processing method performed by an information processing device, To obtain the first image including the subject of examination, Using the acquired first image and an anomaly detection model generated by learning images of the subject being inspected in a normal state, an anomaly score representing the degree of anomaly for each region included in the first image is calculated, and an anomaly score map is generated in which the calculated anomaly scores are assigned to the regions. Based on the generated anomaly score map, a first region is extracted from the first image, To generate a first text representing the extracted first region, Based on the first text generated, the nature of the abnormality occurring in the subject of inspection is identified, Output the details of the identified anomaly. An information processing method comprising the following.

12. A program executed by a computer of an information processing device, To the aforementioned computer, To obtain the first image including the subject of examination, Using the acquired first image and an anomaly detection model generated by learning images of the subject being inspected in a normal state, an anomaly score representing the degree of anomaly for each region included in the first image is calculated, and an anomaly score map is generated in which the calculated anomaly scores are assigned to the regions. Based on the generated anomaly score map, a first region is extracted from the first image, To generate a first text representing the extracted first region, Based on the first text generated, the nature of the abnormality occurring in the subject of inspection is identified, Output the details of the identified anomaly. A program to execute.

Images