Cause elucidation support device and cause elucidation support method

Abstract

To realize a technology that is useful for supporting elucidation of cause of a determination error in an AI model that determines whether or not a detection object is photographed in a photographed image.SOLUTION: A cause elucidation support device that supports elucidation of cause of a determination error made by an AI model 10 that determines whether or not a detection object 30 is photographed in a photographed image 20, executes: a first test of generating a setting-changed photographed image obtained by photographing by changing camera setting on the basis of an erroneous image 22, which is a photographed image 20 in which a model to be tested, which is an AI model 10 to be tested, makes a determination error, and inputting the setting-changed photographed image into the model to be tested to determine whether or not the determination is successful; a second test of evaluating lack of learning photographed images on the basis of a learning photographed image group used when the model to be tested is generated and the erroneous image 22; and a third test of performing learning processing using the learning photographed image group for each of a plurality of types of incomplete AI models to generate AI models by type, and inputting the erroneous image 22 into each of the AI models by type to determine whether or not the determination is successful.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a cause elucidation support device that supports elucidation of the cause of judgment errors in AI models. [Background technology]

[0002] Generally, for those who use AI (Artificial Intelligence) models, the decision-making process of the AI ​​model is a black box, and it is difficult to understand what kind of processing and what kind of decision-making process is performed. Here, the AI ​​model includes a machine learning model, and includes pre-processing for inputting data into the machine learning model and the machine learning model. Therefore, the AI ​​model can be said to be a machine learning model in a stricter narrower sense. Therefore, although the AI ​​model and the machine learning model can be said to be equivalent in the present invention, the AI ​​model will be described in the broader sense in this specification.

[0003] The judgment process of an AI model may be a black box for the user. However, for businesses that manufacture and sell products incorporating an AI model in their systems, or businesses that provide services incorporating an AI model in their systems, it is desirable to quickly clarify the cause of an AI model's judgment error when the AI ​​model uses the AI ​​model as part of the system. However, there is currently no universally effective technology for clarifying the cause of an AI model's judgment error. This is because the usage forms of AI models vary depending on the type of AI model used and the type of processing performed in each technical field. For example, Patent Document 1 proposes a method for presenting which area of ​​an image was used to estimate similarity when an image is used as a search key for similar image search in a search for documents including screen data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-96379 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 is a technology specialized for extremely restricted conditions, that is, when an image is used as a search key for similar image search in searching for documents including screen data. Moreover, it is a technology that determines an extremely limited cause, that is, which area of ​​an image was used to estimate similarity. It is difficult to apply this technology to other usage forms using an AI model.

[0006] Research and development of technologies that use AI models is also being conducted in the field of railways. For example, development is underway on a train front monitoring system that detects obstacles on the track using an AI model that determines whether an object is captured in an image captured in front of the train. As mentioned above, when using an AI model as part of a system, it is desirable to be able to clarify the cause when the AI ​​model makes a mistake in judgment, and this is the same in the field of railways. However, there is still no known technology for clarifying the cause when an AI model used for monitoring the front of railway trains makes a mistake in judgment.

[0007] Although we have given an example of a train front monitoring system, the above issue is similar to other examples as long as the AI ​​model judges whether or not a detection target is captured in the captured image. For example, there is a monitoring system that uses a camera to capture and monitor a location on a railroad or road where suspicious objects should not exist, and uses an AI model to judge whether a suspicious object is captured as a detection target in the captured image.

[0008] The problem that this invention aims to solve is to realize technology that is useful for supporting the elucidation of the cause of judgment errors made by AI models that determine whether or not a detection target is captured in a captured image. [Means for solving the problem]

[0009] The first invention for solving the above problem is: A cause elucidation support device that supports elucidation of the cause of a judgment error made by an AI (Artificial Intelligence) model that judges whether or not a detection target is captured in a captured image, A first inspection means (e.g., the first inspection unit 202 in FIG. 11) that generates a setting-changed image when a camera setting is changed and photographed based on a mistake image, which is a photographed image in which a test model, which is the AI ​​model to be inspected, makes a first inspection as to whether the judgment is successful or not by inputting the setting-changed photographed image to the test model; A second inspection means (e.g., the second inspection unit 206 in FIG. 11 ) that performs a second inspection by evaluating a shortage of learning images based on the learning images used when generating the test model and the error images; a third inspection means (e.g., the third inspection unit 210 in FIG. 11 ) for performing a learning process using the group of captured learning images for each of a plurality of types of incomplete AI models to generate type-specific AI models, and inputting the incorrect image into each of the type-specific AI models to perform a third inspection to determine whether or not the judgment is successful; This is a cause elucidation support device equipped with the above.

[0010] Other inventions include: A cause elucidation support method for a computer system to support elucidation of the cause of a judgment error made by an AI (Artificial Intelligence) model that judges whether a detection target is included in a captured image, comprising: A first test is performed to determine whether the AI ​​model to be inspected makes a judgment error by generating a setting-changed image captured by changing the camera settings based on the mistake image, and inputting the setting-changed image into the test model to determine whether the judgment is successful (for example, step S1 in FIG. 4). A second inspection is performed by evaluating a shortage of learning images based on the learning images used when generating the test model and the error image (for example, step S5 of FIG. 4 ); A learning process is performed on each of the multiple types of incomplete AI models using the group of captured learning images to generate type-specific AI models, and the error image is input to each of the type-specific AI models to perform a third test to determine whether or not the judgment is successful (for example, step S9 in FIG. 4). A cause elucidation support method including the above may be configured.

[0011] According to the first invention, etc., it is possible to realize a technology useful for supporting the elucidation of the cause of a judgment error of an AI model that judges whether or not a detection target is captured in a captured image. In other words, three tests can be performed to identify the possible cause of the judgment error of the AI ​​model that is the test model: a first test to find a camera setting that will result in a successful judgment, assuming that the cause is in the camera settings; a second test to evaluate a shortage of captured learning images, assuming that the cause is in the group of captured learning images used to generate the AI ​​model; and a third test to find the type of AI model that will result in a successful judgment, assuming that the cause is in the AI ​​model itself.

[0012] The second invention is the above-mentioned invention, This is a cause elucidation support device that determines whether to continue or terminate the inspection based on the inspection result of the first inspection, and if the inspection is to be continued, performs the second inspection by the second inspection means, and determines whether to continue or terminate the inspection based on the inspection result of the second inspection, and if the inspection is to be continued, performs the third inspection by the third inspection means.

[0013] According to the second invention, the cause of the misjudgment of the AI ​​model can be efficiently identified by performing each test in the order of the first test, the second test, and the third test, and if the cause cannot be identified, the next test is performed. In particular, the order of the first test, the second test, and the third test is the order in which the cost of dealing with the identified cause is relatively low. Therefore, this order of the tests is particularly effective.

[0014] The third invention is the above-mentioned invention, a first presenting means (for example, the first presenting unit 204 in FIG. 11 ) for presenting, as an improvement item, the camera settings related to the setting-changed captured image for which the first test was successfully performed; The cause elucidation support device further comprises:

[0015] According to the third invention, if the first inspection identifies that the cause of the misjudgment of the test model is the camera settings, specific camera settings can be presented as improvement items as a countermeasure to prevent the recurrence of the misjudgment.

[0016] The fourth aspect of the present invention is the above-mentioned invention, The second inspection means is analyzing features of each of the learning images included in the learning image group using a feature analysis process that analyzes features of the captured images, and calculating a feature distribution of the learning image group; analyzing the characteristics of the miss image using the characteristic analysis process; comparing the feature distribution with features of the mismatched images to determine features that are lacking in the learning captured images; is included in the second inspection, It is a cause clarification support device.

[0017] According to the fourth invention, as a second inspection, the feature distribution of the group of learning images analyzed using feature analysis processing is compared with the features of the defective image, thereby making it possible to determine and evaluate the features of the image that are lacking as a learning image from the viewpoint of image features.

[0018] The fifth invention is the above-mentioned invention, The feature analysis process is a process of analyzing the features of an image portion of the detection object in the captured image in which the detection object is captured, including the features of the image portion as features of an analysis target. It is a cause clarification support device.

[0019] According to the fifth aspect of the present invention, the determination and evaluation of the characteristics of an image that is lacking as a learning captured image can be made more meaningful by focusing on the detection object appearing in the captured image.

[0020] The sixth aspect of the present invention is the above-mentioned invention, a second presentation means (e.g., the second presentation unit 208 in FIG. 11 ) for selecting a learning image that satisfies a predetermined similar feature condition based on the missing feature determined by the second inspection from the group of learning images, and presenting the selected learning image and information indicating the missing feature; The cause elucidation support device further comprises:

[0021] According to the sixth aspect of the present invention, when the second inspection identifies that the cause of the misjudgment of the test model is a lack of learning images, it is possible to present the learning images from the learning images that are similar to the image having the missing features together with the features of the missing images, thereby providing a clue as to which learning images should be added as a countermeasure.

[0022] The seventh aspect of the present invention is the above-mentioned invention, The multiple types of unfinished AI models are different in at least one of the following machine learning models: 1) algorithm, 2) structure, 3) number of layers, 4) number of nodes, and 5) number of parameters. It is a cause clarification support device.

[0023] According to the seventh invention, in the third inspection, multiple types of incomplete AI models having different algorithms and structures related to the machine learning models, the number of layers, the number of nodes, and the number of parameters can be used.

[0024] The eighth aspect of the present invention is the above-mentioned invention, a third presentation means (e.g., the third presentation unit 212 in FIG. 11 ) for presenting information on the type-specific AI model that has been determined to be successful in the third test; The cause elucidation support device further comprises:

[0025] According to the eighth invention, when the third inspection identifies that the cause of the judgment error of the test model is an AI model, information on the type-specific AI models that made a successful judgment can be presented.

[0026] The seventh aspect of the present invention is the above-mentioned invention, The captured image is an image of a track on which a vehicle runs, The detection object is an obstacle on the trajectory. The cause elucidation support device according to claim 1 or 2.

[0027] According to the seventh aspect of the present invention, a method suitable for identifying the cause of a judgment error in an AI model used for monitoring the front of a train to detect obstacles on the track is realized. [Brief description of the drawings]

[0028] [Figure 1] An explanatory diagram of the AI ​​model to be inspected. [Diagram 2] An illustration of AI model generation. [Diagram 3] An illustration of an AI model's misjudgment. [Figure 4] 13 is a flowchart of a cause elucidation support process. [Diagram 5] An example of changes to the camera settings for the first inspection. [Figure 6] 11 is a flowchart of a first inspection process. [Figure 7] An example of image features analyzed in the second test. [Figure 8] 13 is a comparison example between the feature distribution of a group of captured learning images and the features of a mistaken image. [Figure 9] 11 is a flowchart of a second inspection process. [Figure 10] 13 is a flowchart of a third inspection process. [Figure 11] 1 shows an example of the functional configuration of a cause elucidation support device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. Note that the form to which the present invention can be applied is not limited to the following embodiments. In addition, in the description of the drawings, the same elements are given the same reference numerals.

[0030] The cause elucidation support device 1 of this embodiment is a device for supporting elucidation of the cause of a judgment error made by an AI model.

[0031] FIG. 1 shows a diagram for explaining an AI model that is a subject model to be inspected by the cause elucidation support device 1. The AI ​​model 10 that is the subject model is used for monitoring the front of a train on a railway, and judges whether or not a detection object 30 is captured in an input photographed image 20 (presence or absence of a detection object 30) and outputs the result. The photographed image 20 is an image of the track in front of the railway vehicle captured by a camera mounted on the railway vehicle. The detection object 30 is an obstacle that affects the running of the railway vehicle, and is, for example, an object such as a person, an animal, a car, or a bicycle on or near the track. In FIG. 1, the photographed image 20a on the left side shows a "maintenance worker" that is the detection object 30 beside the track, and the AI ​​model 10 judges that "the detection object is present". The photographed image 20b on the right side does not show the detection object 30, and the AI ​​model 10 judges that "the detection object is not present".

[0032] 2, the AI ​​model 10 is a trained AI model generated by performing a training process on an untrained AI model 12 using a training image group 40, which is a collection of training captured images 42 in which the captured images are associated (matched) with the presence or absence of a detected object 30. The training captured image group 40 can also be considered as teacher data for training the AI ​​model, with data input to the AI ​​model being the training captured images 42 and data output by the AI ​​model being the presence or absence of a detected object 30.

[0033] An error in judgment by the AI ​​model 10 refers to a case in which the AI ​​model 10 judges that a “detection target is not present” even though a captured image 20 (22) contains a detection target 30, as shown in an example in FIG.

[0034] In order to assist in identifying the cause of a judgment error made by the AI ​​model 10, which is the test model, the cause elucidation support device 1 estimates the cause of the judgment error based on the error image 22, which is the captured image 20 in which the AI ​​model 10 made a judgment error, and the learning captured image group 40 used in the learning process when generating the AI ​​model 10, and presents improvement items that can prevent the recurrence of the judgment error.

[0035] FIG. 4 is a flowchart for explaining the cause elucidation support processing performed by the cause elucidation support device 1. As shown in FIG. 4, the cause elucidation support device 1 first performs a first inspection (step S1). The first inspection is an inspection assuming that the cause is caused by the camera settings, which will be described in detail later. If the cause of the judgment error is not elucidated as a result of the first inspection (step S3: NO), the second inspection is then performed (step S5). The second inspection is an inspection assuming that the cause is caused by the learning captured image group 40, which will be described in detail later. If the cause of the judgment error is not elucidated as a result of the second inspection (step S7: NO), the third inspection is then performed (step S9). The third inspection is an inspection assuming that the cause is caused by the AI ​​model 10, which is a test model, which will be described in detail later. If the cause of the judgment error is not elucidated as a result of the third inspection, a message is displayed to the effect that the cause is unknown (step S13). The cause elucidation support processing is performed as described above.

[0036] Each test will be explained in detail. The first test is a test that assumes that the cause is the camera settings. Specifically, a setting-changed photographed image is generated when the camera settings are changed and photographed based on the error image 22, and the setting-changed photographed image is input to the test model to check whether the judgment is successful or not.

[0037] Even in the same situation, the captured image may differ depending on the camera settings used to capture the image. For this reason, various images that can be captured when the camera settings are changed (images captured with changed settings) are generated from the error image 22 and input to the test model, and the cause of the judgment error and a remedy for it are estimated by searching for a camera setting that results in a successful judgment (determined to be "detected object is present"). The image captured with changed settings when captured with changed camera settings is generated in a simulated manner by performing image processing on the error image 22.

[0038] Figure 5 shows an example of a list of camera settings to be changed in the first inspection. In this camera setting list, each change in the camera settings that is an improvement item is associated with an image processing method that simulates the change in the camera settings, and the image conditions that are changed by the camera settings. The image conditions are the state of the captured image that is thought to have caused the misjudgment of the subject model, such as image blur or blur, contrast, white balance, resolution, aspect ratio, etc., and the changes in the camera settings that correct or change the state of the captured image are associated with each other.

[0039] Fig. 6 is a flow chart for explaining the detailed process flow of the first inspection. First, one of a plurality of pre-prepared camera setting changes, such as the camera setting list shown in Fig. 5 (step S21), is selected. Note that the camera setting changes may be a plurality of items with specific stepwise values, such as "1 / 125" and "1 / 250" for "shutter speed".

[0040] Next, image processing corresponding to the selected camera setting change is performed on the error image 22 to generate a setting-changed photographed image when photographed with the camera setting changed (step S23). Next, the generated setting-changed photographed image is input to the subject model, and it is determined whether the judgment is successful, that is, whether the judgment result is "detection target presence."

[0041] If the judgment is not successful (judgment result is "no object to be detected") (step S27: NO) and there is an unselected change to the camera setting (step S31: YES), the process returns to step S21, an unselected change to the camera setting is selected, and the same process is repeated. If there is no unselected change to the camera setting (step S33: NO), it is presented that the cause of the judgment error of the test model cannot be elucidated by the first inspection, that is, the cause of the judgment error is not in the camera setting (step S33).

[0042] On the other hand, if the judgment is successful (step S27: YES), the system notifies the user that the cause of the judgment error of the subject model is the camera settings, and presents the selected camera setting changes (i.e., the countermeasures for the cause) and the corresponding image conditions as improvement items (step S29). The first inspection is performed in this manner.

[0043] The second test is a test assuming that the cause is the learning image group 40 used to generate the AI ​​model 10. Specifically, based on the learning image group 40 used when generating the test model and the error image 22, it is evaluated whether the learning image 42 used for learning was sufficient. In detail, using a feature analysis process that analyzes the features of an image, the features of each of the learning images 42 included in the learning image group 40 are analyzed, the feature distribution of the learning image group 40 is calculated, and the feature of the error image 22 is analyzed. Then, the feature distribution of the learning image group 40 is compared with the feature of the error image 22 to determine whether the feature of the error image 22 is a feature that is lacking in the learning image group 40. The feature analysis process is a process that analyzes the features of the image part of the detection object 30 of the captured image 20 in which the detection object 30 is captured, including the features of the target of analysis.

[0044] In order for the AI ​​model 10, which is a test model, to correctly judge that the captured image 20 in which the detection target 30 is captured is "detectable object present," it is considered that the learning captured images 42 in which the detection target 30 is captured (associated with "detectable object present") are important among the learning captured images 42 included in the learning captured image group 40 used when generating the test model. Therefore, although various expected variations of the learning captured images 42 are prepared, there is a possibility that the variations are insufficient, such as situations that cannot be anticipated at the time of generating the test model. Since the AI ​​model 10 performs processing based on the feature amount of the image, the cause of the judgment error and the remedy are estimated by comparing the learning captured images 42 and the mis-image 22 from the viewpoint of the features.

[0045] 7 shows an example of a list of image features to be analyzed in the second inspection. Examples of features to be analyzed include "mean, variance, skewness, kurtosis" which are RGB (color) statistics and HSV (color) statistics, "width, height, area" and "fractal dimension" of the image portion of the detection target. In this feature list, combinations of n types of mutually related features are listed, and for each combination, the features of the error image 22 and the learning captured image group 40 are compared.

[0046] FIG. 8 is a diagram showing an example of the feature distribution of the learning captured image group 40 and the features of the error image 22. In FIG. 8, the features to be compared are two types, the width (number of pixels in the horizontal direction) and the height (number of pixels in the vertical direction) of the image portion of the detection object 30 in the image in which the detection object 30 is captured, and the results of the feature analysis process are plotted in a two-dimensional feature space with the horizontal axis representing the width and the vertical axis representing the height. The results of the feature analysis process for each of the learning captured images 42 are plotted with extremely small gray circles, and the set of these is shown in the upper diagram of FIG. 8 as the feature distribution of the learning captured image group 40. In addition, in order to easily show the results of the feature analysis process for the error image 22, the results are plotted with large black squares in this feature distribution.

[0047] When the feature distribution of the learning image group 40 is examined, it can be seen that the features (plots) of each learning image 42 are distributed in a range where the width is half or less of the height, although they are generally dispersed in the height direction. In contrast, the feature (plot) of the incorrect image 22 is outside the range where the feature distribution is concentrated, and is plotted at a relatively large position where the width is more than twice as large as the height. In other words, it is determined that the feature of the incorrect image 22 is insufficient in the feature distribution of the learning image group 40. As a result, there is a shortage of learning images 42 having a feature such as "the size of the image part of the detection object 30 in the width direction is more than twice as large as the size in the height direction" that is captured in the incorrect image 22, and it is presumed that this shortage is the cause of the judgment error of the test model. More specifically, it is presumed that there is a shortage of learning images 42 that have a feature such as "the size of the image part of the detection object 30 in the width direction is more than twice as large as the size in the height direction" captured in the incorrect image 22.

[0048] Fig. 9 is a flow chart for explaining the detailed process flow of the second inspection. First, two or more features are selected as a feature combination from among features prepared in advance, such as the feature list shown in Fig. 7 (step S41). A feature space is set with each selected feature as an axis. Next, feature analysis processing is performed on the error image 22 for each of the selected combinations of features to calculate feature amounts (step S43). Similarly, feature analysis processing is performed on each of the learning images 42 in the learning image group 40 for each of the combinations of features selected in step S41 to calculate feature amounts, and a feature distribution of the learning image group 40 is calculated (step S45).

[0049] Then, in the feature space, the feature distribution of the learning captured images 40 is compared with the features (feature amount) of the error image 22, and it is determined whether the feature amount of the error image 22 is insufficient in the feature distribution (step S47). This determination can be made, for example, by determining whether the distance (Mahalanobis distance, etc.) between the feature distribution of the learning captured images 40 in the feature space and the feature amount of the error image 22 is equal to or smaller than a predetermined threshold value.

[0050] If the feature amount of the error image 22 is sufficient (step S49: NO) and there is an unselected feature combination (step S57: YES), the process returns to step S41, an unselected feature combination is selected, and the same process is repeated. If there is no unselected feature combination (step S57: NO), the second inspection does not reveal the cause of the judgment error of the test model, that is, the cause of the judgment error is not in the learning captured image group 40 (step S59).

[0051] On the other hand, if the feature amount of the error image 22 is insufficient (step S49: YES), it is determined that the feature amount of the error image 22 is insufficient in the learning captured image group 40 (step S51). Then, from among the learning captured images 42 in the learning captured image group 40, a learning captured image 42 having a feature amount similar to the feature amount (which can also be said to be a plot position in the feature space) of the error image 22 in the feature space based on the feature selected in step S41 is selected (step S53). Next, a message is presented as a remedy, which indicates that the cause of the judgment error of the test model is in the learning captured image group 40, as well as the feature or feature amount (plot position in the feature space) determined to be insufficient, and that images such as the learning captured image 42 selected in step S53 (the learning captured image 42 having the insufficient feature amount) are insufficient as a learning amount (step S53). The second inspection is performed in this manner.

[0052] The third inspection is an inspection assuming that the cause lies in the AI ​​model 10, which is the model under test. Specifically, a learning process is performed on each of the multiple types of incomplete AI models using the learning captured image group 40 to generate a type-specific AI model, and the error image 22 is input to each of the type-specific AI models to inspect whether the judgment is successful or not. The multiple types of incomplete AI models are models with different types of machine learning models, and are AI models with different types of machine learning models, for example, AI models with different algorithms, structures, number of layers, number of nodes, and number of parameters related to the machine learning models.

[0053] 10 is a flow chart for explaining the detailed process flow of the third inspection. First, one is selected from a plurality of types of incomplete AI models prepared in advance, such as an incomplete AI list (step S61). Next, a learning process is performed on the selected incomplete AI model using the learning captured image group 40 to generate a learned AI model (type-specific AI model) (step S63). Next, the error image 22 is input to the generated type-specific AI model, and it is determined whether the judgment is successful, that is, whether the judgment result is "detection target presence."

[0054] If the judgment is not successful (judged as "no object to be detected") (step S67: NO) and there is an unselected incomplete AI model (step S71: YES), return to step S61, select an unselected incomplete AI model, and repeat the same process. If there is no unselected incomplete AI model (step S71: NO), it is presented that the cause of the judgment error of the test model cannot be clarified by the third inspection, that is, the cause of the judgment error is not in the test model, AI model 10 (step S73).

[0055] On the other hand, if the judgment is successful (step S67: YES), a message is displayed indicating that the cause of the judgment error of the test model lies in the AI ​​model being tested, and information indicating the type of AI model is presented as a remedial measure (step S69). The third test is performed in this manner.

[0056] Fig. 11 shows an example of the functional configuration of the cause elucidation support device 1. According to Fig. 11, the cause elucidation support device 1 is configured to include an operation unit 102, a display unit 104, a communication unit 106, a processing unit 200, and a storage unit 300, and is realized as a kind of computer system. The cause elucidation support device 1 may be realized by one computer, or may be configured by connecting a plurality of computers. The cause elucidation support device 1 may be constructed as a local system, or some or all of the functions may be constructed and realized on a network server via the Internet.

[0057] The operation unit 102 is realized by input devices such as a keyboard, a mouse, a touch panel, and various switches, and outputs operation signals corresponding to operations performed to the processing unit 200. The display unit 104 is realized by a display device such as a liquid crystal display or a touch panel, and performs various displays based on display signals from the processing unit 200. The communication unit 106 is a communication device realized by, for example, a wireless communication module, a router, a modem, a jack for a wired communication cable, a control circuit, and the like, and connects to a given communication network to perform data communication with an external device.

[0058] The processing unit 200 is a processor realized by an arithmetic device or arithmetic circuit such as a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array), and performs overall control of the cause elucidation support device 1 based on programs and data stored in the memory unit 300, input data from the operation unit 102 and the communication unit 106, etc.

[0059] The processing unit 200 executes a cause elucidation support process (see FIG. 4) that supports elucidation of the cause of a judgment error by an AI model that judges whether a detection target is captured in a captured image by following a cause elucidation support program 302. In this cause elucidation support process, a first inspection is performed by the first inspection unit 202, and it is determined whether to continue or end the inspection based on the inspection result of the first inspection. If the inspection is to be continued, a second inspection is performed by the second inspection unit 206, and it is determined whether to continue or end the inspection based on the inspection result of the second inspection. If the inspection is to be continued, the third inspection is performed by the third inspection unit 210. The AI ​​model 10, which is a subject model in this process, is provided to the cause elucidation support device 1 as subject model data 310, the error image 22 as error image data 314, and the learning captured image group 40 used to generate the AI ​​model 10 as learning captured image group data 312.

[0060] Moreover, the processing unit 200 has, as functional processing blocks, a first inspection unit 202, a first presentation unit 204, a second inspection unit 206, a second presentation unit 208, a third inspection unit 210, and a third presentation unit 212. Each of these functional units of the processing unit 200 can be realized in software by the processing unit 200 executing a program, or can be realized by a dedicated arithmetic circuit. In this embodiment, the former method of realizing the functional units in software will be described.

[0061] The first inspection unit 202 generates an image captured with changed camera settings based on an error image, which is an image captured in which the test model, which is the AI ​​model being inspected, makes a mistake in making a judgment, and inputs the image captured with changed settings to the test model to perform a first inspection to see whether the judgment is successful.

[0062] Specifically, for each of a plurality of camera setting changes prepared in advance using the camera setting list 320 (see FIG. 5) or the like, a setting-changed photographed image is generated when the camera setting is changed and photographed. The setting-changed photographed image is generated by performing image processing corresponding to the camera setting on the error image 22. The generated setting-changed photographed image is then input to the test model to search for camera setting changes that result in a successful judgment (judgment that "detected object is present").

[0063] The first presentation unit 204 presents, as an improvement item, the camera settings related to the setting-changed captured image for which the first inspection was successful.

[0064] Specifically, when there is a change in the camera settings that results in a successful judgment of the test model (judged that "the object to be detected is present") by the first inspection by the first inspection unit 202, the image conditions corresponding to the change in the camera settings and the change in the camera settings are presented as improvement items, along with a message that the cause of the incorrect judgment of the test model is the camera settings. The presentation can be performed, for example, by displaying on the display unit 104.

[0065] The second inspection unit 206 performs a second inspection by evaluating the lack of the learning-purpose captured images based on the learning-purpose captured image group and the error images used when generating the test model. As the second inspection, for example, a feature analysis process for analyzing the features of the captured images is used to analyze the features of each learning-purpose captured image included in the learning-purpose captured image group, and a feature distribution of the learning-purpose captured image group is calculated. The feature analysis process is also used to analyze the features of the error images. Then, it is determined whether the features of the error images in the feature distribution of the learning-purpose captured image group are lacking.

[0066] Specifically, two or more features are selected from features prepared in advance using the feature list 322 (see FIG. 7) or the like to create a feature combination. A feature space is set with each selected feature as an axis. A feature analysis process is performed for each selected feature to calculate the feature amount of the miss image 22. Similarly, the feature amount of each learning image 42 in the learning image group 40 is calculated. Then, the calculated feature amount of each learning image 42 is plotted in the feature space to calculate the feature distribution of the learning image group 40. Next, the feature amount of the miss image 22 is plotted in the same feature space to determine whether the feature distribution of the learning image group 40 lacks features of the miss image 22. The determination of whether there is a lack can be made, for example, by determining whether the distance (Mahalanobis distance, etc.) between the feature distribution of the learning image group 40 and the feature amount of the miss image 22 in the feature space is equal to or less than a predetermined threshold.

[0067] The second presentation unit 208 selects a learning image that satisfies a predetermined similar feature condition based on the missing feature determined by the second inspection from the learning image group, and presents the selected learning image and information indicating the missing feature. Then, the second presentation unit 208 presents, as a remedial measure, a message that the cause of the misjudgment of the test model is the learning image group 40, the features and feature amounts determined to be missing, and a message that an image such as the selected learning image 42 is missing in terms of the learning amount. The presentation can be performed, for example, by displaying on the display unit 104.

[0068] The third inspection unit 210 performs a learning process using a group of captured learning images for each of the multiple types of incomplete AI models to generate type-specific AI models, and inputs an error image into each of the type-specific AI models to perform a third inspection to check whether the judgment is successful. The multiple types of incomplete AI models are, for example, AI models that differ in at least one of the following related to machine learning models: 1) algorithm, 2) structure, 3) number of layers, 4) number of nodes, and 5) number of parameters.

[0069] Specifically, one of a plurality of types of incomplete AI models prepared in advance by the incomplete AI model list 324 or the like is selected, and a learning process is performed on the selected incomplete AI model using the learning captured image group 40 to generate a trained AI model (type-specific AI model). Then, the error image 22 is input to the generated type-specific AI model, and a type-specific AI model that results in a successful judgment (judged as "detection target presence") is searched for.

[0070] The third presentation unit 212 presents information on the type-specific AI model that has been successfully judged in the third inspection. Specifically, when there is a type-specific AI model that has been successfully judged (judged that "detection target object is present") in the third inspection by the third inspection unit 210, information indicating the type of the type-specific AI model (unfinished AI model) is presented as a remedial measure, along with a message to the effect that the cause of the judgment error of the test model is the AI ​​model 10 that is the test model. The presentation can be performed, for example, by displaying on the display unit 104.

[0071] The memory unit 300 is realized by an IC (Integrated Circuit) memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory) or a storage device such as a hard disk, and stores programs, data, etc. that the processing unit 200 uses to comprehensively control the cause elucidation support device 1. The memory unit 300 is also used as a working area for the processing unit 200, and temporarily stores the results of calculations performed by the processing unit 200 and input data from the operation unit 102 and communication unit 106, etc.

[0072] In this embodiment, the memory unit 300 stores a cause elucidation support program 302, test model data 310, learning image group data 312, error image data 314, a camera setting list 320, a feature list 322, and an incomplete AI model list 324.

[0073] [Effects] In this way, according to the present embodiment, a technique useful for supporting the elucidation of the cause of the judgment error of the AI ​​model 10 that judges whether or not a detection target object is captured in a captured image can be realized. In other words, as an inspection to identify the possible cause of the judgment error of the AI ​​model 10, which is the subject model, three inspections can be performed: a first inspection to search for a camera setting that will result in a successful judgment, assuming that the cause is in the camera setting; a second inspection to evaluate a shortage of the learning captured images 42, assuming that the cause is in the learning captured image group 40 used to generate the AI ​​model 10; and a third inspection to search for a type of AI model that will result in a successful judgment, assuming that the cause is in the AI ​​model 10 itself. The cost of dealing with the cause is lowest in the order of the first inspection, the second inspection, and the third inspection. For this reason, the inspections are performed in the order of the first inspection, the second inspection, and the third inspection.

[0074] Incidentally, the applicable embodiments of the present invention are not limited to the above-described embodiments, and can of course be modified as appropriate without departing from the spirit of the present invention.

[0075] For example, in the above-mentioned embodiment, an example of an AI model used for monitoring the front of a train was given. However, the AI ​​model can also be applied to an AI model used in a monitoring system that uses a camera to photograph and monitor a target position on a track (e.g., a track in a station) or a road (e.g., a highway) where no suspicious object should exist, and judges whether a suspicious object is captured in the captured image as a detection target. [Explanation of symbols]

[0076] 1...Cause elucidation support device 200... Processing section 202…First Inspection Department 204…First presentation part 206…Second Inspection Department 208…Second presentation part 210…Third Inspection Department 212…Third presentation part 300...Storage section 302…Cause Identification Support Program 310…Test model data 312…Image data for learning purposes 314... Miss image data 320...Camera setting list 322…Feature List 324…Unfinished AI model list 10…AI model (pre-trained) 12…AI model (untrained) 20...Photo 22... Miss Image 30...Detection target 40...Images taken for study purposes 42...Learn how to take pictures

Claims

1. A cause elucidation support device that supports elucidation of the cause of a judgment error made by an AI (Artificial Intelligence) model that judges whether or not a detection target is captured in a captured image, A first inspection means for generating a setting-changed photographed image when photographed by changing a camera setting based on a mistake image, which is the photographed image in which a test model, which is the AI ​​model to be inspected, makes a judgment error, and inputs the setting-changed photographed image to the test model to perform a first inspection as to whether the judgment is successful or not; a second inspection means for performing a second inspection by evaluating a deficiency of the learning images based on the learning images used when generating the test model and the error image; a third inspection means for performing a learning process using the group of learning captured images for each of a plurality of types of incomplete AI models to generate type-specific AI models, and inputting the error image into each of the type-specific AI models to perform a third inspection to determine whether or not the judgment is successful; A cause identification support device equipped with

2. 2. The cause elucidation support device according to claim 1, further comprising: a determination as to whether to continue or terminate the inspection based on a result of the first inspection; if the inspection is to be continued, the second inspection is performed by the second inspection means; and a determination as to whether to continue or terminate the inspection based on a result of the second inspection; if the inspection is to be continued, the third inspection is performed by the third inspection means.

3. a first presenting means for presenting, as an improvement item, the camera settings related to the setting-changed photographed image for which the first inspection has been successfully performed; The cause elucidation support device according to claim 1 or 2, further comprising:

4. The second inspection means is analyzing features of each of the learning images included in the learning image group using a feature analysis process that analyzes features of the captured images, and calculating a feature distribution of the learning image group; analyzing the characteristics of the miss image using the characteristic analysis process; comparing the feature distribution with features of the mismatched images to determine features that are lacking in the learning captured images; is included in the second inspection, The cause elucidation support device according to claim 1.

5. The feature analysis process is a process of analyzing the features of an image portion of the detection object in the captured image in which the detection object is captured, including the features of the image portion as features of an analysis target. The cause elucidation support device according to claim 4.

6. a second presentation means for selecting a learning image that satisfies a predetermined similar feature condition based on the missing feature determined by the second inspection from the group of learning images, and presenting the selected learning image and information indicating the missing feature; The cause elucidation support device according to claim 4 or 5, further comprising:

7. The plurality of types of unfinished AI models are different in at least one of the following machine learning models: 1) algorithm; 2) structure; 3) number of layers; 4) number of nodes; and 5) number of parameters.

3. The cause elucidation support device according to claim 1 or 2.

8. a third presentation means for presenting information on the type-specific AI model that has been judged to be successful in the third test; The cause elucidation support device according to claim 1 or 2, further comprising:

9. The captured image is an image of a track on which a vehicle runs, The detection object is an obstacle on the trajectory.

3. The cause elucidation support device according to claim 1 or 2.

10. A cause elucidation support method for a computer system to support elucidation of a cause of a judgment error made by an AI (Artificial Intelligence) model that judges whether or not a detection target is included in a captured image, comprising: A setting-changed photographed image is generated based on the mistake image, which is the photographed image in which the test model, which is the AI ​​model to be inspected, makes a judgment error, and inputs the setting-changed photographed image into the test model to perform a first inspection to determine whether the judgment is successful or not. performing a second inspection by evaluating a deficiency of the learning images based on the learning images used in generating the test model and the error image; performing a learning process using the group of learning images for each of a plurality of types of incomplete AI models to generate type-specific AI models, and inputting the error image into each of the type-specific AI models to perform a third inspection to determine whether or not the judgment is successful; A method to assist in identifying the cause.

Images