Device and method for supporting the determination of causes
The cause analysis support device for AI models addresses the challenge of explaining judgment errors by sequentially inspecting camera settings, training data, and AI model types, offering cost-effective error prevention in applications like train monitoring.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- RAILWAY TECHNICAL RESEARCH INSTITUTE
- Filing Date
- 2023-10-26
- Publication Date
- 2026-05-11
AI Technical Summary
Existing technologies lack a universal method to explain the causes of judgment errors in AI models, particularly in applications like train forward monitoring systems where AI models determine the presence of objects in images, as the judgment process is a black box and specific solutions like similar image searches are limited in applicability.
A cause analysis support device that performs three inspections: first, assuming the cause lies in camera settings; second, in the training image set; and third, in the AI model itself, using modified images, feature analysis, and training different AI models to identify the root cause of errors.
Efficiently identifies the causes of judgment errors by sequentially performing inspections with decreasing cost, providing actionable improvements for camera settings, training data, and AI model types, thereby enhancing error prevention.
Smart Images
Figure 0007856620000001 
Figure 0007856620000002 
Figure 0007856620000003
Abstract
Description
Technical Field
[0001] The present invention relates to a cause explanation support device and the like for assisting in explaining the causes of judgment errors in AI models.
Background Art
[0002] Generally, on the side of using an AI (Artificial Intelligence) model, the judgment process of the AI model is a black box, and it is difficult to grasp how it goes through what kind of processing and what kind of judgment process. Here, the AI model means including a machine learning model, and includes pre-processing for data input to the machine learning model and the machine learning model. Therefore, in a more strict and narrow sense, the AI model can be called a machine learning model. Therefore, although the AI model and the machine learning model can be said to be equivalent in the present invention, in this specification, it will be described as an AI model in a broad sense.
[0003] On the side of using the AI model, the judgment process of the AI model may be a black box. However, for businesses that manufacture and sell products incorporating the AI model into the system, or businesses that provide services incorporating the AI model into the system, since the AI model is used as part of the system, early cause explanation when the AI model makes a judgment error is desired. However, there is currently no technology that is universally effective for explaining the causes when the AI model makes a judgment error. This is because the usage forms of AI models vary depending on what kind of AI model is used and what kind of processing is performed in individual technical fields. For example, in Patent Document 1, a method is proposed to present which area of an image is used to estimate similarity when using the image as a search key in a search for a document including screen data for similar image search.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
[0005] However, the technology described in Patent Document 1 is a technology specifically tailored to the extremely limited condition of using an image as a search key for similar image searches in a document search that includes screen data. Moreover, it is a technology that determines the extremely limited cause of which region of the image was used to estimate similarity. It is difficult to apply this technology to other forms of use that utilize AI models.
[0006] Research and development of technologies utilizing AI models are also being conducted in the railway sector. For example, a train forward monitoring system is being developed that uses an AI model to detect obstacles on the track by determining whether or not a target object is visible in the image taken in front of the train. As mentioned above, since an AI model is used as part of the system, it is desirable in the railway sector as well to be able to determine the cause when the AI model makes a judgment error. However, there is currently no known technology for determining the cause when an AI model used for forward monitoring of trains makes a judgment error.
[0007] While I have given the example of a train forward monitoring system, the above challenges are the same in other examples as long as the AI model determines whether or not a target object is captured in the image. For example, one such system would involve using cameras to photograph and monitor locations on railway tracks or roads where suspicious objects should not normally be present, and then using an AI model to determine whether or not a suspicious object is captured in the image as a target object.
[0008] The problem that this invention aims to solve is to realize a technology that is useful in helping to clarify the causes of errors in judgment made by AI models that determine whether or not an object to be detected is present in a captured image. [Means for solving the problem]
[0009] The first invention for solving the above problem is: A cause analysis support device that assists in identifying the cause of errors in judgment made by an AI (Artificial Intelligence) model that determines whether or not a target object is present in a captured image, A first inspection means (for example, the first inspection unit 202 in Figure 11) generates a modified image taken with the camera settings changed based on the erroneous image, which is the erroneous image, taken when the AI model being inspected made a judgment error, and inputs the modified image into the model being inspected to perform a first inspection to determine whether the judgment is successful or not. A second inspection means (for example, the second inspection unit 206 in Figure 11) performs a second inspection by evaluating the deficiencies in training images based on the training image set used when generating the aforementioned test model and the aforementioned error images, A third inspection means (for example, the third inspection unit 210 in Figure 11) performs a learning process using the aforementioned training image set on each of several types of incomplete AI models to generate a type-specific AI model, and then inputs the aforementioned error image into each of the type-specific AI models to perform a third inspection to determine whether the judgment is successful or not. This is a cause-finding support device equipped with the following features.
[0010] Other inventions include, A method for assisting a computer system in identifying the cause of errors in judgment made by an AI (Artificial Intelligence) model that determines whether or not a target object is present in a captured image, Based on the erroneous image, which is the AI model being tested and in which the subject model made a judgment error, a modified image is generated by changing the camera settings and taking a picture. This modified image is then input into the subject model to perform a first test to determine whether the judgment is successful or not (for example, step S1 in Figure 4). A second test is performed by evaluating the deficiencies in training images based on the training image set used to generate the aforementioned test model and the aforementioned error images (for example, step S5 in Figure 4), The process involves performing a learning process on each of several types of incomplete AI models using the aforementioned training image set to generate a type-specific AI model, and then inputting the aforementioned incorrect image into each of these type-specific AI models to perform a third check to determine whether the judgment is successful or not (for example, step S9 in Figure 4), A method for supporting the identification of the cause may be constructed that includes this.
[0011] According to the first invention, it is possible to realize a technology useful for supporting the investigation of the causes of judgment errors in AI models that determine whether or not an object to be detected is captured in a captured image. Specifically, as an inspection to identify possible causes of judgment errors in the AI model under test, three inspections can be performed: a first inspection that assumes the cause lies in the camera settings and searches for camera settings that result in a successful judgment; a second inspection that assumes the cause lies in the group of training images used to generate the AI model and evaluates the deficiency of training images; and a third inspection that assumes the cause lies in the AI model itself and searches for the type of AI model that results in a successful judgment.
[0012] The second invention relates to the above invention, This is a cause identification support device that determines whether to continue or terminate the inspection based on the results of the first inspection, performs the second inspection by the second inspection means if it is to continue, determines whether to continue or terminate the inspection based on the results of the second inspection, and performs the third inspection by the third inspection means if it is to continue.
[0013] According to the second invention, by sequentially performing each test in the order of first test, second test, and third test, and proceeding to the next test if the cause cannot be identified, the causes of judgment errors in the AI model can be efficiently identified. In particular, the order of first test, second test, and third test is in an order in which the cost of addressing the identified cause is relatively low. For this reason, this order of tests is particularly effective.
[0014] The third invention is, in the above invention, A first presenting means (for example, the first presenting unit 204 in FIG. 11) that presents, as improvement items, camera settings related to the set-change captured image that has been successfully determined in the first inspection. It is a cause-explanation support device further comprising
[0015] According to a third invention, when it is specified by the first inspection that the cause of the determination error of the test model lies in the camera settings, specific camera settings can be presented as improvement items as countermeasures for preventing recurrence of the determination error.
[0016] A fourth invention is the above-described invention, in which the second inspection means uses a feature analysis process for analyzing the features of the captured image to analyze the features of each learning captured image included in the learning captured image group, and calculates the feature distribution of the learning captured image group, uses the feature analysis process to analyze the features of the error image, compares the feature distribution with the features of the error image to determine the features lacking in the learning captured image group, and executes the above as included in the second inspection. It is a cause-explanation support device.
[0017] According to a fourth invention, as a second inspection, by comparing the feature distribution of the learning captured image group analyzed using the feature analysis process with the features of the error image, it is possible to determine and evaluate the features of the image lacking as a learning captured image from the perspective of the features of the image.
[0018] A fifth invention is the above-described invention, in which the feature analysis process is a process of analyzing, including, as features to be analyzed, the features of the image portion of the detection target object in the captured image in which the detection target object is captured. It is a cause-explanation support device.
[0019] According to a fifth invention, the determination and evaluation of the features of the image lacking as a learning captured image can be made more significant by focusing on the detection target object shown in the captured image.
[0020] The sixth invention is the invention described above, selecting, from the group of learning captured images, a learning captured image that satisfies a predetermined similar feature condition based on the lacking feature determined by the second inspection, and presenting the selected learning captured image and information indicating the lacking feature, a second presenting means (for example, the second presenting unit 208 in FIG. 11), is a cause explanation support device further comprising.
[0021] According to the sixth invention, when it is specified by the second inspection that the cause of the judgment error of the inspected model lies in the shortage of the group of learning captured images, the learning captured image in the group of learning captured images similar to the image having the feature can be presented together with the feature of the lacking image. Thereby, as a countermeasure, a clue to the learning captured image to be added can be given.
[0022] The seventh invention is the invention described above, at least one of 1) algorithm, 2) structure, 3) number of layers, 4) number of nodes, 5) number of parameters related to the machine learning model is different among the plurality of types of unfinished AI models, is a cause explanation support device.
[0023] According to the seventh invention, in the third inspection, a plurality of types of unfinished AI models in which the algorithm, structure, number of layers, number of nodes, and number of parameters related to the machine learning model are different can be used.
[0024] The eighth invention is the invention described above, a third presenting means (for example, the third presenting unit 212 in FIG. 11) for presenting information on the type-specific AI model that has been determined to be successful in the third inspection, [[ID=3
[0026] The seventh invention is, in the above invention, The aforementioned captured image is an image of the track on which the vehicle is traveling. The object to be detected is an obstacle on the trajectory. The cause identification support device according to claim 1 or 2.
[0027] According to the seventh invention, a method suitable for investigating the causes of judgment errors in AI models used for forward train monitoring to detect obstacles on the track is realized. [Brief explanation of the drawing]
[0028] [Figure 1] Diagram illustrating the AI model to be tested. [Figure 2] A diagram illustrating the generation of an AI model. [Figure 3] Diagram illustrating errors in AI model judgment. [Figure 4] Flowchart for the process of supporting the investigation of the cause. [Figure 5] An example of changes made to camera settings during the first inspection. [Figure 6] Flowchart for the processing of the first inspection. [Figure 7] An example of image features to be analyzed in the second examination. [Figure 8] A comparison of the feature distribution of training images and the features of mis-captured images. [Figure 9] Flowchart for processing the second inspection. [Figure 10] Flowchart for processing the third inspection. [Figure 11] Example of a functional configuration for a device that supports root cause analysis. [Modes for carrying out the invention]
[0029] Preferred embodiments of the present invention will be described below with reference to the drawings. However, the applicable forms of the present invention are not limited to the following embodiments. Furthermore, in the drawings, the same elements are denoted by the same reference numerals.
[0030] The cause identification support device 1 of this embodiment is a device for supporting the identification of the causes of judgment errors made by AI models.
[0031] Figure 1 shows a diagram illustrating the AI model that serves as the test subject for the cause investigation support device 1. The AI model 10, which serves as the test subject, is used for forward train monitoring in railways and determines and outputs whether or not a detection target object 30 is visible in the input captured image 20 (presence or absence of detection target object 30). The captured image 20 is an image of the track in front of the railway vehicle taken by a camera mounted on the railway vehicle. The detection target object 30 is an obstacle that affects the operation of the railway vehicle, such as people, animals, cars, bicycles, etc., on or near the track. In Figure 1, the captured image 20a on the left shows a "maintenance worker," which is the detection target object 30, next to the tracks, and the AI model 10 determines that "detection target object present." The captured image 20b on the right does not show the detection target object 30, and the AI model 10 determines that "detection target object absent."
[0032] As shown in Figure 2 as an example, AI model 10 is a trained AI model generated by performing a training process on an untrained AI model 12 using a set of training images 40, which is a collection of training images 42 that associate (associate) the presence or absence of the object to be detected 30 with the captured images. The set of training images 40 can also be described as training data, where the data input to the AI model is the training images 42, and the data output by the AI model is the presence or absence of the object to be detected 30, allowing the AI model to learn.
[0033] Furthermore, a judgment error by AI model 10 refers to a case where, as shown in Figure 3 as an example, it judged that there was "no object to detect" in the captured image 20 (22) which contains the object to be detected 30.
[0034] The cause identification support device 1 assists in identifying the cause of a judgment error in the AI model 10, which is the model under test. Based on the error image 22, which is the captured image 20 in which the AI model 10 made a judgment error, and the training image group 40 used for the learning process when the AI model 10 was generated, the device estimates the cause of the judgment error and suggests improvement items that can prevent the recurrence of the judgment error.
[0035] Figure 4 is a flowchart illustrating the cause identification support process performed by the cause identification support device 1. As shown in Figure 4, the cause identification support device 1 first performs a first inspection (step S1). The first inspection, as will be described in detail later, is an inspection that assumes the cause lies in the camera settings. If the cause of the judgment error is not identified as a result of the first inspection (step S3: NO), then a second inspection is performed (step S5). The second inspection, as will be described in detail later, is an inspection that assumes the cause lies in the training image group 40. If the cause of the judgment error is not identified as a result of the second inspection (step S7: NO), then a third inspection is performed (step S9). The third inspection, as will be described in detail later, is an inspection that assumes the cause lies in the AI model 10, which is the model under test. If the cause of the judgment error is not identified as a result of the third inspection, it is indicated that the cause is unknown (step S13). The cause identification support process is performed as described above.
[0036] Each test will be explained in detail. The first test assumes that the cause lies in the camera settings. Specifically, based on the faulty image 22, a modified image is generated when the camera settings are changed and a new image is taken. This modified image is then input into the test model to check whether the judgment is successful or not.
[0037] Even under the same circumstances, the resulting images can differ depending on the camera settings used for shooting. Therefore, various images that could be captured when the camera settings are changed (images captured with changed settings) are generated from the error image 22 and input into the test model. By searching for the camera settings that result in a successful judgment ("object detected"), the cause of the judgment error and countermeasures for improvement are estimated. Images captured with changed settings, which would occur when shooting with changed camera settings, are simulated by performing image processing on the error image 22.
[0038] Figure 5 shows an example of a list of camera settings to be changed in the first test. In this camera setting list, each camera setting change that is an improvement item is associated with an image processing method that simulates the change in the camera setting and the image conditions that are changed by that camera setting. The image conditions are the state of the captured image that is thought to have caused the judgment error of the model under test, such as image blur or out-of-focus, contrast, white balance, resolution, and aspect ratio, and the camera setting changes that correct or change the state of the captured image are defined in association with these conditions.
[0039] Figure 6 is a flowchart illustrating the detailed processing flow of the first inspection. First, one of several pre-prepared camera setting changes is selected from the camera setting list, which is an example shown in Figure 5 (step S21). Note that the camera setting changes may consist of multiple items, each with specific stepwise values, such as "1 / 125" or "1 / 250" for "shutter speed".
[0040] Next, image processing corresponding to the changes in the selected camera settings is performed on the misimage 22 to generate a modified image taken with the camera settings changed (step S23). Subsequently, the generated modified image is input to the test model to determine whether the judgment was successful, that is, whether the judgment result is "object detected".
[0041] If the judgment is not successful (the judgment result is "no object detected") (Step S27: NO), and there are unselected camera setting changes (Step S31: YES), return to Step S21, select the unselected camera setting changes, and repeat the same process. If there are no unselected camera setting changes (Step S33: NO), indicate that the cause of the judgment error in the model under test could not be determined by the first test, that is, the cause of the judgment error is not in the camera settings (Step S33).
[0042] On the other hand, if the judgment is successful (Step S27: YES), the system will state that the cause of the judgment error in the model under test lies in the camera settings, and will present the changes to the selected camera settings (i.e., the countermeasures for the cause) and the corresponding image conditions as items for improvement (Step S29). The first test is conducted in this manner.
[0043] The second test assumes that the cause lies in the training image set 40 used to generate the AI model 10. Specifically, it evaluates whether the training images 42 used for training were sufficient, based on the training image set 40 used when generating the model under test and the error image 22. In detail, a feature analysis process is used to analyze the features of each training image 42 included in the training image set 40, calculate the feature distribution of the training image set 40, and analyze the features of the error image 22. Then, the feature distribution of the training image set 40 and the features of the error image 22 are compared to determine whether the features of the error image 22 are features that are lacking in the training image set 40. The feature analysis process is a process that includes the features of the image portion of the object 30 in the image 20 in which the object 30 is captured as features to be analyzed.
[0044] For the AI model 10, which is the model under test, to correctly determine that an image 20 containing an object to be detected 30 has been detected, it is considered important that the training images 42 included in the training image set 40 used when generating the model under test contain the object to be detected 30 (i.e., are associated with "object to be detected"). Therefore, various variations of training images 42 are prepared, but there is a possibility that the variations are insufficient due to situations that could not be anticipated when generating the model under test. Since the AI model 10 performs processing based on the features of the images, it estimates the cause of the judgment error and countermeasures by comparing the training images 42 and the incorrect images 22 from the perspective of features.
[0045] Figure 7 shows an example of a list of image features to be analyzed in the second inspection. Features to be analyzed include, for example, RGB (color) statistics such as "mean, variance, skewness, kurtosis," HSV (color) statistics, "width, height, area" of the image portion of the object to be detected, and "fractal dimension." In this feature list, combinations of n types of related features are enumerated, and for each combination, the features of the misimage 22 and the training image group 40 are compared.
[0046] Figure 8 shows an example of the feature distribution of the training image set 40 and the features of the error image 22. In Figure 8, the features to be compared are two types: the width (number of pixels horizontally) and the height (number of pixels vertically) of the image portion of the detection target 30 in the image in which the detection target 30 is captured. The results of the feature analysis processing are shown as a plot in a two-dimensional feature space with the horizontal axis representing width and the vertical axis representing height. The results of the feature analysis processing for each of the training image sets 42 are plotted as tiny gray dots, and the collection of these dots is shown as the feature distribution of the training image set 40 in the upper part of Figure 8. In addition, for clarity, the results of the feature analysis processing of the error image 22 are plotted as larger black rectangles within this feature distribution.
[0047] Looking at the feature distribution of the training image set 40, we can see that while the features are generally dispersed in the height direction, the features (plots) of each training image 42 are concentrated in a range where the width is less than half the height. In contrast, the features (plots) of the wrong image 22 are outside this concentrated range of feature distribution and are plotted at a relatively large position where the width is more than twice the height. In other words, the features of the wrong image 22 are judged to be lacking in the feature distribution of the training image set 40. As a result, there are lacking training images 42 that have the feature of "the width of the image portion of the object to be detected 30 being more than twice as large as its height," and this lack is presumed to be the cause of the judgment error of the tested model. To explain more specifically, it is presumed that there are lacking training images 42 that are elongated horizontally, such as when the object to be detected 30 is lying down.
[0048] Figure 9 is a flowchart illustrating the detailed processing flow of the second inspection. First, two or more features are selected from pre-prepared features, such as the feature list shown as an example in Figure 7, to form a feature combination (step S41). A feature space is set with each selected feature as an axis. Next, feature analysis processing is performed on each of the features in the selected combination for the mis-image 22 to calculate the feature quantity (step S43). Similarly, feature analysis processing is performed on each of the features in the combination selected in step S41 for each of the training images 42 in the training image group 40 to calculate the feature quantity and calculate the feature distribution of the training image group 40 (step S45).
[0049] Then, in the feature space, the feature distribution of the training image group 40 and the features (feature quantities) of the misimage 22 are compared, and it is determined whether the feature quantities of the misimage 22 are insufficient in the feature distribution (step S47). This determination can be made, for example, by determining whether the distance (Mahalanobis distance, etc.) between the feature distribution of the training image group 40 and the feature quantities of the misimage 22 in the feature space is below a predetermined threshold.
[0050] If the features of the mis-image 22 are not lacking (Step S49: NO) and there are unselected feature combinations (Step S57: YES), return to Step S41, select the unselected feature combinations, and repeat the same process. If there are no unselected feature combinations (Step S57: NO), indicate that the cause of the judgment error in the test model cannot be determined by the second test, that is, the cause of the judgment error is not in the training image group 40 (Step S59).
[0051] On the other hand, if the features of the misimage 22 are insufficient (step S49: YES), it is determined that the features of the misimage 22 are insufficient in the training image set 40 (step S51). Then, from among the training images 42 of the training image set 40, a training image 42 is selected that has features similar to the features of the misimage 22 in the feature space based on the features selected in step S41 (which can also be called the plot position in the feature space) (step S53). Next, along with the fact that the cause of the judgment error of the model under test lies in the training image set 40, the features and features determined to be insufficient (the plot position in the feature space) and images like the training image 42 selected in step S53 (the training image 42 that possesses the missing features) are presented as improvement measures (step S53). The second test is performed in this manner.
[0052] The third test assumes that the cause lies in the AI model 10, which is the model under test. Specifically, a learning process using the training image set 40 is performed on each of several types of incomplete AI models to generate different types of AI models, and then the incorrect image 22 is input to each of these types of AI models to check whether the judgment is successful or not. The several types of incomplete AI models are models of different types of machine learning models, for example, AI models that differ in at least one of the following aspects related to the machine learning model: algorithm, structure, number of layers, number of nodes, number of parameters.
[0053] Figure 10 is a flowchart illustrating the detailed processing flow of the third inspection. First, one is selected from several types of pre-prepared incomplete AI models, such as an incomplete AI list (step S61). Next, the selected incomplete AI model is subjected to a training process using the training image group 40 to generate a trained AI model (type-specific AI model) (step S63). Subsequently, the generated type-specific AI model is input with the error image 22, and it is determined whether the judgment was successful, that is, whether the judgment result is "object detected".
[0054] If the judgment is not successful ("No object detected" is determined) (Step S67: NO), and there is an unselected, incomplete AI model (Step S71: YES), the process returns to Step S61, an unselected, incomplete AI model is selected, and the same process is repeated. If there is no unselected, incomplete AI model (Step S71: NO), the third test indicates that the cause of the judgment error in the model under test cannot be determined, that is, the cause of the judgment error is not in the model under test, AI model 10 (Step S73).
[0055] On the other hand, if the judgment is successful (Step S67: YES), the third test is conducted in the following manner: it is stated that the cause of the judgment error in the model under test lies with the AI model under test, and information indicating the type of AI model is presented as a corrective measure (Step S69).
[0056] Figure 11 shows an example of the functional configuration of the cause analysis support device 1. According to Figure 11, the cause analysis support device 1 is configured with an operation unit 102, a display unit 104, a communication unit 106, a processing unit 200, and a storage unit 300, and is implemented as a type of computer system. The cause analysis support device 1 may be implemented with a single computer, or it may be configured by connecting multiple computers. Furthermore, the cause analysis support device 1 may be built as a local system, or some or all of its functions may be built and implemented on a network server via the Internet.
[0057] The operation unit 102 is implemented by an input device such as a keyboard, mouse, touch panel, or various switches, and outputs an operation signal to the processing unit 200 according to the operation performed. The display unit 104 is implemented by a display device such as a liquid crystal display or touch panel, and displays various information based on the display signal from the processing unit 200. The communication unit 106 is a communication device implemented by a wireless communication module, router, modem, jack or control circuit for wired communication cables, etc., and connects to a given communication network to perform data communication with external devices.
[0058] The processing unit 200 is a processor implemented using arithmetic devices and circuits such as a CPU (Central Processing Unit) or FPGA (Field Programmable Gate Array), and it performs overall control of the cause analysis support device 1 based on programs and data stored in the memory unit 300, input data from the operation unit 102 and the communication unit 106, etc.
[0059] The processing unit 200, following the cause identification support program 302, executes a cause identification support process (see Figure 4) to assist in identifying the cause of a judgment error by the AI model that determines whether or not a target object is captured in the captured image. In this cause identification support process, the first inspection unit 202 performs a first inspection, and based on the results of the first inspection, it determines whether to continue or terminate the inspection. If it decides to continue, the second inspection unit 206 performs a second inspection, and based on the results of the second inspection, it determines whether to continue or terminate the inspection. If it decides to continue, the third inspection unit 210 performs the third inspection. The AI model 10, which is the model under test in this process, is provided to the cause identification support device 1 as the model under test data 310, the error image 22 as the error image data 314, and the group of training captured images 40 used to generate the AI model 10 as the group of training captured images data 312.
[0060] Furthermore, the processing unit 200 includes, as functional processing blocks, a first inspection unit 202, a first presentation unit 204, a second inspection unit 206, a second presentation unit 208, a third inspection unit 210, and a third presentation unit 212. Each of these functional units of the processing unit 200 can be implemented in software by the processing unit 200 executing a program, or by a dedicated arithmetic circuit. In this embodiment, the former, software implementation, will be described.
[0061] The first inspection unit 202 generates a modified image based on the error image, which is an image taken when the AI model under inspection made a judgment error, by changing the camera settings and inputting this modified image into the model under inspection to perform a first inspection to determine whether the judgment is successful or not.
[0062] Specifically, for each of the multiple camera setting changes pre-defined by the camera setting list 320 (see Figure 5), a setting change image is generated as if the camera setting had been changed and a picture had been taken. The setting change image is generated by applying image processing corresponding to the camera setting to the mis-image 22. Then, the generated setting change image is input into the test model to search for the camera setting change that results in a successful judgment ("object detected").
[0063] The first display unit 204 presents the camera settings related to the image captured with the settings changed, which was judged as a success in the first inspection, as items for improvement.
[0064] Specifically, if the first inspection by the first inspection unit 202 reveals changes in camera settings that result in a successful judgment by the model under test ("object detected"), the system will indicate that the cause of the model's judgment error lies in the camera settings, and will also present the corresponding image conditions and the changes in camera settings as improvement items. This presentation can be done, for example, by displaying it on the display unit 104.
[0065] The second inspection unit 206 performs a second inspection by evaluating the deficiencies in training images based on the training image set used to generate the test model and the error images. As part of the second inspection, for example, a feature analysis process is used to analyze the features of each training image included in the training image set, and the feature distribution of the training image set is calculated. The feature analysis process is also used to analyze the features of the error images. Finally, it is determined whether there are any deficiencies in the features distribution of the training image set for the error images.
[0066] Specifically, two or more features are selected from a pre-prepared set of features, such as the feature list 322 (see Figure 7), to form a combination of features. A feature space is set with each selected feature as an axis. Feature analysis processing is then performed on each selected feature to calculate the feature quantities of the misimage 22. Similarly, the feature quantities of each training image 42 in the training image group 40 are calculated. The feature distribution of the training image group 40 is then calculated by plotting the calculated feature quantities of each training image 42 in the feature space. Next, the feature quantities of the misimage 22 are plotted in the same feature space to determine whether the features of the misimage 22 are missing from the feature distribution of the training image group 40. This deficiency can be determined, for example, by checking whether the distance (Mahalanobis distance, etc.) between the feature distribution of the training image group 40 and the feature quantities of the misimage 22 in the feature space is below a predetermined threshold.
[0067] The second presentation unit 208 selects a training image from the training image group that satisfies predetermined similarity feature conditions based on the missing features determined by the second inspection, and presents the selected training image along with information indicating the missing features. It then presents, as a solution, that the cause of the error in the judgment of the model under test lies in the training image group 40, and that the features and feature quantities determined to be missing, as well as images like the selected training image 42, are insufficient as training data. This presentation can be done, for example, by displaying it on the display unit 104.
[0068] The third inspection unit 210 generates different types of AI models by performing a learning process using a set of training images for each of the multiple types of incomplete AI models, and then performs a third inspection to determine whether the judgment is successful or not by inputting a wrong image into each of the multiple types of incomplete AI models.
[0069] Specifically, one is selected from several types of incomplete AI models prepared in advance, such as the incomplete AI model list 324, and a training process is performed on the selected incomplete AI model using the training image set 40 to generate a trained AI model (type-specific AI model). Then, the generated type-specific AI model is input with the error image 22 and a type-specific AI model that makes a successful judgment (determines "object detected") is searched for.
[0070] The third display unit 212 presents information on the type of AI model that was deemed successful in the third inspection. Specifically, if the third inspection by the third inspection unit 210 results in a successful determination (determining "object detected") for a type of AI model, the third display unit 212 presents information indicating the type of the type of AI model (incomplete AI model) as a corrective measure, along with the fact that the cause of the judgment error in the model under inspection lies in the AI model 10 that is under inspection. This presentation can be done, for example, by displaying it on the display unit 104.
[0071] The memory unit 300 is implemented using an integrated circuit (IC) memory such as ROM (Read Only Memory) or RAM (Random Access Memory), or a storage device such as a hard disk. It stores programs and data for the processing unit 200 to comprehensively control the cause analysis support device 1, and is also used as a workspace for the processing unit 200. Calculation results performed by the processing unit 200, as well as input data from the operation unit 102 and the communication unit 106, are temporarily stored in the memory unit 300.
[0072] In this embodiment, the memory unit 300 stores a cause identification support program 302, test model data 310, training image data 312, misimage data 314, camera setting list 320, feature list 322, and incomplete AI model list 324.
[0073] [Effects and Effects] Thus, according to this embodiment, a technology useful for supporting the investigation of the causes of judgment errors in the AI model 10, which determines whether or not a target object is captured in a captured image, can be realized. In other words, as an inspection to identify possible causes of judgment errors in the AI model 10, which is the model under test, three inspections can be performed: a first inspection that assumes the cause lies in the camera settings and searches for camera settings that result in a successful judgment; a second inspection that assumes the cause lies in the training image group 40 used to generate the AI model 10 and evaluates the deficiencies in the training images 42; and a third inspection that assumes the cause lies in the AI model 10 itself and searches for the type of AI model that results in a successful judgment. The cost of addressing the cause decreases in the order of the first inspection, second inspection, and third inspection. For this reason, the inspections are performed in the order of the first inspection, second inspection, and third inspection.
[0074] It should be noted that the applicable embodiments of the present invention are not limited to those described above, and can be modified as appropriate without departing from the spirit of the invention.
[0075] For example, the above-described embodiment explained an example of an AI model used for monitoring the area ahead of a train. However, the same can be applied to AI models used in surveillance systems that use cameras to photograph and monitor target locations on railway tracks (e.g., railway tracks within a station) or roads (e.g., highways) where suspicious objects should not normally exist, and determine whether a suspicious object is captured in the image as a detectable object. [Explanation of Symbols]
[0076] 1…Device for supporting the determination of the cause 200... Processing Unit 202…First Inspection Department 204…First presentation part 206…Second Inspection Department 208…Second presentation part 210...Third Inspection Department 212…Third presentation part 300...Storage section 302…Support Program for Identifying the Cause 310...Test model data 312...Data set of images taken for training 314...Mistake in image data 320...Camera settings list 322…Feature List 324...List of incomplete AI models 10…AI model (pre-trained) 12…AI model (untrained) 20... Photographed images 22...Mistake image 30...Detected object 40…Group of images taken for learning purposes 42…Images taken for learning purposes
Claims
1. A cause analysis support device that assists in identifying the cause of errors in judgment by an AI (Artificial Intelligence) model that determines whether or not a target object is captured in a photographed image, A first inspection means generates a modified image based on the erroneous image, which is the AI model being inspected, in which the camera settings were changed and the image was taken, and inputs the modified image into the model being inspected to perform a first inspection to determine whether the judgment is successful or not. A second inspection means performs a second inspection by evaluating the deficiencies in training images based on the training image set used when generating the aforementioned test model and the aforementioned error images. A third inspection means that performs a learning process using the aforementioned set of training images on each of several types of incomplete AI models to generate type-specific AI models, and then inputs the aforementioned incorrect images into each of the type-specific AI models to perform a third inspection to determine whether the judgment is successful or not. A device for supporting the determination of causes, equipped with the following features.
2. The cause identification support device according to claim 1, wherein a determination is made to continue or terminate the inspection based on the results of the first inspection, and if the inspection is to continue, the second inspection is performed by the second inspection means, and a determination is made to continue or terminate the inspection based on the results of the second inspection, and if the inspection is to continue, the third inspection is performed by the third inspection means.
3. A first presentation means that presents the camera settings related to the image taken with the settings changed, which was judged to be successful in the first inspection, as an item for improvement. The cause determination support device according to claim 1 or 2, further comprising the above.
4. The second inspection means is, Using a feature analysis process that analyzes the features of the aforementioned captured images, the features of each training image included in the training image group are analyzed, and the feature distribution of the training image group is calculated. Using the aforementioned feature analysis process, the characteristics of the misimage are analyzed, The features distribution is compared with the features of the misimages to determine the features missing from the training image set. The second inspection is performed as described above. The cause identification support device according to claim 1.
5. The feature analysis process involves analyzing the features of the image portion of the captured image in which the object to be detected is depicted, and including these features in the features of the object to be analyzed. The cause determination support device according to claim 4.
6. A second presentation means selects a training image from the group of training images that satisfies predetermined similarity feature conditions based on the missing features determined by the second inspection, and presents the selected training image and information indicating the missing features. The cause determination support device according to claim 4 or 5, further comprising the above.
7. The aforementioned multiple types of incomplete AI models differ in at least one of the following aspects of machine learning models: 1) algorithm, 2) structure, 3) number of layers, 4) number of nodes, and 5) number of parameters. The cause identification support device according to claim 1 or 2.
8. A third presentation means that presents information on the type of AI model that was determined to be successful in the third test, The cause determination support device according to claim 1 or 2, further comprising the above.
9. The aforementioned captured image is an image of the track on which the vehicle is traveling. The object to be detected is an obstacle on the trajectory. The cause identification support device according to claim 1 or 2.
10. A method for assisting a computer system in identifying the cause of a judgment error by an AI (Artificial Intelligence) model that determines whether or not a target object is present in a captured image, Based on the erroneous image, which is the AI model being tested and in which the subject model made a judgment error, a modified image is generated by changing the camera settings and taking a picture. This modified image is then input into the subject model to perform a first test to determine whether the judgment is successful or not. A second test is performed by evaluating the deficiencies in training images based on the training images used to generate the aforementioned test model and the aforementioned error images. The process involves performing a learning process on each of several types of incomplete AI models using the aforementioned training image set to generate type-specific AI models, and then inputting the aforementioned error images into each of these type-specific AI models to perform a third check to determine whether the judgment is successful or not. Methods to support the investigation of causes, including those mentioned above.