Evaluation support program, evaluation support method, and information processing device

Abstract

To make it possible to support work for evaluating a learning model.SOLUTION: An information processing device respectively specifies analysis results of a plurality of second data by inputting the plurality of second data to be an analysis object to a learning model which has executed machine learning by using a leaning data set which defines a relation between a plurality of first data and classes of the plurality of first data. When the information processing device receives the selection of any second data among the plurality of second data, it retrieves first data similar to the selected second data from the learning data set. The information processing device outputs an analysis result of the selected second data and information on the retrieved first data collectively.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an evaluation support program and the like. [Background technology]

[0002] In recent years, learning models such as neural networks (NNs) have been used for image recognition. When performing image recognition using a learning model, supervised machine learning is performed using training data as a preparatory step. In the following explanation, supervised machine learning will be referred to simply as machine learning.

[0003] Furthermore, before actually using a machine-learned learning model, workers evaluate the performance of the learning model. For example, workers input multiple pieces of data to be analyzed into the learning model, obtain each analysis result, and evaluate the learning model by comparing the multiple pieces of data to be analyzed with each analysis result. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-201543 [Patent Document 2] International Publication No. 2010 / 119615 [Patent Document 3] Japanese Patent Publication No. 2020-039851 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the task of evaluating a learning model by comparing multiple data sets to be analyzed with the results of each analysis is inefficient and places a heavy burden on workers. For this reason, there is a need for support for the task of evaluating learning models.

[0006] In one aspect, the present invention aims to provide an evaluation support program, an evaluation support method, and an information processing device that can support the task of evaluating a learning model. [Means for solving the problem]

[0007] In the first proposal, a computer is caused to execute the following process. The computer inputs a plurality of second data to be analyzed into a learning model that has performed machine learning using a learning dataset that defines the relationship between a plurality of first data and the classes of the plurality of first data, thereby identifying analysis results for each of the plurality of second data. When the computer accepts selection of one of the plurality of second data, it searches the learning dataset for first data that is similar to the selected second data. The computer outputs the analysis result of the selected second data together with information about the searched first data. [Effects of the Invention]

[0008] It can assist in the task of evaluating learning models. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of an analysis screen. [Figure 2] FIG. 2 is a diagram showing an example of the detailed analysis screen. [Figure 3] FIG. 3 is a functional block diagram showing the configuration of the information processing device according to this embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the data structure of the learning data table. [Figure 5] FIG. 5 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the evaluation support program, evaluation support method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0011] The information processing device according to this embodiment stores a learning model that has been subjected to machine learning in advance using a learning dataset, and supports the user in evaluating the performance of such a learning model.

[0012] The information processing device inputs video data to be analyzed into a learning model and generates an analysis screen based on the analysis results of each image frame. The video data is data consisting of multiple image frames in chronological order. Each image frame is assigned a frame number in chronological order.

[0013] Here, when an image frame is input, the learning model held by the information processing device outputs the likelihood of each class to which the image frame belongs as an analysis result of the image frame. For example, each class corresponds to a type of automobile, ship, airplane, etc. In this embodiment, for convenience, each class is referred to as "AAA," "BBB," and "CCC."

[0014] When the learning model outputs the likelihood of each class, it also outputs a confidence level. For example, the more clearly the input image frame is classified into which class, the higher the confidence level.

[0015] An example of an analysis screen generated by an information processing device will be described below. Fig. 1 is a diagram showing an example of the analysis screen. The analysis screen 10 includes an image display area 11a, a seek bar 11b, control buttons 11c, an accumulation result display area 12, a frame analysis result display area 13, and a reliability display area 14.

[0016] The image display area 11a is an area for displaying video data to be analyzed. The playback position of the video data displayed in the image display area 11a is indicated by an icon 9 on the seek bar 11b. The user can move the position of the icon 9 to any position on the seek bar 11b by operating an input unit (input unit 120, which will be described later).

[0017] The seek bar 11b allows the position on the seek bar 11b to correspond to each frame number of the time-series image data. The left end of the seek bar 11b corresponds to the frame number of the first image data of the video data, and the frame number increases as the position of the icon 9 on the seek bar 11b moves further to the right. The right end of the seek bar 11b corresponds to the frame number of the last image data of the video data. The image frame with the frame number corresponding to the position of the icon 9 is displayed in the image display area 11a. The icon 9 indicates the playback position of the video data.

[0018] The control button 11c is a button for controlling the playback of the video data. The user can play, pause, stop, etc. the video data by operating the input unit and pressing the control button 11c. In this embodiment, the user plays the video data, refers to the image display area 11a, and presses the control button 11c at the timing when the user wants to analyze the video data to pause it. When the video data is temporarily stopped by such a user operation, a detailed analysis screen is displayed (displayed as a pop-up). The detailed analysis screen will be described later.

[0019] The cumulative result display area 12 is an area that displays the analysis results of the class of video data identified based on each image frame from the first image frame of the video data (the image frame with the first frame number) to the image frame with the frame number specified by icon 9. In the following description, the first image frame of the video data will be referred to as the "first frame," and the image frame with the frame number specified by icon 9 will be referred to as the "current frame."

[0020] For example, the information processing device may take the average value of the likelihood of each class when each image frame from the first frame to the current frame is input into the learning model as the cumulative result. In the example shown in FIG. 1, the cumulative result for class "CCC" is 85.07%. The cumulative result for class "AAA" is 8.6%. The cumulative result for class "BBB" is 6.33%. In other words, it can be determined from the output result of the learning model that the class of the video data played up to the position of icon 9 is most likely to be class "CCC."

[0021] Graph 12G shown in cumulative result display area 12 is a graph showing the time series changes in the cumulative results of each class. In graph 12G, the vertical axis indicates likelihood, and the horizontal axis corresponds to frame number (time). Graph 12G displays the cumulative results of likelihood for each class output from the learning model from the first frame to the current frame, in correspondence with the frame number. For example, line 1a indicates the time series changes in the cumulative results of class "CCC." Line 1b indicates the time series changes in the cumulative results of class "AAA." Line 1c indicates the time series changes in the cumulative results of class "BBB."

[0022] The frame analysis result display area 13 is an area that displays the likelihood of each class when the current frame is input to the learning model. In the example shown in FIG. 1, the likelihood of class "CCC" is 45.03%, the likelihood of class "AAA" is 37.94%, and the likelihood of class "BBB" is 17.03%. In other words, it can be determined from the output result of the learning model that the current frame alone is most likely to be class "CCC."

[0023] Graph 13G shown in frame analysis result display area 13 is a graph showing the time series change in likelihood of each class for each image frame. In graph 13G, the vertical axis indicates the likelihood of each class, and the horizontal axis corresponds to the frame number. Graph 13G displays the likelihood of each class (likelihood in image frame units) output from the learning model for each image frame from the first frame to the current frame, in correspondence with the frame number. For example, line 2a indicates the time series change in the likelihood of class "CCC." line 2b indicates the time series change in the likelihood of class "AAA." line 1c indicates the time series change in the likelihood of class "BBB."

[0024] The reliability display area 14 is an area showing reliability information determined based on each image frame from the first frame to the current frame. The video reliability shown in the reliability display area 14 shows the average value of each reliability when each image frame from the first frame to the current frame is input into the learning model. The image reliability shown in the reliability display area 14 shows the reliability when the current frame is input into the learning model.

[0025] Graph 14G shown in reliability display area 14 is a graph showing time-series changes in video reliability and image reliability. In graph 14G, the vertical axis indicates reliability, and the horizontal axis corresponds to frame number. For example, line 3a indicates time-series changes in video reliability. Line 3b indicates time-series changes in image reliability. A reliability threshold Th is set in graph 14G. For example, when an area 4 containing video reliability below reliability threshold Th is included, the information processing device displays a warning screen indicating that the object is an unlearned object.

[0026] Next, an example of a detailed analysis screen displayed by the information processing device will be described. The information processing device displays the detailed analysis screen (displays it as a pop-up) when the user operates the control button 11c to pause the playback of video data. Note that the information processing device may also display the detailed analysis screen when it receives a press of a detailed analysis button (not shown) on the analysis screen 10.

[0027] 2 is a diagram showing an example of a detailed analysis screen 20. The detailed analysis screen 20 includes display areas 20A, 20B, and 20C.

[0028] Information related to the current frame is displayed in the display area 20A. For example, an image 21a, a feature visualization image 21b, and a superimposed image 21c are displayed in the display area 20A. The image 21a is an image corresponding to the current frame. The feature visualization image 21b is an image in which the feature amounts of the image 21a are visualized. Here, the feature visualization image 21b will be described using an example in which the image 21a includes an image of a ship. The information processing device analyzes the image 21a and identifies the positions of the feature parts of the image included in the image 21a. For example, the information processing device identifies the positions of the feature parts of the image, such as the front end, mast, and rear end of the ship. The feature parts may be other parts. The information processing device generates the feature visualization image 21b by color-coding the positions of the feature parts and displaying them with vertical lines. The superimposed image 21c is an image in which the image 21a and the feature visualization image 21b are superimposed.

[0029] Display area 20B displays the cumulative result of the likelihood of each class shown in cumulative result display area 12 of Fig. 1. Display area 20B also displays the video reliability shown in reliability display area 14 of Fig. 1. The information processing device may further display information in frame analysis result display area 13 and image reliability in display area 20B.

[0030] The display area 20C displays a similar learning data table 22. The similar learning data table 22 includes information on similar learning data. The similar learning data indicates learning data that is similar to the current frame among multiple learning data included in a learning dataset used when machine learning a learning model.

[0031] The similar learning data table 22 associates item numbers, classes, data similarities, visualization similarities, and visualization results. The item numbers are numbers that distinguish records in the similar learning data table. The classes indicate the correct labels set for the similar learning data. In this embodiment, the classes are set to "AAA," "BBB," or "CCC."

[0032] The data similarity indicates the degree of similarity between the current frame and the training data (similar training data). The visualization similarity indicates the degree of similarity between the feature visualization image of the current frame and the training data (similar training data). Three types of images are set as visualization results for each similar training data. For example, the visualization results may include an image of the similar training data, a feature visualization image of the similar training data, and a superimposed image of the image and the feature visualization image. The record in the similar training data table 22 is assigned an item number with a smaller numerical value as the data similarity and visualization similarity increase.

[0033] As described above, the information processing device searches the training dataset for similar training data that is similar to the current frame of the analysis target that has been input to the training model, and outputs the analysis results of the data of the analysis target together with information on the searched similar training data. This helps users evaluate the training model that has undergone machine learning.

[0034] For example, the information processing device displays a detailed analysis screen 20. By referring to the detailed analysis screen 20, the user can compare the analysis results displayed in the display areas 20A and 20B with the similar training data displayed in the display area 20C and determine the validity of the analysis results. If the class with the highest likelihood among the classes displayed in the display area 20B matches the class of the similar training data, the user can determine that the analysis results are valid.

[0035] Furthermore, by referring to the detailed analysis screen 20, the user can compare the feature visualization image of the current frame with the feature visualization image of the similar learning data, and determine the validity from the perspective of the feature parts, which is difficult to determine from the image alone. In other words, if the feature parts are similar and the class with the highest likelihood matches the class of the similar learning data, it can be determined that the analysis result is valid.

[0036] Next, an example of the configuration of the information processing device 100 according to this embodiment will be described. Fig. 3 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 3, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0037] The communication unit 110 is connected to an external device or the like by wire or wirelessly, and transmits and receives information to and from the external device or the like. For example, the communication unit 110 is realized by a NIC (Network Interface Card) or the like.

[0038] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. For example, a user uses the input unit 120 to give instructions for analyzing the learning model 142. The user also uses the input unit 120 to operate the icons 9, the control buttons 11c, and various types of information on the detailed analysis screen 20 of the analysis screen 10 displayed on the display unit 130.

[0039] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, etc. For example, the display unit 130 displays the analysis screen 10 and the detailed analysis screen 20.

[0040] The storage unit 140 includes a learning data table 141, a learning model 142, and analysis target data 143. The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0041] The learning data table 141 includes a learning data set used in the machine learning of the learning model 142, data of feature visualization images generated based on the learning data, etc. Fig. 4 is a diagram showing an example of the data structure of the learning data table. As shown in Fig. 4, this learning data table 141 associates item numbers, learning data (image frames), correct labels (classes), and feature visualization images.

[0042] The item number is a number that distinguishes records in the learning data table 141. A pair of learning data and a correct label is data used when executing machine learning of the learning model 142. For example, the parameters of the learning model 142 are adjusted so that the output when learning data is input to the learning model 142 approaches the correct label. Furthermore, the clearer it is to which class the learning data is classified (the higher the reliability), the more the parameters are adjusted so that the output result approaches the correct label.

[0043] The feature visualization image is an image that visualizes the feature amounts of the training data. The description of the feature visualization image is the same as the description of the feature visualization image described in FIG.

[0044] The learning model 142 is a learning model obtained by performing machine learning based on the learning data set included in the learning data table 141. The learning model 142 corresponds to a neural network (NN) or the like.

[0045] Analysis target data 143 indicates video data to be analyzed. The video data has image frames in chronological order, and each video frame is assigned a frame number in chronological order.

[0046] Returning to the explanation of Fig. 3, the control unit 150 includes an acquisition unit 151, an identification unit 152, and a screen information generation unit 153. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be implemented by an integrated circuit such as, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0047] The acquisition unit 151 acquires the data of the learning data table 141 , the data of the learning model 142 , and the data to be analyzed from the communication unit 110 and the input unit 120 , and stores them in the storage unit 140 .

[0048] When the identification unit 152 receives an analysis instruction for the learning model 142 from a user operating the input unit 120, it inputs each image frame of the video data included in the data to be analyzed into the learning model 142 and outputs the analysis result to the screen information generation unit 153. For example, the analysis result includes the frame number of the image frame, the likelihood of each class, and the reliability.

[0049] The identification unit 152 compiles the analysis results for all image frames from the first image frame to the last image frame included in the analysis target data, and outputs them to the screen information generation unit 153. Note that the identification unit 152 may also sequentially output the analysis results obtained by inputting the image frames, starting from the first, into the learning model 142 to the screen information generation unit 153.

[0050] The screen information generating unit 153 generates the analysis screen 10 and the detailed analysis screen 20 based on the analysis results of the analysis target data 143, and outputs them to the display unit 130 for display.

[0051] The analysis screen 10 generated by the screen information generating unit 153 corresponds to the analysis screen 10 described in FIG. 1. The screen information generating unit 153 displays video data included in the analysis target data 143 in the image display area 11a. The screen information generating unit 153 controls playback, pausing, and stopping of the video data in response to pressing of the control button 11c. The screen information generating unit 153 also moves the icon 9 on the seek bar 11b in response to playback of the video data.

[0052] In addition, the screen information generation unit 153 generates information for the cumulative result display area 12, the frame analysis result display area 13, and the reliability display area 14 based on the analysis results of the analysis target data 143, and displays it on the analysis screen 10.

[0053] When generating data for the cumulative result display area 12, the screen information generation unit 153 calculates, as a cumulative result, an average value of the likelihood of each class when each image frame up to the current frame is input to the learning model. The screen information generation unit 153 displays a ranking result based on the magnitude of the likelihood in the cumulative result display area 12. The screen information generation unit 153 generates data for a graph 12G and displays it in the cumulative result display area 12.

[0054] When generating data for the frame analysis result display area 13, the screen information generation unit 153 identifies the likelihood of each class from the analysis result of the current frame. The screen information generation unit 153 displays a ranking result based on the magnitude of the likelihood in the frame analysis result display area 13. The screen information generation unit 153 generates data for a graph 13G and displays it in the frame analysis result display area 13.

[0055] When generating data for the reliability display area 14, the screen information generation unit 153 calculates the video reliability as the average value of the reliability of each image frame when inputting each image frame from the first frame to the current frame into the learning model. The screen information generation unit 153 also identifies the image reliability from the analysis result of the current frame. The screen information generation unit 153 displays the video reliability and the image reliability in the reliability display area 14. The screen information generation unit 153 generates data for a graph 14G and displays it in the reliability display area 14. If the video reliability is included in an area 4 that is below the reliability threshold Th, the screen information generation unit 153 displays a warning screen indicating that the target is not yet learned. The threshold Th is a preset value. The user may adjust the threshold Th by operating the input unit 120.

[0056] When the control button 11c on the analysis screen 10 is operated by the user to pause the playback of the video data, the screen information generation unit 153 generates information for the detailed analysis screen 20 and outputs and displays it on the display unit 130. The screen information generation unit 153 treats the image frame being displayed in the image display area 11a at the time the pause is accepted as the current frame.

[0057] The detailed analysis screen 20 generated by the screen information generating unit 153 corresponds to the detailed analysis screen 20 described in Fig. 2. As described in Fig. 2, the detailed analysis screen 20 includes display areas 20A, 20B, and 20C.

[0058] The screen information generating unit 153 analyzes the current frame, identifies the positions of the characteristic parts, and generates a characteristic visible image. When generating the characteristic visible image, the screen information generating unit 153 may determine the types and colors of the characteristic parts in advance, and highlight important characteristic parts among multiple characteristic parts in red, etc. The screen information generating unit 153 sets the image 21a, the characteristic visualized image 21b, and the superimposed image 21c in the display area 20A.

[0059] The screen information generating unit 153 sets the accumulation result of the likelihood of each class shown in the accumulation result display area 12 of the analysis screen 10 and the information on the video reliability shown in the reliability display area 14 in the display area 20B.

[0060] The screen information generating unit 153 compares the current frame with each piece of learning data in the learning data table 141, and calculates the data similarity for each piece. For example, the screen information generating unit 153 divides the current frame and the learning data into partial regions, and calculates the degree of agreement between the luminance values and pixel values of the partial regions as the data similarity.

[0061] The screen information generating unit 153 calculates the visualization similarity by comparing the characteristic visible image of the current frame with each characteristic visible image in the learning data table 141. For example, the screen information generating unit 153 calculates the degree of agreement between the colors and ranges assigned to the characteristic portions as the visualization similarity.

[0062] The screen information generation unit 153 executes the process of calculating the data similarity and the visualization similarity for each record in the learning data table 141. For example, the screen information generation unit 153 compares the learning data corresponding to item number 1 in the learning data table 141 with the current frame to calculate the data similarity, and compares the feature visualization image corresponding to item number 1 with the visualized image of the current frame to calculate the visualization similarity. The same applies to the other records.

[0063] The screen information generation unit 153 calculates the total value of the data similarity and the visualization similarity for each record number, and sorts the records in descending order of total value. The screen information generation unit 153 generates a similar learning data table 22 based on the top n records among the sorted records, and sets it in the display area 20c. The screen information generation unit 153 reassigns the item numbers from the beginning, and sets the classes, data similarity, visualization similarity, and visualization results in association with each other in the similar learning data table 22. When calculating the total value, the screen information generation unit 153 may calculate the total value by weighting one of the data similarity and the visualization similarity.

[0064] As described above, the screen information generating unit 153 displays the analysis screen 10, and when the user operates the control button 11c on the analysis screen 10 to pause the playback of the video data, the screen information generating unit 153 generates information on the detailed analysis screen 20 and displays it as a pop-up on the display unit 130. When the user pauses at different times, the screen information generating unit 153 generates a detailed analysis screen 20 based on each of the times and displays multiple detailed analysis screens 20 as pop-ups.

[0065] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 5 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 5, the acquisition unit 151 of the information processing device 100 acquires the learning data table 141, the learning model 142, and the analysis target data 143, and registers them in the storage unit 140 (step S101).

[0066] The identification unit 152 of the information processing device 100 receives an analysis instruction from a user who operates the input unit 120 (step S102). The identification unit 152 inputs the analysis target data 143 into the learning model 142 to identify an analysis result (step S103).

[0067] The screen information generating unit 153 of the information processing device 100 generates the analysis screen 10 based on the analysis result and displays it on the display unit 130 (step S104). The screen information generating unit 153 starts playing the video data (analysis target data) (step S105).

[0068] When the screen information generating unit 153 receives the pause instruction, it generates a detailed analysis screen and displays it as a pop-up on the display unit 130 (step S106).

[0069] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 searches the learning data table 141 for similar learning data that is similar to the current frame of the analysis target input to the learning model 142, and outputs the analysis results of the data of the analysis target together with information on the searched similar learning data. This can assist the user in evaluating a learning model that has undergone machine learning.

[0070] For example, the information processing device 100 displays a detailed analysis screen 20. By referring to the detailed analysis screen 20, the user can compare the analysis results displayed in the display areas 20A and 20B with the similar training data displayed in the display area 20C and determine the validity of the analysis results. If the class with the highest likelihood among the classes displayed in the display area 20B matches the class of the similar training data, the user can determine that the analysis results are valid.

[0071] Furthermore, by referring to the detailed analysis screen 20, the user can compare the feature visualization image of the current frame with the feature visualization image of the similar learning data, and determine the validity from the perspective of the feature parts, which is difficult to determine from the image alone. In other words, if the feature parts are similar and the class with the highest likelihood matches the class of the similar learning data, it can be determined that the analysis result is valid.

[0072] Based on the data similarity and the visualization similarity, the information processing device 100 searches the learning data table 141 for learning data that is similar to the current frame. This makes it possible to display information about the learning data that is the basis for the analysis result of the current frame.

[0073] The information processing device 100 generates and displays a cumulative result based on the likelihood of each class when each image frame from the first frame to the current frame is input into the learning model, allowing the user to easily check the overall analysis result from the first frame to the current frame.

[0074] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 6 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0075] 6, computer 200 includes CPU 201 for executing various types of arithmetic processing, input device 202 for receiving data input from a user, and display 203. Computer 200 also includes communication device 204 for transmitting and receiving data to and from external devices via a wired or wireless network, and interface device 205. Computer 200 also includes RAM 206 for temporarily storing various types of information, and hard disk drive 207. Each of devices 201 to 207 is connected to bus 208.

[0076] The hard disk device 207 stores an acquisition program 207a, a specific program 207b, and a screen information generation program 207c. The CPU 201 reads out each of the programs 207a to 207c and loads them into the RAM 206.

[0077] The acquisition program 207a functions as the acquisition process 206a. The identification program 207b functions as the identification process 206b. The screen information generation program 207c functions as the screen information generation process 206c.

[0078] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the identification process 206b corresponds to the processing of the identification unit 152. The processing of the screen information generation process 206c corresponds to the processing of the screen information generation unit 153.

[0079] It should be noted that each of the programs 207a to 207c does not necessarily have to be stored in the hard disk device 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207c.

[0080] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0081] (Supplementary Note 1) A plurality of second data to be analyzed is input into a learning model that has been machine-learned using a learning dataset that defines the relationship between a plurality of first data and classes of the plurality of first data, thereby identifying analysis results of each of the plurality of second data; When a selection of any one of the plurality of second data is received, first data similar to the selected second data is searched for from the training dataset; The analysis results of the selected second data are output together with information about the searched first data. An evaluation support program that causes a computer to execute a process.

[0082] (Supplementary Note 2) The evaluation support program according to Supplementary Note 1, further comprising: a process of generating a second feature visualization image that visualizes a feature of the selected second data; a plurality of first data in the training dataset are associated with the first feature visualization image that visualizes a feature of the first data; and the searching process searches the training dataset for first data similar to the second data by further using a similarity between the first feature visualization image and the second feature visualization image.

[0083] (Supplementary Note 3) The evaluation support program according to Supplementary Note 2, wherein the outputting process displays, as information about the first data, a first feature visualization image of the first data and information about a class of the first data, and displays, as an analysis result of the second data, a second feature visualization image of the second data and information about the class of the second data.

[0084] (Appendix 4) The evaluation support program described in Appendix 3, characterized in that the learning model is trained to further output the reliability of the class of the output result, and the output process further includes the reliability as information about the second data.

[0085] (Appendix 5) The evaluation support program described in Appendix 1, further characterized in that it displays information accumulating analysis results from the first second data to the selected second data among the plurality of second data.

[0086] (Appendix 6) A plurality of second data to be analyzed is input into a learning model that has been machine-learned using a learning dataset that defines the relationship between a plurality of first data and classes of the plurality of first data, thereby identifying analysis results of the plurality of second data; When a selection of any one of the plurality of second data is received, first data similar to the selected second data is searched for from the training dataset; The analysis results of the selected second data are output together with information about the searched first data. An evaluation support method characterized in that processing is executed by a computer.

[0087] (Supplementary Note 7) The evaluation support method according to Supplementary Note 6, further comprising: executing a process of generating a second feature visualization image that visualizes a feature of the selected second data; a plurality of first data in the training dataset are associated with the first feature visualization image that visualizes a feature of the first data; and the searching process searches the training dataset for first data similar to the second data by further using a similarity between the first feature visualization image and the second feature visualization image.

[0088] (Appendix 8) The evaluation support method according to Appendix 7, wherein the outputting process displays a first feature visualization image of the first data and class information of the first data as information about the first data, and displays a second feature visualization image of the second data and class information of the second data as an analysis result of the second data.

[0089] (Appendix 9) The evaluation support method described in Appendix 8, characterized in that the learning model is trained to further output the reliability of the class of the output result, and the output process further includes the reliability as information about the second data.

[0090] (Appendix 10) The evaluation support method described in Appendix 6, further characterized in that it displays information accumulating analysis results from the first second data to the selected second data among the plurality of second data.

[0091] (Supplementary Note 11) An identification unit that inputs a plurality of second data to be analyzed into a learning model that has performed machine learning using a learning dataset that defines a relationship between a plurality of first data and classes of the plurality of first data, and identifies analysis results of the plurality of second data; a screen information generating unit that, when receiving a selection of any one of the plurality of second data, searches the learning dataset for first data similar to the selected second data, and outputs an analysis result of the selected second data together with information related to the searched first data; An information processing device comprising:

[0092] (Supplementary Note 12) The information processing device according to Supplementary Note 11, wherein a plurality of first data in the training dataset are associated with first feature visualization images that visualize features of the first data, and the screen information generation unit further executes processing to generate second feature visualization images that visualize features of selected second data, and searches for first data similar to the second data from the training dataset by further using a similarity between the first feature visualization image and the second feature visualization image.

[0093] (Supplementary Note 13) The information processing device according to Supplementary Note 12, wherein the screen information generation unit displays, as information about the first data, a first feature visualization image of the first data and class information of the first data, and displays, as an analysis result of the second data, a second feature visualization image of the second data and class information of the second data.

[0094] (Appendix 14) The information processing device described in Appendix 13, characterized in that the learning model is trained to further output the reliability of the class of the output result, and the screen information generation unit further includes the reliability as information regarding the second data.

[0095] (Appendix 15) The information processing device described in Appendix 11, characterized in that the screen information generation unit further displays information that accumulates analysis results from the first second data to the selected second data among the plurality of second data. [Explanation of symbols]

[0096] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Training Data Table 142 Learning Model 143 Data to be analyzed

Claims

1. inputting a plurality of second data, which is video data to be analyzed, into a learning model that has been machine-learned using a learning dataset that defines the relationship between a plurality of first data and classes of the plurality of first data; and, when the display of the second data is paused, identifying analysis results from the first frame of the plurality of second data to the paused frame of the second data; When a selection of any one of the plurality of second data is received, first data similar to the selected second data is searched for from the training data set; The analysis result of the selected second data and information about the retrieved first data are output together. An evaluation support program that causes a computer to execute a process.

2. 2. The evaluation support program according to claim 1, further comprising: a process of generating a second feature visualization image that visualizes a feature of the selected second data; wherein a plurality of first data in the training dataset are associated with the first feature visualization image that visualizes a feature of the first data; and wherein the searching process searches the training dataset for first data similar to the second data by further using a similarity between the first feature visualization image and the second feature visualization image.

3. 3. The evaluation support program according to claim 2, wherein the output process displays, as the information about the first data, a first feature visualization image of the first data and class information of the first data, and displays, as an analysis result of the second data, a second feature visualization image of the second data and class information of the second data.

4. The evaluation support program described in claim 3, characterized in that the learning model is trained to further output the reliability of the class of the output result, and the output process further includes the reliability as information regarding the second data.

5. 2. The evaluation support program according to claim 1, further displaying information accumulating analysis results from the first second data to the selected second data among the plurality of second data.

6. inputting a plurality of second data, which is video data to be analyzed, into a learning model that has been machine-learned using a learning dataset that defines the relationship between a plurality of first data and classes of the plurality of first data; and, when the display of the second data is paused, identifying analysis results from the first frame of the plurality of second data to the paused frame of the second data; When a selection of any one of the plurality of second data is received, first data similar to the selected second data is searched for from the training dataset; The analysis result of the selected second data and information about the retrieved first data are output together. An evaluation support method characterized in that processing is executed by a computer.

7. an identification unit that inputs a plurality of second data, which is video data to be analyzed, into a learning model that has been machine-learned using a learning dataset that defines the relationship between a plurality of first data and classes of the plurality of first data, and, when display of the second data is paused, identifies analysis results from a first frame of the plurality of second data to the paused frame of the second data; a screen information generating unit that, when receiving a selection of any one of the plurality of second data, searches the learning dataset for first data similar to the selected second data, and outputs an analysis result of the selected second data together with information related to the searched first data; An information processing device comprising:

Images