Information processing apparatus, visualization method, and visualization program

Abstract

To facilitate comprehensive visual recognition of characteristics of a composite learner including a plurality of learners.SOLUTION: An information processing apparatus includes: an acquisition unit that acquires a first data set, and acquires a first composite learner including a plurality of individual learners; and an output unit that outputs first visualization information including first graphic information obtained by summarizing, composite output information output by the first composite learner, in a case where the first data set is input to the first composite learner, individual output information output by each individual learner, configuration information of each individual learner, and a relationship between different individual learners.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a visualization method, and a visualization program.

Background Art

[0002] Patent Document 1 discloses a technique related to an identification result display device that can display the identification accuracy corresponding to an identification target in a composite identifier combining a plurality of identifiers. The identification result display device according to Patent Document 1 displays an individual identification index indicating the identification accuracy of the identifier included in the composite identifier and a composite identification index indicating the identification accuracy of the composite identifier.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since the technique according to Patent Document 1 displays the individual identification indexes of the plurality of identifiers and the composite identification index side by side, there is a problem that it is difficult to comprehensively grasp the characteristics of the composite identifier and the individual identifiers. For example, in the technique according to Patent Document 1, the problem becomes prominent when the number of individual identifiers and the types of indexes are more than a certain number.

[0005] An object of the present disclosure is to provide an information processing apparatus, a visualization method, and a visualization program for making it easy to comprehensively visually recognize the characteristics of a composite learning device including a plurality of learning devices in view of the above-described problems.

Means for Solving the Problems

[0006] The information processing apparatus according to the present disclosure is Output means for outputting first visualization information including first graphic information summarizing composite output information output by the first composite learning device, individual output information output by each individual learning device, configuration information of each individual learning device, and relationships between different individual learning devices when a first data set is input to the first composite learning device including a plurality of individual learning devices. Comprising.

[0007] The visualization method according to the present disclosure is The computer Acquires a first data set, Outputs first visualization information including first graphic information summarizing composite output information output by the first composite learning device, individual output information output by each individual learning device, configuration information of each individual learning device, and relationships between different individual learning devices when the first data set is input to the first composite learning device including a plurality of individual learning devices.

[0008] The visualization program according to the present disclosure is A process of acquiring a first data set, and A process of outputting first visualization information including first graphic information summarizing composite output information output by the first composite learning device, individual output information output by each individual learning device, configuration information of each individual learning device, and relationships between different individual learning devices when the first data set is input to the first composite learning device including a plurality of individual learning devices, and Causes the computer to execute.

Advantages of the Invention

[0009] According to the present disclosure, it is possible to provide an information processing apparatus, a visualization method, and a visualization program for making it easier to comprehensively visually recognize the characteristics of a composite learning device including a plurality of learning devices.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant descriptions are omitted as necessary for clarity of explanation.

[0012] <Embodiment 1> Hereinafter, the configuration of the information processing apparatus 1 will be described with reference to FIG. 1. The information processing apparatus 1 outputs visualization information including graphic information indicating an outline of characteristics of a predetermined composite learner when a predetermined data set is input to the predetermined composite learner including a plurality of individual learners.

[0013] Here, the learner in this specification is assumed to include a learning function for machine learning a predetermined AI (Artificial Intelligence) model and the AI model to be learned. The learning function includes a computer program in which a predetermined machine learning algorithm is implemented and hyperparameters such as set values. The AI model may be one that takes a feature vector, which is multi-dimensional data, as an input and outputs an estimated value from the input data. Alternatively, the AI model may be one that outputs a result of classifying input data into a predetermined category or label. For example, the AI model used in the present disclosure is applicable to a problem of obtaining an estimated value from input data in a state where there is a correct answer for the input data. Therefore, the AI model used in the present disclosure is one whose parameters can be machine-learned by supervised learning. In the following description, the AI model may sometimes be simply referred to as a "model".

[0014] Also, the "individual learner" shall refer to a single unit of the above-mentioned learner. And the "composite learner" shall refer to one that includes a plurality of individual learners. Note that each of the plurality of individual learners may have different learning functions and some or all of the models. That is, the composite learner may include a plurality of different individual learners. The composite learner includes a plurality of individual learners and a learning function for performing machine learning on the models of each individual learner. The learning function of the composite learner inputs the input data set into each individual learner and obtains output information from each individual learner. Note that the information input into the individual learner may be information output from another individual learner in addition to or instead of the data set. That is, the learning function of the composite learner may input the input data set into some or all of the plurality of individual learners and obtain output information from some or all of the individual learners. The learning function of the composite learner may include a computer program implementing a predetermined machine learning algorithm and hyperparameters such as setting values. Note that the composite learner may also include a model that integrates the models of each of the plurality of individual learners. Note that a predetermined composite learner may also be called a first composite learner. Also, a predetermined data set may also be called a first data set.

[0015] Further, the information processing apparatus 1 includes at least an output unit 10. When a first dataset is input to the first composite learner, the output unit 10 outputs first visualization information including first graphic information summarizing various information regarding the first composite learner and each individual learner. Here, the various information includes composite output information output by the first composite learner, individual output information output by each individual learner, configuration information of each individual learner, and the relationship between different individual learners. Here, the composite output information is the information output by the first composite learner when the first dataset is input to the first composite learner. Also, the individual output information is the information output by the individual learner when the first dataset is input from the first composite learner to the individual learner. Also, the configuration information of the individual learner is the information indicating the configuration of the learned model when the first dataset is input from the first composite learner to the individual learner. Alternatively, the configuration information of the individual learner may be the information indicating the configuration of the model utilized at the time of input of the first dataset when the model has been learned. Alternatively, the configuration information of the individual learner may be the information indicating the configuration of the model due to the relearning of the model in response to the change of the first dataset or hyperparameters. Also, the relationship between different individual learners is the information indicating the commonality and difference of the configuration information and individual output information among two or more individual learners included in the first composite learner.

[0016] Also, the process of "summarizing composite output information, individual output information, configuration information of each individual learning device, and the relationship between different individual learning devices" can be said to be a process of omitting, aggregating, integrating, etc. a part of these four or more types of information and converting them into graphic information. Also, the summarization process may be a process of compressing the amount of information from various types or a large amount of numerical information, etc. into graphic information that can be visually recognized by the user. Also, the summarization process may be a process of converting four or more types of information into information in a low-dimensional space such as three-dimensional space. Note that these summarization processes are merely examples and are not limited thereto. For example, the summarization process may be a process of converting information into graphic information obtained by aggregating information for at least one or more pieces of information including "the relationship between different individual learning devices" among four or more types of information. That is, the output unit 10 may summarize and convert information into graphic information by aggregating information for at least one or more pieces of information including at least "the relationship between different individual learning devices" among four or more types of information.

[0017] Here, the information processing apparatus 1 may acquire an input data set or a first composite learning device to be visualized. Therefore, it can be said that the information processing apparatus 1 includes an acquisition unit that acquires a data set or the first composite learning device.

[0018] The information processing apparatus 1 generates first graphic information summarizing composite output information, individual output information of each individual learning device, configuration information of each individual learning device, and the relationship between different individual learning devices when a first data set is input to the first composite learning device. Note that the information processing apparatus 1 may generate first graphic information summarizing at least one or more pieces of information including the relationship between different individual learning devices when a first data set is input to the first composite learning device. Then, the information processing apparatus 1 generates first visualization information including the generated first graphic information. Therefore, it can be said that the information processing apparatus 1 includes a generation unit that generates the first graphic information and generates the first visualization information including the generated first graphic information.

[0019] Then, the information processing device 1 outputs the generated first visualization information to the display device. The display device displays the first visualization information on the screen. Note that the display device may be either built into the information processing device 1 or an external device connected to the information processing device 1.

[0020] Note that the acquisition unit may be used as a means for acquiring information or data. Also, the generation unit may be used as a means for generating information or data. Further, the output unit 10 may be used as a means for outputting information or data.

[0021] Next, the flow of the visualization method will be described with reference to FIG. 2. First, the information processing device 1 acquires a first data set (S11). Incidentally, the information processing device 1 may also acquire a first composite learning machine to be visualized. Note that the first composite learning machine to be visualized may be pre-set in the information processing device 1. Subsequently, the information processing device 1 generates first graphic information summarizing composite output information, individual output information, configuration information of each individual learning machine, and the relationship between different individual learning machines when the first data set is input to the first composite learning machine. Then, the information processing device 1 outputs first visualization information including the generated first graphic information (S12). As a result, the display device displays the visualization information on the screen. Therefore, the user can visually recognize the visualization information displayed on the display device.

[0022] In this way, the information processing device 1 outputs visualization information in which the characteristics of the composite learning machine are graphed for a specific data set. In particular, the information processing device 1 includes the first graphic information summarizing the composite output information, individual output information, configuration information of each individual learning machine, and the relationship between different individual learning machines in the visualization information. Therefore, the user can easily comprehensively visually recognize the characteristics of the composite learning machine including a plurality of learning machines from the visualization information.

[0023] Furthermore, the information processing apparatus 1 includes a processor, a memory, and a storage device as components not shown in the figures. Also, the storage device stores a computer program in which, for example, the processing of the visualization method in FIG. 2 is implemented. Then, the processor causes the memory to read the computer program and the like from the storage device and executes the computer program. As a result, the processor realizes the functions of the (acquisition unit, generation unit, and) output unit 10.

[0024] Alternatively, each component of the information processing apparatus 1 may be realized by dedicated hardware. Also, some or all of the components of each device may be realized by general-purpose or dedicated circuitry, a processor, or the like, or a combination thereof. These may be configured by a single chip or by a plurality of chips connected via a bus. Some or all of the components of each device may be realized by a combination of the circuitry and the program described above. Also, as the processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a quantum processor (quantum computer control chip), or the like can be used.

[0025] Also, when some or all of the components of the information processing apparatus 1 are realized by a plurality of information processing devices, circuitry, or the like, the plurality of information processing devices, circuitry, or the like may be centrally arranged or may be distributed. For example, the information processing devices, circuitry, or the like may be realized in a form in which each is connected via a communication network, such as a client-server system or a cloud computing system. Also, the functions of the information processing apparatus 1 may be provided in the form of SaaS (Software as a Service).

[0026] <Embodiment 2> Next, the configuration of the visualization system 1000 for the composite learner will be described with reference to FIG. 3. The visualization system 1000 includes at least a storage device 100 and 200 and an information processing device 300. Each of the storage devices 100 and 200 and the information processing device 300 is communicably connected via a communication network (not shown). Note that the communication network is a wired or wireless communication line. Alternatively, the information processing device 300 may incorporate one or both of the storage devices 100 and 200. Also, the storage devices 100 and 200 may be realized by being distributed across multiple storage devices. Alternatively, the storage devices 100 and 200 may be incorporated in a server within a predetermined information system. Note that the storage devices 100 and 200 are non-volatile storage devices such as hard disks and flash memories, for example.

[0027] The storage device 100 stores composite learners 110 to 1i0 (where i is a natural number of 1 or more). The composite learners 110 etc. are examples of the first and second composite learners. For example, the composite learner 110 includes individual learners 111 to 11j (where j is a natural number of 2 or more). Note that the composite learners 120 (not shown) to 1i0 also include two or more individual learners, just like the composite learner 110. For example, the composite learner 1i0 includes individual learners 1i1 to 1ik (where k is a natural number of 2 or more).

[0028] Here, the individual learners 111 etc. may be weak learners. A weak learner refers to one in which the estimation accuracy of the learned model included in the weak learner is better than that when randomly selected, but not so high as to be considered highly accurate. Note that in the following example, the individual learners 111 etc. will be described as including a learned model generated by a decision tree. However, the model included in the individual learners 111 etc. is not limited to a decision tree. For example, the individual learners 111 etc. may include a learned model generated by a method including at least one of a decision tree, a support vector machine, and a neural network that perform multi-class classification or regression.

[0029] The composite learner 110 and the like include a trained model generated by ensemble learning using a plurality of weak learners as a plurality of individual learners. Therefore, the composite learner 110 and the like can also be called an ensemble learner. That is, the composite learner 110 and the like include a computer program in which an ensemble learning machine learning algorithm is implemented, and hyperparameters such as setting values.

[0030] The storage device 200 stores datasets 210 to 2m0 (m is a natural number of 1 or more). The datasets 210 and the like include a set of a feature vector which is multidimensional data and correct data (teacher data) of an estimated value. In particular, the datasets 210 and the like are a set of a plurality of pairs of a feature vector and correct data. The datasets 210 and the like are an example of the first and second datasets.

[0031] The information processing device 300 is an example of the information processing device 1 described above. The information processing device 300 may be a computer device that operates by a processor executing a program stored in a memory. The information processing device 300 may be arranged in a single physical computer device, or may be distributed and arranged in two or more computer devices. The information processing device 300 includes a model acquisition unit 31, a data acquisition unit 32, a model processing unit 33, a generation unit 34, an output unit 35, an operation reception unit 36, and an update unit 37. Note that each or a part of the model acquisition unit 31, the data acquisition unit 32, the model processing unit 33, the generation unit 34, the output unit 35, the operation reception unit 36, and the update unit 37 may be distributed and arranged in a plurality of computer devices. Note that the output unit 35 is an example of the output unit 10 described above.

[0032] The model acquisition unit 31 acquires a composite learner including a model to be visualized. The model acquisition unit 31 may acquire, for example, by reading out a composite learner designated by a user operation from among the composite learners 110 and the like in the storage device 100.

[0033] The data acquisition unit 32 acquires a data set to be input to the composite learner. For example, the data acquisition unit 32 may acquire the data set by reading out the data set designated by the user's operation from among the data sets 210 in the storage device 200 or the like.

[0034] The model processing unit 33 analyzes the operation details and output information of each individual learner and the composite learner when each data of the data set to be input is input to the model included in the composite learner to be visualized. For example, it can be said that the model processing unit 33 analyzes the behavior of each weak learner when data is input to the model. Or, it can be said that the model processing unit 33 analyzes the behavior of the ensemble-learned composite classifier with respect to a predetermined data set. Incidentally, the model processing unit 33 may input the data set acquired by the data acquisition unit 32 to the composite learner acquired by the model acquisition unit 31, acquire composite output information from the composite learner, and acquire individual output information and configuration information from each individual learner. Then, the model processing unit 33 may analyze the individual output information and configuration information of each individual learner to identify the relationship between different individual learners.

[0035] In this case, the composite learner may input each data of the data set input by the model processing unit 33 to each individual learner, and cause each individual learner to execute supervised learning by its respective machine learning method. In response to this, each individual learner learns the model by updating the parameters of the model by supervised learning. Incidentally, each individual learner may relearn the learned model. Also, some or all of the composite learner and each individual learner may relearn when the hyperparameters are changed according to the user's operation.

[0036] The composite output information may be information aggregated by the composite learner from the individual output information. Alternatively, when all or some of the individual learners are connected in series, the composite output information may be the output information of the last-stage individual learner. Note that the model processing unit 33 may obtain the hyperparameters of the composite learner together with the composite output information from the composite learner, and analyze the operation content and output information of the composite learner using the composite output information and the hyperparameters.

[0037] In addition, each individual learner may output individual output information including the result of inputting the input dataset into the learned model to the model processing unit 33. The individual output information of each individual learner may include at least one of the number of data, output result, and error from the target value for each subspace of the input feature space in the individual learner to which the dataset is input. The input feature space is, for example, distribution information or the like for each feature vector of the dataset. Also, the subspace is a subset of the input data or estimated values. For example, when the machine learning method is a decision tree, the subspace corresponds to a leaf node. Therefore, the number of data for each subspace corresponds to, for example, the number of data or estimated values belonging to each leaf node. The output result is the execution result of the model. For example, the output result is a set of data or estimated values belonging to each subspace. The error from the target value is, for example, the prediction error of the data or estimated values of the output result.

[0038] In addition, each individual learner may output configuration information such as a learned model to the model processing unit 33. The configuration information of each individual learner may include at least one or more of the division result of the input feature space into subspaces in the individual learner into which the data set is input, the distance between different subspaces, and the hyperparameters of each individual learner. Here, for example, when the machine learning method is a decision tree, the configuration information of each individual learner may be as follows. For example, the subspace may be a leaf of the decision tree. And, for example, the division result of the input feature space into subspaces may include information such as the information of each node (element) of the decision tree and information indicating the connection relationship between the nodes (parent node ID, etc.). Also, for example, the distance between different subspaces may be a numerical value calculated from information indicating the hierarchy (position or depth) of the nodes. Or, the distance between different subspaces may be a set of similarities (Euclidean distance, KL (Kullback-Leibler) distance, etc.) in the feature space of the samples included in each subspace. Also, for example, the hyperparameters of each individual learner may include the threshold value (classification rule) of the internal node, etc.

[0039] Also, the relationship between different individual learners may include at least one or more of the subspaces in different individual learners and the similarity of the hyperparameters of each individual learner. The similarity of each subspace may be, for example, the degree of commonality of the data belonging to the subspace. For example, when the machine learning method is a decision tree, the similarity of each subspace may be the degree of coincidence of the data IDs belonging to the leaf nodes. Also, the similarity of different subspaces (leaves) may be, for example, a set of similarities (Euclidean distance, KL distance, etc.) in the feature space of the samples included in each subspace. The similarity of the hyperparameters of each individual learner may be, for example, the difference value of the set values between a plurality of individual learners, but is not limited thereto.

[0040] The generation unit 34 generates graphic information that summarizes the output information by each individual learning device and composite learning device, the configuration information of each individual learning device, and the relationship between the individual learning devices, which has been acquired and analyzed by the model processing unit 33. In other words, it can be said that the generation unit 34 converts the output information by each individual learning device and composite learning device, the configuration information of each individual learning device, and the relationship between the individual learning devices into graphic information indicating the characteristics of the composite learning device. Or, it can be said that the generation unit 34 converts the output information by each individual learning device and composite learning device, the configuration information of each individual learning device, and the relationship between the individual learning devices into graphic information representing the characteristics of the individual discriminators, the relationship between the individual discriminators, and the differences between the datasets or models.

[0041] Then, the generation unit 34 generates visualization information including the generated graphic information. Therefore, the visualization information may include numerical information etc. in addition to the graphic information. Here, the generation unit 34 may generate graphic information by summarizing various information as follows, for example.

[0042] For example, the generation unit 34 may summarize by converting the composite output information of the composite learning device, the individual output information of each individual learning device, the configuration information of each individual learning device, and the relationship between the individual learning devices into a graphic in a three-dimensional space. In other words, the graphic information may be information summarized by converting the composite output information of the composite learning device, the individual output information of each individual learning device, the configuration information of each individual learning device, and the relationship between the individual learning devices into a graphic in a three-dimensional space. Thereby, information of four types (four dimensions) or more is compressed into three-dimensional information. Therefore, compared with a list of numerical values of various information and individual graphs, the characteristics of the composite learning device and the individual learning devices can be extracted while reducing the amount of information. Thus, it becomes easier for the user to overview the characteristics of the composite learning device and the individual learning devices.

[0043] Furthermore, for example, the generation unit 34 may summarize, for each individual learner, the planar graphic information converted into a graphic on a plane based on the individual output information and the configuration information by arranging them in layers in the vertical direction with respect to the plane. In other words, the graphic information may be information summarized by arranging, for each individual learner, the planar graphic information converted into a graphic on a plane based on the individual output information and the configuration information in layers in the vertical direction with respect to the plane. Thereby, it becomes easier for the user to grasp by comparing the outline information of a plurality of different individual learners.

[0044] Furthermore, for example, the generation unit 34 may convert, for each individual learner, a plurality of elements included in the configuration information into visual information according to the number of data belonging to each element. In this case, the generation unit 34 may generate planar graphic information by arranging the converted visual information in a manner based on the configuration information. In other words, the planar graphic information may be information in which the visual information converted according to the number of data belonging to each of a plurality of elements included in the configuration information for each individual learner is arranged in a manner based on the configuration information. Here, the "visual information converted according to the number of data belonging to each element" may be such that the size, color, shape, etc. of the visual information corresponding to the element are converted into different ones according to the number of data belonging to the element. For example, the visual information of an element with a larger number of data belonging to it may be graphic information with a larger size, an emphasized shape, or a color compared to the visual information of an element with a smaller number of data belonging to it. Also, "arranging in a manner based on the configuration information" may mean that the generation unit 34 arranges the visual information (graphic information) of each element on a plane according to the positional relationship, distance, etc. of each element. For example, when the distance between two specific elements within specific configuration information is relatively short, the generation unit 34 may arrange the visual information (graphic information) of the two specific elements relatively close to each other on the plane. Thereby, it becomes easier for the user to grasp the outline information of each individual learner from an overhead view without relying on numerical values or graphs.

[0045] Furthermore, for example, the generation unit 34 may connect elements that belong to common data among the planar graphic information in different individual learning devices with lines. In other words, the graphic information may include lines that connect elements that belong to common data among the planar graphic information in different individual learning devices. This makes it easier for the user to visually grasp the relationships between specific elements among a plurality of different individual learning devices, which elements in the data set belong to each individual learning device, and so on.

[0046] The output unit 35 outputs the visualization information generated by the generation unit 34 to the screen. For example, the output unit 35 displays the visualization information on a display device within the information processing apparatus 300 or an external display device connected to the information processing apparatus 300.

[0047] The operation reception unit 36 receives operation inputs from the user. For example, the operation reception unit 36 may receive, as operation inputs, a switching operation between the visualization information and the detailed information of the individual learning device, or an adjustment operation of any parameter, and so on.

[0048] The update unit 37 executes processing according to the operation content received by the operation reception unit 36. Specifically, the update unit 37 may execute at least one of the switching process between the visualization information and the detailed information of the individual learning device, the adjustment process of arbitrary parameters, and the relearning process of the composite learning device in response to an operation input from the user. For example, when the update unit 37 receives a switching operation between the visualization information and the detailed information of the individual learning device, the update unit 37 may switch between the display screen of the visualization information and the display screen of the detailed information of the individual learning device, and cause the output unit 35 to output the switched display screen. Also, for example, when the update unit 37 receives an adjustment operation of arbitrary parameters, the update unit 37 may adjust (update) the corresponding parameters and execute the relearning process of the composite learning device using the updated parameters. Note that the relearning process of the composite learning device may be executed by the model processing unit 33. In this case, the generation unit 34 generates the visualization information again according to the result of the relearning process. Then, the output unit 35 outputs the visualization information generated after the relearning. Thereby, since the adjustment operation of the user's parameters is promptly reflected, it becomes easier for the user to grasp the characteristics of the composite learning device after the adjustment. Therefore, it also becomes easier for the user to finely adjust the parameters, and appropriate tuning can be easily realized.

[0049] The flow of the visualization method will be described with reference to FIG. 4. First, the information processing apparatus 300 acquires a composite learning device including a model to be visualized (S21). Next, the information processing apparatus 300 acquires a data set of input targets for the composite learning device (S22). Note that the information processing apparatus 300 may change the execution order of steps S21 and S22, or may execute them in parallel.

[0050] Subsequently, the information processing apparatus 300 analyzes the operation details and output information of each individual learning device and the composite learning device when each data is input to the model (S23). Then, the information processing apparatus 300 generates graphic information summarizing the output information by each individual learning device and the composite learning device, the configuration information of each individual learning device, and the relationship between the individual learning devices (S24). After that, the information processing apparatus 300 generates visualization information including the graphic information generated in step S24 (S25). Then, the information processing apparatus 300 outputs the visualization information generated in step S25 to the screen (S26).

[0051] After that, the information processing apparatus 300 determines whether it has received an operation input from the user (S27). If it has received an operation input from the user within a certain period of time, the information processing apparatus 300 executes processing according to the operation content (S28). Then, after a certain period of time has elapsed, the information processing apparatus 300 executes step S27 again. In step S27, when a certain period of time has elapsed, or when the information processing apparatus 300 has received an end operation or the like from the user, the information processing apparatus 300 ends the processing of the visualization method.

[0052] <Example of visualization information: characteristics of the composite learning device, characteristics of the individual learning devices, relationship between the individual learning devices> Here, with reference to FIG. 5, visualization information 5 will be described. The visualization information 5 is information visualized using graphic information in order to visually overview the behavior and content of each individual learning device constituting the composite learning device. Also, the visualization information 5 shows an example where the model of the individual learning device is a decision tree. However, the example of the model is not limited to this as described above.

[0053] The visualization information 5 includes individual characteristic information T1, T2, ··· Tj. Here, the individual characteristic information T1 to Tj is an example of planar graphic information obtained by converting, for each individual learner, the individual output information and the configuration information into a graphic on a plane. Also, it can be said that the visualization information 5 is an example of graphic information obtained by arranging the individual characteristic information T1 to Tj in layers in the vertical direction with respect to the plane. For example, among the individual output information and the configuration information, the individual characteristic information such as T1 does not graphically represent the configuration (hierarchical structure) itself of the decision tree, the connection relationship of the nodes, the threshold of the node branch, and the data belonging to each leaf. Note that the individual characteristic information such as T1 may be graphic information that displays an outer frame and makes the inside transparent. Also, the outer shape of the individual characteristic information such as T1 may not display part or all of it. That is, the individual characteristic information such as T1 itself does not necessarily need to be visualized as planar graphic information, as long as at least the elements (such as leaves) included in the same individual characteristic information are displayed on the same layer.

[0054] In the individual characteristic information T1, leaves L11, L12, L13, L14, etc. are arranged. The leaves L11, etc. are an example of visual information corresponding to the elements included in the configuration information of the individual learner 111. Also, the leaves L11, etc. are an example of a subspace of the input feature amount space. The leaves L11, etc. are the elements corresponding to the leaf nodes among the elements included in the configuration information. That is, the leaves L11, etc. indicate a set of data that is the result of classification or estimation for the data set input to the individual learner 111. Note that the visual information does not necessarily need to have the schematic shape of a "leaf" as shown in FIG. 5, and may have other shapes. The visual information corresponding to the element may be, for example, a circle, a double circle, a star, or a triangle, a quadrilateral, or other polygons, etc.

[0055] And the leaves L11, etc. are an example of visual information converted according to the number of data belonging to each element. Specifically, the leaves L11, etc. show an example where the larger the number of data, the larger the size. In FIG. 5, the leaf L14 shows that the number of data is larger than that of the leaves L11, L12, and L13. Thereby, the user can visually and easily grasp the number of data belonging to the element by the size of the visual information (graphical information) corresponding to the element, rather than by a numerical value. Also, the leaves L11, etc. are also an example of visual information converted into different visual information according to the magnitude of the prediction error of the data belonging to the element (leaf). For example, the leaves L11, etc. have different hatching, colors, shapes, or gradations, etc. according to the magnitude of the prediction error of the data belonging to the leaf. Specifically, the leaf L12 shows that the prediction error is larger than that of the leaves L11, L13, and L14. Also, the leaves L11 and L14 show that the prediction errors are equivalent (within a certain range). Further, the leaf L13 shows that the prediction error is smaller than that of the leaves L11, L12, and L14.

[0056] Furthermore, the individual characteristic information T1 is information in which the leaves L11, etc. are arranged in a manner based on the configuration information. Specifically, the leaves arranged within the individual characteristic information T1 are arranged in a positional relationship corresponding to the hierarchy of the elements of the configuration information in the individual learning device 111, the distance on the tree structure, the similarity of the data sets, etc. For example, the leaf L11 is arranged at a position where the relative distance from the leaf L12 is close and the relative distance from the leaf L14 is far. That is, it can be said that the distance between the leaves visualizes the proximity of the similarity between the sub-spaces in the feature space.

[0057] In this way, the individual characteristic information T1 does not graphically represent all elements. For example, it can be said that the internal nodes are omitted information. Also, the individual characteristic information T1 can be said to be information in which the connection relationships between elements are also omitted. Further, the leaves L11, etc. do not display the values of the data (classified data, estimated estimated values, etc.) to which they belong. That is, the individual characteristic information T1 is information in which the internal nodes, the data itself, the connection relationships between nodes, etc. are omitted from among the configuration information and the individual output information of the individual learner 111. Also, based on the configuration information, the individual characteristic information T1 arranges each element in a relative positional relationship. From these facts, the user can easily visually recognize and grasp the outline of the individual learner 111 based on the individual characteristic information T1. Also, the user can overview the learning results, data variations, prediction accuracy, etc. in the individual learner 111 based on the individual characteristic information T1.

[0058] Also, in FIG. 5, since the individual characteristic information T1 to Tj are arranged side by side, the user can easily visually recognize and grasp the relationship between different individual learners by overviewing the individual characteristic information T1 to Tj.

[0059] Note that in the example of FIG. 5, adjacent leaves are arranged evenly, but this is just an example. That is, the positional relationship of each leaf arranged in the individual characteristic information T1, etc. may be in a manner based on the configuration information. Also, in the example of FIG. 5, the leaf sizes are not normalized between different individual characteristic information, but the leaf sizes may be normalized between different individual characteristic information.

[0060] Here, the relationship between individual learners will be explained with reference to FIG. 6. FIG. 6 shows the commonality of data belonging between elements by connection lines C121 and C122 for the individual characteristic information T1 and T2 arranged adjacent to each other in FIG. 5. Specifically, in the leaf L12 within the individual characteristic information T1, four data, namely data1, 3, 7, and 8, belong. Also, in the leaf L21 within the individual characteristic information T2, data1 belongs. Further, in the leaf L22 within the individual characteristic information T2, data3, 7, and 8 belong. Therefore, since data1 is common between the leaf L12 of the individual characteristic information T1 and the leaf L21 of the individual characteristic information T2, it is shown that they are connected by the connection line C121. Similarly, since data3, 7, and 8 are common between the leaf L12 of the individual characteristic information T1 and the leaf L22 of the individual characteristic information T2, it is shown that they are connected by the connection line C122. And since the leaf L22 has more data in common with the leaf L12 than the leaf L21, it is shown that the connection line C122 is displayed more prominently than the connection line C121. In FIG. 6, an example is shown where the connection line C122 is emphasized with a thicker line than the connection line C121. Note that the method of emphasizing the connection line may be a difference in color, a difference in line type, or the like. Note also that since the leaf L22 has a larger number of data belonging to it than the leaf L21, it is also shown that the graphic size is larger.

[0061] In FIG. 5, the identification path R1j2 shows an example of a line tracing which element each individual learner classified or estimated a specific piece of data into. For example, when the information processing apparatus 300 receives an operation to display the identification path in a specific piece of data within the data set from the user, it may display the identification path R1j2 in the visualization information 5. Thereby, the user can easily and visually grasp which element each individual learner classified or estimated a specific piece of data into.

[0062] Also, as described above, when the information processing apparatus 300 receives a switching operation between the visualization information and the detailed information of the individual learner from the user, it may display a screen corresponding to the operation. For example, when the information processing apparatus 300 receives a selection operation of the individual characteristic information in the visualization information 5 from the user, it may display the detailed information of the individual learner corresponding to the selected individual characteristic information. Note that when the information processing apparatus 300 receives a switching operation from the display screen of the detailed information to the display screen of the visualization information 5 from the user, it may display the visualization information 5. Note that the information processing apparatus 300 may receive a switching operation, a selection operation, a parameter change operation, etc. from the user via an arbitrary user interface.

[0063] Using FIG. 7, a display example of the detailed information of the configuration information of the individual learner will be described. The detailed information display screen 51 is an example of a decision tree, and displays the root node N00, internal nodes N10, N11, N20, and N21, and leaves L01 to L06. Specifically, the root node N00 indicates that for each data (feature vector) in the input data set, when an arbitrary feature amount ABC is 0.5 or less (YES), it is classified into the internal node N10, and in other cases (NO), it is classified into the internal node N20. Similarly hereinafter, it indicates that each data is classified. For example, the leaf L01 indicates that data1, 7, 9, 13 belong to it. Note that the detailed information display screen 51 may be generated using a known technique.

[0064] Subsequently, when the information processing apparatus 300 receives a selection operation of an arbitrary internal node from the user on the detailed information display screen 51, it may display a rule adjustment screen. An example of the display of the rule adjustment screen 52 will be described with reference to FIG. 8. The rule adjustment screen 52 is an example of a screen that is displayed when an element that classifies data by a rule (threshold) among the elements of a specific individual learner is selected. In other words, the rule adjustment screen 52 is a screen that displays the branch rule (threshold) of the internal node and the feature histogram of the sample data included in the leaf, and accepts a rule adjustment operation. The rule adjustment screen 52 has the value of each data in the input data set (sample value, specific feature amount of the feature vector) on the horizontal axis and the number of data (number of samples) on the vertical axis. The rule adjustment screen 52 indicates that each data is classified into the sample data group 521 and the sample data group 522 at the threshold 523. And the adjustment 524 indicates that the threshold 523 can be changed by the user. Therefore, for example, when the information processing apparatus 300 receives a movement operation of the threshold 523 to the left or right from the user via an arbitrary user interface, it performs a process of changing the rule (threshold) of the corresponding element (internal node). Specifically, as described above, the information processing apparatus 300 executes an adjustment process of an arbitrary parameter, and executes a relearning process of the composite learner using the adjusted parameter. Then, the information processing apparatus 300 may regenerate the visualization information 5 after the relearning, and display (update) the regenerated visualization information 5. Thereby, since the adjustment operation of the user's parameter is promptly reflected, it becomes easier for the user to grasp the characteristics of the composite learner after the adjustment.

[0065] Also, in FIG. 5, when the information processing apparatus 300 receives a selection operation of a specific element (leaf) of specific individual characteristic information in the visualization information 5 from the user, it may display the feature histogram or prediction error of the selected leaf. Using FIG. 9, a display example of the feature histogram of the samples included in the leaf will be described. The feature histogram group 53 includes feature histograms 531, 532, ··· 53n. For example, the feature histogram 531 shows the feature histogram of the x0 dimension in each feature vector of the input data set. Similarly, the feature histograms 532 to 53n respectively show the feature histograms of the x1 to xn-1 dimensions.

[0066] Using FIG. 10, a display example of the prediction error of the samples included in the leaf will be described. The prediction error distribution diagram 54 has the horizontal axis as each vector data of the feature vector (sample vector) and the vertical axis as RSME (Root Mean Squared Error). Note that FIGS. 9 and 10 are merely examples of a screen that displays detailed information of an element (for example, a leaf node) that is one of the configuration information of a specific individual learner included in the composite learner. Therefore, the detailed information of the element may be other displays. Also, the information processing apparatus 300 may switch from the feature histogram group 53 or the prediction error distribution diagram 54 to the display screen of the visualization information 5 according to the user's operation via an arbitrary user interface. Alternatively, the information processing apparatus 300 may display the feature histogram group 53 or the prediction error distribution diagram 54 together with the visualization information 5 according to the user operation. Thus, the user can arbitrarily switch between the overall overview display of the composite learner such as the visualization information 5 and the detailed display of the individual learner in units of elements, and visually recognize the behavior and results of the composite learner efficiently and intuitively.

[0067] Next, a display example of the visualization information 50 will be described with reference to FIG. 11. The visualization information 50 is an example in which information generated when a predetermined data set is input to a composite learning device including a number of individual learning devices using a gradient boosting decision tree is displayed in a three-dimensional space. The visualization information 50 is an example in which the outer shape of the individual characteristic information is omitted. However, the visualization information 50 may display the outer shape of the individual characteristic information, similarly to the visualization information 5 in FIG. 5 above.

[0068] Also, since the example of the visualization information 50 is a gradient boosting method, it overviewly shows that a plurality of individual learning devices are connected in series. For example, an example is shown in which information corresponding to the individual characteristic information of each individual learning device is evenly arranged in layers on the vertical axis (Z-axis). Note that the interval between the layers is not limited to being uniform. The visualization information 50 is an example in which the elements belonging to each layer indicate the number of data belonging thereto by the size, and the magnitude of the prediction error is indicated by the shading of the elements. Also, between adjacent individual learning devices, elements belonging to common data are connected by lines. And the thickness of the connection line indicates the number of data common between the elements. Note that the way of displaying the difference in the number of data belonging to the elements, the prediction error, and the relationship between different individual learning devices is not limited to these, as described above.

[0069] Also, when the information processing device 300 receives a viewpoint change operation of the visualization information 50 from the user via an arbitrary user interface, it may change the display angle of the three-dimensional display and display it on the screen. For example, the information processing device 300 may display the visualization information 50 with the XY-axis plane as the horizontal direction on the screen and the Z-axis as the vertical direction on the screen in response to the viewpoint change operation. Thereby, since the individual characteristic information is aligned and displayed in the vertical axis direction, the user can also make it easier to see the relationship between each individual learning device. Note that the connection lines between the individual characteristic information are not limited to being between adjacent individual learning devices, and may be connected between any individual learning devices. Also, the information processing device 300 may display by swapping the positions of the individual characteristic information according to the user's operation. Furthermore, the information processing device 300 may change the scale of the visualization information 50 according to the user's operation.

[0070] <Example of Visualization Information: Differences between Datasets> Next, a case where differences between datasets are displayed in the visualization information will be described. The information processing apparatus 300 may display, as visualization information, the differences in the learning results obtained by different datasets for the same composite learning apparatus. That is, it is assumed that the information processing apparatus 300 has previously generated first graphic information summarizing the composite output information, the individual output information, the configuration information of each individual learning apparatus, and the relationship between different individual learning apparatuses when the first dataset is input to the first composite learning apparatus. Thereafter, the information processing apparatus 300 generates second graphic information summarizing the composite output information output by the first composite learning apparatus, the individual output information output by each individual learning apparatus, the configuration information of each individual learning apparatus, and the relationship between different individual learning apparatuses when the second dataset is input to the first composite learning apparatus. Then, the information processing apparatus 300 may generate visualization information further including difference information emphasizing the difference between the second graphic information and the first graphic information. Then, the information processing apparatus 300 displays the generated visualization information. In other words, the first visualization information may further include difference information emphasizing the difference between the second graphic information summarizing the composite output information output by the first composite learning apparatus, the individual output information output by each individual learning apparatus, the configuration information of each individual learning apparatus, and the relationship between different individual learning apparatuses when the second dataset is input to the first composite learning apparatus, and the first graphic information.

[0071] Using FIG. 12, an example of the display of differences between data sets in visualization information will be described. The visualization information 5a has an additional connection line C2j3 between the leaf L23 of the individual characteristic information T2 and the leaf Lj3 of the individual characteristic information Tj compared to the visualization information 5. Further, the connection line C2j3 is an example of a line that highlights the differences between data sets. That is, the connection line C2j3 indicates that it is highlighted compared to the connection lines C2j1 and C2j2 to the leaf L23 with the same connection source. In this example, the connection line C2j3 is a thick dashed line, but the difference from other connection lines may be emphasized by color. For example, when the connection lines C2j1 and C2j2 are black or light green, the connection line C2j3 may be a red line. Alternatively, the connection line C2j3 may be represented by a difference in line type. Further, the differences between data sets are not limited to differences in connection lines, and may be the size, number, distance between elements, etc. of elements (visual information).

[0072] Using FIG. 13, an example of the comparative display between different data sets in the same composite learner will be described. The comparative display screen 55 shows an example of the comparative display of the learning results of the first data set (Train data) and the second data set (Test data). For example, as indicated by the increase 553, in the visualization information 552 compared to the visualization information 551 in a specific region (element), the number of data and the error have increased. Further, as indicated by the additional relationship 554, connection lines are added in the visualization information 552 between elements that did not have connection lines of relationships in the visualization information 551.

[0073] Note that the comparative display screen 55 displays the visualization information 551 and the visualization information 552 side by side, but the comparative display is not limited to this. For example, in a state where learning has been performed on a plurality of data sets for the same composite learner and visualization information has been generated, when the information processing apparatus 300 receives a selection operation of an arbitrary plurality of data sets from the user, the comparative display screen 55 may be displayed.

[0074] As a result, the user can easily visually recognize and understand the differences in the learning results of the first and second data sets in the same composite learning device. For example, the user can easily identify the areas with large prediction errors. Therefore, it becomes easier for the user to focus on the analysis and consideration of the composite learning device. For example, it becomes easier to detect overfitting. Therefore, even non-experts in machine learning can easily understand the differences between data sets. And non-experts can accurately inquire of and provide appropriate information to machine learning experts.

[0075] <Example of visualization information: Differences between composite learning devices> In addition, the information processing device 300 may display, as visualization information, the differences in the learning results of different composite learning devices using the same data set. That is, it is assumed that the information processing device 300 has previously generated first graphic information summarizing the composite output information, individual output information, configuration information of each individual learning device, and the relationship between different individual learning devices when the first data set is input to the first composite learning device. Thereafter, the information processing device 300 generates third graphic information summarizing the composite output information output by the second composite learning device, the individual output information output by each individual learning device, the configuration information of each individual learning device, and the relationship between different individual learning devices when the first data set is input to a second composite learning device different from the first composite learning device. Then, the information processing device 300 may generate visualization information further including difference information emphasizing the difference between the third graphic information and the first graphic information. And the information processing device 300 displays the generated visualization information. In other words, the first visualization information may further include difference information emphasizing the difference between the third graphic information summarizing the composite output information output by the second composite learning device, the individual output information output by each individual learning device, the configuration information of each individual learning device, and the relationship between different individual learning devices when the first data set is input to a second composite learning device different from the first composite learning device, and the first graphic information. Note that the manner of displaying the differences between the composite learning devices may be the same as the manner of displaying the differences between the data sets described above.

[0076] In this way, by the visualization method according to the present disclosure, in a machine learning method (ensemble method) that combines a plurality of individual learners, the interpretability, result interpretability, and maintainability of the learned model can be improved. That is, with the visualization information, it is possible to overview and easily visually recognize the internal configuration of the individual learner and the composite learner, the validity of the output, and the individual learning results and the overall learning results. In particular, focusing on the relationship between weak learners, the entire ensemble learner is visualized so as to provide an overview. Therefore, it becomes possible to visually grasp the peculiar behavior of the model by overviewing it.

[0077] Incidentally, the technology according to Patent Document 1 graphs a single index and displays it in a list for each index. Therefore, when the number of individual discriminators and the types of indices are more than a certain number, it is difficult to display them in a list, and it is difficult to grasp the overall picture. Also, the relationship and role between individual discriminators were difficult to intuitively understand.

[0078] On the other hand, by the visualization method according to the present disclosure, even when the number of individual discriminators and the types of indices are more than a certain number, since it is converted into summarized graphic information, it is possible to overview and easily grasp the overall picture of the composite learner and the plurality of individual learners.

[0079] Also, regarding the configuration of the composite learner, etc., by the intuitive rule adjustment, immediate re-learning, and update and display of visualization information based on the re-learning results as described above, even a non-expert in AI or machine learning can easily make appropriate setting changes (fine-tuning). Also, the user can grasp the influence of the setting change on each individual learner in a short time. Also, non-experts can detect suspicious points at an early stage and can easily consult with experts (data scientists, etc.). Also, the visualization information can be used as a clue for error factor analysis during the actual operation of the learned model group of the composite learner.

[0080] Using FIG. 14, an example of the hardware configuration of the information processing apparatus 300 will be described. The information processing apparatus 300 includes a memory 301, a processor 302, a network interface 303, and a display 104. Note that, as described above, the display 104 may be an external device of the information processing apparatus 300.

[0081] The memory 301 is composed of a combination of a volatile memory and a non-volatile memory. The volatile memory is, for example, a volatile storage device such as a RAM (Random Access Memory), and is a storage area for temporarily holding information during the operation of the processor 302. The non-volatile memory is, for example, a non-volatile storage device such as a hard disk or a flash memory. The memory 301 stores at least a computer program in which the visualization processing of the information processing apparatus 300 according to the present disclosure is implemented. Note that the memory 301 may include a storage disposed separately from the processor 302. In this case, the processor 302 may access the memory 301 via an I / O (Input / Output) interface (not shown).

[0082] The processor 302 is a control device that controls each component of the information processing apparatus 300. The processor 302 reads and executes software (computer program) from the memory 301. Thereby, the processor 302 realizes the functions of the model acquisition unit 31, the data acquisition unit 32, the model processing unit 33, the generation unit 34, the output unit 35, the operation reception unit 36, and the update unit 37. That is, the processor 302 performs the visualization processing according to the present disclosure. The processor 302 may be, for example, a microprocessor, an MPU (Multi Processing Unit), or a CPU (Central Processing Unit). Also, the processor 302 may include a plurality of processors.

[0083] The network interface 303 may be used to communicate with a network node. The network interface 303 may include, for example, a network interface card (NIC) compliant with the IEEE 802.3 series. IEEE represents the Institute of Electrical and Electronics Engineers.

[0084] The display 104 is a display device that displays information instructed by the processor 302. The display 104 is, for example, a screen such as a liquid crystal display or an organic electro-luminescence (EL) display.

[0085] <Other embodiments> Note that the visualization method according to the present disclosure is applicable to both regression problems and classification problems. That is, the output unit 35 outputs, as the first visualization information, the result of applying the composite learner to either a predetermined classification problem or a regression problem.

[0086] Note that the above-described embodiment targets boosting among ensemble learning methods, and is configured by connecting decision trees (weak learners) in series. However, the present disclosure is also applicable to other learning methods (bagging and stacking).

[0087] Note that in the case of bagging, the order of the decision trees (individual characteristic information) may be arbitrary. The information processing apparatus 300 may be configured to be able to change the order of the individual characteristic information in response to a user's designated operation. Also, in the case of stacking, since there may be branching and aggregation of decision trees, other displays may be possible.

[0088] Note that in the above-described embodiment, the configuration has been described as hardware, but the present disclosure is not limited thereto. The present disclosure can also be realized by causing a CPU to execute a computer program for any processing.

[0089] In the above example, when the program is loaded into a computer, it includes a set of instructions (or software code) for causing the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD), or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0090] The present disclosure has been described with reference to the embodiments, but the present disclosure is not limited to the above-described embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. And each embodiment can be combined with other embodiments as appropriate.

[0091] Each drawing is merely an illustration for explaining one or more embodiments. Each drawing is not associated with only one specific embodiment, but may be associated with one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with the features or steps shown in one or more other drawings to create, for example, embodiments not explicitly illustrated or described. Not all of the features or steps shown in any one drawing for explaining exemplary embodiments are necessarily essential, and some features or steps may be omitted. The order of the steps described in any drawing may be changed as appropriate.

[0092] Some or all of the above embodiments may be described as follows in the appended claims, but are not limited thereto. (Appended Claim A1) Output means for outputting first visualization information including first graphic information summarizing composite output information output by the first composite learning device, individual output information output by each individual learning device, configuration information of each individual learning device, and the relationship between different individual learning devices, when a first dataset is input to the first composite learning device including a plurality of individual learning devices; An information processing apparatus comprising the same. (Appended Claim A2) The first graphic information is information summarized by converting the composite output information, the individual output information, the configuration information, and the relationship into a graphic on a three-dimensional space. The information processing apparatus according to Appended Claim A1. (Appended Claim A3) The first graphic information is information summarized by arranging, in a layer in the vertical direction with respect to the plane, planar graphic information obtained by converting, for each individual learning device, the individual output information and the configuration information into a graphic on a plane. The information processing apparatus according to Appended Claim A2. (Appended Claim A4) The planar graphic information is information in which visual information converted according to the number of data belonging to each element for a plurality of elements included in the configuration information is arranged in a manner based on the configuration information, for each individual learning device. The information processing apparatus according to Appended Claim A3. (Appended Claim A5) The first graphic information includes a line connecting elements having common data among the planar graphic information in different individual learning devices. The information processing apparatus according to Appended Claim A4. (Appended Claim A6) The output means outputs, as the first visualization information, a result of applying the first composite learning device to any one of a predetermined classification problem or a regression problem. The information processing apparatus according to any one of Appended Claims A1 to A5. (Appendix A7) The first visualization information further includes difference information that emphasizes the difference between the second graphic information summarizing the composite output information output by the first composite learning device, the individual output information output by each individual learning device, the configuration information of each individual learning device, and the relationship between different individual learning devices when a second dataset is input to the first composite learning device, and the first graphic information The information processing apparatus according to any one of Appendices A1 to A6 (Appendix A8) The first visualization information further includes difference information that emphasizes the difference between the third graphic information summarizing the composite output information output by the second composite learning device, the individual output information output by each individual learning device, the configuration information of each individual learning device, and the relationship between different individual learning devices when a first dataset is input to a second composite learning device different from the first composite learning device, and the first graphic information The information processing apparatus according to any one of Appendices A1 to A6 (Appendix A9) The composite learning device includes a trained model generated by ensemble learning using a plurality of weak learning devices as the plurality of individual learning devices The information processing apparatus according to any one of Appendices A1 to A8 (Appendix A10) The configuration information includes at least one of a division result of the input feature space of the individual learning device into subspaces when the first dataset is input, a distance between different subspaces, and hyperparameters of each individual learning device The information processing apparatus according to any one of Appendices A1 to A9 (Appendix A11) The individual output information includes at least one of the number of data, the output result, and the error from the target value for each subspace of the input feature space of the individual learning device when the first dataset is input The information processing apparatus according to any one of Appendices A1 to A10 (Appendix A12) The relationship includes at least one or more of the partial spaces in different individual learners and the similarity of the hyperparameters of each individual learner. The information processing apparatus according to any one of Appendices A1 to A11. (Appendix A13) The individual learner includes a learned model generated by a method including at least one or more of a decision tree, a support vector machine, and a neural network that perform multi-class classification or regression. The information processing apparatus according to any one of Appendices A1 to A12. (Appendix A14) The information processing apparatus Performs at least one of the switching process between the first visualization information and the detailed information of the individual learner, the adjustment process of arbitrary parameters, and the re-learning process of the composite learner in response to an operation input from a user. The information processing apparatus according to any one of Appendices A1 to A13. (Appendix B1) A computer Obtains a first dataset, Outputs first visualization information including first graphic information summarizing composite output information output by the first composite learner, individual output information output by each individual learner, configuration information of each individual learner, and the relationship between different individual learners when the first dataset is input to the first composite learner including a plurality of individual learners. Visualization method. (Appendix C1) The process of obtaining a first dataset, The process of outputting first visualization information including first graphic information summarizing composite output information output by the first composite learner, individual output information output by each individual learner, configuration information of each individual learner, and the relationship between different individual learners when the first dataset is input to the first composite learner including a plurality of individual learners, A visualization program for causing a computer to execute.

[0093] Some or all of the elements (e.g., configuration and function) described in Supplementary Notes A2 to A14 that are subordinate to Supplementary Note A1 {e.g., device} may be subordinate to Supplementary Note B1 {e.g., method} and Supplementary Note C1 {e.g., program} in the same subordinate relationship as Supplementary Notes A2 to A14. Some or all of the elements described in any supplementary note may be applicable to various hardware, software, recording means for recording software, systems, and methods.

Explanation of Reference Signs

[0094] 1 Information processing apparatus 10 Output unit 1000 Visualization system 100 Storage device 110 Composite learner 111 Individual learner 11j Individual learner 1i0 Composite learner 1i1 Individual learner 1ik Individual learner 200 Storage device 210 Dataset 2m0 Dataset 300 Information processing apparatus 31 Model acquisition unit 32 Data acquisition unit 33 Model processing unit 34 Generation unit 35 Output unit 36 Operation reception unit 37 Update unit 5 Visualization information 5a Visualization information T1 Individual characteristic information T2 Individual characteristic information Tj Individual characteristic information L11 Leaf L12 Leaf L13 Leaf L14 Leaf L21 Leaf L22 Leaf L23 Leaf Lj1 Leaf Lj2 Leaf Lj3 Leaf C121 Connection Line C122 Connection Line C2j1 Connection Line C2j2 Connection Line C2j3 Connection Line R1j2 Identification Route 51 Detailed Information Display Screen N00 Root Node N10 Internal Node N11 Internal Node N20 Internal Node N21 Internal Node L01 Leaf L02 Leaf L03 Leaf L04 Leaf L05 Leaf L06 Leaf 52 Rule Adjustment Screen 521 Sample Data Group 522 Sample Data Group 523 Threshold Value 524 Adjustment 53 Feature Quantity Histogram Group 531 Feature Quantity Histogram 532 Feature Quantity Histogram 53n Feature Quantity Histogram 54 Prediction Error Distribution Diagram 50 Visualization Information 55 Comparison Display Screen 551 Visualization Information 552 Visualization Information 553 Increase 554 Addition of Relationship 301 Memory 302 Processor 303 Network Interface 304 Display

Claims

1. Output means for outputting first visualization information including first graphic information summarizing composite output information output by the first composite learning device, individual output information output by each individual learning device, configuration information of each individual learning device, and the relationship between different individual learning devices when a first dataset is input to the first composite learning device including a plurality of individual learning devices; An information processing apparatus comprising the same.

2. The first graphic information is information summarized by converting the composite output information, the individual output information, the configuration information, and the relationship into a graphic on a three-dimensional space. The information processing apparatus according to claim 1.

3. The first graphic information is information summarized by arranging, in a layered manner in a vertical direction with respect to the plane, planar graphic information obtained by converting, for each individual learning device, into a graphic on a plane based on the individual output information and the configuration information. The information processing apparatus according to claim 2.

4. The planar graphic information is information in which visual information converted according to the number of data belonging to each of a plurality of elements included in the configuration information is arranged in a manner based on the configuration information for each individual learning device. The information processing apparatus according to claim 3.

5. The first graphic information includes a line connecting elements having common data among the planar graphic information in different individual learning devices. The information processing apparatus according to claim 4.

6. The output means outputs, as the first visualization information, a result of applying the first composite learning device to any one of a predetermined classification problem or a regression problem. The information processing apparatus according to claim 1 or 2.

7. The first visualization information further includes difference information emphasizing the difference between second graphic information summarizing composite output information output by the first composite learning device, individual output information output by each individual learning device, configuration information of each individual learning device, and the relationship between different individual learning devices when a second dataset is input to the first composite learning device and the first graphic information. The information processing apparatus according to claim 1 or 2.

8. The first visualization information further includes difference information that emphasizes the difference between the third graphic information summarizing the composite output information output by the second composite learning device, the individual output information output by each individual learning device, the configuration information of each individual learning device, and the relationship between different individual learning devices when the first dataset is input to a second composite learning device different from the first composite learning device, and the first graphic information. The information processing apparatus according to claim 1 or 2. **Claim 9** A computer acquires a first dataset, and outputs first visualization information including first graphic information summarizing the composite output information output by the first composite learning device, the individual output information output by each individual learning device, the configuration information of each individual learning device, and the relationship between different individual learning devices when the first dataset is input to the first composite learning device including a plurality of individual learning devices. Visualization method. **Claim 10** A process of acquiring a first dataset, and a process of outputting first visualization information including first graphic information summarizing the composite output information output by the first composite learning device, the individual output information output by each individual learning device, the configuration information of each individual learning device, and the relationship between different individual learning devices when the first dataset is input to the first composite learning device including a plurality of individual learning devices, A visualization program that causes a computer to execute.

Images