Artificial intelligence structure learning system, method, and program

Abstract

To enable generation of an artificial intelligence model of a favorable artificial intelligence structure.SOLUTION: A system includes a storage device and a processing device. The storage device stores artificial intelligence structure database, artificial intelligence structure constitution component database, and a data set for learning. The processing device sequentially selects the artificial intelligence structure model, and calculates accuracy or an index which replaces the accuracy, and when this is equal to or greater than a threshold value, it adopts a corresponding learned artificial intelligence model, and, when, reversely, this is not, it sequentially generates an artificial intelligence structure model to which a selected artificial intelligence structure constitution component is added, performs re-learning using the data set for learning, and registers the artificial intelligence structure model corresponding to the learned artificial intelligence model in the artificial intelligence structure database. When the learned artificial intelligence model in which the index for measuring the performance of the prediction is equal to or greater than a predetermined threshold value is generated, the processing device adopts the learned artificial intelligence model.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This disclosure relates to a technique for learning a suitable artificial intelligence structure. [Background technology]

[0002] There are systems that select a suitable structure for artificial intelligence according to the purpose of analysis, rather than fixing the structure of the artificial intelligence in advance. For example, the artificial intelligence module development system disclosed in Patent Document 1 has a library of pre-trained neural networks such as AE (Auto Encoder), CNN (Convolutional Neural Network), and LSTM (Long Short-Term Memory), and the user selects a pre-trained neural network from the library that is suitable for the analysis purpose and target selected by the user. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2019-003603 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Artificial intelligence has a structure that connects various components, and the parameters of each part are determined through learning. For example, a convolutional neural network has a structure that connects components such as convolutional layers, pooling layers, recurrent layers, fully connected layers, and activation layers such as the softmax function. The weights of the connections are then determined through learning. If the structure of artificial intelligence itself is different, the performance that can be achieved as a result of learning will also be different.

[0005] However, the system disclosed in Patent Document 1 only prepares basic pre-learning neural networks such as AE, CNN, and LSTM as a library in advance, and selects one of them according to the analysis purpose and analysis target, and cannot necessarily generate an artificial intelligence model with a suitable artificial intelligence structure.

[0006] One object included in the present disclosure is to provide a technology that enables generation of a suitable artificial intelligence model by learning the artificial intelligence structure itself. [Means for Solving the Problems]

[0007] An artificial intelligence structure learning system according to one aspect of the present disclosure comprises a storage device and a processing device, the storage device storing an artificial intelligence structure database containing an artificial intelligence structure model having a predetermined artificial intelligence structure, an artificial intelligence structure component database containing artificial intelligence structure components that can constitute a part of the artificial intelligence structure, and a training dataset and a validation dataset (hereinafter referred to as the learning dataset), the processing device sequentially selects an artificial intelligence structure model from the artificial intelligence structure database, performs a test on judgment data (unknown data) using the artificial intelligence structure model, calculates a first evaluation index for the result, and if the first evaluation index is equal to or greater than a predetermined first evaluation index threshold, and a second evaluation index obtained from the learning result of the artificial intelligence structure model is equal to or greater than a predetermined second evaluation index threshold, the artificial intelligence structure model is adopted, and the If there is no artificial intelligence structure model in the artificial intelligence structure database in which the first and second evaluation indicators are equal to or greater than the thresholds for the first and second evaluation indicators, respectively, an artificial intelligence structure model is generated by adding artificial intelligence structure components sequentially selected from the artificial intelligence structure component database based on any of the artificial intelligence structure models. The generated artificial intelligence structure model is then trained using the training dataset to generate a trained artificial intelligence model. The artificial intelligence structure model corresponding to the trained artificial intelligence model is registered in the artificial intelligence structure database, and the second evaluation indicator of the artificial intelligence structure model is calculated. The testing of the judgment data (unknown data) is then performed again using the artificial intelligence structure model. If a trained artificial intelligence model is generated in which each evaluation indicator is equal to or greater than the predetermined evaluation indicator threshold, the trained artificial intelligence model is adopted. If either evaluation indicator is below the evaluation indicator threshold, the training of the artificial intelligence structure model itself is repeated. If, after repeating training for a certain period of time or a certain number of times, a result equal to or greater than the evaluation indicator threshold is not obtained, the threshold is either revised downwards, or the artificial intelligence model in which the product of the first and second evaluation indicators is maximized during the repeated training is adopted.

[0008] An artificial intelligence structure learning method according to one aspect included in the present disclosure is an artificial intelligence structure learning method by an apparatus having a storage device and a processing device, wherein the storage device stores an artificial intelligence structure database in which artificial intelligence structure models each having a predetermined artificial intelligence structure are stored, an artificial intelligence structure component database in which artificial intelligence structure components capable of constituting a part of an artificial intelligence structure are stored, and a learning data set. The processing device sequentially selects an artificial intelligence structure model from the artificial intelligence structure database, performs a test on determination data (unknown data) using the artificial intelligence structure model, calculates a first evaluation index related to the result, adopts the artificial intelligence structure model if the first evaluation index is equal to or greater than a predetermined first evaluation index threshold, and a second evaluation index obtained from the learning result of the artificial intelligence structure model is equal to or greater than a predetermined second evaluation index threshold. If there is no artificial intelligence structure model in the artificial intelligence structure database for which the first evaluation index and the second evaluation index are respectively equal to or greater than the first evaluation index threshold and the second evaluation index threshold, an artificial intelligence structure model added with artificial intelligence structure components sequentially selected from the artificial intelligence structure component database is generated based on any one of the artificial intelligence structure models, the generated artificial intelligence structure model is trained using the learning data set to generate a trained artificial intelligence model, the artificial intelligence structure model corresponding to the trained artificial intelligence model is registered in the artificial intelligence structure database, the second evaluation index of the artificial intelligence structure model is calculated, the test on the determination data (unknown data) is re-performed using the artificial intelligence structure model, and when a trained artificial intelligence model whose respective evaluation indices are equal to or greater than predetermined evaluation index thresholds is generated, the trained artificial intelligence model is adopted. If either of the evaluation indices is less than the evaluation index threshold, the learning of the artificial intelligence structure model itself is repeated. If a result equal to or greater than the evaluation index threshold is not obtained even after repeating the learning for a certain period of time or a certain number of times, the threshold is corrected downward, or the artificial intelligence model with the maximum product of the first evaluation index and the second evaluation index among the repeated processes is adopted.

[0009] A program according to one aspect of this disclosure is a program for causing a device having a storage device and a processing device to learn an artificial intelligence structure, wherein the storage device stores an artificial intelligence structure database containing an artificial intelligence structure model having a predetermined artificial intelligence structure, an artificial intelligence structure component database containing artificial intelligence structure components that can constitute a part of the artificial intelligence structure, and a learning dataset, and when executed by the processing device, it sequentially selects an artificial intelligence structure model from the artificial intelligence structure database, performs a test on judgment data (unknown data) using the artificial intelligence structure model, calculates a first evaluation index for the result, and if the first evaluation index is equal to or greater than a predetermined first evaluation index threshold, and a second evaluation index obtained from the learning result of the artificial intelligence structure model is equal to or greater than a predetermined second evaluation index threshold, the artificial intelligence structure model is adopted, and If no artificial intelligence structure model exists in the artificial intelligence structure database in which the first and second evaluation indicators are equal to or greater than the thresholds for the first and second evaluation indicators, respectively, an artificial intelligence structure model is generated by adding artificial intelligence structure components sequentially selected from the artificial intelligence structure component database based on any existing artificial intelligence structure model. The generated artificial intelligence structure model is then trained using the training dataset to generate a trained artificial intelligence model. The artificial intelligence structure model corresponding to the trained artificial intelligence model is registered in the artificial intelligence structure database, and the second evaluation indicator for the artificial intelligence structure model is calculated. The testing of the judgment data (unknown data) is then performed again using the artificial intelligence structure model. If a trained artificial intelligence model is generated in which each evaluation indicator is equal to or greater than the predetermined evaluation indicator threshold, the trained artificial intelligence model is adopted. If either evaluation indicator is below the evaluation indicator threshold, the training of the artificial intelligence structure model itself is repeated. If, after repeating training for a certain period of time or a certain number of times, a result equal to or greater than the evaluation indicator threshold is not obtained, the threshold is either lowered, or the artificial intelligence model in which the product of the first and second evaluation indicators is maximized during the repeated training is adopted. [Effects of the Invention]

[0010] According to this disclosure, it is possible to generate an artificial intelligence model with a suitable artificial intelligence structure. [Brief explanation of the drawing]

[0011] [Figure 1] This is a functional block diagram showing an example configuration of an artificial intelligence structure learning system. [Figure 2] This is a conceptual diagram illustrating an overall processing example of an artificial intelligence structure learning system. [Figure 3] This is a flowchart illustrating the overall processing of an artificial intelligence structure learning system. [Figure 4] This is the first flowchart illustrating the process of automatically generating structures using artificial intelligence. [Figure 5] This is a second flowchart illustrating the process of automatically generating structures using artificial intelligence. [Figure 6] This is the first conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure. [Figure 7] This is the second conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure. [Figure 8] This is a third conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure. [Figure 9] This is the fourth conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure. [Figure 10] This is the fifth conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure. [Figure 11] This is the sixth conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure. [Figure 12] This is the seventh conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure. [Figure 13] This diagram shows how to add another node between any two nodes. [Figure 14] This is the first conceptual diagram showing the addition of an edge. [Figure 15] This diagram shows the path from input (I) to output (O) in Figure 14. [Figure 16] This is a second conceptual diagram showing the addition of an edge. [Figure 17] This diagram shows the path from input (I) to output (O) in Figure 16. [Figure 18] This is a conceptual diagram illustrating a method for combining the outputs from multiple nodes into a single output. [Figure 19] This graph illustrates the ROC curve and its AUC value, which is the area under the lower part of the curve. [Figure 20] This is a conceptual diagram illustrating a management table for managing evaluation metrics for each artificial intelligence structure. [Figure 21] This graph illustrates the results of the sample data using a histogram. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the drawings. Note that the embodiments described below are not intended to limit the claims, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0013] In the diagrams illustrating the embodiments, the same reference numerals are used to denote parts with the same function, and repeated explanations of such parts are omitted.

[0014] Figure 1 is a functional block diagram showing an example configuration of an artificial intelligence structure learning system.

[0015] The artificial intelligence structure learning system 1 comprises a processing unit 11, a main memory 12, an auxiliary memory 13, a communication device 14, an input device 15, and an output device 16. The processing unit 11, the main memory 12, the auxiliary memory 13, the communication device 14, the input device 15, and the output device 16 are connected to each other via a bus 17 so as to be able to communicate with each other.

[0016] The processing unit 11 is composed of, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), etc. The processing unit 11 reads and executes various programs stored in the main memory 12 or auxiliary memory 13, thereby realizing various functions of the artificial intelligence structure learning system 1.

[0017] The main memory 12 is a device for storing programs and data, and is, for example, Random Access Memory (RAM), Read Only Memory (ROM), or Non-Volatile RAM (NVRAM).

[0018] The auxiliary storage device 13 is, for example, a Hard Disc Drive (HDD), Solid State Drive (SSD), storage system, Integrated Circuit (IC) card, Secure Digital (SD) memory card, or a reading and writing device for optical recording media (Compact Disc (CD), Digital Versatile Disc (DVD), etc.), or the storage area of ​​a cloud server.

[0019] The cache and registers of the processing unit 11, the main memory 12, and the auxiliary memory 13 are sometimes collectively referred to as memory. Information processing in the artificial intelligence structure learning system 1 is performed when the necessary programs and data are loaded into memory and executed. Various thresholds, such as the first evaluation index threshold and the second evaluation index difference threshold described later, may also be stored in memory. Programs, databases, datasets, etc., may also be stored in memory.

[0020] The communication device 14 is a wired or wireless communication interface that enables communication with other devices via communication means such as a Local Area Network (LAN) or the Internet, and is, for example, a Network Interface Card (NIC), a wireless communication module, a Universal Serial Interface (USB) module, or a serial communication module.

[0021] The input device 15 is a device that receives input from the user. The input device 15 may be, for example, a keyboard, mouse, touch panel, card reader, or voice input device.

[0022] The output device 16 is a device that provides the user with various information such as processing progress and processing results. The output device 16 may be, for example, a screen display device (Liquid Crystal Display (LCD), Head Mounted Display (HMD), etc.), an audio output device, a printing device, etc.

[0023] Furthermore, the artificial intelligence structure learning system 1 may be configured to input and output information to and from other devices via the communication device 14.

[0024] Figure 2 is a conceptual diagram showing an overall processing example of artificial intelligence structure learning system 1.

[0025] The artificial intelligence structure learning system 1 includes an artificial intelligence structure database 100 and an artificial intelligence structure component database 106. These databases may be stored in the main memory 12 or auxiliary memory 13, or they may be stored in an external memory device that is communicably connected to the artificial intelligence structure learning system 1 via a communication device 14.

[0026] An artificial intelligence structure model refers to, for example, a neural network model (NN model). In this case, the artificial intelligence structure corresponds to a neural network. An artificial intelligence structure model has a predetermined artificial intelligence structure. In Figure 2, n artificial intelligence structure models, from model 1 to model n, are stored in the artificial intelligence structure database 100. New artificial intelligence structures may be added to the artificial intelligence structure database 100.

[0027] Artificial intelligence structure components refer to the parts that make up an artificial intelligence structure. In other words, they refer to components that can constitute a part of an artificial intelligence structure. An artificial intelligence structure is formed by combining one or more artificial intelligence structure components. The artificial intelligence structure is stored in the artificial intelligence structure database 100. On the other hand, artificial intelligence structure components are stored in the artificial intelligence structure component database 106.

[0028] An artificial intelligence structural component (module) 106b is composed of one or more nodes 106a connected by edges (directed arrows that connect components). Nodes here include functions, etc. Examples of nodes 106a include 3x3 convolution, 3x1 convolution, 3x3 max pooling, 3x3 average pooling, etc. However, it is not limited to these. The artificial intelligence structural component itself may be a module composed of multiple nodes.

[0029] The artificial intelligence structural component database 106 stores node 106a and artificial intelligence structural component (module) 106b. New nodes and new artificial intelligence structural components may be added to the artificial intelligence structural component database 106.

[0030] A training dataset 101, a validation dataset 102, a test dataset 103, and a decision dataset (unknown data) 111 are prepared for machine learning. The training dataset 101, the validation dataset 102, the test dataset 103, and the decision dataset (unknown data) 111 may be stored in the main memory 12 or auxiliary memory 13, or they may be stored in an external memory device that is communicably connected to the artificial intelligence structure learning system 1 via a communication device 14.

[0031] The artificial intelligence structure learning system 1 loads the artificial intelligence structure model 104 from the artificial intelligence structure database 100. The processing unit 11 performs machine learning using the training dataset 101 on the artificial intelligence structure model 104. The processing unit 11 verifies the learning using the validation dataset 102 on the artificial intelligence structure model 104. The processing unit 11 tests the machine learning using the test dataset 103 on the artificial intelligence structure model 104. Note that the training dataset 101, validation dataset 102, and test dataset 103 may be freely divided and used from the datasets from which data collection and processing have been performed.

[0032] Refer to Figure 1 as well. The processing unit 11 reads and executes a program stored in the main memory 12 or auxiliary memory 13 to realize the determination unit 105, the artificial intelligence structure automatic generation unit 107, the artificial intelligence structure consistency check unit 108, the artificial intelligence structure input / output dimension adjustment unit 109, and the artificial intelligence structure component database registration and addition unit 110.

[0033] The determination unit 105 inputs the training dataset 101 and the validation dataset 102 into the artificial intelligence structure model 104 and performs training 112 to output a trained artificial intelligence model 113, a second evaluation index 114 representing the performance of the trained model, and a first evaluation index 115 obtained as a result of inputting determination data (unknown data) 111 into the trained model. The determination is made, for example, by comparing the first evaluation index 115 calculated for each set of determination data (unknown data), the second evaluation index 114 calculated when training the artificial intelligence structure model 104 112, and predetermined thresholds for each evaluation index. The comparison between the first evaluation index and the second evaluation index and their respective thresholds may be performed independently, or the threshold may be the product of both. Here, the first evaluation index 115 is an index calculated for each set of determination data (unknown data), and for example, probability or the distance from the cluster centroid (center) in the Euclidean space of cluster analysis may also be used as an index. The second evaluation indicator, 114, will be discussed later.

[0034] The artificial intelligence structure automatic generation unit 107 automatically generates a new artificial intelligence structure by adding artificial intelligence structure components stored in the artificial intelligence structure component database 106 to the artificial intelligence structure model 104, etc. An example of the automatic generation process of the artificial intelligence structure will be described later with reference to Figure 4 and subsequent figures.

[0035] The artificial intelligence structure consistency check unit 108 checks that the automatically generated artificial intelligence structure is not the same as an artificial intelligence structure previously tried. An artificial intelligence structure can be represented by a directed acyclic graph (DAG) in which artificial intelligence structure components (nodes) are connected by directed edges. The artificial intelligence structure consistency check unit 108 states that the processing device topologically sorts the artificial intelligence structures obtained by adding artificial intelligence structure components to the artificial intelligence structure model, and skips any structure that is the same as any of the artificial intelligence structures previously tried, and makes a judgment using a second evaluation index if it is not the same.

[0036] Topological sorting means ordering each node so that every node precedes the node that lies beyond its output edge. Topological sorting can transform complex graph structures into a unique form, allowing us to check whether we have tried the same structure before.

[0037] Furthermore, the artificial intelligence structure consistency check unit 108 may also have a function to check the consistency between the artificial intelligence structure to be combined and the artificial intelligence structure components. When adding artificial intelligence structure components to an artificial intelligence structure model, the addition may not be successful if there is an inconsistency between the format of the input data and the format of the output data of the combined destination. Therefore, the artificial intelligence structure consistency check unit 108 has a function to check whether the artificial intelligence structure components can be added without problems from the perspective of the consistency of the input and output data.

[0038] The artificial intelligence structure input / output dimension adjustment unit 109 adjusts the data format between the input of the artificial intelligence structure component and the output of the artificial intelligence structure model connected to that input if the data formats of the input and the output of the artificial intelligence structure model connected to that input do not match when an artificial intelligence structure component is added to the artificial intelligence structure model. It also adjusts the data format between the input and the output of the artificial intelligence structure component and the input of the artificial intelligence structure model connected to that output if the data formats of the input and the output do not match.

[0039] The artificial intelligence structure input / output dimension adjustment unit 109 adjusts the input or output data format to enable processing of data with different shapes or layer depths when adding artificial intelligence structure components. Adjustment methods include, but are not limited to, compression using discrete cosine transform, convolution, and pooling.

[0040] The artificial intelligence structure component database registration addition unit 110 adds a portion (module) composed of multiple artificial intelligence structure components (nodes) to the artificial intelligence structure component database if the second evaluation index is better than or equal to a predetermined second evaluation index difference threshold than the second evaluation index when the portion (module) is absent.

[0041] The second evaluation metric is an indicator of the capabilities of each artificial intelligence model, and specific examples will be discussed later.

[0042] The artificial intelligence structure component database registration addition unit 110 adds a second evaluation index (σ) for the artificial intelligence structure after adding a new node (component) or edge (directed arrow connecting components). n ) is the second evaluation metric (σ) of the pre-trained artificial intelligence structure before adding the node or edge. n-1 ) is compared with the difference value (σ) of a certain second evaluation index. Δ Improvement exceeding σ Δ <σ n -σ n-1 If a combination of the added node and the nodes before or after it, or the added edge and the nodes at both ends of it, is obtained, the combination is extracted as a new module and added to the artificial intelligence structural component database. However, if the module is equivalent to another module that has already been registered, the additional registration is not performed to avoid duplicate registration. Modules added to the artificial intelligence structural component database are used as components for generating new artificial intelligence structures, just like other artificial intelligence structural components.

[0043] Figure 3 is a flowchart illustrating the overall processing of the artificial intelligence structure learning system 1. In this overall processing, the decision of whether or not to adopt the artificial intelligence structure is made based on a first evaluation metric of the judgment data (unknown data).

[0044] Furthermore, the artificial intelligence structure database 100 may store a second evaluation index obtained by a trained artificial intelligence model corresponding to the artificial intelligence structure model, and the average judgment time required for the trained artificial intelligence model to calculate the second evaluation index.

[0045] The processing unit 11 receives judgment data (unknown data) (S101). The processing unit 11 selects one artificial intelligence structure model from the artificial intelligence structure database 100 (S102). The processing unit 11 may select from the artificial intelligence structure database 100 models in which the second evaluation index (σ) is greater than the second evaluation index threshold (σ1), starting with models with a small number of layers (or nodes) or models with a small average judgment time required to calculate the second evaluation index, and then sequentially selecting models with a larger number of layers (or nodes) or models with a larger average judgment time required to calculate the second evaluation index. This allows the system to search for an artificial intelligence structure that can obtain the desired evaluation index in the shortest possible time.

[0046] The processing unit 11 calculates a first evaluation index when determining the judgment data (unknown data) for the loaded artificial intelligence structure model. The processing unit 11 determines whether the first evaluation index (ρ) is equal to or greater than the first evaluation index threshold (ρC) (S103). The evaluation index and its threshold may be stored in memory in advance.

[0047] If the first evaluation metric is above the threshold (S103: YES), the process proceeds to step S107. If the first evaluation metric is not above the threshold (S103: NO), the process proceeds to step S104.

[0048] In step S104, the processing unit 11 determines whether all artificial intelligence structural models have been tried. If all artificial intelligence structural models have been tried (S104: YES), the process proceeds to step S105. If not all artificial intelligence structural models have been tried (S104: NO), the process returns to step S102.

[0049] In step S105, the processing device 11 determines the initial artificial intelligence structure model with the highest first evaluation index as the initial model NN0.

[0050] In step S106, the processing device 11 executes an artificial intelligence structure automatic generation process. Details of the artificial intelligence structure automatic generation process will be described later.

[0051] In step S107, the processing device 11 adopts the trained artificial intelligence model corresponding to the artificial intelligence structure model of which the first evaluation index is equal to or greater than the first evaluation index threshold.

[0052] Figure 4 is a first flowchart illustrating the artificial intelligence structure automatic generation process. In this artificial intelligence structure automatic generation process, certainty and accuracy are adopted as the first evaluation index and the second evaluation index respectively.

[0053] First, the processing device 11 performs machine learning on the artificial intelligence structure model 104 using the datasets 101, 102, and 103 (S201). The artificial intelligence structure model after this machine learning is defined as a trained model 104a.

[0054] The processing device 11 inputs determination data (unknown data) 111 to the trained model 104a. The processing device 11 calculates the first evaluation index ρ based on the output data from the trained model 104a. The processing device 11 determines that the first evaluation index ρ is a predetermined first evaluation index threshold ρ C or more, and whether the second evaluation index σ is a second evaluation index threshold σ C or more (S202). It is determined that the first evaluation index ρ is a predetermined first evaluation index threshold ρ C or more, and the second evaluation index σ is a predetermined second evaluation index threshold σ CIf the above is true, the process proceeds to step S203. In step S203, the result of the determination in step S202 is output. As a result, the trained model 104a at this point, which is the trained artificial intelligence model corresponding to the artificial intelligence structural model, is adopted. At this time, the first evaluation index and the second evaluation index may be compared with the threshold independently, but the product of the first evaluation index and the second evaluation index (ρ·σ) may be compared with the product of their respective thresholds (ρ C ·σ C ) can be compared to this.

[0055] In step S202, the first evaluation index ρ is the predetermined first evaluation index threshold ρ C If it is less than or the second evaluation index σ is less than the second evaluation index threshold σ C If the value is less than the given value, a new structure (node) is added to the previous (nth) artificial intelligence structure model to generate the (n+1th) artificial intelligence structure model. Then, the processing unit 11 reads data again from datasets 101, 102, and 103 onto the generated (n+1th) artificial intelligence structure model and performs machine learning. This artificial intelligence structure model after machine learning is designated as the trained model 104b.

[0056] The processing unit 11 may register the (n+1)th artificial intelligence structure model, i.e., the artificial intelligence structure model corresponding to the trained artificial intelligence model, in the artificial intelligence structure database 100.

[0057] The processing unit 11 inputs the judgment data (unknown data) 111 into the trained model 104b. The processing unit 11 calculates the first evaluation index ρ based on the output data from the trained model 104b. The processing unit 11 determines that the first evaluation index ρ is equal to a predetermined first evaluation index threshold ρ C Whether or not the second evaluation index σ is above the second evaluation index threshold σ C It is determined whether or not the first evaluation index ρ is equal to the predetermined first evaluation index threshold ρ. C The above conditions are met, and the second evaluation index σ is the predetermined second evaluation index threshold σ CIf the above conditions are met, the process proceeds to step S203. In step S203, the result of the determination in step S206 is output. As a result, the trained model 104b at this point, which is the trained artificial intelligence model corresponding to the artificial intelligence structural model, is adopted.

[0058] In step S206, the first evaluation index ρ is set to a predetermined first evaluation index threshold ρ C If it is less than or the second evaluation index σ is less than the second evaluation index threshold σ C If the value is less than the specified value, the process in steps S204 to S206 is repeated as described above.

[0059] In the above, the first evaluation metric may be accuracy, and the second evaluation metric may be precision, but the first and / or second evaluation metric may be changed dynamically. For example, the processing unit 11 may obtain information from the user regarding the nature of the prediction problem and set the first and / or second evaluation metric based on this information. Information regarding the nature of the prediction problem refers to information indicating what and how to predict using the trained model. For example, when using a trained model for food inspection or cancer screening, the prediction targets are "food defects" or "presence or absence of cancer," and the problem has the characteristic that predictions should be made in a way that does not overlook anything. On the other hand, in a problem such as the discovery of mineral veins, the drilling costs would be enormous if all possible locations were detected, so it is necessary to determine only the locations that are truly likely to be found. It is also possible to dynamically change the evaluation metric according to the characteristics of each problem. The F-score may also be a weighted F-measure rather than just a simple F-score.

[0060] Furthermore, if the first evaluation metric is the accuracy of the prediction for the judgment data (unknown data) to be predicted, the processing unit 11 may use the trained artificial intelligence model for the prediction process for the prediction target if the accuracy of the prediction for the judgment data (unknown data) by the trained artificial intelligence model corresponding to the artificial intelligence structure model selected from the artificial intelligence structure database 100 is equal to or greater than the threshold of the first evaluation metric.

[0061] Figure 5 is a second flowchart illustrating an artificial intelligence-based automatic structure generation process. In this artificial intelligence-based automatic structure generation process, the F-score is used as the second evaluation metric.

[0062] First, let's explain the F-score. The F-score (F-measure) represents the harmonic mean of recall and precision. The formula for calculating the F-score is as follows:

[0063] F=2·Recall·Precision / (Recall+Precision)

[0064] The F-score is calculated using the test dataset 103.

[0065] In step S301, the processing unit 11 loads the initial model (NN0) from the artificial intelligence structure database 100 and performs machine learning using the initial model (NN0). The processing unit 11 inputs the test dataset 103 into the trained initial model (NN0) and calculates the F-score based on its output values. This F-score is F 0_max Let's proceed as follows. We will repeat the training and evaluation using F-scores for n=1 and m=1 to k.

[0066] In step S302, the processing unit 11 retrieves the mth node from the artificial intelligence structural component database 106 and the model NN (n-1) Add it to the nth position. The model after this addition is called NN. n This will serve as a model.

[0067] In step S303, the processing unit 11 processes the (n-1)*k+m-th model NN (n-1)*k+m Machine learning is performed using the following. The processing unit 11 uses the trained model NN. (n-1)*k+m Input the test dataset 103 and calculate the F-score based on its output values. (n-1)*k+m Let's assume that.

[0068] In step S304, the processing unit 11 Fn_max and F (n-1)*k+m Compare with F. n_max <F (n-1)*k+m If this is the case (S304:YES), the process proceeds to step S305. n_max ≥F (n-1)*k+m If this is the case (S304:NO), the process proceeds to step S306.

[0069] In step S305, the processing unit 11 F n_max Update the value of F at this point. More specifically, first update the value of F at this point. n_max F n_max_old Evacuate to F. n_max The value of F (n-1)*k+m The value is updated. Then the processing unit 11 processes the model NN. n_max is NN n_max_old Save as a model. Model NN n Model NN n_max Save it as.

[0070] In step S306, the processing unit 11 increases the value of m by 1.

[0071] In step S307, the processing unit 11 determines whether m > k. If m > k (S307: YES), the process proceeds to step S308. If m > k is not true (S307: NO), the process returns to step S302.

[0072] In step S308, the processing device 11 F n_max >F (n-1)_max Determine whether or not it is true. F n_max >F (n-1)_max If this is the case (S308:YES), the process proceeds to step S309. n_max ≤F (n-1)_max If this is the case (S308:NO), the process proceeds to step S310.

[0073] In step S309, the processing unit 11 sets the value of m to 1. It also increases the value of n by 1. Then the process returns to step S302.

[0074] In step S310, we return to the next shallower model (n-1). This is the model with the next best F-score, i.e., F (n-1)_次点 Model NN (n-1) Repeat the process from steps S302 to S309. If there is no improvement even after going back to n=1, the process is terminated.

[0075] Furthermore, if, as a result of reverting to a shallower model (n-1), there is a model in which the F-score has improved by a certain value or more, that model may be modularized and registered in the artificial intelligence structure component database 106. Then, the artificial intelligence structure automatic generation process may be re-executed with the module added.

[0076] Figure 6 is the first conceptual diagram illustrating a pattern of adding nodes to an existing artificial intelligence structure.

[0077] In the diagram, (I) represents INPUT, i.e., input data. Input data is, for example, an array of numerical information. (O) represents OUTPUT, i.e., output data. Output data is, for example, an array of numerical information. The nodes between the input and output are functions that process the data, and the nodes are connected by edges. The edges are represented by arrows to show the upstream and downstream of the data flow. The structure from input to output is such that there is no reverse flow or the formation of a closed space and cycle. In other words, the structure from input to output forms a directed acyclic graph. The above is the same for Figures 6 to 16, so in order to avoid duplication of explanation, similar explanations for Figure 7 and beyond will be omitted.

[0078] Reference number AISa00 shows the structure before adding a node. AISa01 to AISa05 are patterns for adding a new node (2) to a structure that already has node (1). AISa01 and AISa02 show patterns for adding node (2) in series. AISa03 to AISa05 show patterns for adding node (2) in parallel. AISa01 shows a pattern where node (2) is added in series between node (1) and the output. AISa02 shows a case where node (2) is added in series between the input and node (1). AISa03 shows a pattern where node (2) is added in parallel between the input and the output. AISa04 shows a pattern where node (2) is added in parallel between the input and node (1). AISa05 shows a pattern where node (2) is added in parallel between node (1) and the output.

[0079] Figure 7 is a second conceptual diagram illustrating patterns for adding nodes to an existing artificial intelligence structure. The structure before adding nodes is shown by reference number AISb00. Examples of patterns for adding a new node (3) in series to a structure that has nodes (1) and (2) are shown by AISb01 to AISb03. AISb01 shows a pattern in which node (3) is added in series between node (2) and the output. AISb02 shows a pattern in which node (3) is added in series between node (1) and node (2). AISb03 shows a pattern in which node (3) is added in series between the input and node (1).

[0080] Figure 8 is a third conceptual diagram illustrating patterns for adding nodes to an existing artificial intelligence structure. Reference number AISc00 shows the structure before adding the node. AISc01 is an example of a pattern where a new node (3) is added in parallel to a structure in which nodes (1) and (2) are connected in series. AISc01 shows a pattern where node (3) is added in parallel between nodes (1) and (2).

[0081] Figure 9 is a fourth conceptual diagram illustrating patterns for adding nodes to an existing artificial intelligence structure. Reference number AISd00 shows the structure before adding a node. AISd01 to AISd03 are examples of patterns for adding a new node (4) in parallel to a structure in which nodes (1), (2), and (3) are connected in series. AISd01 shows a pattern in which node (4) is added in parallel between node (1) and node (2). AISd02 shows a pattern in which node (4) is added in parallel between node (1) and node (3). AISd03 shows a pattern in which node (4) is added in parallel between node (2) and node (3).

[0082] Figure 10 is the fifth conceptual diagram illustrating patterns for adding nodes to an existing artificial intelligence structure. The structure before adding nodes is shown by reference number AISe00. AISe01 to AISe03 are examples of patterns for adding a new node (4) in parallel to a structure in which nodes (1) and (2) are connected in series, and node (3) is connected in parallel to nodes (1) and (2). AISe01 shows a pattern in which node (4) is added between node (1) and node (2) in parallel, similar to node (3). AISe02 shows a pattern in which node (4) is added between node (1) and node (3) in parallel. AISe03 shows a pattern in which node (4) is added between node (3) and node (2) in parallel.

[0083] In Figure 10, node (3) is upstream in the flow of information compared to node (2). Therefore, when adding node (4), it is necessary to add node (4) in a way that maintains the flow of information from node (3) to node (2). In this way, in artificial intelligence structures, it is necessary to control the flow of data so that it does not flow in reverse and become cyclical. The network structure created in this way is called a directed acyclic graph (DAG) structure. In the case of an artificial intelligence structure, there is no point in having nodes that have no effect on the output node of the judgment result. Therefore, all nodes are positioned as intermediate components of a DAG structure that starts from an input and ends at an output.

[0084] Figure 11 is the sixth conceptual diagram illustrating patterns for adding nodes to an existing artificial intelligence structure. Reference number AISf00 shows the structure before adding nodes. AISf01 to AISf05 are examples of patterns for adding a new node (4) in parallel to a structure where nodes (1) and (2) are connected in series, and node (3) is connected in parallel to nodes (1) and (2). AISf01 shows a pattern in which node (4) is added in series between node (2) and the output. AISf02 shows a pattern in which node (4) is added in series between the input and node (1). AISf03 shows a pattern in which node (4) is added in series between node (1) and node (2). AISf04 shows a pattern in which node (4) is added in series between node (1) and node (3). AISf05 shows that node (4) can be added in series between node (3) and node (2) (S4g05). In the example in Figure 11, in order to adhere to the rules of DAG, we do not add node (4) so ​​that the flow is from node (2) to node (3).

[0085] Figure 12 is the seventh conceptual diagram illustrating patterns for adding nodes to an existing artificial intelligence structure. Reference number AISg00 shows the structure before adding a node. AISg01 and AISg02 are examples of patterns for adding a new node (5) to a structure where nodes (1) and (2) are connected in series, and nodes (3) and (4) are connected in parallel to nodes (1) and (2), respectively. In this case, there is no upstream / downstream relationship in the flow of information between nodes (3) and (4). Therefore, node (5) can be added between nodes (3) and (4) while adhering to the rules of DAG. For example, AISg01 shows a pattern in which node (5) is added between nodes (3) and (4) so ​​that information flows in the order of node (3), node (5), node (4). For example, AISg02 shows a pattern in which node (5) is added between nodes (3) and (4) so ​​that information flows in the order of node (4), node (5), node (3).

[0086] As illustrated with reference to Figures 6 to 12, the artificial intelligence structure is a network structure that constitutes a directed acyclic graph. Referring to Figure 13, there are two main ways to add another node between any two nodes in the above network structure. The first method is to add a new node Z in series between any two nodes X and Y. As shown by reference numerals AISh00 and AISh01, the first method is achieved by adding node Z to the edge E01 between node X and node Y.

[0087] The second method involves connecting a new node Z in parallel between nodes X and Y. This second method is achieved by adding node Z outside the edge between nodes X and Y, as shown in reference numerals AISh02 and AISh03.

[0088] If there is an upstream-downstream relationship between nodes X and Y regarding the flow of information, the added node Z should be added so that the information flows from upstream to downstream. On the other hand, if there is no upstream-downstream relationship, both cases are possible: adding node Z so that information flows from node X to node Y, and adding node Z so that information flows from node Y to node X.

[0089] As mentioned above, another way to change the structure of artificial intelligence is to add new nodes, or to add new edges between existing nodes. Figure 14 is a first conceptual diagram illustrating the addition of edges. In a state where the input (I) to the output (O) is connected by several nodes and edges, a new artificial intelligence structure can be generated by adding a new edge, for example, between node (2) and node (5). Thus, selecting two arbitrary nodes and connecting them with an edge can also be considered one variation of changing the artificial intelligence structure. However, it is necessary to check whether or not there is an upstream-downstream relationship between the two nodes being connected.

[0090] For example, in Figure 14, the artificial intelligence structure can be changed by adding a new edge between node 2 and node 5. There are two possible ways to connect node 2 and node 5 with a new edge, as shown by the dashed line: edge E02 going from node 2 to node 5, and edge E03 going from node 5 to node 2.

[0091] Figure 15 shows the path from input (I) to output (O) in Figure 14. There is no path in which node 2 is downstream of node 5, nor is there a path in which node 5 is downstream of node 2. Therefore, as mentioned above, we can add an edge from node 2 to node 5, and we can also add an edge from node 5 to node 2.

[0092] Figure 16 is a second conceptual diagram illustrating the addition of an edge. Similar to Figure 14, if we consider adding a new edge between node (2) and node (5), it is possible to add a new edge E04 from node (2) to node (5). However, it is not possible to add an edge from node (5) to node (2).

[0093] Here, Figures 14 and 15 show different orientations for the edges connecting node (4) and node (5). Figure 17 is a diagram showing the path from input (I) to output (O) in Figure 16. As shown in the diagram, there is a path where node (5) is downstream of node (2). Therefore, if we were to add a new edge from node (5) to node (2), the flow of information would reverse, forming a circuit. In other words, adding a new edge from node (5) to node (2) would mean that it is no longer a directed acyclic graph. Thus, as mentioned above, it is not possible to add an edge from node (5) to node (2). When adding an edge, the processing unit 11 extracts information showing the upstream-downstream relationship between nodes in the existing network structure and can add a new edge with an appropriate orientation based on this information.

[0094] Figure 18 is a conceptual diagram illustrating a method for combining the outputs of multiple nodes into a single output. Due to reasons such as skipped connections, the outputs of multiple nodes may merge into one output that becomes the input to the next node. Methods for this merging include the `add` method, which adds or averages each element of the data, and the `concatenate` method, which concatenates the data along a specific dimension. These two methods can also be considered variations in how the artificial intelligence structure is modified.

[0095] (Second evaluation indicator) Now, let's discuss the second evaluation indicator.

[0096] First, the significance of the second evaluation index will be explained. In the technology disclosed herein, the processing unit 11 inputs a test dataset 103 into the artificial intelligence structure model 104 and calculates the second evaluation index based on the information output from the artificial intelligence structure model 104. The processing unit 11 automatically generates a more suitable artificial intelligence structure by changing the artificial intelligence structure, such as by adding artificial intelligence structural components, so that this second evaluation index improves.

[0097] Examples of second evaluation metrics that the technology disclosed herein may employ include accuracy and F-score. However, the second evaluation metrics that the technology disclosed herein may employ are not limited to accuracy and F-score. The following are examples of elements that constitute the second evaluation metrics, or the second evaluation metrics themselves.

[0098] TPR (True Positive Rate) refers to the percentage of all positive results that were actually positive and were correctly identified as positive. It is also called recall.

[0099] FPR (False Positive Rate) refers to the percentage of all negatives that were actually negative but were mistakenly classified as positive.

[0100] TP (True Positive) refers to the number of cases where an event was predicted to be positive, and actually turned out to be positive.

[0101] FP (False Positive) refers to the number of cases where an event was predicted to be positive, but actually turned out to be negative.

[0102] FN (False Negative) refers to the number of cases where an event was predicted to be negative, but was actually positive.

[0103] TN (True Negative) refers to the number of cases where an event was predicted to be negative, and in reality, it was negative.

[0104] Based on the numbers described above, the processing unit 11 calculates secondary evaluation metrics such as Accuracy, which indicates how accurately the predictions were made; Recall, which indicates how accurately the predictions were made without any omissions; and Precision, which indicates how correct the predictions that were deemed positive were. The processing unit 11 also calculates the F-score, which is the harmonic mean of Recall and Precision, as a secondary evaluation metric. The formulas for calculating these secondary evaluation metrics are as follows.

[0105] Correct answer rate: Accuracy=(TP+TN) / (TP+FP+FN+TN)

[0106] Precision rate (precision): Precision=TP / (TP+FP)

[0107] Recall rate:Recall=(TP / (TP+FN))

[0108] F value:F-Measure=2(Precision*Recall) / (Precision+Recall)

[0109] Additionally, the AUC value may be used as a second evaluation metric. AUC stands for Area Under the Curve and represents the area under the ROC curve. Figure 19 is a graph illustrating the ROC curve and the AUC value, which is the area under its lower portion.

[0110] The ROC curve is a curve obtained by plotting the values ​​based on each trial with FPR on the horizontal axis and TPR on the vertical axis. The AUC value, which is the area under the ROC curve, approaches 1 as the classification into Positive and Negative is performed more effectively, and approaches 0.5 as the classification is random. In other words, the closer the AUC value is to 1, the better the classification is evaluated. Therefore, the processing unit 11 can use the AUC value as a second evaluation metric.

[0111] The AUC value is calculated using test dataset 103.

[0112] Figure 20 is a conceptual diagram illustrating a management table for managing evaluation metrics for each artificial intelligence structure. The management table is stored in memory. The items in the management table include a number, model information indicating the artificial intelligence structure, Time, TP, FN, FP, TN, Accuracy, Precision, Recall, F-Measure, ρ1, and AUC. Here, Time is the average decision time for the artificial intelligence structure (model), and ρ1, as shown in Figure 21, represents the lower limit of the confidence level at which a result can be confidently judged as Positive.

[0113] The model is a new artificial intelligence structure generated by the AI ​​structure automatic generation unit, and is a directed acyclic graph (DAG) composed of nodes and edges. To prevent identical structures from being registered in the management table, the AI ​​structures are sorted topologically using the AI ​​structure as the key and arranged in order.

[0114] The processing unit 11 performs machine learning on the artificial intelligence structure using the training dataset 101. The processing unit 11 calculates the values ​​of TP, FN, FP, and TN using the validation dataset 102. Based on these values, the processing unit 11 calculates evaluation metrics such as Accuracy, Precision, Recall, F-Measure, and ρ1. The processing unit 11 further generates an ROC curve for each model and calculates the AUC value.

[0115] (modified version) The evaluation metrics in the artificial intelligence structural learning system 1 disclosed herein can be changed as needed depending on the nature of the problem that the artificial intelligence is trying to solve. For example, in the case of problems such as food inspections where no defective products can be produced, or cancer screenings where no cancer patients can be overlooked, it is necessary to learn in a way that maximizes recall, even at the expense of precision. Conversely, in the case of problems where it is necessary to narrow down the correct answer, such as surveys to select candidate mining sites, it is necessary to increase precision even at the expense of recall. Furthermore, there are problems where both recall and precision need to be at a high level in a well-balanced manner. In some problems, it is preferable to have a high accuracy of 99% in determining a result rather than a high accuracy of 60% in determining a result as positive. By changing the evaluation metrics and evaluation metric thresholds during learning according to the characteristics of such problems, it is possible to generate an optimal learning model that matches the characteristics of the problem.

[0116] The artificial intelligence structure learning system 1 disclosed herein selects a model that requires the least amount of computation depending on the difficulty of the problem and makes a decision based on that model. This speeds up the system's response time and increases the overall system throughput when processing large amounts of data.

[0117] To achieve the above, depending on the difficulty of the problem and the amount of processing required per unit time, the system will avoid using AI models with an unnecessarily deep number of layers for judgment, and instead use AI models with an appropriate number of layers that allow for faster computation. For example, as shown in Figure 20, the management table will include items such as the average judgment time Time for each model and the lower limit of confidence ρ1, which indicates that a result can be judged as positive without any doubt. This allows the processing unit to immediately adopt the judgment of a model if, for example, there is a model with a shallow number of layers that can ensure that ρ is above a certain value ρ1. Even if ρ cannot be ensured to be above a certain value ρ1, the first evaluation index threshold ρ C If there are models that can ensure the above, the model with the shortest average decision time (Time) can be selected for the decision. In the unlikely event that the decision data (unknown data) is difficult to determine and the decision accuracy ρ is constant ρC If the above conditions are not met, the processing unit 11 will use an AI model with a deeper number of layers and a constant value ρ for accuracy ρ. C The judgment is repeated until it exceeds a certain value. A sufficient accuracy ρ is achieved even when the judgment is performed on all registered models. C If it is not possible to secure the necessary resources, the process proceeds to a dynamic learning process that modifies the model itself. By performing the above processing, for decision problems that simply need to be processed within a time limit, the decision can be made using a simple model with a shallow number of layers, and only irregular cases require re-evaluation or retraining. Therefore, the overall throughput of the artificial intelligence structure learning system 1 can be improved. Here ρ C Set the value to be equal to or less than ρ1. For example, the probability of a cumulative result being judged as Negative can be set to less than 1% or less than 3%, within an acceptable range depending on the problem characteristics. C Like ρ1, this can also be dynamically changed for each model.

[0118] Furthermore, if there are multiple high-accuracy models with a high number of layers in the management table shown in Figure 20, and both have a high level of learning and similar decision times, the processing unit 11 will select the model with the higher accuracy ρ and use it for decision-making.

[0119] Similarly, in the management table shown in Figure 20, if there are multiple models with similar levels of learning and similarly high judgment accuracy (ρ), the processing unit 11 will select the model with the higher evaluation index value and use it for judgment.

[0120] As described above, the selection of a model by the processing unit 11 is performed according to the difficulty and nature of the problem. For example, in a problem where the user has to identify a dog from several photographs, the difficulty of the problem is relatively low, so the processing unit 11 may select a model with a shallow level of training to make the determination. On the other hand, in a problem where the user has to identify the breed of dog from a photograph, the difficulty of the problem is higher, so the processing unit 11 may select a model with a deeper level of training to make the determination. Similarly, in a problem where the user has to determine from a medical image whether the affected area is cancerous, the difficulty of the problem is high, so it is necessary to select a model with a deep level of training to make the determination.

[0121] When using artificial intelligence to make judgments, it's not simply a matter of processing one case at a time; there are cases where multiple problems must be processed within a unit of time. In such cases, the processing unit 11 dynamically changes the model selection criteria according to conditions such as the difficulty level and nature of the problem (e.g., whether suitability or reproducibility is prioritized), and the acceptable threshold for accuracy. This allows the user to obtain the optimal processing time and the optimal judgment result.

[0122] Furthermore, if the processing time is short but the judgment result does not yield satisfactory accuracy, the processing unit 11 will either dynamically select a more deeply learned model or proceed to a process of automatically retraining the artificial intelligence structure itself.

[0123] The embodiments of the present invention described above are illustrative for the purpose of explaining the present invention and are not intended to limit the scope of the present invention to those embodiments only. Those skilled in the art can implement the present invention in various other forms without departing from the scope of the present invention. For example, this embodiment includes the following; however, the matters included in this embodiment are not limited to those listed below.

[0124] As described above, the artificial intelligence structure learning system comprises memory devices 12 and 13 and a processing device 11. The memory device stores an artificial intelligence structure database 100 containing artificial intelligence structure models having a predetermined artificial intelligence structure, an artificial intelligence structure component database 106 containing artificial intelligence structure component parts that can constitute a part of the artificial intelligence structure, and a training dataset 101. The processing device sequentially selects an artificial intelligence structure model from the artificial intelligence structure database, calculates a first evaluation index for the artificial intelligence structure model, and if there is an artificial intelligence structure model whose first evaluation index is equal to or greater than a predetermined first evaluation index threshold, it adopts the trained artificial intelligence model corresponding to that artificial intelligence structure model. If no artificial intelligence structure model exists in the artificial intelligence structure database whose first evaluation metric is equal to or greater than the first evaluation metric threshold, the processing unit generates an artificial intelligence structure model by sequentially adding artificial intelligence structure components selected sequentially from the artificial intelligence structure component database to the locations of the artificial intelligence structure where they can be added, based on any artificial intelligence structure model, performs training on the generated artificial intelligence structure model using a training dataset to generate a trained artificial intelligence model, registers the artificial intelligence structure model corresponding to the trained artificial intelligence model in the artificial intelligence structure database, and adopts the trained artificial intelligence model if a trained artificial intelligence model is generated whose second evaluation metric is better than other models registered in the artificial intelligence structure database and equal to or greater than a predetermined second evaluation metric threshold.

[0125] This is an artificial intelligence structure learning method using a device having a memory device and a processing device. The memory device stores an artificial intelligence structure database containing artificial intelligence structure models having a predetermined artificial intelligence structure, an artificial intelligence structure component database containing artificial intelligence structure component parts that can constitute a part of the artificial intelligence structure, and a training dataset. The processing device sequentially selects an artificial intelligence structure model from the artificial intelligence structure database, calculates a first evaluation index for the artificial intelligence structure model, and if there is an artificial intelligence structure model whose first evaluation index is equal to or greater than a predetermined first evaluation index threshold, it adopts the trained artificial intelligence model corresponding to that artificial intelligence structure model. If the processing unit does not have an artificial intelligence structure model in the artificial intelligence structure database whose first evaluation metric is equal to or greater than the first evaluation metric threshold, it generates an artificial intelligence structure model by sequentially adding artificial intelligence structure components selected sequentially from the artificial intelligence structure component database to the locations of the artificial intelligence structure where they can be added, based on any artificial intelligence structure model, performs training on the generated artificial intelligence structure model using a training dataset to generate a trained artificial intelligence model, registers the artificial intelligence structure model corresponding to the trained artificial intelligence model in the artificial intelligence structure database, and adopts the trained artificial intelligence model if a trained artificial intelligence model is generated whose second evaluation metric is better than other models registered in the artificial intelligence structure database and is equal to or greater than a predetermined second evaluation metric threshold.

[0126] This is a program for training an artificial intelligence structure in a device having a memory device and a processing device. The memory device stores an artificial intelligence structure database containing artificial intelligence structure models having a predetermined artificial intelligence structure, an artificial intelligence structure component database containing artificial intelligence structure components that can constitute part of the artificial intelligence structure, and a training dataset. When executed by the processing device, the program sequentially selects an artificial intelligence structure model from the artificial intelligence structure database, calculates a first evaluation index for the artificial intelligence structure model, and if there is an artificial intelligence structure model whose first evaluation index is equal to or greater than a predetermined first evaluation index threshold, it adopts the trained artificial intelligence model corresponding to that artificial intelligence structure model. When executed by the processing unit, the program generates an artificial intelligence structure model by sequentially adding artificial intelligence structure components selected sequentially from the artificial intelligence structure component database to the locations of the artificial intelligence structure based on any existing artificial intelligence structure model, if there is no artificial intelligence structure model in the artificial intelligence structure database whose first evaluation metric is equal to or greater than the first evaluation metric threshold when executed by the processing unit. It then generates a trained artificial intelligence model by performing training on the generated artificial intelligence structure model using a training dataset, registers the artificial intelligence structure model corresponding to the trained artificial intelligence model in the artificial intelligence structure database, and adopts the trained artificial intelligence model if a trained artificial intelligence model is generated whose second evaluation metric is better than other models registered in the artificial intelligence structure database and equal to or greater than a predetermined second evaluation metric threshold.

[0127] As described above, an artificial intelligence structure database and an artificial intelligence structure component database are prepared. If an artificial intelligence structure model that achieves the desired performance is found in the artificial intelligence structure database, it is adopted. If no such model exists, an artificial intelligence structure model is generated by adding artificial intelligence structure components to any existing artificial intelligence structure and registered in the artificial intelligence structure database. Additionally, if a trained artificial intelligence model that achieves the desired performance is obtained, it is adopted. This makes it possible to generate suitable artificial intelligence structure models and trained artificial intelligence models that achieve the desired performance. In this process, if an artificial intelligence structure model that achieves the desired performance exists in the artificial intelligence structure database, it is adopted; otherwise, an artificial intelligence structure model that achieves the desired performance is generated by adding artificial intelligence structure components, thus efficiently generating suitable artificial intelligence models. Furthermore, since the newly generated artificial intelligence structures are accumulated in the artificial intelligence structure database, the number of artificial intelligence structures that have the potential to achieve the desired performance increases, further streamlining the search for suitable artificial intelligence structures.

[0128] The processing unit registers a portion of an artificial intelligence structure as an artificial intelligence structure component in the artificial intelligence structure component database if the second evaluation index, when it contains a portion composed of multiple artificial intelligence structure components, is better than the second evaluation index when it does not contain such a portion, by a predetermined second evaluation index difference threshold. As a result, portions that are expected to significantly improve performance are registered in the database as artificial intelligence structure component components, allowing for efficient performance improvement when generating new artificial intelligence structures thereafter.

[0129] The processing unit obtains information from the user regarding the nature of the prediction problem and sets a first and / or second evaluation metric based on that information. This makes it easy to set appropriate evaluation metrics according to the characteristics of the prediction target.

[0130] The artificial intelligence structure model is a neural network, and the processing unit sequentially selects artificial intelligence structure models from the artificial intelligence structure database whose second evaluation metric is above a predetermined second evaluation metric threshold, starting with those with the fewest node layers. This allows for the search for a neural network that achieves the desired performance with the simplest possible structure.

[0131] The first evaluation metric is the accuracy of the prediction result for the judgment data, which is unknown data to be predicted, or a metric that substitutes for accuracy. If the value of the accuracy of the prediction result for the judgment data by the trained artificial intelligence model corresponding to the artificial intelligence structure model selected from the artificial intelligence structure database, or a metric that substitutes for accuracy, is greater than or equal to the threshold of the first evaluation metric, the processing unit adopts the trained artificial intelligence model for the prediction process for the data to be predicted. This allows the system to adopt a trained artificial intelligence model with a suitable artificial intelligence structure without performing any training if the desired performance can be obtained with the trained artificial intelligence model.

[0132] The artificial intelligence structure database further stores the average value of the first evaluation metric obtained by a trained artificial intelligence model corresponding to the artificial intelligence structure model, and the average judgment time required for the second evaluation metric to be calculated by the trained artificial intelligence model. The processing unit sequentially selects artificial intelligence structure models from those with an average value of the first evaluation metric above a certain value, starting with those with the shortest average judgment time. This allows the system to search for an artificial intelligence structure that can obtain the desired evaluation metric in the shortest possible time.

[0133] The memory device further stores the test dataset 103, and the second evaluation metric, which measures the performance of predictions, is calculated using the test dataset. In addition to the F-score, other possible metrics include weighted F-score, AUC, accuracy, precision, and recall. By deriving and evaluating the second evaluation metric for each artificial intelligence structure model generated by the AI ​​structure automatic generation unit using the test dataset, it is possible to obtain an artificial intelligence structure model that achieves the desired performance.

[0134] An artificial intelligence (AI) structure can be represented by a directed acyclic graph in which AI structure components are connected by directed edges. The processing unit topologically sorts the AI ​​structures obtained by adding AI structure components to the AI ​​structure model. If a structure is the same as any of the AI ​​structures tried in the past, it is skipped. Otherwise, a judgment is made using a second evaluation metric. This prevents the essentially same AI structure from being tried repeatedly, and allows for the efficient generation of a suitable AI model.

[0135] The artificial intelligence structure is a structure in which artificial intelligence structural components are connected by directed edges. When the processing unit adds an artificial intelligence structural component to the artificial intelligence structural model, if the data formats of the input of the artificial intelligence structural component and the output of the artificial intelligence structural model connected to that input are incompatible, the processing unit will align the data formats of the output and the input. The processing unit will align the data formats of the input and the output of the artificial intelligence structural model connected to that output if they are incompatible. This allows artificial intelligence structural components to be added to the artificial intelligence structural model without any problems. [Explanation of symbols]

[0136] 1...Artificial intelligence structure learning system, 11...Processing device, 12...Storage device, 12...Main memory, 13...Auxiliary memory, 14...Communication device, 15...Input device, 16...Output device, 17...Bus, 100...Artificial intelligence structure database, 101...Training dataset, 102...Verification dataset, 103...Test dataset, 104...Artificial intelligence structure model, 105...Decision unit, 106...Artificial intelligence structure component database, 107...Automatic artificial intelligence structure generation unit, 108...Artificial intelligence structure consistency check unit, 109...Artificial intelligence structure input / output dimension adjustment unit, 110...Artificial intelligence structure component database registration and addition unit, 111...Decision data (unknown data), 112...Training, 113...Trained artificial intelligence model, 114...Second evaluation index, 115...First evaluation index

Claims

1. It has a memory device and a processing device, The storage device stores an artificial intelligence structure database containing an artificial intelligence structure model having a predetermined artificial intelligence structure, an artificial intelligence structure component database containing artificial intelligence structure components that can constitute a part of the artificial intelligence structure, and a training dataset. The processing device sequentially selects artificial intelligence structure models from the artificial intelligence structure database, calculates a first evaluation index for the artificial intelligence structure model, and if there is an artificial intelligence structure model for which the first evaluation index is equal to or greater than a predetermined first evaluation index threshold, it adopts the trained artificial intelligence model corresponding to the artificial intelligence structure model. If no artificial intelligence structure model exists in the artificial intelligence structure database whose first evaluation index is equal to or greater than the first evaluation index threshold, an artificial intelligence structure model is generated by sequentially adding artificial intelligence structure components selected sequentially from the artificial intelligence structure component database to the locations of the artificial intelligence structure where they can be added, training is performed on the generated artificial intelligence structure model using the training dataset to generate a trained artificial intelligence model, and the artificial intelligence structure model corresponding to the trained artificial intelligence model is registered in the artificial intelligence structure database. Furthermore, if a trained artificial intelligence model is generated whose second evaluation index is better than that of other models registered in the artificial intelligence structure database and is equal to or greater than a predetermined second evaluation index threshold, the trained artificial intelligence model is adopted. Artificial intelligence structure learning system.

2. The processing device registers the portion composed of multiple artificial intelligence structural components in the artificial intelligence structural component database if the second evaluation index when the artificial intelligence structure has a portion composed of multiple artificial intelligence structural components is better than the second evaluation index when the portion does not have a predetermined second evaluation index difference threshold. The artificial intelligence structure learning system according to claim 1.

3. The processing device obtains information from the user regarding the nature of the prediction problem and sets the first evaluation index and / or the second evaluation index based on the information. The artificial intelligence structure learning system according to claim 1.

4. The aforementioned artificial intelligence structural model is a neural network, The processing device sequentially selects artificial intelligence structure models from among the artificial intelligence structure models in the artificial intelligence structure database whose second evaluation index is equal to or greater than a predetermined second evaluation index threshold, starting with the artificial intelligence structure models with the fewest node layers. The artificial intelligence structure learning system according to claim 1.

5. The first evaluation index is the accuracy of the prediction result for the judgment data, which is the unknown data to be predicted, or an index that substitutes for accuracy. The processing device adopts the trained artificial intelligence model for prediction processing on the prediction target if the accuracy of the prediction result on the judgment data by the trained artificial intelligence model corresponding to the artificial intelligence structure model selected from the artificial intelligence structure database, or the value of an index that replaces the accuracy, is equal to or greater than the first evaluation index threshold. The artificial intelligence structure learning system according to claim 3.

6. The aforementioned artificial intelligence structure database further stores the average value of the accuracy obtained by the trained artificial intelligence model corresponding to the aforementioned artificial intelligence structure model, and the average value of the judgment time required for the second evaluation index to be calculated by the trained artificial intelligence model. The processing device sequentially selects from among artificial intelligence structural models whose average accuracy is above a certain value, starting with those whose average determination time is shortest. The artificial intelligence structure learning system according to claim 5.

7. The aforementioned storage device further stores a test dataset, The second evaluation index is a measure of prediction performance, such as the F-score, weighted F-score, AUC score, accuracy, precision, and recall, calculated using the test dataset. The artificial intelligence structure learning system according to claim 6.

8. An artificial intelligence structure can be represented by a directed acyclic graph in which the components of the artificial intelligence structure are connected by directed edges. The processing device topologically sorts the artificial intelligence structures obtained by adding artificial intelligence structural components to the artificial intelligence structural model, skips the structure if it is the same as any of the artificial intelligence structures tried in the past, and makes a determination using the second evaluation index if it is not the same. The artificial intelligence structure learning system according to claim 1.

9. The artificial intelligence structure is a structure in which artificial intelligence structural components are connected by directed edges. When the processing device adds an artificial intelligence structural component to an artificial intelligence structural model, if the data formats of the input of the artificial intelligence structural component and the output of the artificial intelligence structural model connected to the input are incompatible, the processing device will align the data formats of the output and the input; if the data formats of the output of the artificial intelligence structural component and the input of the artificial intelligence structural model connected to the output are incompatible, the processing device will align the data formats of the input and the output. The artificial intelligence structure learning system according to claim 1.

10. An artificial intelligence structure learning method using a device having a memory device and a processing device, The aforementioned storage device stores an artificial intelligence structure database containing an artificial intelligence structure model having a predetermined artificial intelligence structure, an artificial intelligence structure component database containing artificial intelligence structure components that can constitute a part of the artificial intelligence structure, and a training dataset. The aforementioned processing apparatus From the aforementioned AI structure database, an AI structure model is selected sequentially, a first evaluation index is calculated for the AI ​​structure model, and if there is an AI structure model whose first evaluation index is equal to or greater than a predetermined first evaluation index threshold, the trained AI model corresponding to that AI structure model is adopted. If no artificial intelligence structure model exists in the artificial intelligence structure database whose first evaluation index is equal to or greater than the first evaluation index threshold, an artificial intelligence structure model is generated by sequentially adding artificial intelligence structure components selected sequentially from the artificial intelligence structure component database to the locations of the artificial intelligence structure where they can be added, training is performed on the generated artificial intelligence structure model using the training dataset to generate a trained artificial intelligence model, and the artificial intelligence structure model corresponding to the trained artificial intelligence model is registered in the artificial intelligence structure database. Furthermore, if a trained artificial intelligence model is generated whose second evaluation index is better than that of other models registered in the artificial intelligence structure database and is equal to or greater than a predetermined second evaluation index threshold, the trained artificial intelligence model is adopted. Artificial intelligence structural learning method.

11. A program for causing a device having a memory device and a processing device to learn an artificial intelligence structure, The aforementioned storage device stores an artificial intelligence structure database containing an artificial intelligence structure model having a predetermined artificial intelligence structure, an artificial intelligence structure component database containing artificial intelligence structure components that can constitute a part of the artificial intelligence structure, and a training dataset. When executed by the aforementioned processing unit, From the aforementioned AI structure database, an AI structure model is selected sequentially, a first evaluation index is calculated for the AI ​​structure model, and if there is an AI structure model whose first evaluation index is equal to or greater than a predetermined first evaluation index threshold, the trained AI model corresponding to that AI structure model is adopted. If no artificial intelligence structure model exists in the artificial intelligence structure database whose first evaluation index is equal to or greater than the first evaluation index threshold, an artificial intelligence structure model is generated by sequentially adding artificial intelligence structure components selected sequentially from the artificial intelligence structure component database to the locations of the artificial intelligence structure where they can be added, training is performed on the generated artificial intelligence structure model using the training dataset to generate a trained artificial intelligence model, and the artificial intelligence structure model corresponding to the trained artificial intelligence model is registered in the artificial intelligence structure database. Furthermore, if a trained artificial intelligence model is generated whose second evaluation index is better than that of other models registered in the artificial intelligence structure database and is equal to or greater than a predetermined second evaluation index threshold, the trained artificial intelligence model is adopted. program.

Images