Analytical programs, analytical methods, computers and machine learning programs

By analyzing deep learning model accuracy in a multidimensional space using discrete thresholds, the method optimizes AI system performance and reduces costs by accurately setting accuracy requirements, addressing the challenge of resource inefficiency in AI systems.

JP7581992B2Active Publication Date: 2024-11-13FUJITSU LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2021042573
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-16
Publication Date
2024-11-13
Estimated Expiration
2041-03-16

AI Technical Summary

Technical Problem

Deep learning models in AI systems require significant computational resources, and it is challenging to predict the impact of individual model accuracy on the final system's accuracy, leading to potential overestimation and increased costs.

Method used

A discrete threshold evaluation method is used to analyze the accuracy of deep learning models in a multidimensional space, allowing for the quantification of required accuracy and reducing unnecessary computational resources by optimizing model combinations.

Benefits of technology

This approach enables efficient training of deep learning models by accurately setting accuracy thresholds, reducing system costs and computational demands while maintaining or improving performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581992000001
    Figure 0007581992000001
  • Figure 0007581992000002
    Figure 0007581992000002
  • Figure 0007581992000003
    Figure 0007581992000003
Patent Text Reader

Abstract

To provide a program, an analysis method, a calculator, and a machine learning program that quantitatively analyze a required accuracy of a deep-learning model combined in an AI (Artificial Intelligence) system.SOLUTION: An analysis program causes a computer to perform processing of generating accuracy reference information for a plurality of deep-learning models in a space in which accuracy evaluation information is projected in multi-dimension using a discrete threshold evaluation for an AI system in which the deep-learning models are combined.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an analysis program, an analysis method, a computer, and a machine learning program. [Background technology]

[0002] Deep learning models are typically trained using a training dataset that includes training data and label information.

[0003] In addition, the recognition accuracy of a deep learning model after training is generally calculated using a test data set that includes test data and label information. Specifically, the inference made by the deep learning model on the test data is compared with the label information, and the recognition accuracy is calculated from the accuracy rate.

[0004] In addition, because a deep learning model is a processing module that uses AI (Artificial Intelligence), it is possible to build a system that combines multiple deep learning models (hereinafter referred to as an AI system).

[0005] For example, in recent years, a behavior estimation system that detects various human behaviors from video images has been realized by combining multiple processing modules (deep learning models). This behavior estimation system is configured by combining processing modules for, for example, person recognition, posture recognition, tracking, and behavior estimation. Among these processing modules, person recognition and posture recognition are realized by deep learning models.

[0006] In person recognition, a person is identified from one frame of a video using an object recognition model. In posture recognition, the posture of the person identified in person recognition is identified using a posture recognition model. After that, the posture of the identified person is represented in a time series in tracking, and the behavior of the person is estimated in behavior estimation. The user evaluates the results of such estimations by an AI system as to whether they are acceptable (OK) or not (NG).

[0007] In addition, the multiple deep learning models used in such an AI system are trained individually, meaning that multiple processing modules (deep learning models) are not trained together as an entire AI system. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] US Patent Application Publication No. 2017 / 0230394 [Patent Document 2] US Patent Application Publication No. 2017 / 0372023 [Patent Document 3] Special table 2013-520714 publication Summary of the Invention [Problem to be solved by the invention]

[0009] Deep learning requires a huge amount of calculation to improve accuracy. There is also a strong relationship between accuracy and the computational resources required for calculation in deep learning. For example, Residual Network (ResNet), a known neural network model, requires twice as many layers to reduce the error rate by 1%. Furthermore, the number of layers may be increased by 1.5 times to reduce the error rate by 0.4%.

[0010] In an AI system that combines multiple deep learning models, if the minimum accuracy (or error rate) required for training each deep learning model can be accurately estimated, the amount of computation can be significantly reduced. This reduction in the amount of computation can then be used to reduce the cost of the system, for example by replacing the required computing resources with lower-cost ones.

[0011] However, at the stage of training a deep learning model, it is difficult to predict the impact of the accuracy of each individual deep learning model on the accuracy of the final AI system. For example, in an AI system that combines multiple deep learning models as described above, the minimum accuracy required for a deep learning model for a user to judge the inference result as OK is unknown, and it is not possible to consider relaxing the accuracy threshold. As a result, there is a risk that the accuracy of each deep learning model combined with the AI ​​system will be increased more than necessary, resulting in excessive quality and increasing costs. In one aspect, the present invention aims to enable quantitative analysis of the required accuracy of a deep learning model to be combined with an AI system. [Means for solving the problem]

[0012] For this reason, this analysis program causes a computer to execute a process of creating accuracy standard information for multiple deep learning models in a space where accuracy evaluation information is projected multidimensionally, using discrete threshold evaluation, for an AI (Artificial Intelligence) system that combines multiple deep learning models. Effect of the Invention

[0013] According to one embodiment, the required accuracy of a deep learning model to be combined with an AI system can be quantitatively analyzed. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating a functional configuration of an information processing apparatus as an example of an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an AI system generated by an information processing device as an example of an embodiment. [Diagram 3] 1 is a diagram illustrating a snapshot acquired in an information processing apparatus as an example of an embodiment; [Figure 4]FIG. 2 is a diagram for explaining the processing of an AI system generation unit in an information processing device as an example of an embodiment. [Diagram 5] FIG. 1 is a diagram illustrating a multi-dimensional space representing AI system accuracy. [Figure 6] FIG. 1 is a diagram illustrating a multi-dimensional space representing AI system accuracy. [Figure 7] 1 is a flowchart for explaining an overview of a method for constructing an AI system in an information processing device as an example of an embodiment. [Figure 8] 1 is a flowchart for explaining an overview of a method for constructing an AI system in an information processing device as an example of an embodiment. [Figure 9] 11 is a diagram illustrating an example of an effect of an AI system generated by an information processing device as an embodiment. [Figure 10] FIG. 1 is a diagram illustrating a hardware configuration of an information processing apparatus according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present analysis program, analysis method, computer, and machine learning program will be described with reference to the drawings. However, the embodiments shown below are merely examples, and there is no intention to exclude the application of various modified examples and technologies not explicitly stated in the embodiments. In other words, the present embodiment can be implemented with various modifications within the scope of the gist of the embodiment. In addition, each figure does not intend to include only the components shown in the figure, but can include other functions, etc.

[0016] (A) Configuration FIG. 1 is a diagram illustrating a functional configuration of an information processing device 1 as an example of an embodiment. The information processing device 1 has a function of generating an AI system equipped with multiple deep learning models and analyzing the accuracy of the system. FIG. 2 is a diagram illustrating an AI system generated by the information processing device 1 as an example of an embodiment.

[0017] The AI ​​system shown in Fig. 2 is a behavior estimation system that estimates human behavior from input video, and is composed of four processing modules: person recognition, posture recognition, tracking, and behavior estimation. Of these four processing modules, the person recognition and posture recognition processing modules are realized by deep learning models.

[0018] In person recognition, an object recognition model (first deep learning model) is used to identify a person from one frame of a video. In posture recognition, a posture recognition model (second deep learning model) is used to identify the posture of the person identified in person recognition. The identified posture is represented in a time series in tracking, and the behavior is estimated in behavior estimation.

[0019] In response to such an estimation result by an AI system, the user inputs, for example, a two-choice (binary) evaluation of whether it is acceptable (OK) or not (NG). The user inputs the evaluation using, for example, a keyboard 15a or a mouse 15b described later in FIG. As shown in FIG. 1, the information processing device 1 includes a deep learning model generation unit 101, an AI system generation unit 105, and an AI system accuracy analysis unit 106.

[0020] The deep learning model generation unit 101 generates a plurality of types of deep learning models. The deep learning models generated by the deep learning model generation unit 101 are incorporated into an AI system for use.

[0021] In this embodiment, an example is shown in which the deep learning model generation unit 101 generates two types of deep learning models: an object recognition model (first deep learning model) and a posture recognition model (second deep learning model). The deep learning model generation unit 101 has functions as a training execution unit 102, an inference execution unit 103, and a snapshot saving processing unit 104.

[0022] The training execution unit 102 trains the deep learning model using the training data. The training data may be data with label information. The training execution unit 102 generates multiple types (two types in this embodiment) of deep learning models to be used in the AI ​​system, and trains these deep learning models using the training data.

[0023] A deep learning model may be generated using various techniques, such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Long Short Term Memory (LSTM) method.

[0024] Training of the deep learning model is performed, for example, until the validation accuracy of the deep learning model obtained by the inference execution unit 103 described later becomes equal to or greater than a predetermined threshold. In addition, training may be performed until the number of repetitions (number of epochs) of machine learning using training data becomes equal to or greater than a predetermined threshold. It is desirable for the training execution unit 102 to use the same test data for training multiple types of deep learning models.

[0025] The inference execution unit 103 performs inference using the deep learning model generated and trained by the training execution unit 102. The inference execution unit 103 inputs test data to be inferred to the deep learning model, and obtains an inference result output by the deep learning model.

[0026] In addition, the inference execution unit 103 calculates the validation accuracy of the inference result by comparing the inference result with the label information of the test data. Hereinafter, the accuracy of the estimation result of the deep learning model may be simply referred to as the accuracy of the deep learning model.

[0027] The snapshot saving processing unit 104 stores the snapshot of the deep learning model generated by the deep learning model generating unit 101 in a model DB (Data Base) (not shown). Information constituting the model DB is stored in a predetermined storage area such as the storage device 13 (see FIG. 10).

[0028] A snapshot is information that represents a deep learning model, and may include, for example, information about intermediate layers that constitute the deep learning model, and information such as weights, activations, and gradients obtained by training. A snapshot represents a deep learning model at the time the snapshot is created, and can reproduce the deep learning model at the time the snapshot is created.

[0029] The snapshot saving processing unit 104 saves a snapshot of the deep learning model, for example, every time the accuracy of the deep learning model calculated by the inference execution unit 103 exceeds a predetermined threshold (prescribed accuracy threshold).

[0030] In this embodiment, for example, a specified accuracy threshold is set for the accuracy of the deep learning model in 5% increments, which allows multiple deep learning models to be obtained with different accuracies in 5% increments. FIG. 3 is a diagram for explaining a snapshot acquired in the information processing device 1 as an example of an embodiment.

[0031] Figure 3 shows a graph of the relationship between the number of epochs in the training process of the object recognition model and the model accuracy (validation accuracy).

[0032] In the example shown in FIG. 3, the accuracy of the deep learning model improves as the number of training epochs increases. Also, in FIG. 3, a plurality of shaded circles indicate the timing at which a snapshot is saved. That is, a snapshot of the object recognition model is saved each time the accuracy of the deep learning model increases by 5%. In other words, a snapshot of the object recognition model is saved each time the accuracy of the deep learning model becomes equal to or greater than a specified accuracy threshold. It can be said that the snapshot saving processing unit 104 saves snapshots discretely at the accuracy of the deep learning model. The snapshot storage processing unit 104 acquires a snapshot for each of the multiple deep learning models.

[0033] The snapshot saving processing unit 104 creates a first snapshot group consisting of multiple first snapshots by saving a snapshot (first snapshot) each time the accuracy of an object recognition model (first deep learning model) becomes equal to or greater than a unit threshold during the training process of the object recognition model.

[0034] In addition, the snapshot saving processing unit 104 creates a second snapshot group consisting of multiple second snapshots by saving a snapshot (second snapshot) each time the accuracy of the posture recognition model (second deep learning model) becomes equal to or greater than a unit threshold during the training process of the posture recognition model.

[0035] For example, with respect to the AI ​​system shown in FIG. 2, the snapshot saving processing unit 104 saves a snapshot of the pose recognition model as well as the object recognition model during the training process each time the accuracy of the deep learning model calculated by the inference execution unit 103 exceeds a specified accuracy threshold (e.g., 5%).

[0036] Snapshots of deep learning models during training are stored in the model DB along with information about the accuracy at the time the snapshot was taken.

[0037] The AI ​​system generation unit 105 generates an AI system by combining a plurality of deep learning models generated by the deep learning model generation unit 101. The AI ​​system generation unit 105 generates an AI system by combining a plurality of processing modules including deep learning models, for example, as illustrated in FIG.

[0038] The AI ​​system generation unit 105 uses multiple snapshots of each deep learning model stored by the snapshot storage processing unit 104 to reproduce each deep learning model at each point in time when the snapshots were obtained. FIG. 4 is a diagram for explaining the processing of the AI ​​system generation unit 105 in the information processing device 1 as an example of an embodiment. FIG. 4 illustrates a method for generating the AI ​​system illustrated in FIG.

[0039] The AI ​​system generation unit 105 reproduces a plurality of (n) deep learning models (snapshot models) that realize human recognition, using a plurality of (n) snapshots (first snapshot group) acquired during the training course of the object recognition model, where n is a natural number.

[0040] In addition, the AI ​​system generation unit 105 reproduces a plurality of (m) deep learning models (snapshot models) that realize posture recognition, using a plurality of (m) snapshots (second snapshot group) acquired in the training course of the posture recognition model, where m is a natural number.

[0041] Then, the AI ​​system generation unit 105 generates multiple (n x m) types of AI systems by combining one deep learning model extracted from multiple (n) deep learning models that realize human recognition with one deep learning model extracted from multiple (m) deep learning models that realize posture recognition. Hereinafter, an AI system constructed by combining deep learning models in the training process may be referred to as a sub-AI system.

[0042] Based on the first snapshot group and the second snapshot group, the AI ​​system generation unit 105 creates multiple sub-AI systems by combining a first snapshot extracted from the first snapshot group and a second snapshot extracted from the second snapshot group.

[0043] In other words, the AI ​​system generation unit 105 combines one object recognition model selected from multiple snapshots of object recognition models saved by the snapshot saving processing unit 104 with one posture recognition model selected from multiple snapshots of posture recognition models, and constructs a sub-AI system including this combination.

[0044] The AI ​​system generation unit 105 constructs such a sub-AI system for all combinations of multiple snapshots of the object recognition model and multiple snapshots of the pose recognition model stored by the snapshot storage processing unit 104. The AI ​​system accuracy analysis unit 106 analyzes the accuracy of the sub-AI system generated by the AI ​​system generation unit 105. As shown in FIG. 1, the AI ​​system accuracy analysis unit 106 has the functions of an accuracy verification unit 107 and an accuracy threshold setting unit 108. The accuracy verification unit 107 verifies the accuracy of each of the multiple types of AI systems (sub-AI systems) generated by the AI ​​system generation unit 105.

[0045] Specifically, the accuracy verification unit 107 inputs input data (e.g., image data) to each of the sub-AI systems generated by the AI ​​system generation unit 105, and obtains the inference results output by the sub-AI systems.

[0046] The accuracy verification unit 107 presents the inference result of the sub-AI system to the user and accepts the user's evaluation input for this inference result. For example, the accuracy verification unit 107 presents the inference result to the user by displaying it on the monitor 14a (see FIG. 10). The user inputs an evaluation for the inference result using an input device such as the keyboard 15a or the mouse 15b (see FIG. 10).

[0047] The evaluation of the inference result is, for example, a binary judgment result (threshold evaluation) that indicates whether the inference result by the sub-AI system is acceptable to the user. This judgment result may be called accuracy evaluation information.

[0048] The user inputs "OK" if the inference result is acceptable, and "NG" if it is unacceptable. Such a user evaluation of the inference result can be said to be a rough indicator of the quality of the sub-AI system. The user's evaluation of the inference result also functions as an acceptable threshold for the accuracy axis of the AI ​​system.

[0049] The accuracy verification unit 107 associates the user's evaluation (OK or NG) inputted for the estimation result of the sub-AI system with information on the multiple deep learning models used in configuring the sub-AI system (e.g., information on snapshots of the object recognition model and information on snapshots of the posture recognition model) and stores it in a predetermined storage area such as the memory 12 or the storage device 13. The snapshot information may be the accuracy of the deep learning model at the time when the snapshot was acquired. The accuracy threshold setting unit 108 sets an accuracy threshold for verifying the accuracy of the AI ​​system (AI system accuracy).

[0050] The accuracy of a deep learning model is determined by the accuracy rate for test data, and the deep learning model can be selected arbitrarily. Therefore, it can be said that the accuracy of a deep learning model depends on the test data, not on the deep learning model.

[0051] The accuracy of an AI system that has multiple such deep learning models can be expressed in a multidimensional space centered on the test data (one or more for each deep learning model). FIG. 5 is a diagram illustrating a multi-dimensional space representing AI system accuracy.

[0052] The multidimensional space shown in Figure 5 is composed of an accuracy axis for object recognition test data, an accuracy axis for pose recognition test data, and an AI system accuracy axis. This multidimensional space is a space in which accuracy evaluation information (the accuracy axis for object recognition test data and the accuracy axis for pose recognition test data) is projected in multiple dimensions.

[0053] In the multidimensional space shown in Figure 5, the evaluation of the inference of the AI ​​system (sub-AI system) (see symbol P02 in Figure 5) is shown on a straight line (see symbol P01 in Figure 5) that passes through the point that combines the accuracy for the object recognition test data and the accuracy for the pose recognition test data, and is parallel to the AI ​​system accuracy axis. The multidimensional space shown in Figure 5 can be said to be a multidimensional space with each of the test data for each deep learning model as a dimensional axis.

[0054] In addition, in this multidimensional space, it is possible to virtually represent a threshold surface (see symbol P03 in Figure 5) that serves as a standard for the user's evaluation of the inference results of the AI ​​system. In other words, if the inference result is above this threshold surface, the inference result by the AI ​​system is acceptable to the user (OK), and if the inference result is below this threshold surface, the inference result by the AI ​​system is not acceptable to the user (NG).

[0055] Therefore, rough information about the AI ​​system (user evaluation of the system) can be estimated from the accuracy of the object recognition test data and the accuracy of the pose recognition test data. FIG. 6 is a diagram illustrating a multi-dimensional space representing AI system accuracy.

[0056] In the multidimensional space shown in Figure 6, in the multidimensional space illustrated in Figure 5, the user's evaluation (OK or NG) obtained by the accuracy verification unit 107 for the estimation results of each sub-AI system is plotted as the accuracy of the sub-AI system, corresponding to the snapshot information of the object recognition model and the snapshot information of the pose recognition model used in configuring the sub-AI system.

[0057] Here, the accuracy of the AI ​​system increases as the accuracy of the object recognition test data increases, and increases as the accuracy of the pose recognition test data increases. Therefore, when the accuracy of the sub-AI system is plotted in a multidimensional space as shown in Figure 6, a right-angled triangular pyramid is virtually formed, with the plane formed by the accuracy axis for the object recognition test data and the accuracy axis for the pose recognition test data as the base, and standing along the AI ​​system accuracy axis. In this right-angled pyramid, the side parallel to the AI ​​system accuracy axis (perpendicular line) is the side farthest from the AI ​​system accuracy axis.

[0058] Then, by plotting the evaluations of each AI system acquired by the AI ​​system accuracy analysis unit 106 in a multidimensional space, multiple points representing the user's evaluation (OK) of the inference results of the AI ​​system are placed on the threshold surface. Multiple points representing the user's evaluation (OK) of the inference results of these AI systems are placed on the surface where the right-angled pyramid and the threshold surface overlap. The surface where the virtual right-angled pyramid representing the accuracy of the AI ​​system and the threshold surface overlap (see P04 in Figure 6) represents the boundary surface of the threshold of the user's evaluation of the inference results of the AI ​​system. In the example shown in Figure 6, the triangle formed by overlapping the right-angled pyramid representing the accuracy of the AI ​​system and the threshold plane may be referred to as the threshold boundary.

[0059] In the multidimensional space illustrated in Figure 6, by referring to the accuracy axis for the object recognition test data and the accuracy axis for the pose recognition test data corresponding to the threshold boundaries, it is possible to grasp the accuracy for the object recognition test data and the accuracy for the pose recognition test data used to configure the sub-AI system that is required to obtain an accuracy of the sub-AI system that the user will evaluate as OK.

[0060] In other words, the threshold boundary serves as a performance indicator of the multiple deep learning models used to configure the sub-AI system, as well as an accuracy indicator (accuracy standard information) for the object recognition test data and the pose recognition test data, which are necessary to obtain an accuracy of the sub-AI system that meets the user's requirements.

[0061] The accuracy threshold setting unit 108 presents the accuracy axis for the object recognition test data and the accuracy axis for the pose recognition test data, which correspond to the threshold boundary in the multidimensional space, to the user as reference information for setting the accuracy threshold. This reference information may be displayed on the monitor 14a shown in FIG. 10, for example.

[0062] (B) Operation An overview of a method for constructing an AI system in the information processing device 1 as an example of an embodiment configured as described above will be described with reference to the flowcharts shown in Figures 7 and 8. Note that Figure 7 is a flowchart (steps A1 to A4) showing an overview of the method for constructing an AI system, and Figure 8 is a flowchart (steps B1 to B15) showing the details of the processing of step A2.

[0063] In step A1 of Fig. 7, the user considers the configuration of the AI ​​system. For example, the user considers a deep learning model to be used in the configuration of the AI ​​system and determines the number of epochs for training the deep learning model. In step A2 in Fig. 7, an AI system is constructed and its accuracy is analyzed. The details of the process in step A2 will be described with reference to the flowchart shown in Fig. 8.

[0064] In step B1 of FIG. 8, a loop process is started in which the control up to step B9 of FIG. 8 is repeatedly performed for a plurality (s) of deep learning models used in the AI ​​system.

[0065] In step B2 in FIG. 8, a loop process is started in which the control up to step B7 in FIG. 8 is repeatedly performed until the number of loops t performed in training the deep learning model is reached. In step B3 of FIG. 8, the training execution unit 102 trains the deep learning model using the training data. In step B4 of FIG. 8, the inference execution unit 103 inputs the test data to be inferred to the deep learning model and obtains the inference result output by the deep learning model.

[0066] In step B5 of FIG. 8, the snapshot saving processing unit 104 determines whether the accuracy of the inference result acquired by the inference execution unit 103 is equal to or greater than a predetermined threshold (prescribed accuracy threshold).

[0067] If the accuracy of the inference result is equal to or higher than the specified accuracy threshold (see YES route in step B5), in step B6, the snapshot storage processing unit 104 stores a snapshot of the deep learning model. The snapshot (deep learning model) is stored in the model DB. Then, the process proceeds to step B7 in FIG. 8.

[0068] On the other hand, if the confirmation result in step B5 indicates that the accuracy of the inference result is less than the specified accuracy threshold (see NO route in step B5), step B6 is skipped and the process proceeds to step B7.

[0069] In step B7, a loop end process corresponding to step B2 is performed. When training of the deep learning model for a predetermined number of loops t is completed, control proceeds to step B8 in FIG. In step B8, the deep learning model generation unit 101 stores the created trained deep learning model in the model DB.

[0070] In step B9 of Fig. 8, a loop end process corresponding to step B1 is performed. When the creation and training of all s deep learning models is completed, control proceeds to step B10 of Fig. 8.

[0071] In step B10, a loop process is started in which the control up to step B15 in FIG. 8 is repeatedly executed for all combinations of snapshots saved for each of the s deep learning models. In this embodiment, processing is performed for two deep learning models (s=2), an object recognition model and a posture recognition model. That is, a loop process is started in which the control up to step B15 in FIG. 8 is repeatedly executed for all combinations of a plurality of snapshots of the object recognition model and a plurality of snapshots of the posture recognition model stored by the snapshot storage processing unit 104.

[0072] 8, the AI ​​system generation unit 105 extracts one combination from among all combinations of multiple snapshots of the object recognition model and multiple snapshots of the pose recognition model. Then, the AI ​​system generation unit 105 acquires a snapshot model corresponding to this combination from the model DB.

[0073] In step B12 of FIG. 8, the AI ​​system generation unit 105 constructs a sub-AI system using a combination of the acquired snapshot models (object recognition model, pose recognition model).

[0074] In step B13 of Figure 8, the AI ​​system accuracy analysis unit 106 (accuracy verification unit 107) verifies the accuracy of the sub-AI system generated by the AI ​​system generation unit 105. That is, the AI ​​system accuracy analysis unit 106 inputs input data (test data) for each sub-AI system and obtains an inference result output by the sub-AI system. Then, the accuracy verification unit 107 presents the inference result of the AI ​​system to the user and accepts an evaluation input of the inference result from the user.

[0075] In step B14 of Fig. 8, the accuracy threshold setting unit 108 determines the threshold to be set for the accuracy of the sub-AI system. In a multidimensional space, the accuracy threshold setting unit 108 plots the user's evaluation (OK or NG) of the estimation result of each sub-AI system acquired by the accuracy verification unit 107 as the accuracy of the sub-AI system, corresponding to the snapshot information of the object recognition model and the snapshot information of the pose recognition model used in configuring the sub-AI system.

[0076] The accuracy threshold setting unit 108 presents to the user an accuracy axis for the object recognition test data and an accuracy axis for the pose recognition test data that correspond to the threshold boundaries in the multidimensional space as reference information (accuracy reference information) for setting the accuracy threshold.

[0077] In step B15 of Fig. 8, a loop end process corresponding to step B10 is performed. When the processes for all combinations of snapshots are completed, the process proceeds to step A3 of Fig. 7.

[0078] In step A3, optimization is considered based on the result of the accuracy analysis of the AI ​​system performed in step A2. For example, changing the deep learning model used in the AI ​​system to another deep learning model, changing the device (processor or memory) used for calculation, changing the accuracy of numerical expression, etc. Such optimization consideration may be performed by a user, or may be realized by the information processing device 1 according to predefined rules. Thereafter, in step A4 of FIG. 7, optimization is performed from the viewpoint of aspects other than the AI ​​system, and the process ends.

[0079] (C) Effects Thus, according to the information processing device 1 as an example of the embodiment, the snapshot saving processing unit 104 saves a snapshot for each of the multiple types of deep learning models used in the AI ​​system every time the accuracy of the deep learning model exceeds a specified accuracy threshold in each training course. This makes it possible to obtain multiple deep learning models (accuracy patterns) in which the accuracy in the training course for the training data changes stepwise for each deep learning model.

[0080] Furthermore, the AI ​​system generation unit 105 generates a sub-AI system using multiple types of deep learning models generated by the deep learning model generation unit 101. Then, the AI ​​system generation unit 105 constructs such a sub-AI system for all combinations of multiple snapshots between each deep learning model (for example, an object recognition model and a posture recognition model).

[0081] Then, by verifying the estimation results by the sub-AI system (user evaluation input) for all sub-AI systems equipped with a combination of these multiple types of deep learning models, it is possible to verify all possible accuracy patterns for multiple deep learning models.

[0082] Furthermore, the accuracy threshold setting unit 108 generates a threshold boundary by plotting the user's evaluation results in a multidimensional space having the dimensions of the test data of each deep learning model, using the verification results of the accuracy of all these AI systems. By referring to this threshold boundary and projecting the accuracy required by the user for the AI ​​system onto the accuracy for the test data set of each deep learning model, it is possible to clarify the target accuracy of each deep learning model that can realize the AI ​​system desired by the user. This makes it possible to reduce excessive quality for the AI ​​system and reduce the costs of calculations, devices, etc.

[0083] By referring to the threshold boundaries, it is possible to understand the impact that the accuracy of each deep learning model has on the accuracy of the final AI system, and to train each deep learning model efficiently.

[0084] Users only need to provide feedback on the sub-AI system in the form of binary values ​​such as OK / NG or true / false, and do not need to input quantitative accuracy information. Even if the test data for the AI ​​system does not contain label information, the accuracy of each of multiple deep learning models can be quantitatively measured. In other words, there is no need to prepare label information for test data for the AI ​​system, which reduces the significant cost of preparing label information. FIG. 9 is a diagram illustrating an effect of an AI system generated by the information processing device 1 as an example of an embodiment.

[0085] The example shown in Fig. 9 shows an example of constructing an AI system that recognizes objects in an image. In Fig. 9, symbol A shows an image of test data for object recognition, and symbol B shows an example of the relationship between a machine learning model, a recognition rate (car_detection), and processing time.

[0086] When performing object recognition on an image such as that shown in the example of symbol A, the processing time required for object recognition varies depending on the required recognition accuracy for cars, for example. Generally, the smaller the car, the harder it is to detect and the lower the accuracy, so the "size of the recognizable subject" or the like may be given as the discrimination threshold.

[0087] In the example shown by symbol B, Yolov3_normal, Yolov3_light_image, and Tiny_yolo are shown as deep learning models (machine learning models) used for object recognition.

[0088] Regarding the car recognition rate for the same test data, Yolov3_normal performs calculations using normal input image settings, while Yolov3_light_image performs calculations by reducing the input image. Also, Tiny_yolo is a different machine learning model from Yolov3 and has a very light calculation load.

[0089] As shown by symbol B, when car recognition was performed on the test data using Yolov3_normal, the processing time was 23.675 ms and the recognition rate (recognition accuracy) was 88.52%.

[0090] In addition, when using Yolov3_light_image to recognize cars on the same test data, the processing time was 18.412 ms and the recognition rate (recognition accuracy) was 86.96%. In other words, by allowing a 1.56% decrease in the recognition rate compared to Yolov3_normal, the processing time performance was improved by 1.28 times.

[0091] In addition, when Tiny_yolo was used to recognize cars on the same test data, the processing time was 2.220 ms and the recognition rate (recognition accuracy) was 84.06%. In other words, by allowing a 3.92% decrease in the recognition rate compared to Yolov3_normal, the processing time performance was improved by 10.66 times. In this way, by accurately setting the accuracy threshold required for the deep learning model (machine learning model), it is possible to achieve significant improvements in processing time performance.

[0092] 9, an example of changing a deep learning model (machine learning model) used for object recognition is shown, but the present invention is not limited to this. For example, a hardware configuration such as a processor that executes the deep learning model may be changed, and the model can be implemented by appropriately changing the hardware configuration.

[0093] (D) Other FIG. 10 is a diagram illustrating a hardware configuration of an information processing device 1 as an example of an embodiment.

[0094] The information processing device 1 is a computer and includes, as components, a processor 11, a memory 12, a storage device 13, a graphics processing device 14, an input interface 15, an optical drive device 16, a device connection interface 17, and a network interface 18. These components 11 to 18 are configured to be able to communicate with each other via a bus 19.

[0095] The processor (control unit) 11 controls the entire information processing device 1. The processor 11 may be a multiprocessor. The processor 11 may be, for example, any one of a CPU, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). The processor 11 may also be a combination of two or more types of elements among the CPU, MPU, DSP, ASIC, PLD, and FPGA.

[0096] Then, when the processor 11 executes a control program (analysis program, machine learning program, not shown) for the information processing device 1, the functions of the deep learning model generation unit 101, the AI ​​system generation unit 105, and the AI ​​system accuracy analysis unit 106 illustrated in Figure 1 are realized.

[0097] The information processing device 1 realizes the functions of the deep learning model generation unit 101, the AI ​​system generation unit 105, and the AI ​​system accuracy analysis unit 106, for example, by executing a program (machine learning program, OS program) recorded on a computer-readable non-transient recording medium. The information processing device 1 also realizes the functions of the AI ​​system generation unit 105 and the AI ​​system accuracy analysis unit 106, for example, by executing a program (analysis program, OS program) recorded on a computer-readable non-transient recording medium.

[0098] The programs describing the processing contents to be executed by the information processing device 1 can be recorded on various recording media. For example, the programs to be executed by the information processing device 1 can be stored in the storage device 13. The processor 11 loads at least a part of the programs in the storage device 13 into the memory 12 and executes the loaded programs.

[0099] The programs to be executed by the information processing device 1 (processor 11) may also be recorded in a non-transitory portable recording medium such as the optical disk 16a, the memory device 17a, or the memory card 17c. The programs stored in the portable recording medium are executable after being installed in the storage device 13, for example, under the control of the processor 11. The processor 11 may also read and execute the programs directly from the portable recording medium.

[0100] The memory 12 is a storage memory including a ROM (Read Only Memory) and a RAM (Random Access Memory). The RAM of the memory 12 is used as a main storage device of the information processing device 1. The RAM temporarily stores at least a part of the program to be executed by the processor 11. The memory 12 also stores various data necessary for processing by the processor 11.

[0101] The storage device 13 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a storage class memory (SCM), and stores various data. The storage device 13 may store various data such as training data, test data, and snapshots generated by the deep learning model generation unit 101, the AI ​​system generation unit 105, and the AI ​​system accuracy analysis unit 106. The storage device 13 may also store data constituting a model DB. An OS program, a control program, and various data are stored in the storage device 13. The control program includes an analysis program and a machine learning program.

[0102] As the auxiliary storage device, a semiconductor storage device such as an SCM or a flash memory can also be used. Furthermore, a plurality of storage devices 13 may be used to configure a RAID (Redundant Arrays of Inexpensive Disks).

[0103] A monitor 14a is connected to the graphics processing device 14. The graphics processing device 14 displays an image on the screen of the monitor 14a in accordance with an instruction from the processor 11. Examples of the monitor 14a include a display device using a CRT (Cathode Ray Tube) and a liquid crystal display device.

[0104] A keyboard 15a and a mouse 15b are connected to the input interface 15. The input interface 15 transmits signals sent from the keyboard 15a and the mouse 15b to the processor 11. The mouse 15b is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

[0105] The optical drive device 16 uses a laser beam or the like to read data recorded on an optical disk 16a. The optical disk 16a is a portable, non-transient recording medium on which data is recorded so that it can be read by the reflection of light. Examples of the optical disk 16a include a DVD (Digital Versatile Disk), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), and a CD-R (Recordable) / RW (ReWritable).

[0106] The device connection interface 17 is a communication interface for connecting peripheral devices to the information processing device 1. For example, a memory device 17a or a memory reader / writer 17b can be connected to the device connection interface 17. The memory device 17a is a non-transitory recording medium equipped with a communication function with the device connection interface 17, such as a Universal Serial Bus (USB) memory. The memory reader / writer 17b writes data to a memory card 17c or reads data from the memory card 17c. The memory card 17c is a card-type non-transitory recording medium.

[0107] The network interface 18 is connected to a network. The network interface 18 transmits and receives data via the network. Other information processing devices, communication devices, etc. may be connected to the network.

[0108] The disclosed technology is not limited to the above-described embodiment, and may be modified in various ways without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment may be selected as necessary, or may be combined as appropriate.

[0109] For example, in the above-described embodiment, an example was shown in which the AI ​​system is a behavior estimation system that combines two types of deep learning models, an object recognition model and a posture recognition model, but the present invention is not limited to this. The AI ​​system is not limited to the behavior estimation system, and may include a deep learning model other than the object recognition model and the posture recognition model.

[0110] The AI ​​system may also include one or three or more deep learning models. In the above-described embodiment, an example is shown in which the accuracy threshold setting unit 108 generates a threshold boundary using a three-dimensional (three-axis) multidimensional space in correspondence with two deep learning models included in the AI ​​system. Therefore, for example, when the AI ​​system includes one deep learning model, the threshold boundary may be set using a two-dimensional (two-axis) multidimensional space. When the AI ​​system includes three or more deep learning models, the threshold boundary may be set using a four-dimensional (four-axis) or more multidimensional space. Furthermore, the above disclosure enables a person skilled in the art to implement and manufacture the present embodiment.

[0111] (E) Notes Regarding the above embodiment, the following supplementary notes are further disclosed. (Appendix 1) For an AI (Artificial Intelligence) system that combines multiple deep learning models, a discrete threshold evaluation is used to create accuracy standard information for the multiple deep learning models in a space in which accuracy evaluation information is projected in multiple dimensions. An analysis program that causes a computer to carry out processing.

[0112] (Appendix 2) The AI ​​system is configured by combining a first deep learning model and a second deep learning model, based on a first snapshot group consisting of a plurality of first snapshots saved each time the accuracy of the first deep learning model becomes equal to or greater than a unit threshold during a training process of the first deep learning model, and a second snapshot group consisting of a plurality of second snapshots saved each time the accuracy of the second deep learning model becomes equal to or greater than a unit threshold during a training process of the second deep learning model, creating a plurality of sub-AI systems each combining a first deep learning model corresponding to a first snapshot extracted from the first snapshot group and a second deep learning model corresponding to a second snapshot extracted from the second snapshot group; The threshold evaluations for the plurality of sub-AI systems are projected onto the space to generate accuracy reference information for the first deep learning model and the second deep learning model. 2. The analysis program according to claim 1, which causes the computer to execute a process.

[0113] (Appendix 3) For an AI (Artificial Intelligence) system that combines multiple deep learning models, a discrete threshold evaluation is used to create accuracy standard information for the multiple deep learning models in a space in which accuracy evaluation information is projected in multiple dimensions. An analytical method characterized in that the processing is carried out by a computer.

[0114] (Appendix 4) The AI ​​system is configured by combining a first deep learning model and a second deep learning model, based on a first snapshot group consisting of a plurality of first snapshots saved each time the accuracy of the first deep learning model becomes equal to or greater than a unit threshold during a training process of the first deep learning model, and a second snapshot group consisting of a plurality of second snapshots saved each time the accuracy of the second deep learning model becomes equal to or greater than a unit threshold during a training process of the second deep learning model, creating a plurality of sub-AI systems each combining a first deep learning model corresponding to a first snapshot extracted from the first snapshot group and a second deep learning model corresponding to a second snapshot extracted from the second snapshot group; The threshold evaluations for the plurality of sub-AI systems are projected onto the space to generate accuracy reference information for the first deep learning model and the second deep learning model. The analytical method according to claim 3, characterized in that the processing is performed by the computer.

[0115] (Appendix 5) For an AI (Artificial Intelligence) system that combines multiple deep learning models, a discrete threshold evaluation is used to create accuracy standard information for the multiple deep learning models in a space in which accuracy evaluation information is projected in multiple dimensions. A computer having a processing unit.

[0116] (Appendix 6) The AI ​​system is configured by combining a first deep learning model and a second deep learning model, The processing unit, based on a first snapshot group consisting of a plurality of first snapshots saved each time the accuracy of the first deep learning model becomes equal to or greater than a unit threshold during a training process of the first deep learning model, and a second snapshot group consisting of a plurality of second snapshots saved each time the accuracy of the second deep learning model becomes equal to or greater than a unit threshold during a training process of the second deep learning model, creating a plurality of sub-AI systems each combining a first deep learning model corresponding to a first snapshot extracted from the first snapshot group and a second deep learning model corresponding to a second snapshot extracted from the second snapshot group; The threshold evaluations for the plurality of sub-AI systems are projected onto the space to generate accuracy reference information for the first deep learning model and the second deep learning model. 6. The computer of claim 5,

[0117] (Appendix 7) During a training process of a first deep learning model, a first snapshot group consisting of a plurality of first snapshots saved each time the accuracy of the first deep learning model becomes equal to or greater than a unit threshold is stored in a storage device; During a training process of a second deep learning model, a second snapshot group consisting of a plurality of second snapshots saved each time the accuracy of the second deep learning model becomes equal to or greater than a unit threshold is stored in the storage device; Creating a plurality of sub-AI systems each combining a first deep learning model corresponding to a first snapshot extracted from the first snapshot group and a second deep learning model corresponding to a second snapshot extracted from the second snapshot group; By projecting the threshold evaluations for the plurality of sub-AI systems into a space in which accuracy evaluation information is projected multidimensionally, accuracy standard information for the first deep learning model and the second deep learning model is created. A machine learning program that causes a computer to carry out processing. [Explanation of symbols]

[0118] 1. Information processing device 11 Processor (processing unit) 12 RAM 13 HDD 14 Graphics Processing Unit 14a Monitor 15 Input Interface 15a Keyboard 15b Mouse 16 Optical drive device 16a Optical disc 17 Device connection interface 17a Memory Device 17b Memory Reader / Writer 17c Memory Card 18 Network Interface 18a Network 19 Bus 101 Deep learning model generation unit 102 Training Execution Department 103 Inference execution unit 104 Snapshot storage processing unit 105 AI System Generation Department 106 AI System Accuracy Analysis Department 107 Accuracy Verification Department 108 Accuracy threshold setting unit

Claims

1. For an AI (Artificial Intelligence) system that combines multiple deep learning models, accuracy standard information for the multiple deep learning models is created in a space in which accuracy evaluation information is projected multidimensionally using discrete threshold evaluation. An analysis program that causes a computer to carry out processing.

2. The AI ​​system is configured by combining a first deep learning model and a second deep learning model, based on a first snapshot group consisting of a plurality of first snapshots saved each time the accuracy of the first deep learning model becomes equal to or greater than a unit threshold during a training process of the first deep learning model, and a second snapshot group consisting of a plurality of second snapshots saved each time the accuracy of the second deep learning model becomes equal to or greater than a unit threshold during a training process of the second deep learning model, creating a plurality of sub-AI systems each combining a first deep learning model corresponding to a first snapshot extracted from the first snapshot group and a second deep learning model corresponding to a second snapshot extracted from the second snapshot group; By projecting the threshold evaluations for the plurality of sub-AI systems into the space, accuracy reference information for the first deep learning model and the second deep learning model is created.

2. The analysis program according to claim 1, which causes the computer to execute a process.

3. For an AI (Artificial Intelligence) system that combines multiple deep learning models, accuracy standard information for the multiple deep learning models is created in a space in which accuracy evaluation information is projected multidimensionally using discrete threshold evaluation. An analytical method characterized in that the processing is carried out by a computer.

4. For an AI (Artificial Intelligence) system that combines multiple deep learning models, accuracy standard information for the multiple deep learning models is created in a space in which accuracy evaluation information is projected multidimensionally using discrete threshold evaluation. A computer having a processing unit.

5. During a training process of a first deep learning model, a first snapshot group consisting of a plurality of first snapshots saved each time the accuracy of the first deep learning model becomes equal to or greater than a unit threshold is stored in a storage device; During a training process of a second deep learning model, a second snapshot group consisting of a plurality of second snapshots saved each time the accuracy of the second deep learning model becomes equal to or greater than a unit threshold is stored in the storage device; Creating a plurality of sub-AI systems each combining a first deep learning model corresponding to a first snapshot extracted from the first snapshot group and a second deep learning model corresponding to a second snapshot extracted from the second snapshot group; By projecting the threshold evaluations for the plurality of sub-AI systems into a space in which accuracy evaluation information is projected multidimensionally, accuracy standard information for the first deep learning model and the second deep learning model is created. A machine learning program that causes a computer to carry out processing.

Citation Information

Patent Citations

  • JP1977000714A

  • Object detector, learning device, and object detection system, method, and program

    JP2006268825A

  • Person estimation system and estimation program

    JP2018173707A

  • Image processing unit, image processing method

    JP2019016298A

  • Protection system including machine learning snapshot evaluation

    US20170230394A1