Setting assistance device, setting assistance method, and setting assistance program

Abstract

When the inference result of a first inference with a light processing load satisfies a threshold condition, a setting assistance device (100) varies a threshold value used for two-layer inference for performing a second inference with a processing load heavier than that of the light inference, and collates, with correct answer data, the result of two-layer inference for each varied threshold value, the result being obtained by performing two-layer inference for each varied threshold value. The setting assistance device (100) then creates a correlation graph between the varied threshold value and the accuracy and performance of two-layer inference for each varied threshold value, the accuracy and performance being calculated as the result of collation.

Description

Setting assistance device, setting assistance method, and setting assistance program

[0001] The present invention relates to a setting assistance device, a setting assistance method, and a setting assistance program.

[0002] In recent years, advances in deep learning and the widespread use of IoT (Internet of Things) devices have led to the development of applications used for automatic analysis of surveillance camera footage, autonomous driving, and other applications. These applications perform predetermined processing based on the results of analyzing video and images, so real-time performance is important. Therefore, to ensure real-time performance, it is desirable for deep neural network (DNN) inference and other processes to be performed on edge devices such as IoT devices.

[0003] However, due to significant resource constraints on edge devices, it is difficult to run a highly accurate DNN model resident on the device. Therefore, in order to achieve both accuracy and real-time performance, a framework has been proposed in which low-accuracy but low-load inference (hereinafter sometimes referred to as "light inference") is performed on the edge device, and high-accuracy inference (hereinafter sometimes referred to as "heavy inference") is performed on the cloud side when it is determined to be necessary. For example, when performing DNN inference, a technique is known in which the necessity of the next inference is determined based on the output result, such as the confidence level of the light inference, and a threshold value, and heavy inference processing is performed when it is determined to be necessary, thereby reducing the frequency of heavy inference and improving the performance of the entire system (see, for example, Non-Patent Document 1).

[0004] NoScope: Optimizing Neural Network Queries over Video at Scale <URL: https: / / www.vldb.org / pvldb / vol10 / p1586-kang.pdf> <Retrieved September 25, 2023>

[0005] However, the above-mentioned conventional technology has a problem in that it is not easy to tune the threshold value used for inference. For example, the threshold value used to determine whether or not to perform heavy inference is a delicate value that affects accuracy and performance, so tuning is required to achieve the required accuracy and performance depending on the inference use case. However, the process of verifying each threshold that balances accuracy and performance and determining the optimal threshold is a heavy load.

[0006] Therefore, in order to solve the above-mentioned problems and achieve the object, the setting assistance device of the present invention is characterized by having a matching unit that varies the threshold value used in two-layer inference, which performs a second inference that has a heavier processing load than the first inference, when the inference result of a first inference that has a lighter processing load satisfies the threshold value condition, and compares the results of the two-layer inference for each varied threshold value obtained by performing the two-layer inference for each varied threshold value with correct answer data, and a creation unit that creates a correlation graph between the varied threshold value and the accuracy and performance of the two-layer inference for each varied threshold value calculated as the matching result by the matching unit.

[0007] The present invention has the effect of facilitating tuning of a threshold value used in inference.

[0008] FIG. 1 is a diagram illustrating an overall view of processing by the setting assistance device according to this embodiment. FIG. 2 is a diagram illustrating an example of processing by the setting assistance device according to this embodiment. FIG. 3 is a diagram illustrating an example of the configuration of the setting assistance device according to this embodiment. FIG. 4 is a table diagram illustrating an example of a matching result according to this embodiment. FIG. 5 is a table diagram illustrating an example of an inference condition according to this embodiment. FIG. 6 is a diagram illustrating an example of output setting assistance information according to this embodiment. FIG. 7 is a diagram illustrating an example of setting assistance information generation according to this embodiment. FIG. 8 is a diagram illustrating an example of setting assistance information generation according to this embodiment. FIG. 9 is a diagram illustrating an example of setting assistance information generation according to this embodiment. FIG. 10 is a diagram illustrating an example of setting assistance information generation according to this embodiment. FIG. 11 is a diagram illustrating an example of setting assistance information generation according to this embodiment. FIG. 12 is a diagram illustrating an example of setting assistance information generation according to this embodiment. FIG. 13 is a diagram illustrating an example of setting assistance information generation according to this embodiment. FIG. 14 is a flowchart illustrating an example of a setting assistance information generation procedure according to this embodiment. FIG. 15 is a diagram illustrating an example of a computer that executes a setting assistance program according to this embodiment.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the embodiments are not limited to the following description.

[0010] <Explanation of an Overall Picture of Processing by the Setting Assistance Device> Figure 1 is a diagram illustrating an overall picture of processing by the setting assistance device 100 according to this embodiment. The setting assistance device 100 shown in Figure 1 is an example of a computer that provides a technology that repeatedly performs two-layer inference while changing a threshold value (hereinafter, may be simply referred to as "threshold value") used to determine whether or not to perform heavy inference in two-layer inference, compares the inference results of the two-layer inference with ground truth data, and creates a correlation graph between the threshold value, accuracy, and performance.

[0011] As described above, it is desirable to perform DNN inference and the like on the edge side of IoT devices, etc., in order to ensure real-time performance. However, due to resource constraints, it is difficult for edge devices to have a highly accurate DNN model resident and operational.

[0012] Therefore, a reference technology is known that divides inference into two stages (two-layer inference) to reduce the frequency of heavy inference, which requires a large processing load, and thereby improve the performance of the entire two-layer inference system. For example, as shown in Figure 1, a two-layer inference system 10 outputs predetermined numerical values ​​such as a difference degree and a certainty degree by performing light inference on the edge side using a lightweight model with a low processing load, such as a small amount of calculation, and compares the results with a preset threshold. Then, the two-layer inference system 10 performs heavy inference processing on the cloud side only if, as a result of the comparison, it is determined that heavy inference (high-precision DNN inference) should be performed.

[0013] The thresholds used in the two-layer inference system 10 described above are sensitive values ​​that affect accuracy, performance, and the like, and therefore require tuning to suit the purpose of the inference. However, such tuning is not easy. For example, the performance required for two-layer inference is defined by the application of the user using the two-layer inference. The thresholds are then determined to meet the required performance. However, accuracy and performance are in a trade-off relationship that fluctuates depending on the threshold setting, such that emphasizing performance results in a decrease in accuracy. Therefore, users using two-layer inference must tune the thresholds according to the accuracy and performance required for each use case of two-layer inference. However, when tuning, comparing the balance between accuracy and performance for each threshold and verifying the optimal threshold is a significant amount of work and is not easy.

[0014] Therefore, the setting assistance device 100 according to this embodiment varies the threshold value used in the second-layer inference, which performs an inference (second inference) that has a heavier processing load than the light inference when the inference result of the inference (first inference) with a lighter processing load satisfies the threshold value condition, and performs the second-layer inference for each varied threshold value. The result of the second-layer inference for each varied threshold value is compared with the correct answer data. The setting assistance device 100 then creates a correlation graph between the varied threshold value and the accuracy and performance of the second-layer inference calculated as the comparison result for each varied threshold value.

[0015] Specifically, as shown in (1) of Fig. 1, the setting assistance device 100 performs two-layer inference using moving images, still images, etc. as input data. At this time, the setting assistance device 100 repeatedly performs two-layer inference while varying the threshold value used for judgment based on predetermined conditions, and outputs the result of the two-layer inference for each varied threshold value.

[0016] As shown in (2) of Figure 1, the setting assistance device 100 compares the output inference results of the two-layer inference with correct answer data such as data generated by heavy inference using a high-precision second-layer model or generated data generated in advance by hand (e.g., ground truth).

[0017] As shown in (3) of Figure 1, the setting assistance device 100 creates a correlation graph or the like in which predetermined indicators indicating the accuracy calculated by comparing the inference results of the two-layer inference with correct data and the performance calculated from the results of the two-layer inference are associated with each of the varied thresholds.

[0018] In this way, the setting assistance device 100 automatically creates a correlation graph showing the relationship between accuracy, performance, and thresholds based on the results of matching the results of two-layer inference with correct data, thereby easily enabling tuning of thresholds to suit the purpose and needs of the inference.

[0019] <Description of Setting Assistance Device> Next, the setting assistance device 100 will be described. First, the processing by the setting assistance device 100 according to this embodiment will be described. FIG. 2 is a diagram showing an example of processing by the setting assistance device 100 according to this embodiment. FIG. 2 shows an example in which the setting assistance device 100 outputs setting assistance information (hereinafter, sometimes simply referred to as "setting assistance information") such as a correlation graph correlating varied thresholds with the accuracy and performance of two-layer inference based on the results of comparing the results of two-layer inference for each threshold with the correct answer data, recommended thresholds, and comparison results of changes in inference results, and the output setting assistance information is used to set thresholds, etc. of the two-layer inference system 10.

[0020] First, as shown in (1) of Figure 2, the setting assistance device 100 accepts specifications such as input data to be input into the model used for two-layer inference, models to be used for light and heavy inference of two-layer inference, correct answer data to be used in the matching process, and inference conditions for two-layer inference.

[0021] As shown in Figure 2 (2), the setting assistance device 100 inputs specified input data into the model used for two-layer inference and performs light inference while varying the thresholds used for judgment based on specified inference conditions.The setting assistance device 100 then performs heavy inference when the difference degree, certainty factor, etc. obtained as a result of the light inference satisfy the judgment conditions for each of the varied thresholds, and outputs the result obtained from the heavy inference as the result of two-layer inference.The setting assistance device 100 repeatedly performs the above-mentioned process according to the specified inference conditions.

[0022] On the other hand, as shown in (3) of Figure 2, in order to prepare correct answer data, the setting assistance device 100 performs heavy inference to generate correct answer data to be used in the matching process, and accepts pre-generated data (ground truth).

[0023] The setting assistance device 100 compares the inference results of the two-layer inference with the correct answer data, as shown in (4) of Fig. 2. Then, based on the comparison results, the setting assistance device 100 creates setting assistance information such as a correlation graph that associates accuracy and performance for each threshold value, recommended threshold values, and comparison results of changes in inference results, as shown in (5) of Fig. 2.

[0024] The setting assistance device 100 outputs to the user, etc., setting assistance information such as a correlation graph that associates the created accuracy with the performance of the two-layer inference, recommended thresholds, and comparison results of changes in inference results, as shown in (6) of Fig. 2. Then, as shown in (7) of Fig. 2, the user, etc., uses the output setting assistance information to set thresholds, etc. for the two-layer inference system 10, etc.

[0025] (Setting Assistance Device 100) Next, the configuration of the setting assistance device 100 and the functions of each functional unit will be described. FIG. 3 is a diagram showing an example of the configuration of the setting assistance device 100 according to this embodiment. As shown in FIG. 3, the setting assistance device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Although not shown in FIG. 3, the setting assistance device 100 can also have an input unit such as a keyboard or mouse for accepting input such as operations by the user, etc. Furthermore, the setting assistance device 100 can also have a display unit such as a display for displaying setting assistance information to the user, etc.

[0026] (Communication Unit 110) The communication unit 110 performs data communication related to the reception of input data and correct answer data, the output of information related to calculated thresholds, etc. The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and controls communication via electric communication lines such as a LAN (Local Area Network) or the Internet. The communication unit 110 is connected to a network via a wired or wireless connection as necessary, and can transmit and receive information bidirectionally.

[0027] (Storage Unit 120) The storage unit 120 stores data and programs used for various processes by the control unit 130, as well as various data acquired through operation of the control unit 130. The storage unit 120 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 3 , the storage unit 120 has an input data DB 121, a correct answer data DB 122, a matching result DB 123, and an inference condition DB 124.

[0028] (Input Data DB 121) The input data DB 121 is a database that stores input data to be input to a model used in two-layer inference. For example, the input data DB 121 stores still images, moving images (videos), etc. as input data to be input to the model.

[0029] (Supervised Data DB 122) The supervised data DB 122 is a database that stores supervised data used in a matching process performed by the matching unit 134, which will be described later. For example, the supervised data DB 122 stores, as supervised data, results of object detection (e.g., bounding boxes, etc.) generated by the generating unit 132, which will be described later, results of posture estimation, etc. (e.g., data on key points and bones, etc.), generated data (ground truth) generated in advance by hand or by another information processing device, etc.

[0030] (Matching result DB 123) The matching result DB 123 is a database that stores the results of the matching process performed by the matching unit 134, which will be described later. Specifically, the matching result DB 123 stores predetermined indices of accuracy and performance associated with each varied threshold value for each combination of models to be used that is included in the specified inference conditions.

[0031] Here, an example of the matching result stored in the matching result DB 123 will be described. Fig. 4 is a table diagram showing an example of the matching result according to this embodiment. As shown in Fig. 4, the matching result DB 123 stores the accuracy, performance, recommended threshold value, etc. in association with each varied threshold value.

[0032] The thresholds mentioned above include one or more thresholds varied based on inference conditions specified by a user or the like. Accuracy is an evaluation index related to the result of matching the inference result of two-layer inference with correct data, and includes accuracy rate, recall rate, F-measure, etc. Performance is an index representing the performance of two-layer inference, such as the proportion of heavy inferences performed using the second-layer model in two-layer inference, and latency, which is an index of real-time performance. The recommended threshold is a threshold that meets the criteria specified by parameters included in the inference conditions described below, and is information identifying whether the threshold is recommended to a user or the like.

[0033] 4, the matching result DB 123 stores matching results such as accuracy of 80%, performance of 65%, and recommended threshold of 0 (recommended), which correspond to the threshold of 0.3 among the varied thresholds. Note that the recommended threshold of x indicates that the threshold is not recommended.

[0034] (Inference Condition DB 124) The inference condition DB 124 is a database that stores inference conditions for creating auxiliary setting information. The inference condition DB 124 stores the inference conditions specified by a user, etc., in association with the used model, input data, correct answer data, evaluation index, and parameters.

[0035] Here, an example of an inference condition stored in the inference condition DB 124 will be described. Fig. 5 is a table diagram showing an example of an inference condition according to this embodiment. For example, the inference condition DB 124 stores models to be used, such as a first-layer model used for two-layer inference and a second-layer model used for heavy inference, which are associated with a unique number or the like that identifies the inference condition, input data, correct answer data, evaluation indexes such as accuracy and performance, and parameters such as target values ​​for the evaluation indexes, threshold ranges, and the number of inference iterations.

[0036] The above-mentioned model to be used is information indicating which model is to be used, such as a model used for moving object detection, a model used for object detection, or a model used for pose estimation. The input data is information indicating still images, moving images, etc. to be input into a model for performing two-layer inference. The correct answer data is information indicating the correct answer data used in the matching process by the matching unit 134 (described later). The target value among the parameters is a parameter for determining a standard used when the recommended threshold is generated by the creation unit 135 (described later). The threshold range among the parameters is a threshold variation range when light inference is performed by varying the threshold by the inference unit 133 (described later). The number of inference repetitions among the parameters is a parameter indicating how many times light inference is performed within the threshold variation range. The accuracy and performance among the evaluation indexes are the same as those described in FIG. 4, and therefore will not be described here.

[0037] For example, the inference condition DB 124 stores inference conditions such as a first layer model "motion detection model," a second layer model "object detection model," input data "verification video A," correct answer data "correct answer data A," accuracy "correct answer rate," performance "proportion of frames processed by the first layer model," target values ​​"accuracy: 70%, performance: 20%," threshold range "0 to 1," and number of inference repetitions "10 times," all associated with No. "1."

[0038] (Control Unit 130) The control unit 130 has an internal memory for temporarily storing programs and processing data that define various processing procedures of the setting assistance device 100, and is realized by electronic circuits such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), and integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). As shown in FIG. 3 , the control unit 130 has a reception unit 131, a generation unit 132, an inference unit 133, a comparison unit 134, a creation unit 135, and an output unit 136.

[0039] The receiving unit 131 receives the specification of inference conditions for creating auxiliary setting information. Specifically, the receiving unit 131 receives the usage model, input data, correct answer data, evaluation index, and parameters via the input unit and communication unit 110, and stores them in the inference condition DB 124.

[0040] Furthermore, the receiving unit 131 receives input data used in two-layer inference by the inference unit 133 (described later) via the communication unit 110 or the like, and stores the input data in the input data DB 121. Furthermore, when supervised answer data to be used in the matching process by the matching unit 134 (described later) is not generated by the generation unit 132 (described later), the receiving unit 131 receives separately generated supervised answer data via the input unit or communication unit 110 and stores the supervised answer data in the supervised answer data DB 122. For example, in the case of the supervised answer data "supervised answer data C" shown in FIG. 5 , the receiving unit 131 receives data generated in another information processing device or manually generated data as supervised answer data (ground truth) and stores the data in the supervised answer data DB 122.

[0041] (Generation Unit 132) The generation unit 132 generates correct answer data using a second-layer model used for heavy inference based on inference conditions specified by a user, etc. For example, in the case of the correct answer data "correct answer data A" shown in FIG. 5, the generation unit 132 generates correct answer data including an equal number of positive and negative examples, with frames in which at least one object is detected being positive examples and frames in which no object is detected being negative examples, among the results of object detection based on the second-layer model. Also, in the case of the correct answer data "correct answer data B" shown in FIG. 5, the generation unit 132 generates a bounding box as a result of object detection based on the second-layer model.

[0042] (Inference unit 133) The inference unit 133 compares a predetermined value obtained by performing light inference with each of the varied thresholds, and performs heavy inference if the determination condition for whether to perform heavy inference is met, and outputs the inference result of the two-layer inference. For example, the inference unit 133 performs heavy inference if the difference degree, certainty factor, etc. obtained as a result of performing inference using the first layer model used for light inference meets the determination condition for each of the varied thresholds, and outputs the obtained two-layer inference result.

[0043] As an example, the inference unit 133 performs heavy inference when the determination condition "if the difference degree calculated by light inference exceeds a threshold (lower threshold)," is satisfied, and outputs the obtained result of the two-layer inference to the matching unit 134. Also, the inference unit 133 performs heavy inference when the determination condition "if the certainty degree calculated by light inference is below a threshold (upper threshold)," is satisfied, and outputs the obtained result of the two-layer inference to the matching unit 134.

[0044] In addition, if the obtained difference degree, certainty degree, etc. do not satisfy the judgment conditions related to the threshold value, the inference unit 133 may output information such as ``Judgment conditions not satisfied, no two-layer inference output'' as the result of the two-layer inference.

[0045] (Matching unit 134) The matching unit 134 varies the threshold used in the second-layer inference, performs the second-layer inference for each varied threshold, and matches the results of the second-layer inference for each varied threshold with the correct answer data. Specifically, the matching unit 134 compares the results of the second-layer inference with pre-created correct answer data and calculates predetermined indicators such as the accuracy of the second-layer inference as matching results. Furthermore, the matching unit 134 uses the results of the second-layer inference to calculate predetermined indicators such as the proportion of heavy inferences using the second-layer model and performance indicators such as latency, which is an indicator of real-time performance. The matching unit 134 then stores the matching results in the matching result DB 123.

[0046] For example, when the light inference in the two-layer inference is moving object detection and the heavy inference is object detection, the matching unit 134 calculates the accuracy rate as an index of precision and the ratio of frames processed by the first-layer model as an index of performance based on the inference condition No. "1" shown in FIG. The matching unit 134 then stores the calculated results as matching results in the matching result DB 123. Here, the accuracy rate is the rate of agreement between the inference result by the two-layer inference and the correct data. In other words, it is an index indicating how accurately the two-layer inference (moving object detection, object detection, etc.) is performed on the input data. The ratio of frames processed by the first-layer model is the number of data (frames) processed using only light inference relative to the number of input data (frames).

[0047] The accuracy rate described above is calculated, for example, by the formula "Accuracy rate = (TP + TN) ÷ (TP + TN + FP + FN)." TP is the number of frames in the positive examples where the inference result of light inference is "moving object present." FN is the number of frames in the positive examples where the inference result of light inference is "no moving object." FP is the number of frames in the negative examples where the inference result of light inference is "moving object present." TN is the number of frames in the positive examples where the inference result of light inference is "no moving object." Positive examples here are "frames in which at least one object is detected." Negative examples are "frames in which no object is detected."

[0048] Specifically, the collation unit 134 compares the result of the two-layer inference "moving object present (heavy inference performed)" with the correct answer data, and calculates the number of frames that fall into the categories of "TP (object present)" and "FP (no object present)" among the "frames determined to have a moving object." The collation unit 134 also compares the result of the two-layer inference "no moving object (heavy inference not performed)" with the correct answer data, and calculates the number of frames that fall into the categories of "FP (moving object present)" and "FN (no moving object)" among the "frames determined to have no moving object." The collation unit 134 then calculates the accuracy rate by applying the number of frames that fall into the categories of "TP, TN, FP, FN" to the formula for calculating the accuracy rate described above.

[0049] Furthermore, the matching unit 134 can calculate the recall rate of positive examples and the recall rate of negative examples. The recall rate of positive examples here is an index indicating the degree of missed detections, and for example, a higher recall rate of positive examples indicates fewer missed detections. The recall rate of negative examples is an index indicating the accuracy rate of frames in which it is determined that an object is not detected, and a higher value indicates a more accurate determination (i.e., fewer missed detections).

[0050] The recall rate of the positive examples described above is calculated, for example, by the formula "recall rate of positive examples = TP ÷ (TP + FN)." The recall rate of negative examples is calculated, for example, by the formula "recall rate of negative examples = TN ÷ (TN + FP)."

[0051] On the other hand, when the light inference is object detection and the heavy inference is object detection, the matching unit 134 calculates the recall as an index of accuracy and the ratio of frames for which processing has ended in the first-layer model as an index of performance based on the condition No. "2" of the inference conditions shown in Fig. 5. Then, the matching unit 134 stores the calculated results in the matching result DB 123 as matching results.

[0052] The above-mentioned recall is calculated based on IoU (Intersection over Union). Specifically, when the result of object detection using heavy inference is regarded as correct data, the matching unit 134 calculates the IoU of a bounding box (hereinafter, sometimes referred to as "BB") obtained using light inference and the BB obtained using heavy inference. For example, the matching unit 134 calculates the IoU using the formula "IoU = (intersection area of ​​the BB obtained using light inference and the BB obtained using heavy inference) / (total area obtained by combining the BB obtained using light inference and the BB obtained using heavy inference)."

[0053] The matching unit 134 determines that detection is successful when the IoU is equal to or greater than a threshold value for matching processing, and determines that detection is unsuccessful when the IoU is less than the threshold value for matching processing. Then, using the above-mentioned determination results of "detection is successful" and "unsuccessful detection," the matching unit 134 calculates the recall rate based on the formula "recall rate = (number of successful detections) / (number of successful detections + number of unsuccessful detections)." Note that the recall rate here indicates the ratio of the number of frames detected (object detection) by heavy inference to the number of frames detected (object detection) by light inference. Furthermore, the IoU is an index that indicates the proportion of image overlap, with a larger IoU indicating greater image overlap.

[0054] Furthermore, the matching unit 134 can calculate a precision rate, etc. For example, the matching unit 134 can calculate the precision rate using the formula "precision rate = (number of successful detections) / (number of BBs obtained by light inference)." Note that the precision rate here is the number of BBs that were also detected by heavy inference (number of successful detections) out of the number of BBs obtained by light inference, and is an index that indicates the degree to which the inference results of light inference and heavy inference match.

[0055] (Creation unit 135) The creation unit 135 creates a correlation graph between the varied thresholds and the accuracy and performance for each varied threshold calculated as the matching result by the matching unit 134. Note that the creation unit 135 can also create a correlation graph in which at least one of the accuracy and performance calculated as a predetermined index is associated with each varied threshold. Furthermore, the creation unit 135 can create the correlation graph using the matching results stored in the matching result DB 123.

[0056] When a predetermined index falls within a predetermined standard range, the creation unit 135 calculates a threshold value associated with the predetermined index as a recommended threshold value. For example, the creation unit 135 identifies accuracy and performance that satisfy standards defined by target values ​​of accuracy and performance specified by a user or the like. Then, the creation unit 135 calculates a threshold value associated with the identified accuracy and performance as a recommended threshold value.

[0057] Furthermore, when the average of a predetermined index to which a predetermined weighting is applied falls within a predetermined reference range, the creation unit 135 calculates a threshold value corresponding to the predetermined index as a recommended threshold value. For example, the creation unit 135 weights the accuracy and performance by multiplying them by a predetermined coefficient, and calculates a harmonic mean for each of the weighted accuracy and performance. Then, when the calculated harmonic mean satisfies a condition such as exceeding a predetermined reference, the creation unit 135 calculates a threshold value corresponding to the accuracy and performance related to the harmonic mean as a recommended threshold value.

[0058] The creation unit 135 identifies the results of the two-layer inference in which a predetermined change has occurred due to the change in the threshold. Specifically, when a change in the inference result of the two-layer inference (light inference or heavy inference) occurs due to the change in the threshold, the creation unit 135 calculates the rate of change between the inference results before and after the change, and identifies the inference result that exceeds a predetermined standard as an inference result in which a change has occurred. The creation unit 135 then creates a result of comparing the inference result before the change with the inference result after the change as auxiliary setting information. Note that an example of the auxiliary setting information created by the creation unit 135 will be described later in the section on the output unit 136.

[0059] (Output Unit 136) The output unit 136 outputs auxiliary setting information, such as the correlation graph, recommended threshold, and comparison results of inference result changes created by the creation unit 135, to a user or the like via the input unit or communication unit 110 described above. Here, an example of auxiliary setting information output by the output unit 136 will be described. FIG. 6 is a diagram showing an example of auxiliary setting information output according to this embodiment. FIG. 6 shows a correlation graph ((1) in FIG. 6), recommended thresholds ((1-1) in FIG. 6), and comparison results of inference result changes ((2) in FIG. 6). The correlation graph shown in (1) in FIG. 6 is a graph in which accuracy and performance associated with each varied threshold are plotted. The correlation graph displays the accuracy values ​​and performance values ​​associated with each varied threshold connected by lines.

[0060] The X-axis of the correlation graph represents the varied threshold value, and is set based on the "threshold range" and "number of repeated inference executions" of the inference conditions specified by the user, etc. For example, when the threshold range is "0 to 1" and the number of repeated inference executions is "10 times," the X-axis of the correlation graph is set with the interval for varying the threshold value as "0.1."

[0061] The Y axis on the left side of the correlation graph represents precision values ​​such as the accuracy rate and recall rate calculated based on the matching results of the matching unit 134. On the other hand, the Y axis on the right side of the correlation graph represents performance values ​​such as the ratio of frames for which processing is completed using the first-layer model, the ratio at which second-layer inference is performed, and latency, all calculated based on the matching results of the matching unit 134.

[0062] Furthermore, the output unit 136 can display a recommended threshold on the correlation graph and output the correlation graph. For example, when the creation unit 135 calculates a recommended threshold based on the inference conditions, the output unit 136 can output a correlation graph with a display indicating that the recommended threshold is "0.3", as shown in (1-1) of FIG.

[0063] The comparison result of the inference result change as shown in (2) of Figure 6 is information presented to the user, etc., that a change has occurred in the inference result of the two-layer inference due to the change in the threshold. For example, in the upper diagram of the comparison result of the inference result change as shown in (2) of Figure 6, three objects (people) are detected ((2-1) to (2-3) of Figure 6). On the other hand, in the lower diagram of the comparison result of the inference result change as shown in (2) of Figure 6, two objects (people) are detected ((2-4) to (2-5) of Figure 6). In this way, the output unit 136 can output the comparison result of the upper diagram and the lower diagram, in which the detection results of objects (people) differ due to the change in the threshold.

[0064] (First Example: When Determining Whether Heavy Inference Needs to Be Performed Using a Lower Threshold) An example of the setting assistance process realized by the above-described functions of the setting assistance device 100 will now be described with reference to Figures 7 to 13. Figures 7 to 13 are diagrams showing an example of setting assistance information generation according to this embodiment.

[0065] First, an example of "determining whether or not to perform heavy inference in two-layer inference using a lower threshold" will be described with reference to Figures 7 to 9. Figure 7 shows a two-layer inference system 10 that uses a motion detection model as a first-layer model for light inference and an object detection model as a second-layer model for heavy inference, and a setting assistance device 100 that creates setting assistance information for setting thresholds used in the two-layer inference system 10.

[0066] In the example shown in Figure 7, the two-layer inference system 10 performs heavy inference on the cloud side when the degree of difference obtained by light inference on the edge side exceeds a threshold ("High degree of difference" at the top of Figure 7). In other words, the two-layer inference system 10 shown in Figure 7 performs heavy inference only when a difference of a certain amount or more occurs for each piece of input data, and detects objects that have been previously set as detection targets, such as people, animals, and automobiles, from among the input data. In other words, when the tasks processed by the first-layer model and the second-layer model are different, the two-layer inference system 10 reduces the frequency of heavy inference, which places a heavy processing load, thereby reducing the processing load.

[0067] 7, the setting assistance device 100 performs two-layer inference using the received inference conditions and input data, and compares the inference results of the two-layer inference with the correct answer data to create setting assistance information. In the first example, the setting assistance device 100 outputs a correlation graph, recommended thresholds (recommended thresholds), and the like as setting assistance information.

[0068] Next, an example of auxiliary setting information output by the setting assistance device 100 in the first example will be described. Fig. 8 shows an example of a correlation graph output by the setting assistance device 100. In the correlation graph shown in Fig. 8, the X axis represents the varied threshold value, the Y axis (left) represents the accuracy rate as an index of precision, and the Y axis (right) represents the ratio of frames for which processing has ended in the first layer model as an index of performance.

[0069] As described above, in the first example, the two-layer inference system 10 performs heavy inference when the degree of difference obtained by light inference exceeds a threshold (the lower threshold). That is, in the first example, as the threshold value increases, the frequency with which the degree of difference obtained by light inference exceeds the threshold decreases, and the frequency with which heavy inference is performed decreases. In this way, as the threshold value increases, the frequency with which heavy inference is performed by the second-layer model decreases, so the correlation graph shown in Figure 8 shows a tendency for the accuracy rate to gradually decrease and the proportion of frames for which processing is completed by the first-layer model to gradually increase.

[0070] Furthermore, when outputting a correlation graph, the setting assistance device 100 can also display an operation screen 20 for accepting "selection of an evaluation index." This operation screen 20 includes check boxes for selecting evaluation indexes to be displayed in the correlation graph, such as "accuracy rate," "recall rate," and "proportion of frames for which processing is completed in the first-layer model." In the example of FIG. 8 , "accuracy rate" and "proportion of frames for which processing is completed in the first-layer model" are selected.

[0071] The setting assistance device 100 also outputs a correlation graph displaying recommended thresholds to be presented to the user, etc., as shown in Fig. 9. Specifically, the setting assistance device 100 can calculate, as the recommended threshold, a threshold that satisfies the criteria determined by the target value of the accuracy and the target value of the performance set by the user, etc.

[0072] For example, when the target value for the accuracy rate is set to "70% (or higher)" as shown on the operation screen 21 in FIG. 9 , the setting assistance device 100 identifies, as a recommended threshold candidate, a threshold value of "0 to 0.5" corresponding to the section indicated by the solid line on the Y-axis (left) where the accuracy rate is 70% or higher. Furthermore, when the target value for the proportion of processing-end frames in the first-layer model is set to "20% (or higher)," the setting assistance device 100 identifies, as a recommended threshold candidate, a threshold value of "0.3 to 1" corresponding to the section indicated by the solid line on the Y-axis (right) where the proportion of processing-end frames in the first-layer model is 20% or higher. Then, as shown in FIG. 9 , the setting assistance device 100 calculates, as a recommended threshold, a threshold value of "0.3 to 0.5" that satisfies both the target value for the accuracy rate and the target value for the proportion of processing-end frames in the first-layer model.

[0073] In addition, the setting assistance device 100 can calculate the recommended threshold value as "0.3" which provides higher accuracy in the case of "preferential accuracy", and "0.5" which provides higher performance in the case of "preferential performance".

[0074] Furthermore, the setting assistance device 100 can output a correlation graph displaying a breakdown value for each index. Specifically, the setting assistance device 100 can display a "reproduction rate of positive examples" and a "reproduction rate of negative examples" as breakdown values ​​at each point of any accuracy rate. For example, as shown in FIG. 9 , the setting assistance device 100 can output a correlation graph displaying breakdown values ​​such as a positive example reproduction rate of "80%" and a negative example reproduction rate of "40%" as indicators of an accuracy rate of "80%" at a threshold value of 0.3.

[0075] (Second Example: When Determining Whether Heavy Inference in Two-Layer Inference Needs to Be Performed Using an Upper Threshold Value) Next, an example of "When Determining Whether Heavy Inference in Two-Layer Inference Needs to Be Performed Using an Upper Threshold Value" will be described with reference to Figures 10 and 11. Figure 10 shows a two-layer inference system 10 that uses an object detection model as a first-layer model for light inference and an object detection model as a second-layer model for heavy inference, and a setting assistance device 100 that creates setting assistance information for setting threshold values ​​used in the two-layer inference system 10.

[0076] In the example shown in Figure 10, the two-layer inference system 10 does not perform heavy inference on the cloud side if the confidence obtained by light inference on the edge side exceeds a threshold, but performs heavy inference on the cloud side if the confidence is below the threshold. In other words, the two-layer inference system 10 shown in Figure 10 uses the inference result of light inference as is as the final output result of the two-layer inference when the confidence of the inference result of light inference exceeds a threshold ("High Confidence" at the top of Figure 10). On the other hand, when the confidence is below the threshold ("Low Confidence" at the top of Figure 10), it determines that the inference result of light inference is insufficient, performs heavy inference, and outputs the result of heavy inference as the result of the two-layer inference. In other words, when the tasks processed by the first-layer model and the second-layer model are the same, the two-layer inference system 10 reduces the frequency of heavy inference, which has a high processing load, thereby reducing the processing load.

[0077] 10, the setting assistance device 100 performs two-layer inference using the accepted inference conditions and input data, and compares the results of the two-layer inference with the correct answer data to create setting assistance information. In the second example, the setting assistance device 100 outputs, as setting assistance information, a correlation graph that associates the accuracy and performance of the two-layer inference for each threshold, a recommended threshold (recommended threshold), comparison information on changes in inference results, etc.

[0078] Next, an example of auxiliary setting information output by the setting assistance device 100 in the second example will be described. Fig. 11 shows an example of a correlation graph output by the setting assistance device 100. In the correlation graph shown in Fig. 11, the X axis represents the varied threshold value, the Y axis (left) represents the recall as an index of precision, and the Y axis (right) represents the ratio of frames processed in the first layer model as an index of performance.

[0079] As described above, in the second example, the two-layer inference system 10 performs heavy inference when the confidence obtained by light inference falls below a threshold (upper threshold). That is, in the second example, as the threshold value increases, the frequency with which the confidence obtained by light inference falls below the threshold increases, and the frequency with which heavy inference is performed increases. In this way, as the threshold value increases, the frequency with which heavy inference is performed by the second-layer model increases, and therefore the correlation graph shown in Figure 11 shows a tendency for the recall rate to gradually increase and the proportion of frames for which processing is completed by the first-layer model to gradually decrease.

[0080] Furthermore, when outputting a correlation graph, the setting assistance device 100 can also display an operation screen 22 for accepting "selection of evaluation index." This operation screen 22 includes check boxes for selecting evaluation indexes to be displayed in the correlation graph, such as "accuracy rate," "recall rate," and "proportion of frames for which processing is completed in the first-layer model." In the example of FIG. 11 , "recall rate" and "proportion of frames for which processing is completed in the first-layer model" are selected.

[0081] The setting assistance device 100 also outputs a correlation graph displaying recommended thresholds to be presented to a user, as shown in FIG. 12 . For example, when the target value for the reproducibility is set to "75% (or higher)" as shown on the operation screen 23 in FIG. 12 , the setting assistance device 100 identifies a threshold value of "0.7 to 1" corresponding to the section indicated by the solid line on the left side of the Y axis where the reproducibility is 75% or higher, as a recommended threshold candidate. On the other hand, when the target value for the proportion of frames processed in the first-layer model is set to "30% (or higher)," as shown in FIG. 12 , the setting assistance device 100 identifies a threshold value of "0 to 0.7" corresponding to the section indicated by the solid line on the right side of the Y axis where the proportion of frames processed in the first-layer model is 30% or higher, as a recommended threshold candidate. Then, as shown in FIG. 12 , the setting assistance device 100 calculates a threshold value of "0.7" that satisfies both the target value for accuracy and the target value for the proportion of frames processed in the first-layer model as a recommended accuracy-oriented threshold.

[0082] The "comparison information of inference result change" according to the second example described above has the same content as that described in FIG. 6, and therefore will not be described here.

[0083] (Third Example: When Determining, Using a Lower Threshold, Whether to Implement Heavy Inference That Processes Light Inference Results in Two-Layer Inference) Next, an example of "When Determining, Using a Lower Threshold, Whether to Implement Heavy Inference That Processes Light Inference Results in Two-Layer Inference" will be described with reference to Fig. 13. Fig. 13 shows a two-layer inference system 10 that uses an object detection model as a first-layer model for light inference and a pose estimation model as a second-layer model for heavy inference, and a setting assistance device 100 that creates setting assistance information used to set thresholds used in the two-layer inference system 10.

[0084] In the example shown in Figure 13, the two-layer inference system 10 performs heavy inference on the cloud side when the confidence level obtained by light inference on the edge side exceeds a threshold ("High Confidence Level" at the top of Figure 13). In other words, the two-layer inference system 10 shown in Figure 13 performs heavy inference when the confidence level for an object extracted from input data exceeds a threshold, and estimates the bones of the object detected by light inference, etc. In other words, when the tasks processed by the first-layer model and the second-layer model are different, the two-layer inference system 10 reduces the frequency of heavy inference, which has a high processing load, thereby reducing the processing load.

[0085] 13, the setting assistance device 100 performs two-layer inference using the received inference conditions and input data, and compares the two-layer inference result with the correct answer data to create the setting assistance information. In the third example, the setting assistance device 100 outputs a correlation graph and a recommended threshold (recommended threshold) as the setting assistance information. In the third example, since the result of the light inference is used to perform the heavy inference, the inference result of the heavy inference cannot be used as the correct answer data, and therefore, generated data separately generated as the correct answer data is used.

[0086] The auxiliary setting information generated in the third example is similar to that in the first or second example, and therefore a detailed description thereof will be omitted.

[0087] (Procedure of Setting Support Processing) Hereinafter, a procedure of the setting support processing realized by the setting support device 100 according to this embodiment will be described. Fig. 14 is a diagram showing an example of a flowchart of the setting support processing according to this embodiment.

[0088] The receiving unit 131 receives the specification of an inference condition (S101).

[0089] If the supervised answer data is not to be generated (No in S102), the receiving unit 131 receives the supervised answer data created in advance (S103). On the other hand, if the supervised answer data is to be generated (Yes in S102), the generating unit 132 generates the supervised answer data using the output result of the heavy inference (S104).

[0090] The inference unit 133 performs two-layer inference while varying the threshold based on the inference conditions (S105). Next, the matching unit 134 matches the inference result of the two-layer inference with the correct answer data (S106). Then, the creation unit 135 creates a correlation graph (S107).

[0091] If a recommended threshold is to be created (Yes in S108), the creation unit 135 creates the recommended threshold (S109). On the other hand, if a recommended threshold is not to be created (No in S108), the creation unit 135 skips the step of S109.

[0092] Furthermore, when generating a comparison result regarding a change in the inference result of the two-layer inference due to a change in the threshold value (Yes in S110), the creation unit 135 creates the comparison result (S111). On the other hand, when not generating a comparison result (No in S110), the creation unit 135 skips the step of S111.

[0093] The output unit 136 outputs the result generated by the generation unit 135 (S112), and the setting assistance device 100 then ends the process.

[0094] (Effects) The effects of the setting assistance device 100 according to this embodiment will now be described. The comparison unit 134 according to this embodiment varies the thresholds used in two-layer inference, and performs two-layer inference for each varied threshold. The comparison unit 134 compares the two-layer inference results for each varied threshold with the correct answer data. The creation unit 135 then creates a correlation graph between the varied thresholds and the accuracy and performance of the two-layer inference for each varied threshold calculated as the comparison result. Therefore, the setting assistance device 100 according to this embodiment has the effect of facilitating tuning of the thresholds used in two-layer inference.

[0095] Specifically, the inference unit 133 compares a predetermined value obtained by performing light inference with each of the varied thresholds, and performs heavy inference if a determination condition for whether to perform heavy inference is met, and outputs the obtained two-layer inference result. The collation unit 134 compares the result of the two-layer inference with previously generated correct answer data and calculates a predetermined index including the accuracy and performance of the two-layer inference as a collation result. The creation unit 135 creates a correlation graph in which the accuracy and performance calculated as the predetermined index are associated with each varied threshold. In this way, the setting assistance device 100 automatically varies the thresholds based on inference conditions specified by a user, etc., compares the result of the two-layer inference at each varied threshold with the correct answer data, calculates indexes such as accuracy and performance, and creates a correlation graph. Therefore, the setting assistance device 100 eliminates the need for a user, etc., to individually check the balance between accuracy and performance when tuning the thresholds, thereby facilitating tuning of the thresholds used in two-layer inference.

[0096] Furthermore, the creation unit 135 calculates a threshold value corresponding to a predetermined index as a recommended threshold value when the predetermined index falls within a predetermined reference range. Furthermore, the creation unit 135 calculates a threshold value corresponding to a predetermined index as a recommended threshold value when the average of the predetermined weighted index falls within a predetermined reference range. In this way, the setting assistance device 100 calculates a candidate threshold value desired by the user as a recommended threshold value. Therefore, the setting assistance device 100 reduces the amount of work required for tuning threshold values ​​performed by the user, etc., and facilitates tuning of threshold values ​​used in two-layer inference.

[0097] The creation unit 135 also identifies the results of the two-layer inference in which a predetermined change occurred due to varying the threshold. In this way, the setting assistance device 100 identifies the changed inference result when a change in the inference result occurs as a result of varying the threshold, thereby allowing the user to visually understand what change occurred in the inference result due to the threshold. As a result, the setting assistance device 100 has the effect of making it easy for the user to select an appropriate threshold.

[0098] As described above, the setting assistance device 100 according to this embodiment enables users to easily tune thresholds appropriate for their inference use cases. As a result, the setting assistance device 100 improves the availability and usability of the two-tier inference system, and enables easy use of the two-tier inference system without relying on the user's knowledge or proficiency in inference.

[0099] Furthermore, while previously users had to tune thresholds by checking the balance between accuracy and performance one by one, the setting assistance device 100 automatically creates a correlation graph that associates thresholds with accuracy and performance, eliminating the need for individual verification for tuning thresholds and reducing the labor and computer processing required for the tuning.

[0100] <Modifications> Modifications realized by the setting assistance device 100 according to this embodiment will be described below.

[0101] (Data, etc.) The input data, correct answer data, matching results, inference conditions, light inference, heavy inference, performance, accuracy, names of functional parts of the setting assistance device 100, steps, processes, names of steps or processes, etc. used in the description of the above embodiments are merely examples and can be changed as desired.

[0102] The input data DB 121 can store data other than still images and moving images, such as audio data and text data, as long as the data can be used for model-based inference.

[0103] Although the supervised data DB 122 has been described as storing supervised data including positive examples or negative examples, it is not limited to this and can store any data that can be used in the matching process by the matching unit 134.

[0104] Although the matching result DB 123 has been described as storing a threshold, accuracy, performance, recommended threshold, etc. in association with each other, it may store a matching result that does not include a recommended threshold, for example.

[0105] Although the inference condition DB 124 has been described as storing a moving object detection model, an object detection model, and a specified estimation model as usage models, the inference condition DB 124 is not limited to these and can store other inference models. Furthermore, the contents of the correct answer data, accuracy, performance, target value, threshold range, number of inference iterations, etc. stored by the inference condition DB 124 may be changed as appropriate depending on the actual usage situation. Furthermore, the "target value" among the inference conditions stored by the inference condition DB 124 may include any condition such as "greater than or equal to the target value," "less than or equal to the target value," "exceeding the target value," or "less than the target value."

[0106] As an example, it has been explained that the inference unit 133 outputs a light inference result when a judgment condition such as "when the degree of difference exceeds a threshold (lower threshold), a heavy inference is performed, which is an inference using the second layer model" or "when the confidence level is below a threshold (upper threshold), a light inference is performed, which is an inference using the second layer model" is met. However, the above-mentioned judgment conditions are not limited to these, and it is possible to determine whether or not to output a light inference result based on any condition.

[0107] (Flowcharts, etc.) The steps in a flowchart, etc. may be interchanged and performed as long as there is no contradiction, and some steps may not be performed. In addition, conjunctions such as "next," "continue," "further," "at this time," and "on this occasion" used in the description of a flowchart do not limit the order or timing of performing the processes in the flowchart.

[0108] (System) The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0109] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown. In other words, all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0110] <Hardware Configuration> The components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0111] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can also be performed manually using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the drawings can be changed as desired unless otherwise specified.

[0112] <Program> In one embodiment, the various devices constituting the setting assistance device 100 can be implemented by installing a setting assistance program as package software or online software on a desired computer. For example, by executing the setting assistance program on an information processing device, the various devices constituting the setting assistance device 100 can function. The information processing device referred to here includes desktop and notebook personal computers. In addition, information processing devices also include mobile communication terminals such as smartphones and mobile phones, and even slate terminals such as PDAs (Personal Digital Assistants).

[0113] 15 is a diagram showing an example of a computer that executes the setting assistance program according to this embodiment. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0114] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0115] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes of the various devices that make up the setting assistance device 100 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, on the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configurations of the various devices that make up the setting assistance device 100 are stored on the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0116] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0117] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN or a WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0118] Although the present embodiment has been described above, the present embodiment is not limited by the descriptions and drawings that form a part of the disclosure. In other words, other embodiments, examples, operational techniques, etc. that are made by those skilled in the art based on the present embodiment are all included in the scope of the present embodiment.

[0119] REFERENCE SIGNS LIST 100 Setting assistance device 110 Communication unit 120 Storage unit 121 Input data DB 122 Correct answer data DB 123 Matching result DB 124 Inference condition DB 130 Control unit 131 Receiving unit 132 Generating unit 133 Inference unit 134 Matching unit 135 Creating unit 136 Output unit

Claims

1. A setting assistance device characterized by having: a comparison unit that varies a threshold value used in a second layer inference in which a second inference in which a processing load is heavier than that of a first inference is performed when the inference result of a first inference in which the processing load is lighter satisfies a threshold condition, and performs the second layer inference for each varied threshold value, thereby comparing the results of the second layer inference for each varied threshold value obtained by performing the second layer inference for each varied threshold value with correct answer data; and a creation unit that creates a correlation graph between the varied threshold value and the accuracy and performance of the second layer inference for each varied threshold value calculated as the comparison result by the comparison unit.

2. The setting assistance device described in claim 1, further comprising an inference unit that compares a predetermined value obtained by performing the first inference with each of the varied threshold values, and performs the second inference and outputs the result of the two-layer inference if a judgment condition for whether or not to perform the second inference is satisfied, and the matching unit compares the result of the two-layer inference with the correct answer data created in advance, and calculates a predetermined index including the accuracy and performance of the two-layer inference as the matching result.

3. The setting assistance device according to claim 2, characterized in that the creation unit calculates the threshold corresponding to the specified index as the recommended threshold when the specified index is within a range of a preset standard.

4. The setting assistance device described in claim 2, characterized in that the creation unit calculates the threshold corresponding to the specified index as the recommended threshold when the average for the specified index weighted by a specified weight falls within a range of a preset standard.

5. The setting assistance device described in any one of claims 1 to 4, characterized in that the creation unit identifies the result of the two-layer inference in which a specified change has occurred by varying the threshold value.

6. A setting assistance method to be executed by a setting assistance device, comprising: a comparison process for varying a threshold value used in a second layer inference in which a second inference with a heavier processing load than a first inference is performed when the inference result of a first inference with a lighter processing load satisfies a threshold condition, and performing the second layer inference for each varied threshold value, thereby comparing the results of the second layer inference for each varied threshold value obtained by performing the second layer inference for each varied threshold value with correct answer data; and a creation process for creating a correlation graph between the varied threshold value and the accuracy and performance of the second layer inference for each varied threshold value calculated as the comparison result by the comparison process.

7. A configuration assistance program that causes a computer to execute the following steps: a comparison step in which a threshold used in a second layer inference that performs a second inference that performs a heavier processing load than the first inference is varied when the inference result of a first inference that performs a lighter processing load satisfies a threshold condition, and the results of the second layer inference for each varied threshold obtained by performing the second layer inference for each varied threshold are compared with correct answer data; and a creation step in which a correlation graph is created between the varied threshold and the accuracy and performance of the second layer inference for each varied threshold calculated as the comparison result by the comparison step.

Images