Network scanning device, a program for causing a computer to execute the program, and a computer-readable recording medium having the program recorded thereon

Abstract

To provide a network scanning device capable of precisely estimating a result of network scanning executed in an actual communication environment.SOLUTION: Estimation means 16 detects, from among a plurality of scanning response delays when a plurality of known estimated network scanning results are obtained, a similar scanning response delay having the highest similarity with a scanning response delay acquired in a communication environment where a network scanning result is unknown, and trains an estimation algorithm by using similar standard sample data including the detected similar scanning response delay and an optimum communication characteristic that is optimum to estimate a network scanning result, and updates the estimation algorithm so as to make the estimation algorithm close to an optimum algorithm by using an evaluation index which indicates the accuracy of an estimated result obtained when the network scanning result is estimated. Then, the estimation means 16 estimates a network scanning result on the basis of the updated algorithm.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a network scanning device, a program to be executed by a computer, and a computer-readable recording medium on which the program is recorded. [Background technology]

[0002] With the practical application of the fifth generation mobile communication (5G) system, many devices such as mobile terminals, computers, sensors, and web cameras will be connected to the Internet. Such mobile terminals and computers connected to the Internet are called IoT (Internet of Things) devices.

[0003] In the IoT era, the number of IoT devices connected to communication networks is expected to increase rapidly. Of these, IoT devices that communicate wirelessly are expected to be used illegally without sufficient security, and there are likely to be an increasing number of cases where they are exploited in cyber attacks such as DDoS (Distributed Denial of Service) attacks.

[0004] To avoid such situations, it is necessary to conduct network scans of IoT devices connected to communication networks and investigate whether security measures are being implemented for each IoT device.

[0005] Conventionally, there is known a network scan described in Non-Patent Document 1. The network scan described in Non-Patent Document 1 is a scan for applications in the IPv4 space, and is performed by the following method.

[0006] First, a port scan is performed on the IPv4 space to check for responses. Then, for hosts that respond, a handshake is performed using various protocols (such as HTTP (Hypertext Transfer Protocol), HTTPS (Hypertext Transfer Protocol Secure), and POP3 (Post Office Protocol version 3)) to obtain messages. Keywords that are useful for vulnerability assessment are then extracted from the obtained messages. Metadata and tag information are then assigned to the keywords. The keywords with the metadata and tag information assigned are then registered in the database as structured data. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Z. Durumeric, et al., “A Search Engine Backed by Internet-Wide Scanning”, CCS15, October 12-16, 2015. [Non-Patent Document 2] Katsunari Yoshioka, Kosuke Murakami, Tsutomu Matsumoto, “Network Scanning Method and Automatic Generation of Detection Signatures for Detecting Malware-Infected Hosts,” Transactions of the Information Processing Society of Japan, Vol. 51, No. 9, 1633-1644 (Sep. 2010). Summary of the Invention [Problem to be solved by the invention]

[0008] In a network scan, the round-trip time (RTT), which is the time between sending a scan packet and receiving a scan response, has very similar characteristics for terminal devices when the network is congested or when communication quality is insufficient.

[0009] As a result, when a network scan is performed in an actual communication environment, there is a problem that the rate at which the cause of the network scan failure is mistakenly assumed to be insufficient communication quality increases, when in fact the cause is network congestion.

[0010] Therefore, according to an embodiment of the present invention, a network scanning device capable of accurately estimating the results of a network scan executed in an actual communication environment is provided.

[0011] Furthermore, according to an embodiment of the present invention, a program for causing a computer to accurately estimate the results of a network scan performed in an actual communication environment is provided.

[0012] Furthermore, according to an embodiment of the present invention, a computer-readable recording medium having a program recorded thereon for causing a computer to perform an accurate estimation of the results of a network scan performed in an actual communication environment is provided. [Means for solving the problem]

[0013] (Configuration 1) According to an embodiment of the present invention, a network scanning device includes a scanning means, a detecting means, a generating means, an updating means, and an estimating means. The scanning means executes a network scan in a first communication environment where the results of a network scan for M (M is an integer equal to or greater than 2) terminal devices are unknown, and acquires first scan response delays of the network scan for all of the M terminal devices. The detecting means executes a detection process to detect, as similar scan response delays, M' second scan response delays that are most similar to the M first scan response delays among a plurality of sets of M' (M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment where the results of the network scan are known are a plurality of known estimated results. The generating means acquires, for all of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating a known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays. The updating means trains an estimation algorithm that estimates a result of a network scan performed in the first communication environment based on the similar standard sample data, and updates the estimation algorithm using an evaluation index so as to approach the optimal algorithm. The estimation means estimates a result of a network scan performed in the first communication environment using the estimation algorithm updated by the updating means. The optimal algorithm is an algorithm that estimates a result of a network scan such that the estimated result when the result of the network scan is estimated matches a known estimation result. The evaluation index is an index that evaluates the accuracy of an estimation result when the result of the network scan is estimated by the trained algorithm.

[0014] (Configuration 2) In configuration 1, the generating means further generates training data and test data from the similar standard sample data and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of each of a plurality of standard sample data corresponding to a plurality of known estimation results and each of which is composed of M' second scan response delays and M' optimal communication characteristics, and generates provisional sample data by replacing a portion of the initial sample data composed of the same type of standard sample data as the training set with a portion of the training set. The updating means updates the estimation algorithm with the provisional sample data when a first evaluation index for evaluating an estimation result obtained by training the estimation algorithm using the initial sample data and estimating the result of the network scan using the trained estimation algorithm using the validation set is equal to or smaller than a second evaluation index for evaluating an estimation result obtained by training the estimation algorithm using the provisional sample data and estimating the result of the network scan using the trained estimation algorithm using the validation set.

[0015] (Configuration 3) In configuration 2, the multiple known estimation results include a first estimation result indicating that the result of the network scan is normal, a second estimation result indicating that the network scan has failed due to insufficient communication quality, and a third estimation result indicating that the network scan has failed due to network congestion. The multiple standard sample data include first standard sample data, second standard sample data, and third standard sample data. The first standard sample data is standard sample data for estimating that the result of the network scan is the first estimation result, and includes M' second scan response delays and M' optimal communication characteristics acquired in a communication environment in which the network communication quality is insufficient. The second standard sample data is standard sample data for estimating that the result of the network scan is the second estimation result, and includes M' second scan response delays and M' optimal communication characteristics acquired in a communication environment in which the network communication quality is insufficient. The third standard sample data is standard sample data consisting of M' second scan response delays and M' optimal communication characteristics acquired in a communication environment with a congested network, and is used to estimate that the result of the network scan is the third estimated result. The first estimated result is a result estimated by an estimation algorithm using the first standard sample data, the second estimated result is a result estimated by an estimation algorithm using the second standard sample data, and the third estimated result is a result estimated by an estimation algorithm using the third standard sample data.

[0016] (Configuration 4) In configuration 2 or 3, the generating means executes a process of replacing a part of sample data consisting of standard sample data of the same type as the training set among the initial sample data with one of the plurality of subset data of the training set for all of the plurality of subset data to generate a plurality of provisional sample data constituting the provisional sample data. The updating means executes a first process of training an estimation algorithm using the initial sample data and generating a first confusion matrix based on an estimation result when a result of a network scan is estimated by the estimation algorithm after training using the validation set, a second process of training an estimation algorithm using one of the plurality of provisional sample data constituting the provisional sample data and generating a second confusion matrix based on an estimation result when a result of a network scan is estimated by the estimation algorithm after training using the validation set for all of the plurality of provisional sample data, and a first evaluation index calculated based on the first confusion matrix and a plurality of second evaluation indexes. a third process of updating the estimation algorithm with optimal sample data consisting of one provisional sample data among the provisional sample data when the maximum evaluation index of the second evaluation index is obtained, which is greater than the first evaluation index, based on a plurality of second evaluation indexes calculated based on the second confusion matrix; a fourth process of generating a third confusion matrix based on an estimation result obtained when the result of the network scan is estimated by the updated estimation algorithm using test data after the third process; and a fifth process of approving the updated estimation algorithm and the optimal sample data when the third evaluation index calculated based on the third confusion matrix is ​​equal to or greater than the maximum evaluation index.

[0017] (Configuration 5) In configuration 4, when the third evaluation index is smaller than the maximum evaluation index, the update means repeatedly executes the first process to the fifth process using the estimation algorithm updated with the optimal sample data until the third evaluation index becomes equal to or greater than the maximum evaluation index.

[0018] (Configuration 6) In the configuration 4 or 5, the network scanning device further includes a calculation means for calculating an evaluation index. Each of the first to third confusion matrices includes a first element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event does not actually occur, a third element indicating a rate at which the occurrence of an assumed event is erroneously estimated as another event when the assumed event occurs, and a fourth element indicating a rate at which the occurrence of an assumed event is erroneously estimated when the assumed event does not actually occur. Then, the calculation means calculates Precision, which is the ratio of the proportion of the proportion classified into the first element or the fourth element that was actually classified into the first element, and Recall, which is the ratio of the proportion of the proportion classified into the first element or the third element that was actually classified into the first element, and executes a third calculation process that calculates 2×Precision / (Precision+Recall) based on the Precision and Recall based on the first confusion matrix to calculate a first evaluation index, executes the third calculation process based on a plurality of second confusion matrices to calculate a plurality of second evaluation indexes, and executes the third calculation process based on the third confusion matrix to calculate a third evaluation index.

[0019] (Configuration 7) In the configuration 4 or 5, the network scanning device further includes a calculation means for calculating an evaluation index. Each of the first to third confusion matrices includes a first element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event does not actually occur, a third element indicating a rate at which the occurrence of an assumed event is erroneously estimated as another event when the assumed event occurs, and a fourth element indicating a rate at which the occurrence of an assumed event is erroneously estimated when the assumed event does not actually occur. Then, the calculation means calculates a first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates a plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates a third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix.

[0020] (Configuration 8) In any of configurations 1 to 7, the detection means, in a detection process, detects M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays from the multiple sets of M' second scan response delays as similar scan response delays based on multiple distributions of the multiple sets of M' second scan response delays and the distribution of the M first scan response delays.

[0021] (Configuration 9) In configuration 8, the detection means performs a first calculation process in the detection process to calculate a first mode indicating a value of the first scan response delay that is most likely to occur based on M first scan response delays, and calculates a first mean absolute deviation that is the average value of the absolute values ​​of the differences between the M first scan response delays and the first mode, and after the first calculation process, calculates a second mode indicating a value of the second scan response delay that is most likely to occur based on M' second scan response delays, calculates a second mean absolute deviation that is the average value of the absolute values ​​of the differences between the M' second scan response delays and the second mode, calculates a difference in mean absolute deviation consisting of the absolute value of the difference between the first mean absolute deviation and the second mean absolute deviation, and subtracts the difference in mean absolute deviation from 1 to calculate a similarity, for all of the multiple known estimation results to obtain multiple similarities, and detects the M' second scan response delays that were used to calculate the highest similarity among the multiple obtained similarities as the similar scan response delays.

[0022] (Configuration 10) According to an embodiment of the present invention, the program further comprises: A first step in which a scanning means executes a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquires a first scan response delay of the network scan for each of the M terminal devices; a second step of executing a detection process in which the detection means detects, as similar scan response delays, M' second scan response delays that are most similar to the M first scan response delays among a plurality of sets of M' (M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are a plurality of known estimated results; a third step in which the generating means acquires, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating a known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays; a fourth step in which an updating means trains an estimation algorithm for estimating a result of a network scan performed in the first communication environment based on similar standard sample data, and updates the estimation algorithm using an evaluation index so as to approach an optimal algorithm; a fifth step in which the estimation means estimates a result of the network scan performed in the first communication environment using the estimation algorithm updated in the fourth step; The optimal algorithm is an algorithm that estimates the results of a network scan such that the estimated results match known estimated results, and the evaluation index is an index that evaluates the accuracy of the estimated results when the results of a network scan are estimated by the trained algorithm.

[0023] (Configuration 11) In the configuration 10, in the third step, the generating means further generates training data and test data from the similar standard sample data, and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of each of a plurality of standard sample data corresponding to a plurality of known estimation results and each of which is composed of M' second scan response delays and M' optimal communication characteristics, and generates temporary sample data by replacing a portion of the sample data of the initial sample data composed of the same type of standard sample data as the training set with a portion of the training set; In the fourth step, the updating means updates the estimation algorithm with the hypothetical sample data when a first evaluation index that evaluates an estimation result when the estimation algorithm is trained using initial sample data and the network scan result is estimated by the trained estimation algorithm using a validation set is equal to or less than a second evaluation index that evaluates an estimation result when the estimation algorithm is trained using hypothetical sample data and the network scan result is estimated by the trained estimation algorithm using a validation set.

[0024] (Configuration 12) In configuration 11, the plurality of known estimated results include a first estimated result indicating that the result of the network scan is normal, a second estimated result indicating that the network scan has failed due to insufficient communication quality, and a third estimated result indicating that the network scan has failed due to network congestion. The plurality of standard sample data include a first standard sample data, a second standard sample data, and a third standard sample data.

[0025] The first standard sample data consists of M' second scan response delays and M' optimal communication characteristics acquired in a communication environment in which the network is normal, and is standard sample data for estimating that the result of the network scan is the first estimated result.

[0026] The second standard sample data consists of M' second scan response delays and M' optimal communication characteristics obtained in a communication environment where the network communication quality is insufficient, and is standard sample data for estimating that the result of the network scan is the second estimated result.

[0027] The third standard sample data consists of M' second scan response delays and M' optimal communication characteristics acquired in a communication environment where the network is congested, and is standard sample data for estimating that the result of the network scan is the third estimated result.

[0028] The first estimation result is a result estimated by an estimation algorithm using the first standard sample data, the second estimation result is a result estimated by an estimation algorithm using the second standard sample data, and the third estimation result is a result estimated by an estimation algorithm using the third standard sample data.

[0029] (Configuration 13) In the configuration 11 or 12, the generating means, in the third step, replaces some sample data of the initial sample data, which is made of standard sample data of the same type as the training set, with one of the plurality of subset data of the training set, for all of the plurality of subset data, to generate a plurality of provisional sample data constituting the provisional sample data; The fourth step is A first sub-step in which an update means trains an estimation algorithm using initial sample data and generates a first confusion matrix based on an estimation result when the estimation algorithm after training uses a validation set to estimate a result of a network scan; a second sub-step in which the update means trains the estimation algorithm using a first provisional sample data among a plurality of provisional sample data constituting the provisional sample data, and generates a second confusion matrix based on an estimation result obtained when the result of the network scan is estimated by the trained estimation algorithm using a validation set; a third sub-step of an update means setting the second evaluation index to the first evaluation index when the second evaluation index calculated based on the second confusion matrix is ​​greater than the first evaluation index calculated based on the first confusion matrix, and updating the estimation algorithm with the first provisional sample data; a fourth sub-step in which the update means does not set the second evaluation index to the first evaluation index when the second evaluation index is not greater than the first evaluation index, and selects second provisional sample data different from the first provisional sample data from the plurality of provisional sample data; a fifth sub-step in which the updating means repeatedly executes the second to fourth sub-steps for all of the plurality of virtual sample data; a sixth sub-step of generating a third confusion matrix based on an estimation result obtained by estimating the result of the network scan by the estimation algorithm finally updated in the fifth sub-step using the test data by the update means; The seventh sub-step includes an updating means approving the finally updated estimation algorithm when the third evaluation index calculated based on the third confusion matrix is ​​equal to or greater than the first evaluation index finally obtained in the fifth sub-step, and the first provisional sample data when the second evaluation index constituting the finally obtained first evaluation index is obtained.

[0030] (Configuration 14) In configuration 13, the fourth step further includes an eighth substep in which, when the third evaluation index is smaller than the first evaluation index finally obtained in the fifth substep, the updating means repeatedly executes the second substep to the seventh substep using the finally updated estimation algorithm until the third evaluation index becomes equal to or larger than the first evaluation index finally obtained in the fifth substep.

[0031] (Configuration 15) In the configuration 13 or 14, the calculation means further causes the computer to execute a sixth step of calculating an evaluation index; Each of the first to third confusion matrices includes a first element indicating a proportion of cases in which the occurrence of an assumed event was correctly estimated when the assumed event occurred, a second element indicating a proportion of cases in which the occurrence of an assumed event was correctly estimated when the assumed event did not actually occur, a third element indicating a proportion of cases in which the occurrence of an assumed event was erroneously estimated as another event when the assumed event occurred, and a fourth element indicating a proportion of cases in which the occurrence of an assumed event was erroneously estimated when the assumed event did not actually occur. In the sixth step, the calculation means calculates Precision, which is the ratio of the proportion of the proportion of the patterns classified into the first element or the fourth element that is actually classified into the first element, and Recall, which is the ratio of the proportion of the proportion of the patterns classified into the first element or the third element that is actually classified into the first element, and calculates 2×Precision / (Precision+Recall) based on Precision and Recall, based on the first confusion matrix to calculate a first evaluation index, based on a plurality of second confusion matrices to calculate a plurality of second evaluation indexes, and based on a third confusion matrix to calculate a third evaluation index. (Configuration 16) In the configuration 13 or 14, the calculation means further causes the computer to execute a sixth step of calculating an evaluation index; Each of the first to third confusion matrices includes a first element indicating a proportion of cases in which the occurrence of an assumed event was correctly estimated when the assumed event occurred, a second element indicating a proportion of cases in which the occurrence of an assumed event was correctly estimated when the assumed event did not actually occur, a third element indicating a proportion of cases in which the occurrence of an assumed event was erroneously estimated as another event when the assumed event occurred, and a fourth element indicating a proportion of cases in which the occurrence of an assumed event was erroneously estimated when the assumed event did not actually occur. In the sixth step, the calculation means calculates a first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates a plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates a third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix.

[0032] (Configuration 17) In any of configurations 10 to 16, in the detection process of the second step, the detection means detects M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays from the multiple sets of M' second scan response delays as similar scan response delays based on multiple distributions of the multiple sets of M' second scan response delays and the distribution of the M first scan response delays.

[0033] (Configuration 18) In configuration 17, in the detection process of the second step, the detection means performs a first calculation process to calculate a first mode indicating a value of the first scan response delay that is most likely to occur based on M first scan response delays and calculate a first mean absolute deviation that is the average value of the absolute values ​​of the differences between the M first scan response delays and the first mode, and after the first calculation process, calculates a second mode indicating a value of the second scan response delay that is most likely to occur based on M' second scan response delays, calculates a second mean absolute deviation that is the average value of the absolute values ​​of the differences between the M' second scan response delays and the second mode, calculates a difference in mean absolute deviation consisting of the absolute value of the difference between the first mean absolute deviation and the second mean absolute deviation, and subtracts the difference in mean absolute deviation from 1 to calculate a similarity, for all of the multiple known estimation results to obtain multiple similarities, and detects the M' second scan response delays that were used to calculate the highest similarity among the multiple obtained similarities as similar scan response delays.

[0034] (Configuration 19) Furthermore, according to an embodiment of the present invention, a recording medium is a computer-readable recording medium having the program according to any one of the tenth to eighteenth configurations recorded thereon. Effect of the Invention

[0035] It is possible to accurately estimate the results of network scans performed in a real communication environment. [Brief description of the drawings]

[0036] [Figure 1]1 is a schematic diagram of a communication system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram of the network scanning device shown in FIG. [Diagram 3] 3 is a conceptual diagram of a correspondence table TBL1 stored in the host information database shown in FIG. 2. [Figure 4] 3 is a conceptual diagram of another correspondence table TBL2 stored in the host information database shown in FIG. 2. [Diagram 5] FIG. 4 is a conceptual diagram of yet another correspondence table TBL3 stored in the host information database shown in FIG. [Figure 6] FIG. 2 is a conceptual diagram of a scan schedule. [Figure 7] FIG. 1 illustrates the time dependency of traffic. [Figure 8] FIG. 13 is a diagram illustrating a correlation coefficient of communication characteristics. [Figure 9] FIG. 13 is a diagram illustrating another correlation coefficient of communication characteristics. [Figure 10] FIG. 13 is a diagram showing still another correlation coefficient of communication characteristics. [Figure 11] FIG. 13 is a diagram showing still another correlation coefficient of communication characteristics. [Figure 12] FIG. 13 is a diagram showing an example of original data when selecting communication characteristic CMbest. [Figure 13] FIG. 13 is a diagram for explaining a method for selecting communication characteristic CMbest. [Figure 14] FIG. 11 is a diagram for explaining a method for calculating a score of a communication characteristic. [Figure 15] FIG. 13 is a diagram showing scores of communication characteristics. [Figure 16] FIG. 13 is a diagram showing the calculation results of rank weight Rk. [Figure 17] FIG. 13 is a diagram showing a specific distribution of scan response delays. [Figure 18] FIG. 13 is a diagram showing a specific distribution of scan response delays. [Figure 19] FIG. 13 is a diagram showing a specific distribution of scan response delays. [Figure 20]FIG. 13 is a diagram showing the correspondence between PDF regions and the number of samples generated. [Figure 21] FIG. 13 is a diagram showing an example distribution of a probability density function of a scan response delay. [Figure 22] FIG. 13 illustrates another exemplary distribution of scan response delays. [Figure 23] FIG. 1 shows data when training the estimation algorithm ALG. [Figure 24] FIG. 1 is a conceptual diagram of sample data when training the estimation algorithm ALG. [Diagram 25] FIG. 13 is a conceptual diagram showing updating of virtual sample data for modeling. [Figure 26] FIG. 1 is a conceptual diagram of training and evaluation of the estimation algorithm ALG. [Figure 27] FIG. 26 is a diagram showing a specific example of initial sample data D_sample_initial for modeling shown in FIG. 25. [Figure 28] FIG. 24 is a diagram showing a specific example of the training set shown in FIG. 23. [Figure 29] FIG. 24 is a diagram showing a specific example of the validation set shown in FIG. 23. [Diagram 30] FIG. 24 is a diagram showing a specific example of the test data shown in FIG. 23. [Diagram 31] FIG. 1 is a conceptual diagram of a confusion matrix. [Diagram 32] FIG. 13 is a diagram for explaining a method for generating a confusion matrix. [Diagram 33] 10 is a flowchart illustrating a network scan operation. [Diagram 34] 34 is a first flowchart illustrating a detailed operation of step S10 in FIG. 33. [Diagram 35] 34 is a second flowchart illustrating the detailed operation of step S10 in FIG. 33. [Diagram 36] 34 is a first flowchart illustrating a detailed operation of step S11 in FIG. 33. [Figure 37] 34 is a second flowchart illustrating the detailed operation of step S11 in FIG. 33. [Figure 38] 34 is a third flowchart illustrating a detailed operation of step S11 in FIG. 33. [Figure 39] 37 is a flowchart for explaining a detailed operation of step S130 in FIG. 36. [Diagram 40] 38 is a first flowchart illustrating a detailed operation of step S143 in FIG. 37. [Diagram 41] 38 is a second flowchart illustrating the detailed operation of step S143 in FIG. 37. [Diagram 42] 38 is a third flowchart illustrating a detailed operation of step S143 in FIG. 37. [Diagram 43] 41 is a flowchart illustrating a detailed operation of step S1432 in FIG. 40. [Diagram 44] 41 is a flowchart for explaining detailed operations of step S1436 in FIG. 40. [Diagram 45] 38 is a flowchart for explaining a detailed operation of step S145 in FIG. 37. [Diagram 46] 38 is a flowchart for explaining a detailed operation of step S145 in FIG. 37 when the Matthews correlation coefficient MCC is calculated in step S145. [Figure 47] FIG. 13 is a diagram illustrating an example of a confusion matrix. [Figure 48] FIG. 13 is a diagram showing the simulation results of D-score. [Figure 49] FIG. 13 is a diagram showing a probability density function of a scan response delay RTTm. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and their description will not be repeated.

[0038] 1 is a schematic diagram of a communication system according to an embodiment of the present invention. Referring to FIG. 1, a communication system 10 according to the embodiment of the present invention includes a network scanning device 1, base stations BS1 and BS2, and an Internet Service Provider (ISP). ) and terminal devices TM1 to TM18.

[0039] A network scanning device 1, base stations BS1 and BS2, an Internet provider ISP, and terminal devices TM1 to TM18 are located in a communication space.

[0040] The base station BS1 is a base station of the wireless communication network CNW1, the base station BS2 is a base station of the wireless communication network CNW2, and the Internet provider ISP is a carrier of the wired communication network CNW3.

[0041] The wireless communication networks CNW1 and CNW2 are communication networks that perform wireless communication in accordance with mutually different wireless communication standards.

[0042] In the embodiment of the present invention, each of the terminal devices TM1 to TM13 is a wireless terminal, and each of the terminal devices TM14 to TM18 is a wired terminal.

[0043] The terminal devices TM1 to TM6 are located outside the wireless communication network CNW2 and inside the wireless communication network CNW1.

[0044] The terminal device TM7 is located within the overlapping area of ​​the wireless communication network CNW1 and the wireless communication network CNW2.

[0045] The terminal devices TM8 to TM13 are located outside the wireless communication network CNW1 and inside the wireless communication network CNW2.

[0046] The terminal devices TM14 to TM18 are arranged within the wired communication network CNW3 and are connected to the Internet provider ISP via wires.

[0047] Each of the terminal devices TM1 to TM13 includes two wireless communication modules capable of wireless communication with the base stations BS1 and BS2. Therefore, each of the terminal devices TM1 to TM13 is capable of wireless communication with the base station BS1 or the base station BS2.

[0048] The terminal devices TM1 to TM6 perform wireless communication with the base station BS1, the terminal device TM7 performs wireless communication with the base station BS1 or base station BS2, and the terminal devices TM8 to TM13 perform wireless communication with the base station BS2.

[0049] The terminal devices TM14 to TM18 perform wired communication via an Internet provider ISP.

[0050] The network scanning device 1 performs a network scan on the terminal devices TM1 to TM18 by a method to be described later.

[0051] Fig. 2 is a schematic diagram of the network scanning device 1 shown in Fig. 1. Referring to Fig. 2, the network scanning device 1 includes a network scanner 11, a scan analysis manager 12, a host information database 13, an extraction means 14, a calculation means 15, an estimation means 16, and a scan scheduler 17.

[0052] The network scanner 11 receives scan timing from the scan scheduler 17, and based on the received scan timing, performs a network scan of the ports of terminal devices connected to the network 20 to be scanned, for example, using the well-known Nmap (see non-patent document 2).

[0053] Then, the network scanner 11 receives a scan response of the network scan from a terminal device connected to the network 20, and outputs the received scan response to a user (comprised of network security, etc.) 30 and the scan analysis manager 12.

[0054] In addition, the network scanner 11 may detect a scan response delay (RTT) when the network scan is performed without monitoring background traffic. m (m is an integer satisfying 1≦m≦M, and M is the total number of terminal devices TM1 to TM18 and is an integer of 2 or more), and the packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m Measure the scan response delay RTT m , Packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m to the scan analysis manager 12. That is, when a network scan is performed without monitoring background traffic, the network scanner 11 outputs a scan response delay RTT m , Packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m Communication characteristics CM1 = [scan response delay RTT m / Packet Loss Rate PLR m / Scan Maximum Delay DLmax m / Scan minimum delay DLmin m ] is measured, and the measured communication characteristic CM1 = [scan response delay RTT m / Packet Loss Rate PLR m / Scan Maximum Delay DLmax m / Scan minimum delay DLmin m ] is output to the scan analysis manager 12.

[0055] Furthermore, the network scanner 11 may be configured to detect the scan response delay (RTT) when the network scan is performed by monitoring background traffic. m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLminm Measure the scan response delay RTT m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m to the scan analysis manager 12. That is, when the network scan is performed by monitoring background traffic, the network scanner 11 outputs the scan response delay RTT m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m Communication characteristic CM2 = [scan response delay RTT m / Packet Loss Rate PLR m / Delayed DL m / Jitter JT m / Background Max Delay BGDLmax m / Background minimum delay BGDLmin m ] is measured, and the measured communication characteristic CM2 = [scan response delay RTT m / Packet Loss Rate PLR m / Delayed DL m / Jitter JT m / Background Max Delay BGDLmax m / Background minimum delay BGDLmin m ] is output to the scan analysis manager 12.

[0056] The scan analysis manager 12 receives a scan response from the network scanner 11, and receives a communication log during a network scan. The scan analysis manager 12 also receives the communication characteristic CM1 or the communication characteristic CM2 from the network scanner 11.

[0057] Then, when the scan analysis manager 12 receives the scan response and the communication characteristics CM1, it generates a scan result SC1 that associates the address of the terminal device to be scanned, the scan response, and the communication characteristics CM1 based on the scan response and the communication characteristics CM1, and creates or updates the host information database 13 by storing the generated scan result SC1 in the host information database 13.

[0058] In addition, when the scan analysis manager 12 receives the scan response and the communication characteristics CM2, it generates a scan result SC2 that associates the address of the terminal device to be scanned, the scan response, and the communication characteristics CM2 based on the scan response and the communication characteristics CM2, and creates or updates the host information database 13 by storing the generated scan result SC2 in the host information database 13.

[0059] The host information database 13 stores the scan response delay (RTT) obtained when a network scan is performed in a communication environment where the result of the network scan is "normal." m’ _known}_1(m'=1'~M') and the communication characteristic that is most suitable for estimating that the result of the network scan is "normal" {CMbest m’ _known}_1 (m'=1' to M') is stored in advance in association with m', where m' is an integer satisfying 1'≦m'≦M', and M' is the total number of terminal devices that are targets of network scans performed in a communication environment where the results of network scans are known. Furthermore, m' may be the same as m or may be different from m.

[0060] In addition, the host information database 13 stores the scan response delay {RTT m’ _known}_2(m'=1'~M') and the most suitable communication characteristic {CMbest m’_known}_2 (m'=1' to M') are stored in advance in correspondence with each other.

[0061] Furthermore, the host information database 13 stores the scan response delay {RTT m’ _known}_3(m'=1'~M') and the most suitable communication characteristic {CMbest m’ _known}_3 (m'=1' to M') are stored in advance in correspondence with each other.

[0062] Furthermore, the host information database 13 stores the scan result SC1 or the scan result SC2.

[0063] The extraction means 14 executes an extraction process to extract the communication characteristic CMbest_unknown that is most suitable for estimating the result of the network scan, by a method to be described later, based on the communication characteristic CM1 (or the communication characteristic CM2) stored in the host information database 13. Then, the extraction means 14 extracts the scan response delay RTT m (hereinafter referred to as "Scan Response Delay RTT m _unknown).) and the detected scan response delay RTT m _unknown(m=1~M) and communication characteristics CMbest m The estimation means 16 generates sample data D_unknown consisting of _unknown (m=1 to M) and outputs the generated sample data D_unknown to the calculation means 15 and the estimation means 16.

[0064] The extraction means 14 also extracts from the correspondence table TBL3 stored in the host information database 13 the scan response delay {RTT m’ _known}_1(m'=1'~M') and communication characteristics {CMbest m’_known}_1(m'=1'~M') and the detected scan response delay {RTT m’ _known}_1(m'=1'~M') and communication characteristics {CMbest m’ _known}_1 (m′=1′ to M′), and outputs the generated standard sample data D_standard_1 to the calculation means 15.

[0065] Furthermore, the extraction means 14 extracts from the correspondence table TBL3 stored in the host information database 13 the scan response delay {RTT m’ _known}_2(m'=1'~M') and communication characteristics {CMbest m’ _known}_2(m'=1'~M') and the detected scan response delay {RTT m’ _known}_2(m'=1'~M') and communication characteristics {CMbest m’ _known}_2(m'=1'~M') and standard sample data D_s The standard sample data D_standard_2 is generated, and the generated standard sample data D_standard_2 is output to the calculation means 15.

[0066] Furthermore, the extraction means 14 extracts from the correspondence table TBL3 stored in the host information database 13 the scan response delay {RTT m’ _known}_3(m'=1'~M') and communication characteristics {CMbest m’ _known}_3(m'=1'~M') and the detected scan response delay {RTT m’ _known}_3(m'=1'~M') and communication characteristics {CMbest m’ _known}_3 (m′=1′ to M′), and outputs the generated standard sample data D_standard_3 to the calculation means 15.

[0067] Furthermore, the extraction means 14 extracts the scan response delay {RTT m’ _known}_1(m'=1'~M')~{RTT m’ _known}_3 (m'=1'~M') and the scan response delay RTT included in the sample data D_unknown m 3 similarities with _unknown P S1 ,P S2 ,P S3 from the calculation means 15, and the three similarities P S1 ,P S2 ,P S3 The scan response delay when the maximum value (highest similarity) of m’ _known}_similarity(m'=1'~M')(=scan response delay {RTT m’ _known}_1(m'=1'~M')~{RTT m’ _known}_3 (m'=1' to M'). Then, the extraction means 14 detects the scan response delay {RTT m’ _known}_similarity (m'=1'~M') and scan response delay {RTT m’ _known}_similarity (m'=1'~M') corresponding communication characteristics {CMbest m’ _known}similarity(=communication characteristics{CMbest m’ _known}_1(m'=1'~M')~{CMbest m’ _known}_3 (m′=1′ to M′)) to the estimation means 16.

[0068] The calculation means 15 receives the sample data D_unknown and the standard sample data D_standard_1 to D_standard_3 from the extraction means 14. Then, the calculation means 15 calculates the scan response delay {RTT m’ _known}_1(m'=1'~M')~{RTT m’_known}_3 (m'=1'~M') and the scan response delay RTT included in the sample data D_unknown m 3 similarities P with unknown(m=1~M) S1 ,P S2 ,P S3 The similarity P S1 ,P S2 ,P S3 is output to the extraction means 14.

[0069] The calculation means 15 also receives the estimation results (confusion matrices CFM, nCFM, tCFM) obtained by estimating the results of the network scan using the estimation algorithm ALG from the estimation means 16. The calculation means 15 then calculates an evaluation index EVD representing the accuracy of the estimation result based on the confusion matrix CFM, calculates an evaluation index nEVD representing the accuracy of the estimation result based on the confusion matrix nCFM, and calculates an evaluation index tEVD representing the accuracy of the estimation result based on the confusion matrix tCFM, using a method to be described later. The calculation means 15 then outputs the evaluation indexes EVD, nEVD, and tEVD to the estimation means 16.

[0070] The estimation means 16 combines the standard sample data D_standard_1 to D_standard_3 to generate initial sample data D_sample_initial for modeling.

[0071] The estimation means 16 also estimates the standard sample data D_similarity (= scan response delay {RTT m’ _known}_similarity(m'=1'~M') and communication characteristics {CMbest m’_known}_similarity (m'=1' to M'). The estimation means 16 receives standard sample data D_similarity from the extraction means 14. The estimation means 16 then divides the standard sample data D_similarity into training data and test data, and divides the training data into a training set and a validation set. The estimation means 16 then generates a plurality of tentative sample data D_sample_tentative based on the training set and initial sample data D_sample_initial by a method described later.

[0072] Then, the estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial, estimates the results of the network scan by the estimation algorithm ALG (the trained estimation algorithm ALG) using the validation set, generates a confusion matrix CFM based on the estimated results of the network scan, and outputs it to the calculation means 15.

[0073] Next, the estimation means 16 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative, estimates the results of the network scan by the estimation algorithm ALG (the trained estimation algorithm ALG) using the validation set, and generates a confusion matrix nCFM based on the estimated results of the network scan and outputs it to the calculation means 15.

[0074] The estimation means 16 outputs the confusion matrix CFM to the calculation means 15, and then receives the evaluation index EVD from the calculation means 15, and outputs the confusion matrix nCFM to the calculation means 15, and then receives the evaluation index nEVD from the calculation means 15. When the evaluation index nEVD is greater than the evaluation index EVD, the estimation means 16 sets the evaluation index nEVD to the evaluation index EVD. On the other hand, when the evaluation index nEVD is not greater than the evaluation index EVD, the estimation means 16 does not set the evaluation index nEVD to the evaluation index EVD.

[0075] The estimation means 16 compares the evaluation index nEVD with the evaluation index EVD for all of the multiple tentative sample data D_sample_tentative, and replaces the variables of the estimation algorithm ALG with the tentative sample data D_sample_tentative when the largest evaluation index nEVD is obtained.

[0076] Thereafter, the estimation means 16 estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG in which the variables have been replaced) using the test data, generates a confusion matrix tCFM based on the estimation result of the network scan, and outputs it to the calculation means 15. Then, the estimation means 16 receives the evaluation index tEVD from the calculation means 15, and when the evaluation index tEVD is equal to or greater than the evaluation index EVD, approves the estimation algorithm ALG in which the variables have been replaced and the tentative sample data D_sample_tentative when the largest evaluation index nEVD is obtained.

[0077] In this way, the estimation means 16 trains the estimation algorithm ALG and evaluates the accuracy of the estimation result when the result of the network scan is estimated by the trained estimation algorithm ALG.

[0078] Furthermore, the estimation means 16 estimates the result of the network scan by using the estimation algorithm ALG in the following manner. m The terminal devices having _unknown (m=1 to M) are classified into a group CLS1 in which the network scan is normal, and groups CLS2 and CLS3 in which the network scan is unsuccessful due to a cause attributable to the communication channel with the terminal device.

[0079] The estimation means 16 then calculates a D score D_2 for distinguishing between the groups CLS1 and CLS3, or a D score D_3 for distinguishing between the groups CLS2 and CLS3, and estimates that the network scan is normal or unsuccessful based on the calculated D score D_2 or D score D_3, and when it estimates that the network scan is unsuccessful, executes an estimation process for estimating the cause of the failure of the network scan. The estimation means 16 then outputs the estimation result estimated in the estimation process to the scan scheduler 17.

[0080] The scan scheduler 17 reads the scan results from the host information database 13 and receives scan requirements from the user 30. The scan scheduler 17 also holds in advance the global IP addresses and domain names of all terminal devices present in the area (e.g., Japan) where the network scan is to be performed.

[0081] Then, the scan scheduler 17 generates a scan schedule SCH_scan that associates the scan timings of the network scan with the terminal devices that are targets of the network scan at each scan timing based on the scan requirements and the scan results, using a method to be described later. Then, the scan scheduler 17 outputs the scan schedule SCH_scan to the network scanner 11, and controls the network scanner 11 to execute a network scan in accordance with the scan schedule SCH_scan.

[0082] Furthermore, when the scan scheduler 17 receives the estimation result from the estimation means 16, the scan scheduler 17 reschedules the scan schedule based on the cause of the failure of the network scan in the estimation result, and outputs the rescheduled scan schedule RESCH_scan to the network scanner 11 to control the network scanner 11 to execute the network scan in accordance with the scan schedule RESCH_scan.

[0083] Fig. 3 is a conceptual diagram of the correspondence table TBL1 stored in the host information database 13 shown in Fig. 2. Note that the correspondence table TBL1 shown in Fig. 3 is a correspondence table used when a network scan is performed without monitoring background traffic.

[0084] Referring to FIG. 3, the correspondence table TBL1 is a table of time t and IP address Add b_v (v=1 to V, where V is the total number of base stations.) and IP address Add m and the domain name DN m and port number PN r (r is a positive integer) and the scan response SR m and scan response delay RTT m and packet loss rate (PLR) m and the maximum scan delay DLmax m and the minimum scan delay DLmin m Includes:

[0085] Time t is expressed by YYYY / MM / DD / hh / mm / ss (year / month / day / hour / minute / second). Time t is a time within one scan timing having a certain time length, which will be described later, and represents the time when a test input in a network scan is transmitted to a terminal device that is the target of the network scan.

[0086] IP Address Add b_v is the address of the base station. IP address Add m consists of the global IP address of the terminal device that is the target of the network scan.

[0087] Domain Name DN m Add IP address m It is converted using the PTR (PoTeR record) based on the above.

[0088] Port number PN r is the port number that each terminal device uses for communication.

[0089] Scan Response SRm is "1" when a scan response is returned, and is "0" when no scan response is returned.

[0090] Scan Response Delay RTT m is the time from the time the test input was sent to the time the scan response SR m The time it takes for the command to return.

[0091] Packet Loss Rate (PLR) m sends a packet consisting of the test input and returns a scan response SR m This is the rate at which packets fail to be sent or received when receiving a packet.

[0092] Scan maximum delay DLmax m is the scan response delay RTT m is the maximum value of the minimum scan delay DLmin m is the scan response delay RTT m is the minimum value of.

[0093] And the scan response SR m and scan response delay RTT m is the port number PN r It corresponds to the packet loss rate (PLR). m , maximum scan delay DLmax m and scan minimum delay DLmin m Add is the IP address of the terminal device to be scanned. m can be associated with

[0094] In addition, addresses Add1 to Add3 are b_1 is associated with,..., address Add M-1 ~Add M Add Address b_V As a result, it is possible to grasp the terminal devices that belong to each base station.

[0095] Fig. 4 is a conceptual diagram of another correspondence table TBL2 stored in the host information database 13 shown in Fig. 2. Note that the correspondence table TBL2 shown in Fig. 4 is a correspondence table used when a network scan is executed by monitoring background traffic.

[0096] 4, the correspondence table TBL2 is the maximum scan delay DLmax of the correspondence table TBL1 shown in FIG. m and scan minimum delay DLmin m Delay DL m , Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m The rest is the same as in Correspondence Table TBL1.

[0097] Delayed download m , Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Add the IP address of each terminal device. m can be associated with

[0098] Delayed download m is the delay from the standard time in one-way communication from the terminal device.

[0099] Jitter JT m is the fluctuation in packet transmission time in one-way communication from the terminal device.

[0100] Background Max Delay BGDLmax m is the maximum transmission delay in one-way communication from the terminal device, and the background minimum delay BGDLmin m is the minimum transmission delay in one-way communication from the terminal device.

[0101] In the correspondence table TBL2, the scan response SR m and scan response delay RTT mis the port number PN of each terminal device r can be associated with

[0102] FIG. 5 is a conceptual diagram of yet another correspondence table TBL3 stored in the host information database 13 shown in FIG.

[0103] 5, the correspondence table TBL3 includes a known estimation result, a time t', and an IP address Add b’_w (w=1 to W, where W is the total number of base stations.) and IP address Add m’ and scan response delay RTT m’ _known and communication characteristics CMbest m’ _known. Known estimation result, time t', IP address Add b’_w , IP address Add m’ , Scan Response Delay RTT m’ _known and communication characteristics CMbest m’ _known are associated with each other.

[0104] The known estimated results include "normal", "poor communication quality", and "network congestion" as estimated results of a network scan.

[0105] The time t' has the same format as the time t described in FIG. b’_w is the address of the base station. IP address Add m’ IP address Add consists of the global IP address of the terminal device that is the target of the network scan. 1’ ~IP Address Add 3’ is the IP address of the base station. b’_1 Similarly, the IP address Add M’ is the IP address of the base station. b’_W can be associated with

[0106] Scan Response Delay {RTT 1’ _known}_1, Scan Response Delay {RTT 2’_known}_1, Scan Response Delay {RTT 3’ _known}_1, ···, and Scan Response Delay {RTT M’ _known}_1 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ , , IP addressAdd M’ , and is the scan response delay RTT obtained in a communication environment where the network scan result is "normal."

[0107] Communication characteristics {CMbest 1’ _known}_1, communication characteristics {CMbest 2’ _known}_1, communication characteristics {CMbest 3’ _known}_1, ···, and communication characteristics {CMbest M’ _known}_1 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ , , IP addressAdd M’ The communication characteristic CMbest is associated with the communication characteristic CMbest obtained in a communication environment where the result of the network scan is "normal."

[0108] Scan Response Delay {RTT 1’ _known}_2, Scan Response Delay {RTT 2’ _known}_2, Scan Response Delay {RTT 3’ _known}_2, ···, and Scan Response Delay {RTT M’ _known}_2 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ , , IP addressAdd M’ , and is the scan response delay RTT obtained in a communication environment where the network scan result is "insufficient communication quality."

[0109] Communication characteristics {CMbest 1’_known}_2, communication characteristics {CMbest 2’ _known}_2, communication characteristics {CMbest 3’ _known}_2, ···, and communication characteristics {CMbest M’ _known}_2 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ , , IP addressAdd M’ The communication characteristic CMbest is associated with the communication environment in which the result of the network scan is "insufficient communication quality."

[0110] Scan Response Delay {RTT 1’ _known}_3, Scan Response Delay {RTT 2’ _known}_3, Scan Response Delay {RTT 3’ _known}_3, ···, and Scan Response Delay {RTT M’ _known}_3 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ , , IP addressAdd M’ , and is the scan response delay RTT obtained in a communication environment where the result of the network scan is "network congestion."

[0111] Communication characteristics {CMbest 1’ _known}_3, communication characteristics {CMbest 2’ _known}_3, communication characteristics {CMbest 3’ _known}_3, ···, and communication characteristics {CMbest M’ _known}_3 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ , , IP addressAdd M’ The communication characteristic CMbest is associated with the communication characteristic CMbest obtained in a communication environment where the result of the network scan is "network congestion."

[0112] Scan Response Delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 constitutes the standard sample data D_standard_1.

[0113] Also, the scan response delay (RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 constitutes the standard sample data D_standard_2.

[0114] In addition, the scan response delay (RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ _known}_3 constitutes the standard sample data D_standard_3.

[0115] The scan response delay (RTT 1’ _known}_1~{RTT M’ Each of the {_known}_1 actually represents one IP address Add m’ There are multiple ports in the same network. 1’ _known}_1~{CMbest M’ Each of the {_known}_1 actually represents one IP address Add m’ There are multiple entries corresponding to multiple ports in the network. Scan Response Delay {RTT 1’ _known}_2~{RTT M’ _known}_2;{RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’_known}_2~{CMbest M’ _known}_2;{CMbest 1’ _known}_3~{CMbest M’ The same is true for _known}_3.

[0116] Also, the scan response delay (RTT 1’ _known}_1~{RTT M’ _known}_1's "1" and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ The “1” in _known}_1 represents the identification code D_code=1 given to the known estimated result of the network scan, “normal,” and the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2's "2" and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ The “2” in _known}_2 represents the identification code D_code=2 given to the known estimated result of the network scan, “poor communication quality,” and the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3's "3" and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ The "3" in {_known}_3 represents the identification code D_code=3 assigned to the known estimated result of the network scan, "network is congested."

[0117] In the embodiment of the present invention, when a network scan is performed without monitoring background traffic, correspondence tables TBL1 and TBL3 are stored in the host information database 13, and when a network scan is performed while monitoring background traffic, correspondence tables TBL2 and TBL3 are stored in the host information database 13. Correspondence table TBL3 is stored in the host information database 13 before the network scanner 11 starts a network scan in a communication environment where the results of the network scan are unknown.

[0118] 6 is a conceptual diagram of a scan schedule. Referring to FIG. 6(a), the scan schedule SCH_scan has a total scan time length T S _total, T (T is an integer equal to or greater than 1) scan timings 1 S ~Scan timing T S Includes.

[0119] Total scan time T S _total is set to 1 day, for example, and scan timing 1 S ~Scan timing T S Each of the scan timings t has a duration of, for example, one hour. S (t S =1 S ~T S ) is the start time t start and the end time t end and the target addresses Add1 to Add M Includes:

[0120] start time t start is the scan timing t S represents the time when the network scan starts in t end is the scan timing t S t represents the time to end the network scan. Therefore, the network scanner 11 S At end time t end When this occurs, the network scan ends.

[0121] Target address Add1~Add M Each of the target addresses Add1 to Add M represents M terminal devices that are targets of the network scan.

[0122] In addition, the scan timing t S Scan timing other than 1 S ~(t-1) S ,(t+1) S ~T S Each of the scan timings t S It has the same configuration as

[0123] Scan Timing 1 S ~T S are the same time period (=t end -t start ), and may have different time lengths (=t end -t start ) may be included. Also, the scan timing 1 S ~T S may have the same number of IP addresses as each other, or may have a different number of IP addresses from each other.

[0124] When performing a network scan for the first time, the scan timing 1 shown in Figure 6 (a) is used. S ~T S is used.

[0125] In addition, when a network scan is performed during a period when communication is not busy, the scan timing 1' shown in FIG. 6(b) is used. S ~T' S (T' is an integer equal to or greater than 1). In FIG. 6B, scan timing 1' S ~T' S are set to be spaced apart in time, because multiple time periods with no communication are not adjacent to each other.

[0126] In addition, the scan timing t' S (t' S =1' S ~T' S The configuration of the scan timing t S The configuration is the same as

[0127] Also, the scan timing t S ,t' S does not mean a single time, but a starting time t start From end time t end It is a concept that has a certain time length.

[0128] Scan timing t' S A method for determining the time dependency of traffic will be described. Fig. 7 is a diagram showing the time dependency of traffic. In Fig. 7, the vertical axis represents traffic and the horizontal axis represents time. A curve k1 represents the time dependency of traffic in, for example, a terminal device A, and a curve k2 represents the time dependency of traffic in, for example, a terminal device B different from the terminal device A.

[0129] Referring to (a) of FIG. 7, in terminal device A, traffic is heavier at night than during the day (see curve k1). That is, in terminal device A, communication is slow during the day and congested at night. Referring to (b) of FIG. 7, in terminal device B, traffic is heavier during the day than at night (see curve k2). That is, in terminal device B, communication is congested during the day and congested at night.

[0130] Therefore, the scan scheduler 17 detects a time period during which communication is not busy for the M terminal devices that are the targets of the network scan, and determines the scan timing t S (or scan timing t' S ) to determine

[0131] The network scanner 11 receives the scan schedule SCH_scan from the scan scheduler 17, and executes a network scan for M terminal devices in accordance with the received scan schedule SCH_scan.

[0132] In this case, the network scan is performed without monitoring background traffic or with monitoring background traffic.

[0133] Fig. 8 is a diagram showing correlation coefficients of communication characteristics. The correlation coefficients shown in Fig. 8 are correlation coefficients of communication characteristics when communication quality is insufficient in the case where a network scan is performed without monitoring background traffic. In addition, the scan response delay RTT shown in Fig. 8 m , Packet loss rate PLR m , maximum scan delay DLmax m , and the scan minimum delay DLmin m are multiple communication characteristics of M terminal devices when a network scan is performed on M terminal devices. Note that the scan response delay RTT m indicates a single scan response delay or the average value of multiple scan response delays within a single IP address that is the target of a network scan.

[0134] Scan Response Delay RTT m , Packet loss rate PLR m , maximum scan delay DLmax m , and the scan minimum delay DLmin m The correlation coefficients between the diagonal elements are shown in Fig. 8. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in Fig. 8 because they are the same as the correlation coefficients of the off-diagonal elements below the diagonal elements.

[0135] Referring to FIG. 8, when a network scan is performed without monitoring background traffic, the scan response delay RTT m and maximum scan delay DLmax mCorrelation coefficient with scan response delay RTT m and scan minimum delay DLmin m Correlation coefficient and maximum scan delay DLmax m and scan minimum delay DLmin m The correlation coefficient with is greater than 0.5. And the scan response delay RTT m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and scan minimum delay DLmin m The correlation coefficient with the maximum scan delay DLmax m and scan minimum delay DLmin m Therefore, the correlation coefficient of the scan response delay RTT m is the maximum scan delay DLmax m and scan minimum delay DLmin m It has a high correlation with

[0136] Fig. 9 is a diagram showing another correlation coefficient of communication characteristics. The correlation coefficient shown in Fig. 9 is a correlation coefficient of communication characteristics when the network is congested (network congestion) in the case where a network scan is performed without monitoring background traffic. In addition, the scan response delay RTT m indicates the average value within one IP address targeted for network scanning. Scan response delay RTT m , Packet loss rate PLR m , maximum scan delay DLmax m , and the scan minimum delay DLmin m The correlation coefficients between the off-diagonal elements above the diagonal elements are not shown in FIG. 9 for the same reason as explained in FIG. 8. Also, the scan response delay RTT m , Packet loss rate PLR m , maximum scan delay DLmax m , and the scan minimum delay DLmin m are a plurality of communication characteristics of M terminal devices when a network scan is performed on the M terminal devices.

[0137] Referring to Fig. 9, when a network scan is performed without monitoring background traffic, the scan response delay RTT is smaller than the RTT response delay when the network is congested. m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and scan minimum delay DLmin m Correlation coefficient and maximum scan delay DLmax m and scan minimum delay DLmin m The correlation coefficient with is greater than 0.5. And the scan response delay RTT m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and scan minimum delay DLmin m The correlation coefficient with the scan response delay RTT is the same. m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and scan minimum delay DLmin m The correlation coefficient with the maximum scan delay DLmax m and scan minimum delay DLmin m Therefore, even when the network is congested, the scan response delay RTT m is the maximum scan delay DLmax m and scan minimum delay DLmin m It has a high correlation with

[0138] As shown in Figures 8 and 9, when a network scan is performed without monitoring background traffic, the scan response delay (RTT) is significantly longer when the communication quality is poor than when the network is congested. m , Packet loss rate PLR m , maximum scan delay DLmax m , and the scan minimum delay DLmin m The correlation coefficients of the two are different.

[0139] Fig. 10 is a diagram showing still another correlation coefficient of communication characteristics when the communication quality is insufficient in the case where a network scan is performed by monitoring background traffic.

[0140] Scan Response Delay RTT m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m The correlation coefficients between the off-diagonal elements above the diagonal elements are not shown in FIG. 10 for the same reason as explained in FIG. 8. Also, the scan response delay RTT shown in FIG. m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m are a plurality of communication characteristics of M terminal devices when a network scan is performed on the M terminal devices.

[0141] Referring to FIG. 10, when a network scan is performed by monitoring background traffic, if the communication quality is insufficient, the scan response delay RTT m and packet loss rate (PLR) m , Delayed DL m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with packet loss rate PLR m And delayed download m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with delayed DL m and Jitter JT m, background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with jitter JT m and background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with, and background maximum delay BGDLmax m and background minimum delay BGDLmin m The correlation coefficient with is greater than 0.5. And the scan response delay RTT m is the packet loss rate (PLR) m It has the largest correlation coefficient with

[0142] Fig. 11 is a diagram showing still another correlation coefficient of communication characteristics. The correlation coefficient shown in Fig. 11 is a correlation coefficient of communication characteristics when the network is congested in the case where a network scan is executed by monitoring background traffic.

[0143] Scan Response Delay RTT m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m The correlation coefficients between the diagonal elements are shown in Fig. 11. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in Fig. 11 for the same reason as explained in Fig. 8. Also, the scan response delay RTT m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m are a plurality of communication characteristics of M terminal devices when a network scan is performed on the M terminal devices.

[0144] Referring to FIG. 11, when a network scan is performed by monitoring background traffic, the scan response delay RTT m and packet loss rate (PLR) m , Delayed DL m , Jitter JT m and background maximum delay BGDLmax m Correlation coefficient with packet loss rate PLR m And delayed download m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with delayed DL m and Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with jitter JT m and background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with, and background maximum delay BGDLmax m and background minimum delay BGDLmin m The correlation coefficient with is greater than 0.5. And the scan response delay RTT m is the background maximum delay BGDLmax m It has the largest correlation coefficient with

[0145] As shown in Figures 10 and 11, when network scanning is performed by monitoring background traffic, the scan response delay (RTT) is significantly longer when communication quality is poor than when the network is congested. m , Packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m The correlation coefficients of the two are different.

[0146] Note that the scan response delay RTT shown in Figs. m and packet loss rate (PLR) m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m The correlation coefficient with the scan response delay RTT stored in the correspondence table TBL1 shown in Figure 3 or the correspondence table TBL2 shown in Figure 4 is m Scan response SR in one terminal device m The scan response delay (RTT) is calculated by averaging the number of ports that have m The correlation coefficient may be calculated using the scan response delay RTT m and packet loss rate (PLR) m By using each communication characteristic as it is (i.e., scan response delay RTT m Regarding the scan response delay due to port differences, RTT m The correlation coefficient may be calculated by taking into consideration the difference in communication characteristics between the two devices.

[0147] When a network scan is executed, the results of the network scan are classified as successful or failed. If the network scan fails, the following first, second, third, and fourth causes are assumed to be the causes of the failure of the network scan.

[0148] (First cause) The problem is that the communication quality with the terminal device m that is the target of the network scan is insufficient.

[0149] (Second cause) The network to which the terminal device m, which is the target of the network scan, is connected is congested (that is, network congestion).

[0150] (Third cause) The terminal device m that is the target of the network scan is not connected to the network.

[0151] (Fourth cause) The port of the terminal device m that is the target of the network scan is blocked.

[0152] Therefore, in the embodiment of the present invention, the scan response delay RTT obtained when a network scan is executed is calculated by the method described below. m , Packet loss rate PLR m and delayed DL m Using the communication characteristics such as above, it estimates whether the network scan is normal or unsuccessful, and when it estimates that the network scan is unsuccessful, it estimates the cause of the failure (any of the first cause, the second cause, the third cause, and the fourth cause). Note that the first cause and the second cause are causes attributable to the communication channel with the terminal device.

[0153] The method for estimating the results of a network scan is to use the scan response delay RTT shown in Figures 8 to 11. m , Packet loss rate PLR m and delayed DL m The method includes an extraction process for extracting the communication characteristic CMbest that is most suitable for estimating the result of a network scan from among the communication characteristics such as those listed above, and an estimation process for classifying terminal devices having the communication characteristics extracted in the extraction process into a group CLS1 where the network scan is normal and groups CLS2 and CLS3 where the network scan has failed due to a cause caused by the communication channel with the terminal device, calculating a D-score D_2 for distinguishing between groups CLS1 and CLS3, or a D-score D_3 for distinguishing between groups CLS2 and CLS3, estimating that the network scan is normal or unsuccessful based on the calculated D-score D_2 or D-score D_3, and estimating the cause of the failure of the network scan when it is estimated that the network scan has failed.

[0154] [Communication Characteristics CMbestm Extraction of _unknown When a network scan is performed without monitoring background traffic according to the schedule SCH_scan shown in FIG. 6, the scan analysis manager 12 performs one scan timing t s Scan response delay in RTT m , Packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m from the network scanner 11, and the received scan response delay RTT m , Packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m The host information database 13 is created or updated by storing the above in the correspondence table TBL1 of the host information database 13.

[0155] The extraction means 14 extracts M scan response delays RTT m , M packet loss rates PLR m , M scans maximum delay DLmax m and M scan minimum delays DLmin m from the correspondence table TBL1 of the host information database 13, and m , M packet loss rates PLR m , M scans maximum delay DLmax m and M scan minimum delays DLmin m Scan response delay based on RTT m , Packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m The correlation coefficient r ij is calculated by the following formula:

[0156]

number

[0157] In formula (1), x ai is the correlation coefficient r ij The communication characteristic f i represents the sample of y aj is the correlation coefficient r ij The other communication characteristic f j represents a sample of i = 1 ~ M, a j =1~M.

[0158] The extraction means 14 extracts the communication characteristic f i ,f j Correlation coefficient r ij Among them, different communication characteristics f i ,f j Correlation coefficient r for (i≠j) ij (i ≠ j). For example, the calculated correlation coefficient r ij If the correlation coefficients are as shown in Fig. 8, the communication characteristic f i (i=1~4) are the row-wise scan response delays RTT m , Packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m and the communication characteristic f j (j=1~4) are the row-wise scan response delays RTT m , Packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m is set to.

[0159] Correlation coefficient r 11 ,r 22 ,r 33 ,r 44 are the diagonal elements, and the correlation coefficient r 21 ,r 31 ,r 41 ,r 32 ,r 42 ,r 43 are the off-diagonal elements.

[0160] The extraction means 14 extracts the correlation coefficient r 21 ,r 31 ,r 41 ,r 32 ,r 42 ,r 43 Sequentially extract each correlation coefficient r 21 ,r 31 ,r 41 ,r 32 ,r 42 ,r 43 is the threshold value Corr r It is judged whether it is greater than the threshold value Corr r is set to a real number in the range of 0.5 to 1.0, for example. r =0.5.

[0161] First, the extraction means 14 extracts the correlation coefficient r 21 is the threshold Corr r Then, the extraction means 14 determines whether the correlation coefficient r 21 is the threshold Corr r When it is less than or equal to r 21 The two communication characteristics f2 and f1 that were the basis for the calculation of 21 is the threshold value Corr r When it is greater than r 21 The two communication characteristics f2 and f1 used to calculate are stored.

[0162] In this case, the correlation coefficient r 21 (=0.25) is the threshold Corr r (=0.5) or less, the extraction means 14 extracts the correlation coefficient r 21 The two communication characteristics f2 (= packet loss rate PLR) that were used to calculate m ) and communication characteristic f1 (= scan response delay RTT m )

[0163] Next, the extraction means 14 extracts the correlation coefficient r 31 is the threshold Corr r Determine whether the correlation coefficient r is greater than 31Since the correlation coefficient r is 0.80, the extraction means 14 31 is the threshold Corr r is determined to be greater than the correlation coefficient r 31 The two communication characteristics f3 (= maximum scan delay DLmax) that were used to calculate m ) and communication characteristic f1 (= scan response delay RTT m ) to save the file.

[0164] Subsequently, the extraction means 14 extracts the correlation coefficient r 41 is the threshold Corr r Determine whether the correlation coefficient r is greater than 41 Since the correlation coefficient r is 0.97, the extraction means 14 41 is the threshold Corr r is determined to be greater than the correlation coefficient r 41 The two communication characteristics f3 (= minimum scan delay DLmin m ) and communication characteristic f1 (= scan response delay RTT m ) to save the file.

[0165] Similarly, the extraction means 14 extracts the correlation coefficient r 32 ,r 42 ,r 43 is the threshold Corr r Then, the extraction means 14 sequentially determines whether the threshold value Corr r Correlation coefficient r greater than (=0.5) 43 The two communication characteristics f4 (= minimum scan delay DLmin m ), f3(=Maximum scan delay DLmax m ) is stored. The extraction means 14 also stores the threshold value Corr r A correlation coefficient r of less than or equal to (=0.5) 32 ,r 42 The two communication characteristics f3 (= maximum scan delay DLmax) that were used to calculate m ), f2(=Packet loss rate PLR m ); Two communication characteristics f4 (= minimum scan delay DLmin m ), f2(=Packet loss rate PLR m )

[0166] As a result, the two communication characteristics f3 (= maximum scan delay DLmax m ), f1(= scan response delay RTT m ); Two communication characteristics f3 (= minimum scan delay DLmin m ), f1(= scan response delay RTT m ); Two communication characteristics f4 (= minimum scan delay DLmin m ), f3(=Maximum scan delay DLmax m ) is stored, the extraction means 14 finally obtains the scan response delay RTT m , maximum scan delay DLmax m and scan minimum delay DLmin m The three communication characteristics are stored.

[0167] The extraction means 14 extracts the correlation coefficient r ij 9 to 11, the threshold value Corr r A correlation coefficient greater than r ij The communication characteristics on which the threshold Corr is calculated are saved. r A correlation coefficient r that is less than or equal to ij The communication characteristics on which the calculation of is based are deleted.

[0168] Scan Response Delay RTT m is the RTT from sending a packet of test input to receiving a scan response when the network scan is normal. normal It is the best representation of the results of a network scan because it represents the delay relative to

[0169] Therefore, the extraction means 14 extracts the communication characteristic finally stored (in the above example, the scan response delay RTT m , maximum scan delay DLmax m and scan minimum delay DLmin m ) to obtain the maximum scan delay DLmax m and scan minimum delay DLmin mwas extracted as an explanatory variable, and the scan response delay RTT m Extract as the objective variable, perform multiple regression analysis, and calculate the contribution rate (= R 2 ) is required.

[0170] Then, the extraction means 14 extracts the contribution rate R 2 is the threshold S r It is judged whether it is smaller than the threshold value S r is set to a real number in the range of 0.5 to 1.0, for example.

[0171] The extraction means 14 extracts the contribution rate R 2 is the threshold S r If it is determined that the correlation coefficient is smaller than , the mode of the estimation algorithm ALG that estimates the results of the network scan is set to “uncorrelated” and the contribution rate R 2 is the threshold S r If it is determined that the difference is not smaller than , the mode of the estimation algorithm ALG is set to "correlation".

[0172] When the estimation algorithm ALG mode is set to "uncorrelated", it means that the communication characteristics finally stored are somewhat important. On the other hand, when the estimation algorithm ALG mode is set to "correlated", it means that the communication characteristics finally stored are very important.

[0173] Subsequently, the extraction means 14 selects the most suitable communication characteristic CMbest from the K communication characteristics using the machine learning algorithm "SelectKbest" for estimating the results of the network scan.

[0174] A method for selecting communication characteristics CMbest will be described. Fig. 12 is a diagram showing an example of original data when selecting communication characteristics CMbest. Fig. 13 is a diagram for explaining the method for selecting communication characteristics CMbest.

[0175] Several communication characteristics can be obtained from the scan response of the network scan, including packet loss rate (PLR). m , Jitter JT m, maximum scan delay DLmax m , Scan minimum delay DLmin m and delayed DL m is shown in Fig. 12. In Fig. 12, the packet loss rate PLR m , Jitter JT m , maximum scan delay DLmax m , Scan minimum delay DLmin m and delayed DL m is shown for eight terminal devices.

[0176] Then, the extraction means 14 calculates the threshold value Corr r A correlation coefficient greater than r ij As shown in FIG. 13, for example, the packet loss rate (PLR) is m , Jitter JT m , maximum scan delay DLmax m and scan minimum delay DLmin m Select .

[0177] Thereafter, the extraction means 14 calculates the sum of each row and the sum of each column in FIG. 13, and calculates a grand total that is the sum of the sum of each row and the sum of each column.

[0178] Subsequently, the extraction means 14 calculates the expected value E rc is calculated for every row and every column.

[0179]

number

[0180] In formula (2), e r represents the sum of the rth row, and e c represents the sum of the cth column. Also, N grand represents the grand total.

[0181] Fig. 14 is a diagram for explaining a method for calculating the score of the communication characteristic. Fig. 15 is a diagram showing the score of the communication characteristic. In Fig. 14, the maximum scan delay DLmax m We explain how to calculate the score Q of

[0182] 14, the leftmost column is the sum and grand total of the rightmost column of FIG. 13, and the second leftmost column is the scan maximum delay DLmax shown in FIG. m This is a sample.

[0183] The extraction means 14 calculates the sum of each row shown in FIG. 14 (=0.049, 0.034, 0.023, 0.038, 0.022, 0.022, 0.021, 0.023) by subtracting the sum from the e r Then, the sum of the cth column shown in Figure 14 (=0.232) is added to e in equation (2). c , and the total sum of Figure 14 is N in equation (2) grand Substituting into the expected value E rc Calculate.

[0184] Then, the extraction means 14 calculates the score Q by the following formula.

[0185]

number

[0186] In equation (3), O is the maximum scan delay DLmax shown in FIG. m and E is the expected value E calculated by equation (2). rc It is.

[0187] The score Q is a function of one communication characteristic (e.g., the maximum scan delay DLmax m), the score Q represents the magnitude of deviation of each sample O from the expected value E. In equation (3), when each sample O is significantly greater than the expected value E, or when each sample O is significantly smaller than the expected value E, the score Q becomes large, and when each sample O is slightly greater than the expected value E, or when each sample O is slightly smaller than the expected value E, the score Q becomes small. Thus, the score Q represents the magnitude of deviation of one communication characteristic (e.g., maximum scan delay DLmax m ) represents the characteristics of

[0188] After calculating the expected value E, the extraction means 14 calculates OE, and then: (OE) 2 Calculate the calculated (OE) 2 Divide by the expected value E (OE) 2 Calculate / E.

[0189] Then, the extraction means 14 extracts (OE) 2 Scan the sum of / E Maximum delay DLmax m That is, the extraction means 14 calculates the maximum scan delay DLmax by the formula (3). m Calculate the score Q of

[0190] In addition, the extraction means 14 extracts the packet loss rate PLR ​​shown in FIG. m , Jitter JT m and scan minimum delay DLmin m For the packet loss rate PLR, the score Q is calculated by the above-mentioned method. As a result, as shown in FIG. m , Jitter JT m , maximum scan delay DLmax m and scan minimum delay DLmin m A score Q is obtained.

[0191] Then, the extraction means 14 extracts the packet loss rate PLR m , Jitter JT m , maximum scan delay DLmax m and scan minimum delay DLminm From the score Q, K (K is a positive integer) communication characteristics are selected in descending order of score Q.

[0192] For example, when K=3, in the example shown in FIG. 15, the extraction means 14 extracts the jitter JT m , Scan minimum delay DLmin m and the maximum scan delay DLmax m Select .

[0193] After selecting the K communication characteristics, the extraction means 14 selects a communication characteristic CMbest from the K communication characteristics by the following method.

[0194] First, the extraction means 14 classifies the K communication characteristics using classification algorithms such as LR (Linear Regression) and SVM (Support Vector Machine) into a group GP1 of communication characteristics that are important for estimating the results of a network scan and a group GP2 of communication characteristics that are not important for estimating the results of a network scan.

[0195] Here, we use support vector machine (SVM-train) as the classification algorithm.

[0196] SVM is a pattern recognition model that uses supervised learning, and the training data X0 = [x 01 ,x 02 ,x 0M ] T (T stands for transposition) is learned as the boundary that divides it most boldly. In order to learn this boundary, a space of two or more dimensions consisting of two or more parameters is set.

[0197] Then, training samples arranged in a two-dimensional or higher space are used for training, and the training data X0 = [x 01 ,x 02 ,x 0M ] TThe boundary that divides the image most boldly is determined so that the distance (margin) between the boundary and the closest sample (data) is maximized. This determined boundary is called the maximum margin hyperplane. Note that the specific method for determining this boundary is well known, so it will not be described here.

[0198] The extraction means 14 extracts the teacher data X0=[x 01 ,x 02 ,x 0M ] T (T stands for transpose) to train a support vector machine (SVM-train), and use the boundaries determined by the training to train M sample vectors x m Each sample vector x (m=1 to M) satisfies the following equation: m is classified into either group GP1 or GP2. Here, the sample vector x m x m =[CM 1m ,CM 2m ,···,CM km ,···,CM Km ]. That is, the sample vector x m is K communication characteristics CM 1m ~CM Km As a result, data to be classified into either group GP1 or GP2 is expressed by formula (5).

[0199]

number

[0200]

number

[0201] In equation (5), the communication characteristic C km (k=1 to K, m=1 to M) represents the k-th communication characteristic of terminal device m.

[0202] In equation (4), z is a weight vector and b is a predetermined constant.

[0203] Group GP1 is a group of communication characteristics that are important for estimating the results of a network scan. Group GP2 is a group of communication characteristics that are not important for estimating the results of a network scan. m When y is classified into group GP1, m consists of "1", and the sample vector x m When y is classified into group GP2, m consists of "-1".

[0204] The extraction means 14 calculates the weight vector z by the equation (4) to determine the classification index α of the following equation.

[0205]

number

[0206] In equation (6), X = [x1, x2, , x M ] T (T stands for transpose), and y = [y1, y2, , y M ] T (T stands for transposition).

[0207] The extraction means 14 extracts the sample vector x m When y is classified into group GP1 (i.e., m consists of “1”), z x m The weight vector z is calculated by -b=1, and the calculated weight vector z is set as the classification index α. m When y is classified into group GP2 (i.e., m consists of "-1"), z x m A weight vector z is calculated by −b=−1, and the calculated weight vector z is set as the classification index α.

[0208] The extraction means 14 extracts the k-th communication characteristic C kmWhen the classification index α for is determined, the determined classification index α (=α k ), sample vector x km , and y m (=“1” or “-1”) into the following formula to obtain the weight w k Calculate.

[0209]

number

[0210] In formula (7), x km is the sample vector x m Sample CM of the kth communication characteristic km Therefore, the right side of equation (7) represents the k-th communication characteristic C k1 ,CM k2 ,···,CM km ,···,CM kM Using the weight w k This means calculating

[0211] Then, the extraction means 14 extracts the weight w k Substituting into the following formula, the rank weight R k Calculate.

[0212]

number

[0213] The extraction means 14 extracts rank weights R k Calculate.

[0214] Figure 16 shows the rank weight R k 16 is a diagram showing the calculation result of the maximum scan delay DLmax m , Jitter JT m , Scan minimum delay DLmin mand packet loss rate (PLR) m For rank weight R k and rank are associated.

[0215] Then, the extraction means 14 selects the one with the lowest rank (i.e., the smallest rank weight R k ) communication characteristics (here, packet loss rate PLR m ) is determined as the communication characteristic to be deleted. Then, the extraction means 14 extracts the samples of the communication characteristic into the sample vectors x1 to x M The calculation of equations (4), (6) to (8) and the determination of the communication characteristics to be deleted are repeated until all communication characteristics are deleted. k If there are a plurality of communication characteristics that give the smallest rank weight R k If there are a plurality of communication characteristics that give the scan response delay RTT m Correlation coefficient r ij The communication characteristic with the smallest σ may be selected as the communication characteristic to be deleted.

[0216] Then, the extraction means 14 extracts the last deleted communication characteristic as the communication characteristic CMbest m _unknown and the detected communication characteristics are CMbest m _unknown and scan response delay RTT m _unknown and output to the estimation means 16.

[0217] In addition, when the network scan is executed by monitoring background traffic according to the schedule SCH_scan shown in FIG. 6, the extraction means 14 extracts the M scan response delay RTTs described above. m , M packet loss rates PLR m , M scans maximum delay DLmax m and M scan minimum delays DLmin m Instead, M scan response delays RTT m , M packet loss rates PLR m, M delays DL m , M jitter JT m , M background maximum delays BGDLmax m and M background minimum delays BGDLmin m Based on this, the communication characteristic CMbest is calculated by the above-mentioned method. m Select _unknown.

[0218] [Scan Response Delay RTT m Scan response delay RTT most similar to _unknown m Select _known The extraction means 14 extracts the scan response delays RTT1 to RTT2 from the correspondence table TBL1 (or the correspondence table TBL2) stored in the host information database 13. M Scan response delay RTT1_unknown~RTT M Detected as _unknown. In this case, scan response delay RTT1~RTT M Each of the Add m It consists of one scan response delay or an average of multiple scan response delays within a

[0219] The extraction means 14 also extracts the scan response delay {RTT m’}_known_1(m'=1'~M'), scan response delay {RTT m’}_known_2(m'=1'~M') and scan response delay {RTT m’}Detect_known_3(m'=1'~M').

[0220] Thereafter, the extraction means 14 extracts the scan response delay RTT1_unknown to RTT M _unknown, scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m′=1′ to M′) is output to the calculation means 15.

[0221] The calculation means 15 calculates the scan response delay RTT1_unknown to RTT M _unknown, scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m′=1′ to M′) is received from extraction means 14.

[0222] The calculation means 15 calculates the scan response delay RTT m Based on this, mode m (Mo), which is the value of the scan response delay most likely to occur, is acquired. Note that, when there are two most likely scan response delay values, calculation means 15 calculates mode m (Mo), which is the value of the scan response delay most likely to occur, by the following formula. Note that, when there are three or more most likely scan response delay values, calculation means 15 detects minimum and maximum values ​​from the three or more most likely scan response delays, and calculates mode m (Mo), which is the value of the scan response delay most likely to occur, by the following formula based on the detected minimum and maximum values.

[0223]

number

[0224] In equation (9), L indicates the lower boundary of the probability density function region (=PDF (Probability Density Function) region) that includes two values ​​of the most likely scan response delay when the network scan result is unknown or when the network scan result is known. Also, r indicates the number of samples (=the number of scan response delays) in the PDF region that is most likely to occur when the network scan result is unknown or when the network scan result is known. Furthermore, r1 is the region that is smaller than the scan response delay RTT than the PDF region that is most likely to occur when the network scan result is unknown or when the network scan result is known. m In the direction of decreasing, r2 indicates the number of samples (= the number of scan response delays) in the previous PDF region. Furthermore, r2 is the scan response delay RTT of the PDF region that is most likely to occur when the network scan result is unknown or when the network scan result is known. m indicates the number of samples (= the number of scan response delays) in the next PDF region in the larger direction. Furthermore, h indicates the interval between each PDF region (i.e., the width of each PDF region).

[0225] Figures 17 to 19 are diagrams showing specific distributions of scan response delays. Figure 20 is a diagram showing the corresponding relationship between PDF regions and the number of samples generated. Figure 21 is a diagram showing specific distributions of probability density functions of scan response delays.

[0226] If the number of samples for each specific value in the specific distribution of scan response delay shown in Figure 17 is counted and the number of samples for each specific value is displayed, the result is shown in Figure 18. As a result, it can be seen from Figure 18 that scan response delay is most likely to occur at probability density values ​​of 0.41 and 0.6.

[0227] As shown in FIG. 19, the number of samples generated in each of the PDF regions of 0.0 to 0.2, 0.21 to 0.4, 0.41 to 0.6, 0.61 to 0.8, and 0.81 to 1.0 is “5”, “5”, “14”, “4”, and “2”, respectively.

[0228] The correspondence between the PDF region and the number of samples generated in each region is as shown in Figure 20. The number of samples generated in the PDF region of 0.41 to 0.6 (=14) is r, the interval of 0.41 to 0.6 (=0.2) is h, 0.41 is L, the number of samples generated in the PDF region of 0.21 to 0.4 (=5) is r1, and the number of samples generated in the PDF region of 0.61 to 0.8 (=4) is r2. Therefore, the mode m (Mo) can be calculated by substituting r (=14), r1 (=5), r2 (=4), h (=0.2), and L (=0.41) into formula (9).

[0229] As shown in FIG. 21, r is a PDF region REG surrounded by the two values ​​(=0.41 and 0.6) at which the scan response delay is most likely to occur and the curve showing the probability density function. PDF L is the lower of the two values ​​(=0.41 and 0.6) that are most likely to cause a scan response delay. h is the interval between the two values ​​(=0.41 and 0.6). r1 is the number of samples of the scan response delay in the PDF region REG. PDF The PDF area REG with a smaller scan response delay than PDF and r2 is the number of samples in a region of width h adjacent to the PDF region REG PDF The PDF area REG on the side with a larger scan response delay PDF It consists of the number of samples in a region of width h adjacent to

[0230] FIG. 22 is a diagram showing another specific distribution of scan response delay. Referring to FIG. 22, there are three most likely scan response delay values. In this case, the interval h is adjusted to the width of the PDF region including the three most likely scan response delay values. Then, the minimum value (=0.35) and the maximum value (=0.6) are detected from the three most likely scan response delay values. After that, the minimum value (=0.35) and the maximum value (=0.6) are used as the two most likely scan response delay values ​​to determine r, r1, r2, L, and h by the method described in FIG. 17 to FIG. 21, and the determined r, r1, r2, L, and h are substituted into formula (9) to calculate the mode m (Mo).

[0231] When there are four or more most likely scan response delay values, the mode m (Mo) is calculated in the same manner as when there are three most likely scan response delay values.

[0232] The following is the scan response delay RTT1_unknown~RTT M Based on _unknown, the mode m(Mo) calculated by equation (9) is denoted as “m(Mo)_unknown”, and the scan response delay {RTT m’ _known}_1 (m'=1'~M'), the mode m(Mo) calculated by Equation (9) is denoted as "m(Mo)_known_1", and the scan response delay {RTT m’ _known}_2 (m'=1'~M'), the mode m(Mo) calculated by Equation (9) is denoted as "m(Mo)_known_2", and the scan response delay {RTT m’ The mode m(Mo) calculated by equation (9) based on {_known}_3 (m'=1' to M') is denoted as "m(Mo)_known_3."

[0233] The calculation means 15 calculates the modes m(Mo)_unknown, m(Mo)_known_1, m(Mo)_known_2, and m(Mo)_known_3 by the above-mentioned method, and then calculates the scan response delays RTT1_unknown to RTT MMean absolute deviation of _unknown W unknown Calculate.

[0234]

number

[0235] C(x i_unknown ) is represented by formula (10B), and x i_unknown Scan response delay RTT1_unknown~RTT M _unknown is assigned.

[0236] The calculation means 15 calculates the mean absolute deviation W unknown The scan response delay {RTT m’ _known}_z(m'=1'~M') mean absolute deviation W known_z Calculate.

[0237]

number

[0238] In formula (11), z is an argument indicating the known estimated results of the network scan and is an integer satisfying 1≦z≦Z. Furthermore, Z is the total number of known estimated results of the network scan. In this embodiment of the invention, the known estimated results of the network scan are "normal", "poor communication quality", and "network congestion", so in formula (11), z=1,2,3.

[0239] In addition, the scan response delays corresponding to the known estimated network scan results of "normal", "poor communication quality" and "network congestion" may consist of one type of data or multiple types of data.

[0240] The calculation means 15 substitutes the mode m(Mo)_known_1 into m(Mo)_known_z in the formula (11B) to obtain the scan response delay {RTT 1’_known}_1~{RTT M’ _known}_1 into x in formula (11B) i_known_z Substituting this, we obtain the mean absolute deviation W known_1 Calculate.

[0241] In addition, the calculation means 15 substitutes the mode m(Mo)_known_2 into m(Mo)_known_z in the formula (11B) to obtain the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 into x in equation (11B) i_known_z Substituting this, we obtain the mean absolute deviation W known_2 Calculate.

[0242] Furthermore, the calculation means 15 substitutes the mode m(Mo)_known_3 into m(Mo)_known_z in the formula (11B) to obtain {RTT 1’ _known}_3~{RTT M’ _known}_3 into x in formula (11B) i_known_z Substituting this, we obtain the mean absolute deviation W known_3 Calculate.

[0243] That is, the calculation means 15 calculates the modes m(Mo)_known_1, m(Mo)_known_2, m(Mo)_known_3 and the scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1,{RTT 1’ _known}_2~{RTT M’ _known}_2,{RTT 1’ _known}_3~{RTT M’ Based on _known}_3, the mean absolute deviation W known_1 ,W known_2 ,W known_3 Calculate.

[0244] Then, the calculation means 14 calculates the mean absolute deviation W unknown ,W known_1 ,W known_2 ,W known_3 Substituting into the following formula, the difference in mean absolute deviation SS1 ,S S2 ,S S3 Calculate.

[0245]

number

[0246] In this case, the calculation means 15 calculates the mean absolute deviation W known_1 In equation (12), known_z Substituting the mean absolute deviation W unknown In equation (12), unknown Substituting in, the difference in mean absolute deviation S S1 Calculate.

[0247] In addition, the calculation means 15 calculates the mean absolute deviation W known_2 In equation (12), known_z Substituting the mean absolute deviation W unknown In equation (12), unknown Substituting in, the difference in mean absolute deviation S S2 Calculate.

[0248] Furthermore, the calculation means 15 calculates the mean absolute deviation W known_3 In equation (12), known_z Substituting the mean absolute deviation W unknown In equation (12), unknown Substituting in, the difference in mean absolute deviation S S3 Calculate.

[0249] The calculation means 15 calculates the difference in mean absolute deviation S S1 ,S S2 ,S S3 The difference between the calculated mean absolute deviations S S1 ,S S2 ,S S3 Substituting into the following formula, similarity P Sz Calculate.

[0250]

number

[0251] In this case, the calculation means 15 calculates the difference in mean absolute deviations S S1 S in equation (13) Sz Substituting into the similarity P S1 Calculate the mean absolute deviation difference S S2 S in equation (13) Sz Substituting into the similarity P S2 Calculate the mean absolute deviation difference S S3 S in equation (13) Sz Substituting into the similarity P S3 Calculate.

[0252] The calculation means 15 calculates the similarity P S1 ,P S2 ,P S3 The calculated similarity P S1 ,P S2 ,P S3 is output to the extraction means 14.

[0253] The extraction means 14 extracts the similarity P S1 ,P S2 ,P S3 When the calculation means 15 receives the similarity P S1 ,P S2 ,P S3 The highest similarity P S_highest Then, the extraction means 14 refers to the correspondence table TBL3 stored in the host information database 13 and extracts the host with the highest similarity P S_highest Scan response delay when the scan is obtained {RTT m’ _known}_similarity(m'=1'~M')(=scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M'), and scan response delay {RTT m’ _known}_3 (m'=1' to M'). The extraction means 14 also detects the scan response delay {RTT m’ _known}_similarity (m'=1'~M') and the communication characteristic {CMbest m’_known}_similarity(m'=1'~M')(=communication characteristics {CMbest m’ _known}_1(m'=1'~M'), communication characteristics {CMbest m’ _known}_2(m'=1'~M'), and communication characteristics {CMbest m’ _known}_3 (m'=1'~M').

[0254] Then, the extraction means 14 extracts the scan response delay {RTT m’ _known}_similarity(m'=1'~M') and communication characteristics {CMbest m’ _known}_similarity (m′=1′ to M′), and outputs the generated standard sample data D_similarity to the estimation means 16.

[0255] Further, the extraction means 14 refers to the correspondence table TBL1 (or correspondence table TBL2) stored in the host information database 13 to extract the scan response delay RTT1_unknown to RTT M _unknown (=scan response delay RTT m (m=1 to M)) and detect the detected scan response delay RTT1_unknown to RTT M _unknown (=scan response delay RTT m The estimation means 16 generates sample data D_unknown consisting of the communication characteristics CMbest_unknown_m (m=1 to M) and the communication characteristics CMbest_unknown_m (m=1 to M).

[0256] The estimation means 16 receives the standard sample data D_similarity and the sample data D_unknown from the extraction means 14 .

[0257] Then, the estimation means 16 updates the estimation algorithm ALG to an optimal algorithm using the standard sample data D_similarity in the manner described below.

[0258] Fig. 23 is a diagram showing data when training the estimation algorithm ALG. Referring to Fig. 23, when training the estimation algorithm ALG using the standard sample data D_similarity, the estimation means 16 divides the standard sample data D_similarity into training data and test data. In this case, the estimation means 16 divides the standard sample data D_similarity into training data and test data so that the ratio of training data is 80% and the ratio of test data is 20%, for example. Note that when dividing the standard sample data D_similarity into training data and test data, the ratio of training data may be other than 80%, and the ratio of test data may be other than 20%.

[0259] After that, the estimation means 16 divides the training data into a training set and a validation set. In this case, the estimation means 16 divides the training data into a training set and a validation set so that the proportion of the training set is 50% and the proportion of the validation set is 50%, for example. Note that the proportions of the training set and the validation set may be other than 50%.

[0260] The estimation means 16 divides the training data into a training set and a validation set, and then divides the training set into U subsets d1 to d U Divide into subsets d1~d U Each of the sets of data is n in size, where n is, for example, 8. The estimation means 16 divides the validation set into U′ subsets V 1’ ~V U’ Then, the data size of the training set is D S Then, U = INT(D S / n).

[0261] 24 is a conceptual diagram of sample data when training the estimation algorithm ALG. With reference to FIG. 24, when training the estimation algorithm ALG, the estimation means 16 creates initial sample data for modeling. More specifically, the estimation means 16 obtains the scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 and the read scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ Generate standard sample data D_standard_1 from {_known}_1.

[0262] Also, the estimation means 16 obtains the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest1'_known}_2~{CMbest M’ _known}_2 and the read scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ Generate standard sample data D_standard_2 from {_known}_2.

[0263] Furthermore, the estimation means 16 obtains the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ _known}_3 and the read scan response delay {RTT 1’_known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ Generate standard sample data D_standard_3 from {_known}_3.

[0264] Then, the estimation means 16 generates the initial sample data D_sample_initial as follows (see FIG. 24(a)). In this case, the estimation means 16 generates a plurality of subsets d 1_1 ~d U_1 One subset d from 1_initial Select multiple subsets d of the standard sample data D_standard_2 1_2 ~d U_2 One subset d from 2_initial Select multiple subsets d of the standard sample data D_standard_3 1_3 ~d U_3 One subset d from 3_initial Then, "1", "2", and "3" in the standard sample data D_standard_1, D_standard_2, and D_standard_3 respectively represent the identification code ID_code=1 indicating that the result of the network scan is "normal", the identification code ID_code=2 indicating that the result of the network scan is "inadequate communication quality", and the identification code ID_code=3 indicating that the result of the network scan is "network congestion", so the estimation means 16 selects the subset d 1_initial ,d 2_initial ,d 3_initial When you select the subset d via the identification code ID_code=1,2,3 1_initial ,d 2_initial ,d 3_initial are combined to generate the initial sample data D_sample_initial.

[0265] Note that multiple subsets d 1_1 ~d U_1The “1” in corresponds to the “1” in the standard sample data D_standard_1, and multiple subsets d 1_2 ~d U_2 The “2” in corresponds to the “2” in the standard sample data D_standard_2, and multiple subsets d 1_3 ~d U_3 The “3” in the sample data D_standard_3 corresponds to the “3” in the standard sample data D_standard_3. 1_1 ~d U_1 , multiple subsets d 1_2 ~d U_2 and multiple subsets d 1_3 ~d U_3 One subset d from each 1_initial ,d 2_initial ,d 3_initial may be selected randomly or according to any selection method.

[0266] As shown in FIG. 23, subsets d1 to d U is generated from the standard sample data D_similarity. The standard sample data D_similarity is the sample data that has the highest similarity to the sample data D_unknown among the standard sample data D_standard_1 to D_standard_3.

[0267] As a result, the subsets d1 to d U is generated from the standard sample data D_standard_1 by the method described in FIG. 23 when the standard sample data D_standard_1 has the highest similarity to the sample data D_unknown. U is generated from the standard sample data D_standard_2 by the method described in FIG. 23 when the standard sample data D_standard_2 has the highest similarity to the sample data D_unknown. Uis generated from the standard sample data D_standard_3 by the method described in FIG. 23 when the standard sample data D_standard_3 has the highest similarity to the sample data D_unknown.

[0268] Subset d1~d U When generated from the standard sample data D_standard_1, the standard sample data D_standard_1 in the initial sample data D_sample_initial for modeling is set as a subset d u By replacing the sample data D_sample_tentative(D1) with the standard sample data D_standard_1 in the initial sample data D_sample_initial, the tentative sample data D_sample_tentative(D1) for modeling is generated (see FIG. 24(b)). u It indicates that it has been replaced by.

[0269] Also, subsets d1 to d U When generated from the standard sample data D_standard_2, the standard sample data D_standard_2 in the initial sample data D_sample_initial for modeling is set as a subset d u By replacing the sample data D_sample_tentative(D2) with the standard sample data D_standard_2 in the initial sample data D_sample_initial, the tentative sample data D_sample_tentative(D2) for modeling is generated (see FIG. 24(c)). Note that the “D2” in D_sample_tentative(D2) is a subset of the standard sample data D_standard_2 in the initial sample data D_sample_initial. u It indicates that it has been replaced by.

[0270] Furthermore, subsets d1 to d U When the standard sample data D_standard_3 is generated, the standard sample data D_standard_3 in the initial sample data D_sample_initial for modeling is set as a subset du By replacing the sample data D_sample_tentative with the standard sample data D_standard_3 in the initial sample data D_sample_initial, the tentative sample data D_sample_tentative(D3) for modeling is generated (see FIG. 24(d)). Note that the “D3” in D_sample_tentative(D3) is a subset of the standard sample data D_standard_3 in the initial sample data D_sample_initial. u It indicates that it has been replaced by.

[0271] FIG. 25 is a conceptual diagram showing update of temporary sample data for modeling. In FIG. 25, subsets d1 to d U A method for updating tentative sample data D_sample_tentative (D1) for modeling will be described below using an example in which the tentative sample data D_sample_tentative (D1) is generated from standard sample data D_standard_1.

[0272] 25, the estimation means 16 first creates initial sample data D_sample_initial for modeling by the above-mentioned method. After that, the estimation means 16 replaces the standard sample data D_standard_1 of the sample data D_sample_initial by the subset d1 to update the initial sample data D_sample_initial to tentative sample data D_sample_tentative_1 for modeling. Then, by updating the initial sample data D_sample_initial to the tentative sample data D_sample_tentative_1, the initial sample data D_sample_initial becomes an "old training sample" and the tentative sample data D_sample_tentative_1 becomes a "new training sample".

[0273] Thereafter, if the estimation means 16 determines that the evaluation index when estimating the results of the network scan using the tentative sample data D_sample_tentative_1 (= "new training sample") is better than the evaluation index when estimating the results of the network scan using the initial sample data D_sample_initial (= "old training sample"), the tentative sample data D_sample_tentative_1 becomes the "old training sample."

[0274] Subsequently, the estimation means 16 updates the tentative sample data D_sample_tentative_1 to tentative sample data D_sample_tentative_2 by replacing the subset d1 of the tentative sample data D_sample_tentative_1 with the subset d2. Then, if the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_2 (="new training sample") is better than the evaluation index when the result of the network scan is estimated using the "old training sample", the tentative sample data D_sample_tentative_2 becomes the "old training sample". When comparing the evaluation index between the tentative sample data D_sample_tentative_2 (= "new training sample") and the "old training sample", if the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_1 is better than the evaluation index when the result of the network scan is estimated using the initial sample data D_sample_initial in the previous comparison of evaluation indexes, the "old training sample" is made up of the tentative sample data D_sample_tentative_1, and if the evaluation index when the result of the network scan is estimated using the initial sample data D_sample_initial is better than the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_1, the "old training sample" is made up of the initial sample data D_sample_initial. In other words, the sample data with the better evaluation index becomes the "old training sample" to be compared in the next comparison of evaluation indexes.

[0275] Furthermore, the estimation means 16 subsequently updates the tentative sample data D_sample_tentative_2 to tentative sample data D_sample_tentative_3 by replacing the subset d2 of the tentative sample data D_sample_tentative_2 with the subset d3. If the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_3 is better than the evaluation index when the result of the network scan is estimated using the "old training sample", the tentative sample data D_sample_tentative_3 becomes the "old training sample".

[0276] Thereafter, in a similar manner, the estimation means 16 repeats the process of updating the tentative sample data D_sample_tentative_u to the tentative sample data D_sample_tentative_u+1, and finally updates it to the tentative sample data D_sample_tentative_U.

[0277] In this way, the estimation means 16 divides the standard sample data D_standard_1 of the initial sample data D_sample_initial into U subsets d1 to d U , thereby updating the initial sample data D_sample_initial to the tentative sample data D_sample_tentative_1 to D_sample_tentative_U in turn.

[0278] Then, when the standard sample data D_standard_1 has the highest similarity to the sample data D_unknown, the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are used to train the estimation algorithm ALG.

[0279] In addition, when the standard sample data D_standard_2 has the highest similarity to the sample data D_unknown, the provisional sample data D_sample_tentative (D2) shown in (c) of Figure 24 constitutes the provisional sample data D_sample_tentative_1, and the estimation means 16 sequentially updates the initial sample data D_sample_initial to the provisional sample data D_sample_tentative_1 to D_sample_tentative_U using the method described in Figure 25.

[0280] Furthermore, when the standard sample data D_standard_3 has the highest similarity to the sample data D_unknown, the provisional sample data D_sample_tentative (D3) shown in (d) of Figure 24 constitutes the provisional sample data D_sample_tentative_1, and the estimation means 16 sequentially updates the initial sample data D_sample_initial to the provisional sample data D_sample_tentative_1 to D_sample_tentative_U using the method described in Figure 25.

[0281] As described above, the initial sample data D_sample_initial is composed of the standard sample data D_standard_1 to D_standard_3. The tentative sample data D_sample_tentative_1 to D_sample_tentative_U are each composed of one of the standard sample data D_standard_1 to D_standard_3 in the initial sample data D_sample_initial, depending on which of the standard sample data D_standard_1 to D_standard_3 the standard sample data D_similarity is composed of. U Then, the subsets d1 to d U is composed of any one of the standard sample data D_standard_1 to D_standard_3 that constitute the standard sample data D_similarity.

[0282] Therefore, the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are basically configured by combining parts of the standard sample data D_standard_1 to D_standard_3, respectively.

[0283] The initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are formed by combining the standard sample data D_standard_1 to D_standard_3 for the following reason.

[0284] As described below, the estimation algorithm ALG handles group CLS1 where the network scan result is “normal”, group CLS2 where the network scan result is “insufficient communication quality”, and group CLS3 where the network scan result is “network congestion”. Therefore, the initial sample data D_sample_initial and tentative sample data D_sample_tentative_1 to D_sample_tentative_U, which are the data used to train the estimation algorithm ALG, are composed of standard sample data D_standard_1 obtained in a communications environment where the network scan result is “normal”, standard sample data D_standard_2 obtained in a communications environment where the network scan result is “insufficient communication quality”, and standard sample data D_standard_3 obtained in a communications environment where the network scan result is “network congestion”, so that the network scan result can be accurately estimated as any of “normal”, “insufficient communication quality”, and “network congestion”.

[0285] Fig. 26 is a conceptual diagram of training and evaluation of the estimation algorithm ALG. Referring to Fig. 26, the estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial. In this case, the estimation means 16 trains the estimation algorithm ALG by repeatedly estimating the result of the network scan by the estimation algorithm ALG using the initial sample data D_sample_initial.

[0286] Then, the estimation means 16 estimates the results of the network scan using the validation set. After that, the estimation means 16 evaluates the estimated results of the network scan by an evaluation index EVD.

[0287] Subsequently, the estimation means 16 (I) trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, (II) selects a validation set (= a subset V 1’ ~V U’(III) estimating the result of the network scan using the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, and (III) evaluating the estimated result of the network scan by the evaluation index nEVD_u for all of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U. In this case, the estimation means 16 determines whether or not the evaluation index nEVD_u is larger than the evaluation index EVD each time it executes (I) to (III), and when the evaluation index nEVD_u is larger than the evaluation index EVD, it sets the evaluation index nEVD_u to the evaluation index EVD and updates the evaluation index EVD by the evaluation index nEVD_u, and when the evaluation index nEVD_u is not larger than the evaluation index EVD, it does not update the evaluation index EVD (i.e., it maintains the evaluation index EVD). Then, the estimation means 16 maintains the variables of the estimation algorithm ALG when the evaluation index EVD when estimating the results of the network scan using the initial sample data D_sample_initial is the best, and when the evaluation index nEVD_u when estimating the results of the network scan using tentative sample data (any of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U) is the best, replaces the variables of the estimation algorithm ALG with the tentative sample data (any of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U) when the largest evaluation index nEVD_u is obtained.

[0288] Thereafter, the estimation means 16 estimates the result of the network scan using the test data, and evaluates whether to approve the training model (estimation algorithm ALG after training) and the training dataset based on the evaluation index tEVD. More specifically, when the evaluation index tEVD is equal to or greater than the evaluation index EVD (finally updated evaluation index EVD), the estimation means 16 approves the training model (estimation algorithm ALG after training) and the training dataset, and when the evaluation index tEVD is not equal to or greater than the evaluation index EVD (finally updated evaluation index EVD), the estimation means 16 does not approve the training model (estimation algorithm ALG after training) and the training dataset. Then, when the estimation means 16 does not approve the training model (estimation algorithm ALG after training) and the training dataset, the estimation algorithm ALG with the variables finally replaced is used to repeatedly execute (I) to (IV) shown in FIG. 26.

[0289] The approved training dataset becomes the initial sample data D_sample_initial when the next training and evaluation are performed. The evaluation index EVD is composed of the F1 score F1 or the Matthews correlation coefficient MCC, which will be described later. The evaluation index nEVD_u is composed of the F1 score nF1 or the Matthews correlation coefficient nMCC, which will be described later. The evaluation index tEVD is composed of the F1 score tF1 or the Matthews correlation coefficient tMCC, which will be described later.

[0290] In this way, the estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, and also trains the validation set (=subset V 1’ ~V U’ ) is used to evaluate the estimation results of the network scan using the trained estimation algorithm ALG, using the evaluation indices EVD and nEVD. Finally, using the test data, the evaluation index tEVD is used to evaluate whether the trained estimation algorithm ALG and the training dataset should be approved.

[0291] Fig. 27 is a diagram showing a specific example of the initial sample data D_sample_initial for modeling shown in Fig. 25. Fig. 28 is a diagram showing a specific example of the training set shown in Fig. 23. Fig. 29 is a diagram showing a specific example of the validation set shown in Fig. 23. Fig. 30 is a diagram showing a specific example of the test data shown in Fig. 23.

[0292] 27 to 30, the number of terminal devices targeted for the network scan is eight (i.e., M'=8), and the number of samples of the standard sample data D_similarity is one for the IP address Add m’ As an example, a specific example of the initial sample data D_sample_initial, a specific example of the training set, a specific example of the validation set, and a specific example of the test data are shown for the case where there are 20 pieces of data.

[0293] 27, the standard sample data D_standard_1 is a scan response delay (RTT) when a network scan is performed in a communication environment where the result of the network scan is "normal." m’ and communication characteristics CMbest m’ Therefore, in the standard sample data D_standard_1, there are eight scan response delays RTT m’ Specific values ​​and eight communication characteristics of CMbest m’ The specific value of is stored in the correspondence table TBL3 shown in FIG. 1’ _known}_1~{RTT M’ _known}_1, where M'=8, is a subset of the standard sample data D_standard_1.

[0294] In addition, the standard sample data D_standard_2 is the scan response delay RTT when a network scan is performed in a communication environment where the network scan result is "insufficient communication quality." m’ and communication characteristics CMbest m’Therefore, in the standard sample data D_standard_2, there are eight scan response delays RTT m’ Specific values ​​and eight communication characteristics of CMbest m’ The specific value of is the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2, where M'=8, is a subset of the standard sample data D_standard_2.

[0295] Furthermore, the standard sample data D_standard_3 is a scan response delay (RTT) when a network scan is performed in a communication environment where the result of the network scan is "network congestion". m’ and communication characteristics CMbest m’ Therefore, in the standard sample data D_standard_3, there are eight scan response delays RTT m’ Specific values ​​and eight communication characteristics of CMbest m’ The specific value of is the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3, where M'=8, is a subset of the standard sample data D_standard_3.

[0296] As a result, the initial sample data D_sample_initial for modeling is composed of standard sample data D_standard_1 to D_standard_3 combined with identification code D_code=1 assigned to the network scan result of "normal," identification code D_code=2 assigned to the network scan result of "poor communication quality," and identification code D_code=3 assigned to the network scan result of "network congestion."

[0297] 28, the training set includes eight subsets d1 to d8. Since M′=8 for each of the subsets d1 to d8, eight scan response delay RTTs are included. 1’~RTT 8’ And 8 communication characteristics CMbest 1’ ~CMbest 8’ Therefore, in each of the subsets d1 to d8, the vertical direction is made up of eight scan response delay RTTs corresponding to eight terminal devices. 1’ ~RTT 8’ And 8 communication characteristics CMbest 1’ ~CMbest 8’ This indicates that:

[0298] The number of subsets d1 to d8 is eight because the number of samples in the standard sample data D_similarity is 20, which is 20×0.8×0.5=8.

[0299] Furthermore, when the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the subsets d1 to d8 is determined based on the scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 and the standard sample data D_similarity is the standard sample data D_standard_2, the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 and the standard sample data D_similarity is the standard sample data D_standard_3, the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ It consists of _known}_3.

[0300] Furthermore, each of the subsets d1 to d8 has a scan response delay {RTT 1’ _known}_1~{RTT 8’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest 8’ _known}_1, the standard sample data D_standard_1 of the initial sample data D_sample_initial shown in FIG. 27 is replaced with the subsets d1 to d8 in turn, thereby generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 25 in turn.

[0301] Furthermore, each of the subsets d1 to d8 has a scan response delay {RTT 1’ _known}_2~{RTT 8’ _known}_2 and communication characteristics {CMbest1'_known}_22~{CMbest 8’ 27. In the case where the initial sample data D_sample_initial consists of {_known}_2, the standard sample data D_standard_2 of the initial sample data D_sample_initial shown in FIG. 27 is replaced with the subsets d1 to d8 in turn, thereby generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 25 in turn.

[0302] Furthermore, each of the subsets d1 to d8 has a scan response delay {RTT 1’ _known}_3~{RTT 8’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest 8’ 27. In the case where the initial sample data D_sample_initial consists of {_known}_3, the standard sample data D_standard_3 of the initial sample data D_sample_initial shown in FIG. 27 is replaced with the subsets d1 to d8 in turn, thereby generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 25 in turn.

[0303] Referring to FIG. 29, the validation set is made up of eight subsets V 1’ ~V 8’ Consists of the subset V 1’ ~V 8’ Since M'=8, each of the scan response delays RTTs 1’ ~RTT 8’ And 8 communication characteristics CMbest 1’ ~CMbest 8’ Therefore, the subset V 1’ ~V 8’ In each of the above, the vertical direction is divided into eight scan response delays RTT corresponding to eight terminal devices. 1’ ~RTT 8’ And 8 communication characteristics CMbest 1’ ~CMbest 8’ This indicates that:

[0304] Also, the subset V 1’ ~V 8’ The reason why the number of subsets d1 to d8 is eight is the same as the reason why the number of subsets d1 to d8 is eight.

[0305] Furthermore, the subset V 1’ ~V 8’ When the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the scan response delays {RTT 1’ _known}_1~{RTT M’ _known}_1 part and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 and the standard sample data D_similarity is the standard sample data D_standard_2, the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 part and communication characteristics {CMbest 1’ _known}_2~{CMbest M’_known}_2 and the standard sample data D_similarity is the standard sample data D_standard_3, the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 part and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ It consists of part of _known}_3.

[0306] 30, the test data is made up of four subsets TD1 to TD4. Since M′=8 in each of the subsets TD1 to TD4, eight scan response delay RTTs are generated. 1’ ~RTT 8’ And 8 communication characteristics CMbest 1’ ~CMbest 8’ Therefore, in each of the subsets TD1 to TD4, the vertical direction is made up of eight scan response delay RTTs corresponding to eight terminal devices. 1’ ~RTT 8’ And 8 communication characteristics CMbest 1’ ~CMbest 8’ This indicates that:

[0307] The number of subsets TD1 to TD4 is four because the number of samples in the standard sample data D_similarity is one IP address Add m’ This is because there are 20 samples for each, and the test data consists of 20% (20×0.2=4) of the number of samples (=20) of the standard sample data D_similarity.

[0308] Furthermore, when the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the subsets TD1 to TD4 has a scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 part and communication characteristics {CMbest 1’ _known}_1~{CMbestM’ _known}_1 and the standard sample data D_similarity is the standard sample data D_standard_2, the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 part and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 and the standard sample data D_similarity is the standard sample data D_standard_3, the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 part and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ It consists of part of _known}_3.

[0309] The estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U. When training the estimation algorithm ALG using the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, the estimation means 16 sequentially generates tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) including subsets d1 to d8, respectively, by the method described in Fig. 25 and Fig. 28, and trains the estimation algorithm ALG by repeatedly estimating the results of the network scan by the estimation algorithm ALG using the generated tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8).

[0310] The estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial, and then estimates the results of the network scan by the estimation algorithm ALG using the validation set, and generates a confusion matrix CFM based on the estimation results of the network scan.

[0311] In addition, the estimation means 16 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, estimates the results of the network scan by the estimation algorithm ALG using the validation set, and generates a confusion matrix nCFM based on the estimated results of the network scan. The estimation means 16 executes this operation for all of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U.

[0312] Fig. 31 is a conceptual diagram of a confusion matrix. Since the known estimated results when estimating the results of a network scan are three types, namely, "normal", "poor communication quality", and "network congestion", Fig. 31 shows a confusion matrix for the three known estimated results, namely, "normal", "poor communication quality", and "network congestion".

[0313] 31, TP indicates the percentage of cases where the occurrence of an assumed event (occurrence of an assumed event) was correctly estimated when the assumed event occurred, TN indicates the percentage of cases where the occurrence of an assumed event (non-occurrence of an assumed event) was correctly estimated when the assumed event did not actually occur, FN indicates the percentage of cases where the occurrence of an assumed event (occurrence of an assumed event) was incorrectly estimated as another event when the assumed event occurred, and FP indicates the percentage of cases where the assumed event was incorrectly estimated to have occurred when the assumed event did not actually occur. Also, in (a), (b), and (c) of FIG. 31, "Normal", "Insufficient communication quality", and "Network congestion" in the column direction (vertical direction) represent events that actually occurred, and "Normal", "Insufficient communication quality", and "Network congestion" in the row direction (horizontal direction) represent estimated events. Furthermore, (a) of FIG. 31 shows the confusion matrix when the result of the network scan is “normal”, (b) of FIG. 31 shows the confusion matrix when the result of the network scan is “inadequate communication quality”, and (c) of FIG. 31 shows the confusion matrix when the result of the network scan is “network congestion”.

[0314] In (a) of Figure 31, the TP at the intersection of the "Normal" row and the "Normal" column indicates the percentage of times when an assumed "normal" event (the occurrence of an assumed "normal" event) is correctly estimated as a "normal" event, the FN at the intersection of the "Normal" row and the "Poor communication quality" column indicates the percentage of times when an assumed "normal" event is incorrectly estimated as a different "poor communication quality" event, and the FN at the intersection of the "Normal" row and the "Network congestion" column indicates the percentage of times when an assumed "normal" event is incorrectly estimated as a different "network congestion" event.

[0315] In addition, the FP at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases in which an "inadequate communication quality" event, which is not an assumed "normal" event, was incorrectly assumed to have occurred when an "inadequate communication quality" event occurred (i.e., the assumed "normal" event did not actually occur), and the TN at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases in which an "inadequate communication quality" event, which is not an assumed "normal" event, was incorrectly assumed to have occurred (i.e., the assumed "normal" event did not actually occur, and the TN at the intersection of the row of "inadequate communication quality" and the column of "network congestion" indicates the percentage of cases in which an "inadequate communication quality" event, which is not an assumed "normal" event, was incorrectly assumed to have occurred (i.e., the assumed "normal" event did not actually occur, and the

[0316] Furthermore, the FP at the intersection of the row of "Network Congestion" and the column of "Normal" indicates the percentage of cases where a "network congestion" event occurred that was not an assumed "normal" event (i.e., the assumed "normal" event did not actually occur) and the percentage of cases where an "inadequate communication quality" event occurred when a "network congestion" event occurred that was not an assumed "normal" event (i.e., the percentage of cases where an "inadequate communication quality" event occurred when a "network congestion" event occurred that was not an assumed "normal" event (i.e., the percentage of cases where an "intended "normal" event did not occur) occurred) and the TN at the intersection of the row of "Network Congestion" and the column of "Network Congestion" indicates the percentage of cases where a "network congestion" event occurred that was not an assumed "normal" event (i.e., the percentage of cases where an "intended "normal" event did not actually occur) occurred).

[0317] In FIG. 31(b), the TN at the intersection of the “Normal” row and the “Normal” column indicates the proportion of cases where a “normal” event other than the assumed “inadequate communication quality” occurred and it was estimated that a “normal” event had occurred (i.e., when the assumed “inadequate communication quality” event did not actually occur, it was estimated correctly that the assumed “inadequate communication quality” event had not occurred). The FP at the intersection of the “Normal” row and the “inadequate communication quality” column indicates the proportion of cases where a “normal” event other than the assumed “inadequate communication quality” occurred (i.e., when the assumed “inadequate communication quality” event did not actually occur, it was estimated correctly that the assumed “inadequate communication quality” event had not occurred). The TN at the intersection of the "Normal" row and the "Network Congestion" column indicates the percentage of cases where a "normal" event occurred that was not the assumed "inadequate communication quality" and a "network congestion" event occurred (i.e., cases where the assumed "inadequate communication quality" event did not actually occur and a correct assumption was made that the assumed "inadequate communication quality" event did not occur).

[0318] In addition, the FN at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases where the occurrence of the assumed "inadequate communication quality" event was incorrectly estimated to be a different "normal" event when the assumed "inadequate communication quality" event occurred, the TP at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases where the occurrence of the assumed "inadequate communication quality" event was correctly estimated to be a "inadequate communication quality" event when the assumed "inadequate communication quality" event occurred, and the FN at the intersection of the row of "inadequate communication quality" and the column of "network congestion" indicates the percentage of cases where the occurrence of the assumed "inadequate communication quality" event was incorrectly estimated to be a different "network congestion" event when the assumed "inadequate communication quality" event occurred.

[0319] Furthermore, the TN at the intersection of the row of "Network Congestion" and the column of "Normal" indicates the percentage of cases where a "normal" event was estimated to have occurred when a "network congestion" event occurred that was not the assumed "inadequate communication quality" (i.e., when the assumed "inadequate communication quality" event did not actually occur, the percentage of cases where the event was correctly estimated (that the assumed "inadequate communication quality" event did not occur)), the FP at the intersection of the row of "Network Congestion" and the column of "Inadequate Communication Quality" indicates the percentage of cases where a "network congestion" event occurred that was not the assumed "inadequate communication quality" and the percentage of cases where the event was correctly estimated (that the assumed "inadequate communication quality" event did not occur)) when a "network congestion" event occurred that was not the assumed "inadequate communication quality"

[0320] In (c) of Figure 31, the TN at the intersection of the "Normal" row and the "Normal" column indicates the percentage of cases where a "normal" event that is not the assumed "network congestion" occurred and it was estimated that a "normal" event had occurred (i.e., when the assumed "network congestion" event did not actually occur and it was estimated correctly that the "network congestion" event did not occur), while the TN at the intersection of the "Normal" row and the "Poor Communication Quality" column indicates the percentage of cases where a "normal" event that is not the assumed "network congestion" occurred and it was estimated that an "poor communication quality" event had occurred (i.e., when the assumed "network congestion" event did not actually occur and it was estimated correctly that the assumed "network congestion" event did not occur), while the FP at the intersection of the "Normal" row and the "Network Congestion" column indicates the percentage of cases where a "normal" event that is not the assumed "network congestion" occurred and it was estimated incorrectly that the assumed "network congestion" event had occurred (i.e., when the assumed "network congestion" event did not actually occur and it was estimated correctly that the assumed "network congestion" event did not occur).

[0321] In addition, the TN at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases where a "normal" event was estimated to have occurred when an event of "inadequate communication quality" other than the assumed "network congestion" occurred (i.e., when the assumed event of "network congestion" did not actually occur, the percentage of cases where the event was correctly estimated (that the assumed event of "network congestion" did not occur)), while the TN at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases where a "inadequate communication quality" event other than the assumed "network congestion" occurred (i.e., when the assumed event of "network congestion" did not actually occur, the percentage of cases where the event was correctly estimated (that the assumed event of "network congestion" did not occur)), while the FP at the intersection of the row of "inadequate communication quality" and the column of "network congestion" indicates the percentage of cases where a "inadequate communication quality" event other than the assumed "network congestion" occurred (i.e., when the assumed event of "network congestion" did not actually occur), the percentage of cases where the incorrect estimation was made that the assumed event of "network congestion" occurred.

[0322] Furthermore, the FN at the intersection of the row of "Network Congestion" and the column of "Normal" indicates the percentage of cases where the occurrence of the assumed "network congestion" event was mistakenly estimated to be another "normal" event when the assumed "network congestion" event occurred, the FN at the intersection of the row of "Network Congestion" and the column of "Poor Communication Quality" indicates the percentage of cases where the occurrence of the assumed "network congestion" event was mistakenly estimated to be another "poor communication quality" event when the assumed "network congestion" event occurred, and the TP at the intersection of the row of "Network Congestion" and the column of "Network Congestion" indicates the percentage of cases where the occurrence of the assumed "network congestion" event was correctly estimated to be a "network congestion" event when the assumed "network congestion" event occurred.

[0323] A method for generating a confusion matrix will be described. Fig. 32 is a diagram for explaining a method for generating a confusion matrix. In Fig. 32, the method for generating a confusion matrix will be described by taking as an example a case where a confusion matrix is ​​generated when the result of the network scan shown in Fig. 31(a) is "normal".

[0324] Referring to FIG. 32, since the result of the network scan is "normal", in the column direction, "normal" is an expected event, while "poor communication quality" and "network congestion" are unexpected events.

[0325] The estimation means 16 trains the estimation algorithm ALG using old training data consisting of the initial sample data D_sample_initial, and then uses the validation set (= the subset V shown in FIG. 29) 1’ ~V 8’ The result of the network scan is estimated by the estimation algorithm ALG using , and based on the estimated result of the network scan, the process is performed to count A, which is the number of correctly estimated expected events (=“normal”) when an expected event (=“normal”) occurs, B, which is the number of incorrectly estimated expected events (=“normal”) as unexpected events (=“poor communication quality”) when an expected event (=“normal”) occurs, and C, which is the number of incorrectly estimated expected events (=“normal”) as unexpected events (=“network congestion”) when an expected event (=“normal”) occurs, using the subset V shown in FIG. 1’ ~V 8’ This is done for all lines of .

[0326] Subset V shown in Figure 29 1’ ~V 8’ One row is the scan response delay RTT m’ and communication characteristics CMbest m’ Pair with {RTT m’ ,CMbest m’} is 8 pairs, so if the expected event (="normal") is correctly estimated, A=8, B=C=0 will be obtained. As a result, the counts of A, B, and C are stored in the subset V1’ When executed for all rows of ∼V8', if the expected event (=“normal”) is correctly estimated in every subset, we get A=64, B=C=0.

[0327] The estimation means 16 estimates the validation set (= the subset V shown in FIG. 29 ). 1’ ~V 8’ The process of estimating the results of a network scan using the estimation algorithm ALG is shown in FIG. 1’ ~V 8’ After executing this for all rows of and counting A, B, and C, the counted A, B, and C are assigned to the confusion matrix table as shown in FIG. 32.

[0328] Then, the estimation means 16 takes A as OE, B+C as OIE and A+B+C as ECE.

[0329] Here, OE is the total number of events that occurred as expected but were correctly estimated, ECE is the total number of events that occurred as expected, and OIE is the total number of events that occurred as expected but were incorrectly estimated as unexpected.

[0330] Then, the estimation means 16 calculates TP and FN based on the acquired OE, OIE, and ECE using the following equations.

[0331]

number

[0332] By calculating TP and FN based on OE, OIE, and ECE using formula (14), the TP and FN in the first row of the confusion matrix CFM shown in Fig. 31(a) are obtained. In this case, as shown in Fig. 31(a), FN is obtained as the sum of two elements, "inadequate communication quality" and "network congestion", in the confusion matrix. In this way, TP and FN are obtained based on the estimation results of the network scan.

[0333] Thereafter, the estimation means 16 calculates FP and TN based on the calculated TP and FN by the following equations.

[0334]

number

[0335] The TP at the intersection of the "Normal" row and the "Normal" column shown in (a) of Figure 31 is calculated by substituting the total number OE of correctly estimated "Normal" events that occurred as expected and the total number ECE of "Normal" events that occurred as expected into formula (14A).Then, the FP at the intersection of the "Insufficient communication quality" row and the "Normal" column shown in (a) of Figure 31 and the FP at the intersection of the "Network congestion" row and the "Normal" column shown in (a) of Figure 31 are calculated by substituting the TP calculated by formula (14A) into formula (15A).

[0336] Furthermore, FN (=FN1) at the intersection of the "normal" row and the "poor communication quality" column shown in (a) of Figure 31 is calculated by substituting the total number OIE of "normal" events that occurred as expected but were erroneously estimated as "poor communication quality" events that were not expected, and the total number ECE of "normal" events that occurred as expected, into formula (14B). Then, TN at the intersection of the "poor communication quality" row and the "poor communication quality" column shown in (a) of Figure 31 and TN at the intersection of the "network congestion" row and the "poor communication quality" column shown in (a) of Figure 31 are calculated by substituting FN (=FN1) calculated by formula (14B) into formula (15B).

[0337] Furthermore, FN (=FN2) at the intersection of the "normal" row and the "network congestion" column shown in (a) of Figure 31 is calculated by substituting the total number OIE of "normal" events that occurred as expected but were erroneously estimated as "network congestion" events that were not expected, and the total number ECE of "normal" events that occurred as expected, into formula (14B). Then, TN at the intersection of the "poor communication quality" row and the "network congestion" column shown in (a) of Figure 31 and TN at the intersection of the "network congestion" row and the "network congestion" column are calculated by substituting FN (=FN2) calculated by formula (14B) into formula (15B).

[0338] After calculating TP, FN, TN, and FP by the above-mentioned method, the estimation means 16 generates a confusion matrix CFM shown in FIG. 31(a) that is made up of the calculated TP, FN, TN, and FP.

[0339] The estimation means 16 generates the confusion matrix CFM shown in (b) and (c) of Fig. 31 in the same manner. In this case, since the result of the network scan is "inadequate communication quality" in the confusion matrix CFM shown in (b) of Fig. 31, "inadequate communication quality" is an expected event in the column direction, and "normal" and "network congestion" are unexpected events. As a result, by the above-mentioned method, TP and FN in the second row of the confusion matrix CFM shown in (b) of Fig. 31 are calculated based on OE, OIE, and ECE, and then FP and TN in the first and third rows are calculated based on TP and FN.

[0340] In addition, in the confusion matrix CFM shown in (c) of Fig. 31, since the result of the network scan is "network congestion", in the column direction, "network congestion" is an expected event, and "normal" and "poor communication quality" are unexpected events. As a result, by the above-mentioned method, TP and FN in the third row of the confusion matrix CFM shown in (c) of Fig. 31 are calculated based on OE, OIE, and ECE, and then FP and TN in the first and second rows are calculated based on TP and FN.

[0341] In this way, the estimation means 16 acquires TP and FN from the estimation result of the network scan, calculates FP and TN based on the acquired TP and FN, and generates a confusion matrix CFM consisting of TP, FN, FP, and TN. The estimation means 16 also calculates FN, FP, and TN by adding up multiple elements in the confusion matrix.

[0342] The method of generating this confusion matrix CFM differs from a general method of generating a confusion matrix in which TP, FN, FP, and TN are obtained based on the observation of a subject for which a confusion matrix is ​​to be generated, and a confusion matrix is ​​generated.

[0343] As described above, the estimation means 16 trains the estimation algorithm ALG using old training data consisting of the initial sample data D_sample_initial, and then uses the validation set (= the subset V shown in FIG. 29 ) 1’ ~V 8’ When the results of the network scan are estimated using the estimation algorithm ALG using all rows of , a confusion matrix CFM is generated.

[0344] In addition, the estimation means 16 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, and then similarly calculates a validation set (= a subset V shown in FIG. 29 ). 1’ ~V 8’ ) is used to estimate the results of the network scan using the estimation algorithm ALG, and the confusion matrix nCFM when the results of the network scan are estimated is generated in the same manner as the confusion matrix CFM.

[0345] Furthermore, after generating the confusion matrix nCFM, the estimation means 16 estimates the result of the network scan by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD1 to TD4 shown in FIG. 30), and generates the confusion matrix tCFM when the result of the network scan is estimated by the same method as the confusion matrix CFM.

[0346] In this case, the estimation means 16 estimates the results of the network scan using the test data (= consisting of one row of the subsets TD1 to TD4 shown in Figure 30) with the estimation algorithm ALG, and performs the process of counting A, B, and C based on the estimated network scan results for all rows of the subsets TD1 to TD4 shown in Figure 30.

[0347] One row of the subsets TD1 to TD4 shown in FIG. 30 is the scan response delay RTT m’ and communication characteristics CMbest m’ Pair with {RTT m’ ,CMbest m’Since there are four sets of}, if the expected event (="normal") is correctly estimated, A=4 and B=C=0 are obtained. As a result, when counting A, B, and C for all rows of subsets TD1 to TD4 shown in FIG. 30, if the expected event (="normal") is correctly estimated in all subsets, A=32 and B=C=0 are obtained. The same applies when the expected event is "poor communication quality" or "network congestion."

[0348] When the estimation means 16 generates the confusion matrices CFM, nCFM, tCFM, the estimation means 16 outputs the generated confusion matrices CFM, nCFM, tCFM to the calculation means 15.

[0349] The calculation means 15 receives the confusion matrices CFM, nCFM, and tCFM from the estimation means 16. Then, the calculation means 15 calculates Precision, which is the ratio of the proportion actually classified as TP among the proportion classified as TP or FP, and Recall, which is the ratio of the proportion actually classified as TP among the proportion classified as TP or FN, using the elements TP, TN, FP, and FN of the confusion matrices CFM, nCFM, and tCFM.

[0350] Precision is calculated by equation (16), and Recall is calculated by equation (17).

[0351]

number

[0352]

number

[0353] Then, after calculating Precision and Recall, the calculation means 15 calculates the F1 score F1 by substituting the calculated Precision and Recall into the following formula.

[0354]

number

[0355] The calculation means 15 calculates an F1 score F1 by equations (16) to (18) based on the TP, TN, FP, and FN of the confusion matrix CFM. The calculation means 15 also calculates an F1 score nF1_u by equations (16) to (18) based on the TP, TN, FP, and FN of the confusion matrix nCFM. The calculation means 15 also calculates an F1 score tF1 by equations (16) to (18) based on the TP, TN, FP, and FN of the confusion matrix tCFM. Then, the calculation means 15 outputs the F1 scores F1, nF1_u, and tF1 to the estimation means 16.

[0356] The F1 score F1 is calculated by training the estimation algorithm ALG using the initial sample data D_sample_initial and testing it on the validation set (= the subset V shown in Figure 29). 1’ ~V 8’ It is an evaluation index EVD that represents the accuracy of the estimation result when the network scan result is estimated by the estimation algorithm ALG (the estimation algorithm ALG after training) using all rows of .

[0357] In addition, the F1 score nF1_u is calculated by training the estimation algorithm ALG using the tentative sample data D_sample_tentative_u and calculating the F1 score nF1_u using the validation set (= the subset V shown in Figure 29). 1’ ~V 8’ It consists of all rows of . ) is the evaluation index nEVD_u, which represents the accuracy of the estimation result when the network scan result is estimated by the estimation algorithm ALG (the estimation algorithm ALG after training).

[0358] Furthermore, the F1 score tF1 is an evaluation index tEVD that represents the accuracy of the estimation result when the network scan result is estimated by the estimation algorithm ALG (the estimation algorithm ALG after training) using the test data (= consisting of all rows of the subsets TD1 to TD4 shown in Figure 30).

[0359] In the embodiment of the present invention, Matthews correlation coefficients MCC, nMCC_u, and tMCC may be used as evaluation indices EVD, nEVD_u, and tEVD, respectively, instead of F1 scores F1, nF1_u, and tF1. In this case, the calculation means 15 calculates the Matthews correlation coefficient MCC by equation (19) based on TP, TN, FP, and FN of the confusion matrix CFM, calculates the Matthews correlation coefficient nMCC_u by equation (19) based on TP, TN, FP, and FN of the confusion matrix nCFM, and calculates the Matthews correlation coefficient tMCC by equation (19) based on TP, TN, FP, and FN of the confusion matrix tCFM.

[0360]

number

[0361] The Matthews correlation coefficient MCC calculated by equation (19) is “+1” when the best training data is used, “-1” when the worst training data is used, and “0” when neutral (neither “best” nor “worst”) training data is used.

[0362] After calculating the Matthews correlation coefficients MCC, nMCC_u, and tMCC, the calculation means 15 outputs the Matthews correlation coefficient MCC to the estimation means 16 as an evaluation index EVD, outputs the Matthews correlation coefficient nMCC_u to the estimation means 16 as an evaluation index nEVD_u, and outputs the Matthews correlation coefficient tMCC to the estimation means 16 as an evaluation index tEVD.

[0363] Upon receiving the evaluation index nEVD_u consisting of the F1 score nF1_1 and the evaluation index EVD consisting of the F1 score F1 from the calculation means 15, the estimation means 16 determines whether or not the F1 score nF1_1 is greater than the F1 score F1.

[0364] When the estimation means 16 determines that the F1 score nF1_1 is greater than the F1 score F1, it sets the F1 score nF1_1 to the F1 score F1, and sets tentative sample data D_sample_tentative_1 including the subset d1 at the time when the F1 score nF1_1 was obtained as training data. Then, the estimation means 16 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 16 trains the estimation algorithm ALG using the next subset d2, and sets a validation set (=subset V 1’ ~V 8’ The estimation means 16 estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using all of the above. The estimation means 16 generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_2 to the calculation means 15.

[0365] On the other hand, when the estimation means 16 determines that the F1 score nF1_1 is not greater than the F1 score F1, it does not set the F1 score nF1_1 to the F1 score F1, and trains the estimation algorithm ALG using tentative sample data D_sample_tentative_2 including the next subset d2, and sets the F1 score nF1_1 to the F1 score F1. 1’ ~V 8’ ) is used to estimate the result of the network scan by an estimation algorithm ALG (trained estimation algorithm ALG), and a confusion matrix nCFM_2 is generated when the result of the network scan is estimated, and the confusion matrix nCFM_2 is output to the calculation means 15.

[0366] Thereafter, when the estimation means 16 receives the evaluation index nEVD_u consisting of the F1 score nF1_2 from the calculation means 15, the estimation means 16 determines whether or not the F1 score nF1_2 is greater than the F1 score F1.

[0367] When the estimation means 16 determines that the F1 score nF1_2 is greater than the F1 score F1, it sets the F1 score nF1_2 to the F1 score F1, and sets tentative sample data D_sample_tentative_2 including the subset d2 when the F1 score nF1_2 was obtained as training data. Then, the estimation means 16 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 16 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d3, and sets a validation set (=subset V 1’ ~V 8’ The estimation means 16 estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using all of the above. The estimation means 16 generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

[0368] On the other hand, when the estimation means 16 determines that the F1 score nF1_2 is not greater than the F1 score F1, it does not set the F1 score nF1_2 to the F1 score F1, and trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d3, and sets the F1 score nF1_2 to the F1 score F1. 1’ ~V 8’ The estimation means 16 estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using all of the above. The estimation means 16 generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

[0369] Thereafter, the estimation means 16 estimates the subsets d3 to d4 using the F1 scores F1,nF1_u. U The above-mentioned operations are repeatedly performed for all of the above.

[0370] In addition, when the estimation means 16 receives an evaluation index EVD consisting of the Matthews correlation coefficient MCC and an evaluation index nEVD_1 consisting of the Matthews correlation coefficient nMCC_1 from the calculation means 15, it determines whether the Matthews correlation coefficient nMCC_1 is greater than the Matthews correlation coefficient MCC.

[0371] When the estimation means 16 determines that the Matthews correlation coefficient nMCC_1 is greater than the Matthews correlation coefficient MCC, it sets the Matthews correlation coefficient nMCC_1 to the Matthews correlation coefficient MCC, and sets tentative sample data D_sample_tentative_1 including the subset d1 when the Matthews correlation coefficient nMCC_1 was obtained as training data. Then, the estimation means 16 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 16 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_2 including the next subset d2, and sets a validation set (=subset V 1’ ~V 8’ The estimation means 16 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 16 generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_2 to the calculation means 15.

[0372] On the other hand, when the estimation means 16 determines that the Matthews correlation coefficient nMCC_1 is not greater than the Matthews correlation coefficient MCC, it does not set the Matthews correlation coefficient nMCC_1 to the Matthews correlation coefficient MCC, and trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_2 including the next subset d2, and sets the validation set (=subset V 1’ ~V 8’ The estimation means 16 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 16 generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_2 to the calculation means 15.

[0373] Thereafter, when the estimation means 16 receives the evaluation index nEVD_2 consisting of the Matthews correlation coefficient nMCC_2 from the calculation means 15, it determines whether or not the Matthews correlation coefficient nMCC_2 is greater than the Matthews correlation coefficient MCC.

[0374] When the estimation means 16 determines that the Matthews correlation coefficient nMCC_2 is greater than the Matthews correlation coefficient MCC, it sets the Matthews correlation coefficient nMCC_2 to the Matthews correlation coefficient MCC, and sets tentative sample data D_sample_tentative_2 including the subset d2 when the Matthews correlation coefficient nMCC_2 is obtained as training data. Then, the estimation means 16 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 16 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d3, and sets a validation set (=subset V 1’ ~V 8’ The estimation means 16 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 16 generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

[0375] On the other hand, when the estimation means 16 determines that the Matthews correlation coefficient nMCC_2 is not greater than the Matthews correlation coefficient MCC, it does not set the Matthews correlation coefficient nMCC_2 to the Matthews correlation coefficient MCC, and trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_3 including the next subset d3, and sets the validation set (=subset V 1’ ~V 8’ The estimation means 16 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 16 generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

[0376] Thereafter, the estimation means 16 estimates the subsets d3 to d6 using the Matthews correlation coefficient MCC,nMCC_u. U The above-mentioned operations are repeatedly performed for all of the above.

[0377] [Estimation of network scan results] In the training of the estimation algorithm ALG described above, the estimation means 16 estimates the subsets d1 to d2 generated from the initial sample data D_sample_initial and the standard sample data D_similarity. U Using tentative sample data D_sample_tentative_1 to D_sample_tentative_U each including the above, the result of the network scan is estimated by the following method.

[0378] In the description of the method for estimating the results of a network scan, the scan response delay {RTT 1’ _known}_z~{RTT M’ _known}_z to “Scan Response Delay RTT m” and the communication characteristic CMbest included in each of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U is m _unknown, and the communication characteristics {CMbest1'_known}_z~{CMbest M’ _known}_z is expressed as "communication characteristic CMbest".

[0379] The estimation means 16 estimates the scan response delay RTT m and communication characteristic CMbest from the extraction means 14. Then, the estimation means 16 classifies the terminal devices having the communication characteristic CMbest into G (G is an integer of 3 or more) groups CLS1 to CLS2 using an LDA (Linear Discriminant Analysis) algorithm. G Classify into:

[0380] The LDA algorithm divides terminal devices having communication characteristics CMbest into G groups CLS1 to CLS2 so as to maximize the ratio of variance between groups and variance within a group. G Therefore, the groups CLS1 to CLS G This ensures maximum isolation between the

[0381] The LDA algorithm spheres the data (terminal devices with communication characteristics CMbest) based on the common covariance and classifies the data (terminal devices with communication characteristics CMbest) into the closest group centroid in the transformed space.

[0382] To achieve this, the separability between different groups (also called "inter-group covariance"), which is the distance between the means of different groups, is calculated. Then, the distance between the mean and the sample of each group (also called "intra-group covariance") is calculated. This allows a low-dimensional space to be constructed that maximizes the inter-group covariance and minimizes the intra-group covariance. Then, in the constructed low-dimensional space, data (terminal device having communication characteristic CMbest) is classified into the closest group centroid. The LDA algorithm is described in "T. Hastie, R. Tibshirani, and JH Friedman, "The elements of statistical learning: data mining, inference, and prediction," 2nd ed. Chapter 4. pp.106-118, New York: Springer, 2009," and is well known. Therefore, a detailed classification method using the LDA algorithm will be omitted.

[0383] Also, groups CLS1 to CLS GThe number G of the groups CLS1 to CLS2 is set to the same number as the number of causes of the network scan results caused by the communication channel with the terminal device among the estimated results of the network scan. That is, the three results of "normal", "poor communication quality" and "network congestion" are the causes caused by the communication channel with the terminal device, so the groups CLS1 to CLS2 are set to the same number as the number of causes of the network scan results caused by the communication channel with the terminal device among the estimated results of the network scan. G The number G of terminal devices having communication characteristics CMbest is set to "3". As a result, the estimation means 16 classifies the terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the LDA algorithm. The group CLS1 is a "normal" group, the group CLS2 is a "poor communication quality" group, and the group CLS3 is a "network congestion" group.

[0384] When classifying terminal devices having communication characteristics CMbest into groups CLS1 to CLS3, the estimation means 16 learns an LDA algorithm using scan data when the network scan is "network congested" (communication characteristics CMbest when "network is congested"), and classifies terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm. This makes it possible to most accurately estimate that "network congestion" is the cause of the failure of the network scan.

[0385] The estimation means 16 uses Model1, Model2 and Model3 for estimating the results of the network scan. Model1 handles groups CLS1 and CLS2, Model2 handles groups CLS1 and CLS3, and Model3 handles groups CLS2 and CLS3.

[0386] Then, when using Model 1, the estimation means 16 calculates a D score D_1 that distinguishes between groups CLS1 and CLS2, when using Model 2, it calculates a D score D_2 that distinguishes between groups CLS1 and CLS3, and when using Model 3, it calculates a D score D_3 that distinguishes between groups CLS2 and CLS3.

[0387] When the D scores D_1 to D_3 are calculated, the discriminant function δ is expressed by the following formula: pq (x) is used.

[0388]

number

[0389] In equation (20A), x is the scan response delay RTT m and the communication characteristic CMbest are substituted. Then, x is the scan response delay RTT m and communication characteristics CMbest (best training data), or scan response delay RTT when the results of network scans are unknown m and the communication characteristic CMbest. Also, μ g When calculating the discriminant function for group p, the average μ p When calculating the discriminant function for group q, the average μ q is substituted. Furthermore, the prior average Π g When calculating the discriminant function for group p, the prior average Π p When calculating the discriminant function for group q, the prior average Π of the communication characteristics shown in equation (20F) is q is substituted. The joint covariance matrix C in Eq. (20A) pq is expressed by equation (20B). On the right side of equation (20B), the first term is the covariance matrix in group p, and the second term is the covariance matrix in group q. Then, x mp The communication characteristics (scan response delay RTT m and the communication characteristic CMbest) is substituted into x mq The communication characteristics (scan response delay RTT mand the communication characteristic CMbest) is substituted.

[0390] In formulae (20B), (20C), and (20D), M p is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest), and M q is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest), and m p is one communication characteristic (scan response delay RTT m and the communication characteristic CMbest), m q is one communication characteristic (scan response delay RTT m In formulas (20E) and (20F), M is a vector of communication characteristics (scan response delay RTT m and communication characteristic CMbestt). Equation (20C) is the total number of communication characteristics (scan response delay RTT m and the communication characteristic CMbest) p Equation (20D) expresses the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) q Equation (20E) expresses the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) p Equation (20F) expresses the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) q Represents.

[0391] Equation (20B) is the joint covariance matrix C, which is the joint covariance matrix of two groups (two groups CLS1, CLS2; two groups CLS1, CLS3; two groups CLS2, CLS3). pq The combined covariance matrix C is the sum of the two covariance matrices in the two groups CLS1 and CLS2. 12 In the formula (20A), p represents the group CLS1, and q represents the group CLS2. In addition, the merged covariance matrix C 13 In the formula (20A), p represents the group CLS1, and q represents the group CLS3. Furthermore, the two covariance matrices in the two groups CLS2 and CLS3 are combined to obtain a combined covariance matrix C 23 When calculating, in formula (20A), p represents group CLS2 and q represents group CLS3.

[0392] Joint covariance matrix C in Model 1 12 is expressed by the following equation:

[0393]

number

[0394] In formula (21), M1 is M in formula (20B). p M2 corresponds to M in formula (20B). q m1 corresponds to m in formula (20B). p m2 corresponds to m in formula (20B) q Also, μ1 is equivalent to M1 in formula (20C). p The communication characteristics classified into group CLS1 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mp μ2 is calculated by substituting M2 into M in equation (20D). q The communication characteristics classified into group CLS2 (scan response delay RTT mand the communication characteristic CMbest) is expressed as x mq The joint covariance matrix C 12 is represented by the sum of the covariance matrix in group CLS1 and the covariance matrix in group CLS2.

[0395] Joint covariance matrix C in Model 2 13 is expressed by the following equation:

[0396]

number

[0397] In formula (22), M1 is M in formula (20B). p M3 corresponds to M in formula (20B). q m1 corresponds to m in formula (20B). p m3 corresponds to m in formula (20B) q Also, μ1 is equivalent to M1 in formula (20C). p The communication characteristics classified into group CLS1 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mp μ3 is calculated by substituting M3 into M in equation (20D). q The communication characteristics classified as group CLS3 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mq The joint covariance matrix C 13 is represented by the sum of the covariance matrix in group CLS1 and the covariance matrix in group CLS3.

[0398] Joint covariance matrix C in Model 3 23 is expressed by the following equation:

[0399]

number

[0400] In formula (23), M2 is M in formula (20B). p M3 corresponds to M in formula (20B). q m2 corresponds to m in formula (20B) p m3 corresponds to m in formula (20B) q Also, μ2 is equivalent to M2 in formula (20C). p The communication characteristics classified into group CLS2 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mp μ3 is calculated by substituting M3 into M in equation (20D). q The communication characteristics classified as group CLS3 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mq The joint covariance matrix C 23 is represented by the sum of the covariance matrix in group CLS2 and the covariance matrix in group CLS3.

[0401] When calculating the D score D_1 in Model 1, the discriminant function f 1 and the discriminant function f in group CLS2 2 is expressed by the following equation:

[0402]

number

[0403] In equation (24A), the prior average Π1 is calculated by dividing M1 by M in equation (20E). p and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (20E). Also, in equation (24B), the prior average Π2 is calculated by substituting M2 into M in equation (20F). q and the communication characteristics (scan response delay RTT mand the communication characteristic CMbest) into M in equation (20F). Also, the average μ1 in equation (24A) and the average μ2 in equation (24B) are calculated by the method explained in equation (21).

[0404] In addition, when calculating the D score D_2 in Model 2, the discriminant function f 1 and the discriminant function f in group CLS3 2 is expressed by the following equation:

[0405]

number

[0406] In equation (25A), the prior average Π1 is calculated by dividing M1 by M in equation (20E). p and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (20E). Also, in equation (25B), the prior average Π2 is calculated by substituting M3 into M in equation (20F). q and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (20F). Also, the average μ1 in equation (25A) and the average μ3 in equation (25B) are calculated by the method explained in equation (22).

[0407] Furthermore, when calculating the D score D_3 in Model 3, the discriminant function f 1 and the discriminant function f in group CLS3 2 is expressed by the following equation:

[0408]

number

[0409] In equation (26A), the prior average Π2 is calculated by dividing M2 by M in equation (20E).p and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (20E). In addition, in equation (26B), the prior average Π3 is calculated by substituting M3 into M in equation (20F). q and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (20F). Also, the average μ2 in equation (26A) and the average μ3 in equation (26B) are calculated by the method explained in equation (23).

[0410] The D score D_1 is calculated using the discriminant function f 1 and the discriminant function f 2 Difference from = f 1 -f 2 It is represented by:

[0411] The D score D_2 is calculated using the discriminant function f 1 and the discriminant function f 2 Difference from = f 1 -f 2 It is represented by:

[0412] Furthermore, the D score D_3 is calculated using the discriminant function f 1 and the discriminant function f 2 Difference from = f 1 -f 2 It is represented by:

[0413] Therefore, the D scores D_1 to D_3 can be calculated by using equations (20) to (26).

[0414] The estimation means 16 calculates a D score (any of D_1 to D_3) using any of Model1 to Model3.

[0415] Then, the estimation means 16 judges whether the D score is less than zero (=0). When the estimation means 16 judges that the D score is less than zero (=0), it increments the count value DS_count, which indicates the number of D scores less than zero (=0), by "1".

[0416] On the other hand, when the estimation means 16 determines that the D score is equal to or greater than zero (=0), it holds the count value DS_count unchanged.

[0417] The estimation means 16 executes this process for all terminal devices belonging to one base station.

[0418] After counting the count value DS_count of the D score, the estimation means 16 sets the Model for estimating the result of the network scan to Model2 or Model3 based on the count value DS_count.

[0419] More specifically, the estimation means 16 first sets Model to "1" (i.e., sets it to Model1), and when the count value DS_count is zero (=0) or when the count value DS_count is (N terminal × D_thres), the model is set to “2” (i.e., Model2). terminal is the total number of terminal devices belonging to one base station. Also, D_thres is set to a real number in the range of 0 to 1.

[0420] In addition, the estimation means 16 determines whether the count value DS_count is equal to or greater than zero (=0) and whether the count value DS_count is equal to or greater than (N terminal ×D_thres), the model Model is set to "3" (i.e., set to Model3).

[0421] Then, when the model Model is not "1" (that is, when the model Model is "2" or "3"), the estimation means 16 judges whether the count value DS_count is zero (=0) or not.

[0422] When it is determined that the count value DS_count is zero (=0), the estimation means 16 determines "normal" scan data for the terminal device.

[0423] On the other hand, when it is determined that the count value DS_count is not zero (=0), the estimation means 16 determines whether the count value DS_count is (N terminal ×D_thres) or not.

[0424] And the count value DS_count is (N terminal ×D_thres), the estimation means 16 estimates that the cause of the failure of the network scan for the terminal device having a D score smaller than "0" is "insufficient communication quality."

[0425] On the other hand, if the count value DS_count is (N terminal ×D_thres) (i.e., when it is determined that the count value DS_count is not smaller than (N terminal ×D_thres), the estimation means 16 estimates that the cause of the failure of the network scan for all terminal devices belonging to the base station is "network congestion (i.e., network congested)".

[0426] Furthermore, when it is determined that the count value DS_count is zero (=0) and further that there has been a scan response from all ports of the terminal device, the estimation means 16 estimates that the network scan for the terminal device is “normal”.

[0427] On the other hand, when it is determined that the count value DS_count is zero (=0) and further that a response has been received from some of the ports of the terminal device, the estimation means 16 estimates that the cause of the failure of the network scan for the terminal device is "port closed."

[0428] Furthermore, when it is determined that the count value DS_count is zero (=0) and further that there is no scan response from all ports of the terminal device, it is estimated that the cause of the failure of the network scan for the terminal device is "no network connection."

[0429] The estimation algorithm ALG is an algorithm that estimates the results of a network scan using the method described above.

[0430] [Variable replacement for estimation algorithm] The estimation means 16 generates a validation set (= subset V 1’ ~V U’ ) and the F1 score nF1 when the result of the network scan is estimated by the estimation algorithm ALG using the above-mentioned method is greater than the F1 score F1, or the validation set (= subset V 1’ ~V U’ When the Matthews correlation coefficient nMCC_u is greater than the Matthews correlation coefficient MCC when the network scan result is estimated by the estimation algorithm ALG using the above, the subset d is obtained when the F1 score nF1 is determined to be greater than the F1 score F1. max (Subset d1~d U The tentative sample data D_sample_tentative_max containing either D_sample_tentative_max or D_sample_tentative_max is used to replace the variables of the estimation algorithm ALG, or the subset d where the Matthews correlation coefficient nMCC_max determined to be larger than the Matthews correlation coefficient MCC is obtained. max (Subset d1~d U The variables of the estimation algorithm ALG are replaced with tentative sample data D_sample_tentative_max, which includes either

[0431] As explained in the above formula (20), x in formula (20A) is the scan response delay RTT m and the communication characteristic CMbest. pis the communication characteristic (scan response delay RTT m and the communication characteristic CMbest) p and μ in formula (20D) q is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest) q Furthermore, the prior average Π p is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest), and the prior average Π q is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest).

[0432] Here, the n-size data (data d1 to d2) when the F1 score nF1, which is determined to be greater than the F1 score F1, is obtained. U ) or the subset d for which the Matthews correlation coefficient nMCC_u is finally determined to be greater than the Matthews correlation coefficient MCC. max (Subset d1~d U The tentative sample data D_sample_tentative_max including either of the above) is denoted as "sample data D_sample_optimum".

[0433] The estimation means 16 substitutes the sample data D_sample_optimum into x in the formula (20A) to obtain the scan response delay RTT m and the scan response delay RTT of the communication characteristics CMbest classified into group p m The average of the vector consisting of the communication characteristic CMbest is μ p The scan response delay RTT that constitutes the sample data D_sample_optimum is calculated as m and the scan response delay RTT of the communication characteristics CMbest classified into group qm The average of the vector consisting of the communication characteristic CMbest is μ q The scan response delay RTT that constitutes the sample data D_sample_optimum is calculated as m and the scan response delay RTT of the communication characteristics CMbest classified into group p m The prior average of the vector consisting of and the communication characteristic CMbest is the prior average Π p The scan response delay RTT that constitutes the sample data D_sample_optimum is calculated as m and the scan response delay RTT of the communication characteristics CMbest classified into group q m The prior average of the vector consisting of and the communication characteristic CMbest is the prior average Π q The variables of the estimation algorithm ALG are replaced with the sample data D_sample_optimum by calculating as follows:

[0434] This can improve the accuracy of estimation when the estimation algorithm ALG estimates the results of a network scan.

[0435] [Verification using test data] After the model variables are replaced by the optimal training data, the estimation means 16 uses the test data (=consisting of all of the subsets TD1 to TD4) to estimate the results of the network scan using the model in which the model variables have been replaced (estimation algorithm ALG), and generates a confusion matrix tCFM based on the estimated results of the network scan and outputs it to the calculation means 15.

[0436] The calculation means 15 receives the confusion matrix tCFM from the estimation means 16, calculates the F1 score tF1 based on the elements TP, TN, FP, and FN of the received confusion matrix tCFM using equations (16) to (18), and outputs the calculated F1 score tF1 to the estimation means 16.

[0437] The estimation means 16 receives the F1 score tF1 from the calculation means 15 and determines whether the F1 score tF1 is equal to or greater than the F1 score F1.

[0438] When the estimation means 16 determines that the F1 score tF1 is equal to or greater than the F1 score F1, it approves the model in which the model variables have been replaced (estimation algorithm ALG) and the training data (optimal training data).

[0439] When it is determined that the F1 score tF1 is not equal to or greater than the F1 score F1, the estimation means 16 does not approve the model (estimation algorithm ALG) and training data (optimum training data) in which the model variables have been replaced.

[0440] In this way, the estimation means 16 uses the test data (=consisting of all of the subsets TD1 to TD4) to estimate the results of a network scan by the model (estimation algorithm ALG) in which the model variables have been replaced, and verifies the model (estimation algorithm ALG) in which the model variables have been replaced and the training data (optimum training data) by confirming that the F1 score tF1 is equal to or greater than the F1 score F1. That is, the estimation means 16 uses the test data (=consisting of all of the subsets TD1 to TD4) to verify that the model (estimation algorithm ALG) in which the model variables have been replaced can more accurately estimate the results of a network scan performed in a communication environment in which the results of the network scan are unknown, compared to the model (estimation algorithm ALG) before the model variables were replaced.

[0441] 33 is a flow chart for explaining the network scan operation. With reference to FIG. 33, when the network scan operation is started, network scanner 11 reads addresses Add1 to Add2 of terminal devices to be the targets of the network scan. M and port number PN r The scan schedule and the scan list including the above are obtained from the scan scheduler 17 (step S1).

[0442] Then, the network scanner 11 sets m=1 (step S2) and r =1 (step S3).

[0443] Thereafter, the network scanner 11 executes a network scan with or without monitoring background traffic (step S4), and obtains a scan response and a waiting time (step S5).

[0444] Then, the network scanner 11 will r =PN r _MAX(Port Number PN r It is then determined whether the value is equal to or larger than the maximum value (step S6).

[0445] In step S6, PN r =PN r If it is determined that the number is not MAX, the network scanner 11 r =PN r +1 is set (step S7). After that, the series of operations proceeds to step S4, and in step S6, PN r =PN r Steps S4 to S7 are repeatedly executed until it is determined that the value is _MAX.

[0446] Then, in step S6, PN r =PN r If it is determined that m is MAX, the network scanner 11 determines whether m=M (step S8).

[0447] When it is determined in step S8 that m is not equal to M, the network scanner 11 sets m to m+1 (step S9). After that, the series of operations proceeds to step S3, and steps S3 to S9 are repeatedly executed until it is determined in step S8 that m is equal to M.

[0448] Then, in step S8, if it is determined that m=M, the network scanner 11 obtains the communication characteristic (scan response delay RTT m The extracted communication characteristics and the scan response are output to the scan analysis manager 12 (step S10).

[0449] The scan analysis manager 12 receives scan responses and communication characteristics from the network scanner 11 and creates or updates a host information database 13 based on the received scan responses and communication characteristics.

[0450] Thereafter, the estimation means 16 trains an estimation algorithm ALG that estimates the result of the network scan, and selects training data (step S11). In this case, the estimation means 16 estimates the result of the network scan using standard sample data D_similarity, and trains the estimation algorithm ALG by evaluating the accuracy of the estimated result. In addition, the estimation means 16 selects sample data (sample data D_sample_optimum) when the accuracy of the estimation result is the highest, as training data.

[0451] In step S11, the scan scheduler 17 receives from the estimation means 16 the estimated results of the network scan when accuracy is highest, creates a re-scan list based on the received estimated results of the network scan (step S12), and outputs the created re-scan list to the network scanner 11.

[0452] The reason for creating a rescan list is that if the result of a network scan is "insufficient communication quality" or "network congestion," the result of the network scan may become "normal" if a network scan is performed at a different scan timing.

[0453] After step S12, the network scanner 11 determines whether the rescan list is empty (step S13).

[0454] When it is determined in step S13 that the rescan list is not empty, the series of operations proceeds to step S2, and steps S2 to S13 are repeatedly executed until it is determined in step S13 that the rescan list is empty.

[0455] Then, if it is determined in step S13 that the rescan list is empty, the network scan operation ends.

[0456] 34 and 35 are first and second flowcharts, respectively, for explaining the detailed operation of step S10 in FIG.

[0457] 34, when it is determined in step S8 of FIG. 33 that m=M, the extraction means 14 extracts a correlation coefficient r ij (Step S100), where i is an argument in the column direction in Figs. 8 to 11, and j is an argument in the row direction in Figs.

[0458] Then, the extraction means 14 sets j=1 (step S101) and sets i=1 (step S102). After that, the extraction means 14 extracts the communication characteristic f i ,f j Correlation coefficient r ij (Step S103), and the extracted correlation coefficient r ij is the threshold Corr r It is determined whether it is greater than (step S104).

[0459] In step S104, the correlation coefficient r ij is the threshold Corr r When it is determined that the communication characteristic f is not larger than i ,fj (Step S105).

[0460] On the other hand, in step S104, the correlation coefficient r ij is the threshold Corr r When it is determined that the communication characteristic f is larger than i ,f j is saved (step S106).

[0461] Then, after step S105 or step S106, the extraction means 14 judges whether i=I (I is the maximum number of communication characteristics in the column direction in FIGS. 8 to 11) or not (step S107).

[0462] When it is determined in step S107 that i is not equal to I, the extraction means 14 sets i to i+1 (step S108). After that, the series of operations proceeds to step S103, and steps S103 to S108 are repeatedly executed until it is determined in step S107 that i is equal to I.

[0463] Then, when it is determined in step S107 that i=I, the extraction means 14 determines whether or not j=J (J is the maximum number of communication characteristics in the row direction in FIGS. 8 to 11) (step S109).

[0464] When it is determined in step S109 that j is not equal to J, the extraction means 14 sets j to j+1 (step S110). After that, the series of operations proceeds to step S102, and steps S102 to S110 are repeatedly executed until it is determined in step S109 that j is equal to J.

[0465] Then, in step S109, when it is determined that j=I, the extraction means 14 extracts a scan response delay RTT based on all the stored communication characteristics by the above-mentioned method, using all the stored communication characteristics as explanatory variables. m A multiple linear regression is calculated using the objective variable (step S111), and the contribution rate R 2Get the.

[0466] Then, the extraction means 14 extracts the contribution rate R 2 is the threshold S r It is determined whether it is smaller than (step S112).

[0467] In step S112, the contribution rate R 2 is the threshold S r If it is determined that the difference is smaller than , the extraction means 14 sets the mode of the estimation algorithm ALG to "uncorrelated" (step S113).

[0468] On the other hand, in step S112, the contribution rate R 2 is the threshold S r If it is determined that the above is the case, the extraction means 14 sets the mode of the estimation algorithm ALG to "correlation" (step S114).

[0469] Then, after step S113 or step S114, the extraction means 14 stores the waiting time and communication characteristics in the network scan and the mode of the estimation algorithm ALG (step S115).

[0470] 35, after step S115 in FIG. 34, the extraction means 14 calculates the threshold value Corr r A correlation coefficient greater than r ij Then, the CPU 11 calculates a plurality of scores Q of a plurality of communication characteristics having the above-mentioned characteristic, and selects K communication characteristics from the plurality of communication characteristics in descending order of score Q (step S116).

[0471] Then, the extraction means 14 executes learning of a classification algorithm (for example, SVM-train) using the teacher data (step S117).

[0472] After that, the extraction means 14 sets k=K (step S118). Subsequently, the extraction means 14 extracts (k×M) sample vectors x 11 ~x kMare classified into two groups GP1 and GP2 (step S119).

[0473] Then, the extraction means 14 calculates k weights w1 to w2 of the k communication characteristics by using the equations (4), (6), and (7). k is calculated (step S120).

[0474] Then, the extraction means 14 extracts k weights w1 to w k Based on this, k rank weights R1 to R k is calculated (step S121).

[0475] Then, the extraction means 14 extracts k rank weights R1 to R k Based on the smallest rank weight R k_min The extraction means 14 then determines the sample of the communication characteristic to which the smallest rank weight R k_min Let the sample of communication characteristics with 11 ~x kM (step S123).

[0476] Then, the extraction means 14 judges whether the subset of communication characteristics is empty or not (step S124).

[0477] When it is determined in step S124 that the subset of communication characteristics is not empty, the extracting means 14 sets k=k-1 (step S125). That is, the extracting means 14 decreases the number of samples in the subset of communication characteristics by "1". After that, the series of operations proceeds to step S119, and steps S119 to S125 are repeatedly executed until it is determined in step S124 that the subset of communication characteristics is empty.

[0478] Then, in step S124, if it is determined that the subset of communication characteristics is empty, the extraction means 14 extracts the last deleted communication characteristic as the communication characteristic CMbest. m(Step S126). Then, the series of operations proceeds to Step S11 in FIG.

[0479] After the first execution of steps S119 to S125, steps S119 to S125 are repeatedly executed until the subset of communication characteristics becomes empty, thereby sequentially deleting samples of communication characteristics that are least important for estimating the results of a network scan. At this time, after deleting one sample of a communication characteristic, the remaining samples are reclassified into two groups GP1 and GP2 using the learned classification algorithm (see step S119). As described above, group GP1 is a group into which communication characteristics that are important for estimating the results of a network scan are classified, and group GP2 is a group into which communication characteristics that are not important for estimating the results of a network scan are classified. Therefore, for each sample, the accuracy of determining whether or not the communication characteristic is important for estimating the results of a network scan is increased, and the least important sample for estimating the results of a network scan can be deleted.

[0480] In addition, the communication characteristic CMbest is calculated according to the flowcharts shown in FIG. 34 and FIG. m The process of detecting constitutes a "detection process."

[0481] In addition, the communication characteristic CMbest is calculated according to the flowcharts shown in FIG. 34 and FIG. m The extraction means 14 for detecting the above constitutes the "detection means."

[0482] 36 to 38 are first to third flow charts, respectively, for explaining the detailed operation of step S11 in FIG.

[0483] Referring to Figure 36, after step S10 in Figure 33, the extraction means 14 detects the standard sample data D_standard_z (z = 1 to Z) used to estimate multiple known estimation results of the network scan from the correspondence table TBL3 stored in the host information database 13 (step S127), and outputs the detected standard sample data D_standard_z (z = 1 to Z) to the calculation means 15.

[0484] Then, the extraction means 14 extracts sample data x acquired in a communication environment where the result of the network scan is unknown from the correspondence table TBL1 (or correspondence table TBL2) stored in the host information database 13. i_unknown More specifically, the extraction means 14 detects the communication characteristic CMbest according to the flowcharts shown in FIG. m and selects the communication characteristic CMbest from the correspondence table TBL1 (or correspondence table TBL2). m Scan response delay combined with RTT m By detecting the sample data x i_unknown Then, the extraction means 14 extracts the detected sample data x i_unknown is output to the calculation means 15.

[0485] The calculation means 15 calculates standard sample data D_standard_z (z=1 to Z) and sample data x i_unknown The calculation means 15 receives the scan response delay {RTT m’ _known}_z(m'=1'~M',z=1~Z) and sample data x i_unknown Scan Response Delay RTT m A probability density function PDF of _unknown (m=1 to M) is calculated (step S129).

[0486] Then, the calculation means 15 calculates the sample data x i_unknown The mean absolute deviation of W unknown is calculated (step S130).

[0487] Then, the calculation means 15 sets z=1 (step S131), and calculates the mode m(Mo)_known by equation (9) based on the probability density function of the standard sample data D_standard_z (step S132). In this case, when there are two most likely values ​​of scan response delay, the calculation means 15 substitutes the two most likely values ​​of scan response delay into r1 and r2 in equation (9) to calculate the mode m(Mo)_known, and when there are three or more most likely values ​​of scan response delay, the calculation means 15 detects the minimum and maximum values ​​from the three or more most likely scan response delays, and substitutes the detected minimum and maximum values ​​into r1 and r2 in equation (9) to calculate the mode m(Mo)_known.

[0488] Then, the calculation means 15 calculates the sample data x i_known_z Based on the mode m(Mo)_known_z, the relative sample data C(x i_known_z ) is calculated (step S133), and the calculated relative sample data C(x i_known_z ) into equation (11A) to obtain the mean absolute deviation W known_z (Step S134). That is, the calculation means 15 calculates the sample data x i_known_z Based on the mode m(Mo)_known_z, the mean absolute deviation W is calculated by (11). known_z Calculate.

[0489] Subsequently, the calculation means 15 calculates the mean absolute deviation W unknown and the mean absolute deviation W known_z Based on this, the difference in mean absolute deviation S Sz is calculated (step S135).

[0490] Then, the calculation means 15 calculates the difference in mean absolute deviation S Sz Based on this, the similarity P Sz is calculated (step S136).

[0491] Thereafter, the calculation means 15 judges whether or not z=Z (step S137). When it is judged in step S137 that z=Z is not true, the calculation means 15 sets z=z+1 (step S138). Thereafter, the series of operations proceeds to step S132, and steps S132 to S138 are repeatedly executed until it is judged in step S137 that z=Z is true.

[0492] Then, if it is determined in step S137 that z=Z, the series of operations proceeds to step S139 in FIG.

[0493] In the flowchart shown in FIG. 36, when Z=3, steps S133 to S136 are executed as follows.

[0494] In step S133 of FIG. 36, when z=1, the calculation means 15 calculates the scan response delay {RTT m’ _known}_1(m'=1'~M') to x i_known_z , and the mode m(Mo)_known_1 is substituted for m(Mo)_known_z to obtain the relative sample data C(x i_known_1 ) and when z=2, the scan response delay {RTT m’ _known}_2(m'=1'~M') to x i_known_z , and the mode m(Mo)_known_2 is substituted for m(Mo)_known_z to obtain the relative sample data C(x i_known_2 ) and calculate the scan response delay {RTT m’ _known}_3(m'=1'~M') to x i_known_z , and the mode m(Mo)_known_3 is substituted for m(Mo)_known_z to obtain the relative sample data C(x i_known_3 ) is calculated.

[0495] In step S134 of FIG. 36, when z=1, the calculation means 15 calculates the relative sample data C(x i_known_1 ) to C(x i_known_z) and use equation (11A) to obtain the mean absolute deviation W known_1 Calculate the relative sample data C(x i_known_2 ) to C(x i_known_z ) and use equation (11A) to obtain the mean absolute deviation W known_2 Calculate the relative sample data C(x i_known_3 ) to C(x i_known_z ) and use equation (11A) to obtain the mean absolute deviation W known_3 Calculate.

[0496] Furthermore, in step S135 of FIG. 36, when z=1, the calculation means 15 calculates the mean absolute deviation W known_1 W known_z Substituting the mean absolute deviation W unknown W unknown Substituting this into the equation (12), we obtain the mean absolute deviation difference S S1 Calculate the mean absolute deviation W when z=2. known_2 W known_z Substituting the mean absolute deviation W unknown W unknown Substituting this into the equation (12), we obtain the mean absolute deviation difference S S2 Calculate the mean absolute deviation W when z=3. known_3 W known_z Substituting the mean absolute deviation W unknown W unknown Substituting this into the equation (12), we obtain the mean absolute deviation difference S S3 Calculate.

[0497] Furthermore, in step S136 of FIG. 36, the calculation means 15 calculates the difference S S1 S Sz By substituting into the above, the similarity P S1 Calculate the mean absolute deviation difference S when z=2. S2 S Sz By substituting into the above, the similarity P S2 Calculate the mean absolute deviation difference S when z=3. S3 S Sz By substituting into the above, the similarity P S3 Calculate.

[0498] 37, when it is determined in step S137 of FIG. 36 that z=Z, the calculation means 15 calculates the similarity P S1 ,P S2 ,···,P SZ is output to the extraction means 14.

[0499] The extraction means 14 receives the similarity P S1 ,P S2 ,···,P SZ and the similarity P S1 ,P S2 ,···,P SZ The highest similarity P S_highest and find the highest similarity P S_highest The standard sample data D_similarity is detected when the highest similarity P S_highest When there exists a plurality of highest similarities P S_highest Any similarity P S_highest and find the highest similarity P S_highest Detect the standard sample data D_similarity when

[0500] Then, the extraction means 14 extracts the similarity P S_highest The standard sample data D_similarity when is obtained is output to the estimation means 16.

[0501] The estimation means 16 receives the standard sample data D_similarity from the extraction means 14. After that, the estimation means 16 sets the F1 score F1 to "0" (step S140).

[0502] Then, the estimation means 16 divides the standard sample data D_similarity into training data and test data (step S141), and divides the training data into a training set and a validation set (step S142).

[0503] Then, the estimation means 16 generates initial sample data D_sample_initial for modeling by the above-mentioned method, and trains the estimation algorithm ALG using old training data consisting of the generated initial sample data D_sample_initial (step S143).

[0504] Then, the estimation means 16 estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (step S144), and generates a confusion matrix CFM based on the estimation result of the network scan and outputs it to the calculation means 15.

[0505] The calculation means 15 receives the confusion matrix CFM from the estimation means 16, calculates the F1 score F1 using equations (16) to (18) based on the elements TP, TN, FP, and FN of the received confusion matrix CFM (step S145), and outputs the calculated F1 score F1 to the estimation means 16.

[0506] The estimation means 16 receives the F1 score F1 from the calculation means 15. Then, the estimation means 16 divides the training set into subsets d1 to d U (Step S146). After that, the series of operations proceeds to Step S147 in FIG.

[0507] Referring to FIG. 38, after step S146 in FIG. 37, the estimation means 16 sets u=1 (step S147) and u The estimation algorithm ALG is trained using tentative sample data D_sample_tentative_u including (step S148).

[0508] Then, the estimation means 16 estimates the validation set (= subset V 1’ ~V U’ ) is used to estimate the result of the network scan by the estimation algorithm ALG (step S149), and a confusion matrix nCFM_u is generated based on the estimation result of the network scan and output to the calculation means 15.

[0509] The calculation means 15 receives the confusion matrix nCFM_u from the estimation means 16, calculates the F1 score nF1_u based on the elements TP, TN, FP, and FN of the received confusion matrix nCFM_u using equations (16) to (18) (step S150), and outputs the calculated F1 score nF1_u to the estimation means 16.

[0510] The estimation means 16 receives the F1 score nF1_u from the calculation means 15. Then, the estimation means 16 determines whether or not the F1 score nF1_u is greater than the F1 score F1 (step S151).

[0511] When it is determined in step S151 that the F1 score nF1_u is not greater than the F1 score F1, the series of operations proceeds to step S155.

[0512] On the other hand, when it is determined in step S151 that the F1 score nF1_u is greater than the F1 score F1, the estimation means 16 sets the F1 score nF1_u to F1 (step S152).

[0513] Then, the estimation means 16 estimates the subset d when the F1 score nF1_u is obtained. u The tentative sample data D_sample_tentative_u including the above is set as training data (step S153), and the model variables (variables of the estimation algorithm ALG) are replaced with the training data by the above-mentioned method (step S154).

[0514] Then, when it is determined in step S151 that the F1 score nF1_u is not greater than the F1 score F1, or after step S154, the estimation means 16 determines whether or not u=U (step S155).

[0515] In step S155, when it is determined that u ≠ U, the estimation means 16 sets u = u + 1 (step S156). Thereafter, the series of operations proceeds to step S148, and steps S148 to S156 are repeatedly executed until it is determined in step SS155 that u = U.

[0516] Then, when it is determined in step S155 that u = U, the estimation means 16 estimates the result of the network scan using the test data (which consists of all of subsets TD1 to TD4) by the model (estimation algorithm ALG) in which the model variables are replaced (step S157), generates the confusion matrix tCFM based on the estimated result of the network scan, and outputs it to the calculation means 15.

[0517] The calculation means 15 receives the confusion matrix tCFM from the estimation means 16, calculates the F1 score tF1 according to formulas (16) to (18) based on the elements TP, TN, FP, and FN of the received confusion matrix tCFM (step S158), and outputs the calculated F1 score tF1 to the estimation means 16.

[0518] The estimation means 16 receives the F1 score tF1 from the calculation means 15 and determines whether the F1 score tF1 is greater than or equal to the F1 score F1 (step S159).

[0519] In step S159, when it is determined that the F1 score tF1 is less than the F1 score F1, the series of operations proceeds to step S147, and steps S147 to S159 are repeatedly executed until it is determined in step S159 that the F1 score tF1 is greater than or equal to the F1 score F1.

[0520] Then, when it is determined in step S159 that the F1 score tF1 is greater than or equal to the F1 score F1, the estimation means 16 uses the model (estimation algorithm ALG) in which the model variables are replaced and the training data (finally, the subset d that satisfies F1 < nF1_u) uThe tentative sample data D_sample_tentative_u including the above is approved (step S160). The tentative sample data D_sample_tentative_u approved in step S160 becomes the data sample D_sample_initial when step S11 in FIG. 33 is executed for the second or subsequent times.

[0521] Then, after step S160, the estimation means 16 estimates the sample data x acquired by the network scan in the communication environment where the result of the network scan is unknown. unknown The result of the network scan is estimated by the model in which the model variables have been replaced (estimation algorithm ALG) using (step S161). After that, the series of operations proceeds to step S12 in FIG.

[0522] In steps S140 and S145 in FIG. 37 and S150, S151, S152, S153, S158, and S159 in FIG. 38, Matthews correlation coefficients MCC, nMCC_u, and tMCC may be used instead of the F1 scores F1, nF1_u, and tF1, respectively.

[0523] 38 is executed for the first time, if it is determined in step S159 that the F1 score tF1 is not equal to or greater than the F1 score F1 and the loop of steps S148 to S156 is executed for the second time, the estimation algorithm ALG in which the model variables are finally replaced with optimal training data when the loop of steps S148 to S156 is executed for the first time is used in steps S148 and S149. As a result, when the loop of steps S148 to S156 is executed for the second time, the F1 score nF1_u calculated in step S150 is greater than the F1 score nF1_u calculated in step S150 when the loop of steps S148 to S156 is executed for the first time. Similarly, the F1 score nF1_u calculated in step S150 when the loop from step S148 to step S156 is executed a third, fourth, etc., will be greater than the F1 score nF1_u calculated in step S150 when the loop from step S148 to step S156 is executed a second, third, etc., respectively.

[0524] Therefore, as the number of times the loop of steps S148 to S156 is executed increases, the F1 score nF1_u calculated in step S150 in the Rth (R is an integer equal to or greater than 2)th execution of the loop of steps S148 to S156 becomes larger than the F1 score nF1_u calculated in step S150 in the (R-1)th execution of the loop of steps S148 to S156. Then, in step S154, by replacing the model variables with optimal training data, the estimation algorithm ALG approaches an estimation algorithm when it is estimated that the result of the network scan is a known estimation result (any of "normal", "inadequate communication quality", and "network congestion"). In other words, the estimation algorithm ALG approaches an optimal algorithm.

[0525] Then, in step S159, the F1 score tF1 is likely to be greater than or equal to the F1 score F1, and in step S160, the model with the model variables replaced (estimation algorithm ALG) and the training data (eventually, the subset d that satisfies F1 < nF1_u u including the provisional sample data D_sample_tentative_u) are also likely to be approved.

[0526] Therefore, in step S159, after it is determined that the F1 score tF1 is not greater than or equal to the F1 score F1, even if the validation set and test data are not changed, the estimation algorithm ALG approaches the optimal algorithm (the algorithm for which the estimation result of the network scan becomes the known estimation result (any one of "normal", "insufficient communication quality", and "network congestion")), so in step S160, the model with the model variables replaced (estimation algorithm ALG) and the training data (eventually, the subset d that satisfies F1 < nF1_u u including the provisional sample data D_sample_tentative_u) can have a high probability of being approved.

[0527] And steps S147 to S160 in FIG. 38 correspond to updating the estimation algorithm ALG for estimating the result of the network scan executed in the communication environment where the result of the network scan is unknown so as to approach the optimal algorithm. Here, the optimal algorithm is an algorithm for estimating the result of the network scan such that the estimation result when estimating the result of the network scan becomes the known estimation result.

[0528] After the estimation algorithm ALG approaches the optimal algorithm, in step S161, the sample data x obtained by the network scan in the communication environment where the result of the network scan is unknown unknown is used to estimate the result of the network scan by the estimation algorithm ALG that has approached the optimal algorithm, so the result of the network scan can be accurately estimated.

[0529] In step S127 of Fig. 36, the extraction means 14 extracts standard sample data D_standard_z (z = 1 to Z). Then, in steps S131 to S138, the calculation means 15 calculates the Z number of standard sample data D_standard_1 to D_standard_Z and the sample data x i_unknown Z similarities with P S1 ~P SZ Calculate.

[0530] Thereafter, the extraction means 14 extracts Z similarities P S1 ~P SZ The highest similarity P S_highest Detect.

[0531] Therefore, by executing steps S127 to S139, the extraction means 14 extracts the sample data x i_unknown The standard sample data D_similarity (= highest similarity P S_highest is obtained from the Z pieces of standard sample data D_standard_1 to D_standard_Z.

[0532] Then, the estimation means 16 trains the estimation algorithm ALG using old training data (=initial sample data D_sample_initial) (see step S143), and the calculation means 15 uses the validation set to calculate the F1 score F1 when the network scan results are estimated by the trained estimation algorithm ALG (see steps S144 and S145).

[0533] Furthermore, the calculation means 15 and the estimation means 16 calculate the U subsets d1 to d UBy executing steps S147 to S156 for all of U tentative sample data D_sample_tentative_1 to D_sample_tentative_U each including, respectively, the estimation means 16 acquires the largest F1 score nF1_highest among the U F1 scores nF1_1 to nF1_U. When it is determined in step S151 that the F1 score nF1_u is larger than the F1 score F1, the F1 score nF1_u larger than the F1 score F1 is set as the F1 score F1 (see step S152), so that the F1 score F1 when step S151 is executed the u-th time is larger than the F1 score F1 when step S151 is executed the (u-1)th time. As a result, the F1 score nF1_u determined to be greater than the F1 score F1 by the u-th execution of step S151 is greater than the F1 score nF1_u determined to be greater than the F1 score F1 by the (u-1)-th execution of step S151. U By executing steps S147 to S156 for all of U pieces of tentative sample data D_sample_tentative_1 to D_sample_tentative_U each including D_sample_tentative_1 to D_sample_tentative_U, the largest F1 score nF1_highest among the U F1 scores nF1_1 to nF1_U is obtained.

[0534] Then, the estimation means 16 estimates a subset d when the largest F1 score nF1_highest is obtained among the U F1 scores nF1_1 to nF1_U. u_highest The temporary sample data D_sample_highest including the above is set as training data (see step S153), and the model variables (variables of the estimation algorithm ALG) are replaced with the set training data (see step S154).

[0535] As a result, the F1 score nF1_highest indicates the highest accuracy when estimating the results of network scans, and the variables of the estimation algorithm ALG are the subset d when the F1 score nF1_highest is obtained.u_highest Therefore, the accuracy of estimating the results of the network scan can be maximized by estimating the results of the network scan using the estimation algorithm ALG having variables replaced with the provisional sample data D_sample_highest.

[0536] The standard sample data D_standard_z (z=1 to Z) is the scan response delay {RTT m’ _known}_z(m'=1'~M',z=1~Z) and communication characteristics {CMbest m’ _known}_z (m'=1' to M', z=1 to Z), so the extraction means 14 extracts the sample data x i_unknown The standard sample data D_similarity (= highest similarity P S_highest The process of detecting the standard sample data D_standard_z when is calculated from the Z standard sample data D_standard_1 to D_standard_Z constitutes a "detection process". m’ _known}_similarity (m'=1' to M') constitutes the "similar scan response delay". Furthermore, the extraction means 14 that executes the detection process constitutes the "detection means". Furthermore, since each of M' and Z is an integer equal to or greater than 2, the {RTT m’ _known}_z (m'=1' to M', z=1 to Z) constitute "multiple sets of M' second scan response delays."

[0537] Also, the sample data x i_unknown is the scan response delay RTT m _unknown(m=1~M) and communication characteristics CMbest m_unknown (m=1 to M), which constitutes the "first sample data." Then, the communication characteristic CMbest is calculated according to the flowcharts shown in FIG. 34 and FIG. 35. m and selects the communication characteristic CMbest from the correspondence table TBL1 (or correspondence table TBL2). m Scan response delay combined with RTT m By detecting the sample data x i_unknown The extraction means 14 for detecting the above constitutes the "generation means."

[0538] Furthermore, the standard sample data D_similarity is the scan response delay {RTT m '_known}_similarity(m'=1'~M') and communication characteristics {CMbest m’ _known}_similarity (m'=1' to M'), which constitutes "similar standard sample data." The extraction means 14 that detects the standard sample data D_similarity constitutes the "generation means."

[0539] Furthermore, the F1 scores F1, nF1 (=nF1_1 to nF1_U), tF1 or the Matthews correlation coefficients MCC, nMCC (=nMCC_1 to nMCC_U), tMCC constitute an "evaluation index", the F1 score F1 or the Matthews correlation coefficient MCC constitutes a "first evaluation index", the F1 scores nF1 (=nF1_1 to nF1_U) or the Matthews correlation coefficient nMCC (=nMCC_1 to nMCC_U) constitute a "second evaluation index", and the F1 score tF1 or the Matthews correlation coefficient tMCC constitutes a "third evaluation index".

[0540] Furthermore, the training data (= subset d u_highest The estimation means 16 for replacing variables of the estimation algorithm ALG with tentative sample data D_sample_tentative including the first sample data constitutes an "update means" for updating the estimation algorithm ALG with the first sample data.

[0541] Furthermore, steps S132 to S136 are executed for all z (=1 to Z), and the highest similarity P S_highest The detection of the standard sample data D_similarity when m’ _known}_z (m'=1'~M', z=1~Z) scan response delays {RTT m’ _unknown}_1~{RTT m’ _unknown}_Z distribution and sample data x i_unknown Scan Response Delay RTT m Based on the distribution of _unknown (m=1~M), M scan response delay RTTs are calculated. m The M' scan response delays {RTT 1’ _known}_z~{RTT M’ _known}_z for multiple sets (=Z sets) of M' scan response delays {RTT 1’ _known}_z~{RTT M’ This corresponds to detecting the similar scan response delay from {_known}_z.

[0542] Furthermore, in step S130, the mean absolute deviation W unknown The calculation of this constitutes a "first calculation process."

[0543] Furthermore, the similarity P Sz The calculation of this constitutes a "second calculation process."

[0544] Furthermore, executing step S148 constitutes executing a “first training”, and executing step S143 constitutes executing a “second training”. And the estimation means 16 executing steps S143 and S148 constitutes a “training means”.

[0545] FIG. 39 is a flowchart for explaining the detailed operation of step S130 in FIG.

[0546] 39, after step S129 in FIG. 36, the calculation means 15 calculates the sample data x i_unknown Based on the probability density function of i_unknown Then, the mode m(Mo)_unknown of the vector m is calculated (step S1301).

[0547] Then, the calculation means 15 calculates the sample data x i_unknown Based on the mode m(Mo)_unknown, the relative sample data C(x i_unknown )_unknown is calculated (step S1302), and the calculated relative sample data C(x i_unknown )_unknown into equation (10A) to obtain the mean absolute deviation W unknown (step S1303). That is, the calculation means 15 calculates the sample data x i_unknown Based on the mode m(Mo)_unknown, the mean absolute deviation W unknown After step S1303, the process proceeds to step S131 in FIG.

[0548] 40 to 42 are first to third flowcharts, respectively, for explaining the detailed operation of step S143 in FIG.

[0549] 40, after step S142 in FIG. 37, the estimation means 16 estimates the scan response delay {RTT m’ _known}_similarity and communication characteristics {CMbest m’ Detect {known}_similarity.

[0550] Then, the estimation means 16 sets b=1 (step S1431) and calculates the discriminant function δ gThen, the estimation unit 16 sets the model Model for estimating the result of the network scan to Model=1 (step S1433), and generates m b = 1 (step S1434), and DS_count = 0 (step S1435). b represents a terminal device belonging to one base station b.

[0551] Then, the estimation means 16 calculates the discriminant function δ g Using (x), the communication characteristic {CMbest m’ _known}_similarity is calculated (step S1436).

[0552] After that, the estimation means 16 judges whether or not the D score is smaller than zero (0) (step S1437).

[0553] When it is determined in step S1437 that the D score is smaller than zero (0), the estimation means 16 sets DS_count=DS_count+1 (step S1438). That is, the estimation means 16 increments the count value DS_count by "1".

[0554] Then, in step S1437, when it is determined that the D score is not smaller than zero (0), or after step S1438, the estimation means 16 b =N terminal It is determined whether or not this is the case (step S1439).

[0555] In step S1439, m b =N terminal If it is determined that the above is not the case, the estimation means 16 b =m b +1 is set (step S1440). After that, the series of operations proceeds to step S1436, and in step S1439, m b =N terminal Steps S1436 to S1440 are repeatedly executed until it is determined that the above is the case.

[0556] Then, in step S1439, m b =N terminal If it is determined that Model=1, the estimation means 16 determines whether Model=1 (step S1441).

[0557] When it is determined in step S1441 that Model=1, the estimation means 16 checks whether DS_count=0 or DS_count>(N terminal ×D_thres) (step S1442).

[0558] In step S1442, DS_count is not 0 and DS_count>(N terminal ×D_thres), the estimation means 16 sets Model=3 (step S1443).

[0559] On the other hand, in step S1442, DS_count=0 or DS_count>(N terminal ×D_thres), the estimation means 16 sets Model=2 (step S1444).

[0560] Then, after step S1443 or step S1444, the series of operations proceeds to step S1434.

[0561] On the other hand, if it is determined in step S1441 that Model is not 1, the process proceeds to step S1445 in FIG.

[0562] Referring to FIG. 41, when it is determined in step S1441 that Model is not 1, the estimation means 16 determines whether or not DS_count is 0 (step S1445).

[0563] When it is determined in step S1445 that DS_count=0, the estimation means 16 determines "normal" scan data for the terminal device (step S1446).

[0564] On the other hand, when it is determined in step S1445 that DS_count is not 0, the estimation means 16 determines whether DS_count<(N terminal ×D_thres) (step S1447).

[0565] In step S1447, DS_count<(N terminal ×D_thres), the estimation means 16 estimates that the cause of the failure of the network scan for all terminal devices belonging to base station b is "network congestion" (step S1448).

[0566] On the other hand, in step S1447, DS_count<(N terminal ×D_thres), the estimation means 16 estimates that the cause of the failure of the network scan for the terminal device having a D score smaller than “0” is “insufficient SNR (insufficient communication quality)” (step S1449).

[0567] Then, after any of step S1446, step S1448, and step S1449, the series of operations proceeds to step S1450 in FIG.

[0568] Referring to FIG. 42, after any of step S1446, step S1448 and step S1449 in FIG. 41, the estimation means 16 b =1 (step S1450).

[0569] Then, the estimation means 16 estimates the terminal device m b It is determined whether the result of the network scan has been estimated for each of the nodes (step S1451).

[0570] In step S1451, the terminal device mb When it is determined that the result of the network scan has not been estimated, the estimation means 16 estimates the b It is then determined whether or not there is a scan response from all of the ports (step S1452).

[0571] In step S1452, the terminal device m b When it is determined that there is a scan response in some of the ports of the terminal device m, the estimation means 16 b It is then determined whether or not there is a scan response from all of the ports (step S1453).

[0572] In step S1453, the terminal device m b When it is determined that there is a scan response for all ports of the terminal device m, the estimation means 16 b It is estimated that the network scan is "normal" for (step S1454).

[0573] On the other hand, in step S1453, the terminal device m b When it is determined that there is no scan response from some of the ports of the terminal device m, the estimation means 16 b The estimation means 16 estimates that the cause of the failure of the network scan is "port closed" (step S1455). Then, the estimation means 16 stores the obtained waiting time in the host information database 13 (step S1456).

[0574] On the other hand, in step S1452, the terminal device m b When it is determined that there is no scan response for all ports of the terminal device m, the estimation means 16 b It is estimated that the cause of the failure of the network scan is "no network connection" (step S1457).

[0575] Then, in step S1451, the terminal device m b When it is determined that the result for m has been estimated, or after any of steps S1454, S1456 and S1457, the estimation means 16 b =Nterminal It is determined whether or not this is the case (step S1458).

[0576] In step S1458, m b =N terminal If it is determined that the above is not the case, the estimation means 16 b =m b +1 is set (step S1459). After that, the series of operations proceeds to step S1451, and in step S1458, m b =N terminal Steps S1451 to S1459 are repeatedly executed until it is determined that the above condition is met.

[0577] Then, in step S1458, m b =N terminal If it is determined that b=B, the estimation means 16 determines whether b=B (step S1460).

[0578] When it is determined in step S1460 that b is not equal to B, the estimation means 16 sets b to b+1 (step S1461). After that, the series of operations proceeds to step S1432 in Fig. 40, and steps S1432 to S1461 are repeatedly executed until it is determined in step S1460 that b is equal to B.

[0579] If it is determined in step S1460 that b=B, the process proceeds to step S144 in FIG.

[0580] It should be noted that detailed operations of step S144 in FIG. 37 and steps S148, S149, S157, and S161 in FIG. 38 are also executed according to the flowcharts shown in FIGS.

[0581] When the flowcharts shown in FIGS. 40 to 42 are executed as detailed operations of step S144 in FIG. 37, if the estimation means 16 determines in step S1460 that b=B, it generates a confusion matrix CFM and outputs the generated confusion matrix CFM to the calculation means 15.

[0582] Furthermore, when the flowcharts shown in Figures 40 to 42 are executed as detailed operations of step S149 in Figure 38, if the estimation means 16 determines in step S1460 that b = B, it generates a confusion matrix nCFM_u and outputs the generated confusion matrix nCFM_u to the calculation means 15.

[0583] Furthermore, when the flowcharts shown in FIGS. 40 to 42 are executed as detailed operations of step S157 in FIG. 38, if the estimation means 16 determines in step S1460 that b=B, it generates a confusion matrix tCFM and outputs the generated confusion matrix tCFM to the calculation means 15.

[0584] Steps S1450 to S1459 in FIG. 42 are steps for estimating the result of a network scan (either "normal", "port closed" or "no network connection") for each of all terminal devices belonging to one base station.

[0585] Fig. 43 is a flowchart for explaining the detailed operation of step S1432 in Fig. 40. In Fig. 43, the discriminant function δ is calculated by the trained LDA algorithm trained using the scan data when the network scan is "normal" (the communication characteristic CMbest when the "network is congested"). g Explain how to generate (x).

[0586] 43, after step S1431 in FIG. 40 or step S1461 in FIG. 42, the estimation means 16 classifies terminal devices having communication characteristics CMbest into G (G=3) groups CLS1 to CLS3 using the LDA algorithm (step S1432-1).

[0587] Then, the estimation means 16 calculates the averages μ1, μ2, and μ3 of the groups using the formula (20C) (or the formula (20D)) (step S1432-2).

[0588] Thereafter, the estimation means 16 calculates the prior averages Π1, Π2, and Π3 of each group using the equation (20E) (or the equation (20F)) (step S1432-3).

[0589] Next, the estimation means 16 calculates the covariance matrix C of each group using equation (20B) (step S1432-4).

[0590] Then, the estimation means 16 calculates the joint covariance matrix C of Model 1 by equation (21). 12 (step S1432-5). In addition, the estimation means 16 calculates the joint covariance matrix C of Model 2 by the formula (22). 13 (step S1432-6). Furthermore, the estimation means 16 calculates the joint covariance matrix C of Model 3 by the formula (23). 23 (step S1432-7).

[0591] Then, the estimation means 16 calculates the discriminant function δ pq (x) is generated (step S1432-8). In the formula (20A), “C” is the joint covariance matrix C calculated in steps S1432-5 to S1432-7. 12 ,C 13 ,C 23 is assigned.

[0592] In the flowchart shown in FIG. 43, the estimation means 16 classifies, for example, terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 in step S1432-1.

[0593] As described above, group CLS1 is a "normal" group, group CLS2 is a "poor communication quality" group, and group CLS3 is a "network congestion" group. Model 1 handles groups CLS1 and CLS2, Model 2 handles groups CLS1 and CLS3, and Model 3 handles groups CLS2 and CLS3.

[0594] Therefore, when Model1 is used, the joint covariance matrix C calculated in step S1432-5 12 is substituted for “C” in step S1432-8. When Model 2 is used, the joint covariance matrix C calculated in step S1432-6 is 13 is substituted for “C” in step S1432-8. Furthermore, when Model 3 is used, the joint covariance matrix C calculated in step S1432-7 23 is substituted for "C" in step S1432-8.

[0595] Furthermore, since the failure causes of "inadequate communication quality" and "network congestion" are failure causes caused by the communication channel with the terminal device, as described above, classifying the terminal devices having the communication characteristic CMbest into three groups CLS1 to CLS3 in step S1432-1 corresponds to "classifying into a first group indicating that the network scan is normal, and into second and third groups indicating that the network scan has failed due to a cause caused by the communication channel with the terminal device." In this case, group CLS2 constitutes the second group, and group CLS3 constitutes the third group.

[0596] FIG. 44 is a flowchart for explaining detailed operations of step S1436 in FIG.

[0597] With reference to FIG. 44, after step S1435 or step S1440 in FIG. 40, the estimation means 16 determines whether or not Model=1 (step S1436-1).

[0598] When it is determined in step S1436-1 that Model=1, the estimation means 16 calculates the joint covariance matrix C 12 C in formula (24A) 12 Substituting into the discriminant function f 1 (step S1436-2). This discriminant function f 1is the combined covariance matrix C of group CLS1 (= g1) and group CLS2 (= g2). 12 and a discriminant function expressed using the mean μ1 of the group CLS1 (=g1) and the prior mean Π1 of the group CLS1 (=g1).

[0599] Thereafter, the estimation means 16 calculates the joint covariance matrix C 12 C of formula (24B) 12 Substituting into the discriminant function f 2 (step S1436-3). This discriminant function f 2 is the combined covariance matrix C of group CLS1 (= g1) and group CLS2 (= g2). 12 and the discriminant function expressed using the mean μ2 of group CLS2 (=g2) and the prior mean Π2 of group CLS2 (=g2).

[0600] On the other hand, if it is determined in step S1436-1 that Model=1 is not satisfied, the estimation means 16 determines whether or not Model=2 is satisfied (step S1436-4).

[0601] When it is determined in step S1436-4 that Model=2, the estimation means 16 calculates the joint covariance matrix C 13 C in equation (25A) 13 Substituting into the discriminant function f 1 (step S1436-5). This discriminant function f 1 is the combined covariance matrix C of group CLS1 (= g1) and group CLS3 (= g3). 13 and a discriminant function expressed using the mean μ1 of the group CLS1 (=g1) and the prior mean Π1 of the group CLS1 (=g1).

[0602] Thereafter, the estimation means 16 calculates the joint covariance matrix C 13 C of formula (25B)13 Substituting into the discriminant function f 2 (step S1436-6). This discriminant function f 2 is the combined covariance matrix C of group CLS1 (= g1) and group CLS3 (= g3). 13 and a discriminant function expressed using the mean μ3 of group CLS3 (=g3) and the prior mean Π3 of group CLS3 (=g3).

[0603] On the other hand, when it is determined in step S1436-4 that Model=2 is not true, the estimation stage 15 uses the joint covariance matrix C calculated in step S1432-7 of FIG. 23 C in equation (26A) 23 Substituting into the discriminant function f 1 (step S1436-7). This discriminant function f 1 is the combined covariance matrix C of group CLS2 (= g2) and group CLS3 (= g3). 23 and the discriminant function expressed using the mean μ2 of group CLS2 (=g2) and the prior mean Π2 of group CLS2 (=g2).

[0604] Thereafter, the estimation means 16 calculates the joint covariance matrix C 23 C of formula (26B) 23 Substituting into the discriminant function f 2 (step S1436-8). This discriminant function f 2 is the combined covariance matrix C of group CLS2 (= g2) and group CLS3 (= g3). 23 and a discriminant function expressed using the mean μ3 of group CLS3 (=g3) and the prior mean Π3 of group CLS3 (=g3).

[0605] Then, after any of step S1436-3, step S1436-6, and step S1436-8, the estimation means 16 calculates D score=f 1 -f 2(step S1436-9). After that, the series of operations proceeds to step S1437 in FIG.

[0606] In the flowchart shown in Figure 44, Model = 1 is set in step S1433 of Figure 40, so when step S1436 of Figure 40 is executed first, it is determined in step S1436-1 of Figure 44 that Model = 1, and after steps S1436-2 and S1436-3 are executed sequentially, the D score is calculated using Model 1 in step S1436-9.

[0607] Then, after step S1436-9, in step S1439 of FIG. b =N terminal Steps S1436 to S1440 in FIG. 40 are repeatedly executed until it is determined that Model=1, and then in step S1441 it is determined that Model=1, and steps S1442 and S1443 are executed in sequence, or steps S1442 and S1444 are executed in sequence.

[0608] When steps S1442 and S1443 are executed sequentially, Model=3 is set (see step S1443), so when step S1436 is executed after transitioning from step S1443 to step S1434, it is determined in step S1436-1 of FIG. 44 that Model=1 is not true, and further, in step S1436-4, it is determined that Model=2 is not true, and after steps S1436-7 and S1436-8 are executed sequentially, in step S1436-9, the D score is calculated using Model 3.

[0609] On the other hand, if steps S1442 and S1444 in Figure 40 are executed sequentially, Model = 2 is set (see step S1444), so when step S1436 is executed after transitioning from step S1444 to step S1434, it is determined in step S1436-1 in Figure 44 that Model = 1, and further, in step S1436-4, it is determined that Model = 2, and after steps S1436-5 and S1436-6 are executed sequentially, in step S1436-9, the D score is calculated using Model 2.

[0610] Therefore, the flowchart shown in FIG. 44 is a flowchart for calculating the D score according to the set Model (one of Model 1, Model 2, and Model 3).

[0611] In addition, in the flowcharts shown in Figures 40 to 42 for estimating the results of a network scan, after setting Model=1 in step S1433, the D score is calculated in step S1436 because, by using Model 1, it is possible to first estimate whether the network scan is "normal" or "failed (=failed due to insufficient communication quality)", and when it is estimated that the network scan is "failed (=failed due to insufficient communication quality)", the cause of the failure of the network scan can be further estimated.

[0612] Furthermore, in step S1433, after Model=1 is set, D scores are calculated using Model 1 for all terminal devices belonging to one base station b, and the number of D scores smaller than “0” is counted as a count value DS_count (see steps S1436 to S1440).

[0613] Thereafter, when step S1441 is executed for the first time, Model=1 is set, so that it is determined in step S1441 that Model=1, and Model=2 or Model=3 is set in steps S1442 to S1444.

[0614] Therefore, Model 2 or Model 3 is set according to the number of D scores (=DS_count) calculated using Model 1. In other words, Model 2 or Model 3 is set based on Model 1.

[0615] Then, in steps S1442 to S1444, after Model=2 or Model=3 is set, the D score is calculated using Model2 or Model3, and the number of D scores smaller than "0" is counted as a count value DS_count (see steps S1436 to S1440). More specifically, when Model=2 is set, the communication characteristics CMbest classified into group CLS1, the communication characteristics CMbest classified into group CLS3, and the joint covariance matrix C 13 When the D score is calculated using the above, and Model=3 is set, the communication characteristics CMbest classified into group CLS2, the communication characteristics CMbest classified into group CLS3, and the joint covariance matrix C 23 Then, based on the calculated D scores, the number of D scores smaller than "0" is counted as a count value DS_count.

[0616] Then, in step S1441, it is determined that Model=1 is not satisfied, and in steps S1445 to S1457 shown in FIG. 41 and FIG. 42, the result of the network scan (either “normal”, “poor communication quality”, “network congestion”, “port closed”, or “no network connection”) is estimated using Model 2 or Model 3.

[0617] When Model 2 is set and the results of a network scan are estimated, the number of D scores (=count value DS_count) that are smaller than "0" among the D scores calculated using Model 2 is used.

[0618] In addition, when Model 3 is configured to estimate the results of a network scan, the number of D scores calculated using Model 3 that are smaller than "0" (=count value DS_count) is used.

[0619] In this embodiment of the present invention, the estimation algorithm ALG is made up of steps S1431 to S1461 shown in FIGS. 40 to 42 (including the flowcharts shown in FIGS. 43 and 44).

[0620] FIG. 45 is a flowchart for explaining the detailed operation of step S145 in FIG.

[0621] 45, after step S144 in FIG. 37, calculation means 15 receives OE, ECE and OIE from estimation means 16, and detects the numbers of the received OE, ECE and OIE (step S145-1).

[0622] Then, the calculation means 15 calculates TP based on the number of OEs and the number of ECEs using formula (14A) (step S145-2).

[0623] Thereafter, the calculation means 15 calculates FN based on the number of OIEs and the number of ECEs using formula (14B) (step S145-3).

[0624] Next, the calculation means 15 uses TP to calculate FP according to equation (15A) (step S145-4).

[0625] Then, the calculation means 15 calculates Precision using formula (16) based on the proportion classified as TP and the proportion classified as FP (step S145-5).The calculation means 15 also calculates Recall using formula (17) based on the proportion classified as TP and the proportion classified as FN (step S145-6).

[0626] Thereafter, the calculation means 15 calculates the F1 score F1 by substituting the Precision and Recall into the formula (18) (step S145-7). Then, the series of operations proceeds to step S146 in FIG.

[0627] Note that the detailed operations of steps S150 and S158 in Fig. 38 are also executed according to the flowchart shown in Fig. 45. When the detailed operation of step S150 is executed, the calculation means 15 receives OE, ECE, and OIE from the estimation means 16, receives the received OE, ECE, and OIE from the estimation means 16, receives the received OE, ECE, and OIE from the estimation means 16, and sequentially executes steps S145-1 to S145-7 based on the received OE, ECE, and OIE to calculate the F1 score nF1_u. Also, when the detailed operation of step S158 is executed, the calculation means 15 receives OE, ECE and OIE from the estimation means 16, receives the received OE, ECE and OIE from the estimation means 16, receives the received OE, ECE and OIE from the estimation means 16, and sequentially executes steps S145-1 to S145-7 based on the received OE, ECE and OIE to calculate the F1 score tF1.

[0628] Moreover, calculating the F1 scores F1, nF1_u, tF1 in accordance with steps S145-1 to S145-7 constitutes a "third calculation process."

[0629] FIG. 46 is a flowchart for explaining the detailed operation of step S145 in FIG. 37 when the Matthews correlation coefficient MCC is calculated in step S145.

[0630] The flowchart shown in FIG. 46 is the same as the flowchart shown in FIG. 45, except that steps S144-4 to S145-7 of the flowchart shown in FIG. 45 are replaced with steps S145-4A and S145-5A.

[0631] 46, when the detailed operation of step S145 in FIG. 37 is started, the calculation means 15 sequentially executes steps S145-1 to S145-3 described above. Then, after step S145-3, the calculation means 15 calculates FP by equation (15A) using TP, and calculates TN by equation (15B) using FN (step S145-4A).

[0632] Then, the calculation means 15 substitutes TP, TN, FP, and FN into the formula (19) to calculate the Matthews correlation coefficient MCC (step S145-5A). After that, the series of operations proceeds to step S146 in FIG.

[0633] In addition, when the Matthews correlation coefficient MCC is calculated in steps S150 and S158 in Fig. 38, the detailed operations of steps S150 and S158 are also executed according to the flowchart shown in Fig. 46. When the detailed operation of step S150 is executed, the calculation means 15 sequentially executes steps S145-1 to S145-3, S145-4A, and S145-5A to calculate the Matthews correlation coefficient nMCC_u. In addition, when the detailed operation of step S158 is executed, the calculation means 15 sequentially executes steps S145-1 to S145-3, S145-4A, and S145-5A to calculate the Matthews correlation coefficient tMCC.

[0634] FIG. 47 is a diagram showing an example of a confusion matrix. (a) of FIG. 47 shows the confusion matrix for the validation set (= subset V 1’ ~V 8’ 47(a) shows a confusion matrix obtained when the result of a network scan is estimated by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD1 to TD4), and FIG. 47(b) shows a confusion matrix obtained when the result of a network scan is estimated by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD1 to TD4).

[0635] Referring to (a) of Figure 47, if the result of the network scan is “normal” (known estimated result), the validation set (= subset V 1’ ~V 8’When the results of the network scan were estimated by the estimation algorithm ALG using , for example, the confusion matrix was obtained with TP = 64, FN = 0, TN = 0 and FP = 0.

[0636] Referring to (b) of Figure 47, when the result of the network scan is "normal" (known estimation result), when the result of the network scan is estimated by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD1 to TD4), a confusion matrix consisting of, for example, TP = 32, FN = 0, TN = 0 and FP = 0 is obtained.

[0637] Therefore, in step S145 of FIG. 37 and step S150 of FIG. 38, the confusion matrix (validation set (= subset V 1’ ~V 8’ In step S158 of FIG. 38, a confusion matrix shown in FIG. 47(b) (a confusion matrix obtained when the result of the network scan is estimated by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD1 to TD4)) is generated.

[0638] Fig. 48 is a diagram showing a simulation result of the D score. Referring to Fig. 48, as a result of performing a simulation of the D score, it was found that when the network scan is "normal", the D score is larger than "0" in more than half of the terminal devices (see (a) of Fig. 48).

[0639] In addition, when the cause of the network scan failure was “insufficient communication quality,” it was found that the D score was smaller than “0” in less than half of the terminal devices (see (b) of Figure 48).

[0640] Furthermore, it was found that when the cause of the network scan failure was "network congestion," the D score was smaller than "0" in more than half of the terminal devices (see (c) of Figure 48).

[0641] Therefore, by using the number of D scores less than "0" (=count value DS_count), it is possible to estimate whether the result of the network scan is "normal," "inadequate communication quality," or "network congestion."

[0642] Model 2 handles group CLS1 where the network scan was "normal" and group CLS3 where the network scan failed due to "network congestion," and calculates a D-score D_2 to distinguish between group CLS1 and group CLS3.If the number of D-scores D_2 that are less than "0" (=count value DS_count) among the calculated D-scores D_2 is "0," it can be inferred that the network scan is "normal."If the number of D-scores D_2 that are less than "0" (=count value DS_count) is greater than half of the terminal devices, it can be inferred that the network scan failed due to "network congestion."

[0643] In addition, Model 3 handles group CLS2 where the network scan failed due to "insufficient communication quality" and group CLS3 where the network scan failed due to "network congestion," and calculates a D-score D_3 to distinguish between group CLS2 and group CLS3.If the number of D-scores D_3 less than "0" (= count value DS_count) among the calculated D-scores D_3 is less than half of the terminal devices, it can be inferred that the network scan failed due to "insufficient communication quality," and if the number of D-scores D_3 less than "0" (= count value DS_count) is greater than half of the terminal devices, it can be inferred that the network scan failed due to "network congestion."

[0644] Therefore, first, in step S1445 of FIG. 41, it is determined whether DS_count=0 or not. If it is determined that DS_count=0 is not true, then DS_count<(N terminal × D_thres), and terminal ×D_thres), for terminal devices with D scores smaller than "0", it is assumed that the network scan failed due to "insufficient communication quality", and DS_count<(N terminal × D_thres) (i.e., DS_count is not (N terminal ×D_thres), it is assumed that the network scan for all terminal devices belonging to the base station has failed due to "network congestion" (see steps S1445, S1447 to S1449).

[0645] When Model 2 is set, in steps S1445, S1447 to S1449, the D-score D_2 is used to distinguish between group CLS1 where the network scan is "normal" and group CLS3 where the network scan has failed due to "network congestion", so that, depending on the number of D-scores D_2 smaller than "0" (=count value DS_count), it is possible to accurately estimate whether DS_count=0 (i.e., the network scan is "normal") or DS_count≠0 (i.e., the network scan has failed due to "inadequate communication quality" or "network congestion"). Then, when DS_count=0 (i.e., DS_count≠0), the count value DS_count, which is the number of D-scores D_2 smaller than "0", is (N terminal × D_thres), it can be accurately inferred that the cause of the network scan failure is "insufficient communication quality" or "network congestion."

[0646] On the other hand, when Model 3 is set, in steps S1445, S1447 to S1449, the D-score D_3 is used to distinguish between group CLS2 in which the network scan has failed due to "insufficient communication quality" and group CLS3 in which the network scan has failed due to "network congestion." Therefore, when DS_count=0 is not true (i.e., DS_count≠0), the count value DS_count, which is the number of D-scores D_3 smaller than "0," is (N terminal × D_thres), it can be accurately inferred that the cause of the network scan failure is "inadequate communication quality" or "network congestion." Also, since it is clear whether DS_count=0 or not, it can be accurately inferred that DS_count=0 (i.e., the network scan is "normal") and DS_count≠0 (i.e., the network scan failed due to "inadequate communication quality" or "network congestion").

[0647] Therefore, whether Model 2 or Model 3 is used, it is possible to accurately estimate whether a network scan is “normal” or “failed”. When it is estimated that a network scan is “failed”, it is further possible to accurately estimate that the cause of the failure of the network scan is “insufficient communication quality” or “network congestion”.

[0648] Furthermore, when it is determined in step S1445 that DS_count=0, the network scan is not immediately assumed to be "normal", but is assumed to be either "normal", a failure due to "port closed", or a failure due to "no network connection" depending on the number of ports for which a scan response was received for each terminal device port (see steps S1452 to S1455, S1457). Therefore, by detecting the presence or absence of a scan response for each port, it is possible to accurately estimate whether the network scan was "normal", a failure due to "port closed", or a failure due to "no network connection".

[0649] Furthermore, if the process proceeds from "YES" in step S1447 to step S1449, in step S1449, a D score not smaller than "0" (i.e., "0" or more and (N terminal For each terminal device m whose network scan result has not been estimated, but whose network scan result has not been estimated after step S1449, b Since steps S1452 to S1457 in FIG. 42 are executed for each of the D scores, the D score is not smaller than "0" (i.e., "0" or more and (N terminal The result of the network scan for a terminal device having a D score smaller than (D score × D_thres) is estimated to be either “normal”, the cause of the network scan failure is “closed port”, or the cause of the network scan failure is “no network connection” (see steps S1454, S1455, S1457).

[0650] In addition, the steps consisting of "YES" at step S1445 → "NO" at S1451 → "NO" at S1452 → "YES" at S1453 → S1454, and the steps consisting of "NO" at step S1445 → S1447 to S1449, correspond to estimating that the network scan is successful or unsuccessful based on the D score D_2 or the D score D_3, and when it is estimated that the network scan is unsuccessful, estimating that the network scan has failed due to a cause arising from the communication channel with the terminal device.

[0651] In the embodiment of the present invention, the operation of estimating the result of a network scan may be realized by software. In this case, the network scanning device 1 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The ROM stores a program Prog_A consisting of the steps of the flowchart shown in FIG. 33 (including the flowcharts shown in FIG. 34 to FIG. 44, and the flowchart shown in FIG. 45 or the flowchart shown in FIG. 46).

[0652] The CPU reads out the program Prog_A from the ROM, and executes the read out program Prog_A to perform an operation of estimating the results of the network scan. The RAM temporarily stores the various calculation results described above.

[0653] Furthermore, the program Prog_A may be distributed in a form recorded on a recording medium such as a CD or a DVD. When the recording medium on which the program Prog_A is recorded is attached to a computer, the computer reads out the program Prog_A from the recording medium, executes it, and performs an operation of estimating the results of the network scan.

[0654] Therefore, the recording medium on which the program Prog_A is recorded is a computer-readable recording medium.

[0655] In the above, it has been explained that when classifying terminal devices having communication characteristics CMbest into groups CLS1 to CLS3, the estimation means 16 learns an LDA algorithm using scan data when the network scan is “network congested” (communication characteristics CMbest when “network is congested”), and classifies terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm. However, in an embodiment of the present invention, this is not limited to this, and the estimation means 16 may learn an LDA algorithm using scan data when the result desired to be estimated most accurately is obtained from the results of the network scan (= either “normal”, “network congested”, or “inadequate communication quality”), and classify terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm.

[0656] More specifically, when "normal" is the result that the estimation means 16 desires to estimate with the highest accuracy, it learns an LDA algorithm using scan data when the network scan is "normal" (communication characteristics CMbest when "normal"), and classifies terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm.

[0657] In addition, when "network is congested" is the result that the estimation means 16 desires to estimate most accurately, it learns an LDA algorithm using scan data when the network scan results in "network is congested" (communication characteristics CMbest when "network is congested"), and classifies terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm.

[0658] Furthermore, when "inadequate communication quality" is the result that the estimation means 16 desires to estimate most accurately, it learns an LDA algorithm using scan data when the network scan results in "inadequate communication quality" (communication characteristics CMbest when the communication quality is "inadequate"), and classifies terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm.

[0659] In this way, the estimation means 16 may determine the scan data that is the teacher data according to the result that is desired to be estimated most accurately, and may learn an LDA algorithm using the determined scan data, and may classify terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm.

[0660] In the above, the network scan is performed at the scan timing t S (or scan timing t' S ), but in the embodiment of the present invention, the network scan is not limited to this, and the network scan may be performed every scan period T S The scan may be performed in units of _total, and the scan period T S The network scan may be performed over multiple scan timings within _total. Therefore, in the embodiment of the present invention, it is sufficient that the network scan is estimated as being "normal" or "failed" by the above-mentioned method using communication characteristics acquired by executing the network scan over multiple scan timings, and when it is estimated that the network scan is "failed", the cause of the network scan "failure" is further estimated.

[0661] Figure 49 shows the scan response delay RTT m In FIG. 49, the vertical axis represents the scan response delay RTT. m The horizontal axis represents the scan response delay RTT. mIn FIG. 49, (a) shows the scan response delay RTT obtained in a communication environment where the result of the network scan is “normal”. m (b) shows the probability density function of the scan response delay RTT obtained in a communication environment where the network scan result is “poor communication quality”. m (c) shows the probability density function of the scan response delay RTT obtained in a communication environment where the result of the network scan is “network congestion”. m (d) shows the probability density function of the scan response delay RTT obtained in a communication environment where the result of the network scan is “unknown”. m The probability density function of

[0662] Referring to FIG. 49, the scan response delay RTT obtained in a communication environment where the network scan result is “normal” m Probability density function of (a) and scan response delay RTT obtained in a communication environment where the network scan result is “poor communication quality” m Probability density function (b) of the scan response delay RTT obtained in a communication environment where the network scan result is “network congestion” m The probability density functions (c) of are mutually different.

[0663] Scan Response Delay RTT m When the probability density function of is the probability density function shown in FIG. 49(a), the sample data used to estimate the network scan result (= “normal”) is the standard sample data D_standard_1, and the scan response delay RTT m When the probability density function of is the probability density function shown in FIG. 49(b), the sample data used to estimate the network scan result (= “inadequate communication quality”) is the standard sample data D_standard_2, and the scan response delay RTT mWhen the probability density function of is the probability density function shown in FIG. 49(c), the sample data used to estimate the result of the network scan (=“network congestion”) is the standard sample data D_standard_3.

[0664] Then, the scan response delay RTT shown in (d) of FIG. m The probability density function of is the scan response delay RTT shown in Figure 49(a). m is most similar to the probability density function of

[0665] Therefore, according to steps S129 to S139 shown in FIG. 36 and FIG. 37, the scan response delay RTT shown in FIG. 49(d) is m The scan response delay RTT that is most similar to the probability density function of m The probability density function of is shown in (a) to (c) of the three scan response delay RTTs in Figure 49. m By detecting the probability density function of m The scan response delay RTT that is most similar to the probability density function of m It was found that the probability density function of can be accurately detected.

[0666] In addition, the scan response delay RTT shown in (d) of FIG. m The scan response delay RTT that is most similar to the probability density function of m Probability density function of the scan response delay RTT shown in (a) of Figure 49 m If the probability density function of the network scan is detected, the result of the network scan may be assumed to be “normal”. However, when the result of the network scan is estimated by the estimation algorithm ALG, the scan response delay RTT m In addition, communication characteristics CMbest m Since the results of the network scan are estimated using the above, it is not possible to say with certainty that the estimated results of the network scan will always be "normal."

[0667] Therefore, the estimation algorithm ALG is trained using initial sample data D_sample_initial generated by combining standard sample data D_standard_1 used when the network scan result is estimated to be “normal”, standard sample data D_standard_2 used when the network scan result is estimated to be “inadequate communication quality”, and standard sample data D_standard_3 used when the network scan result is estimated to be “network congestion”, and tentative sample data D_sample_tentative_1 to D_sample_tentative_U generated based on the initial sample data D_sample_initial, and a validation set (= subset V 1’ ~V U’ The estimation algorithm ALG is estimated using the estimation data (=consisting of all of subsets TD1 to TD4) and the test data (=consisting of all of subsets TD1 to TD4), and the estimation results of the network scan are evaluated using the evaluation indexes EVD, nEVD, and tEVD, thereby updating the estimation algorithm ALG to approach the optimal algorithm (see the flowcharts shown in Figures 36 to 38).

[0668] Therefore, by estimating the results of a network scan using the estimation algorithm ALG, which is close to the optimal algorithm, the results of a network scan can be estimated with the highest accuracy.

[0669] 36 to 38 are completed, the execution of the flowcharts shown in Fig. 40 to 42 is completed in step S161 of Fig. 38, so the result of the network scan is estimated to be any one of "normal", "inadequate communication quality", "network congestion", "port closed" and "no connection". If the estimated result of the network scan is "inadequate communication quality" or "network congestion", a rescan list is created in step S12 of Fig. 33, and then a network scan is performed based on the rescan list in step S4. As a result, even if the estimated result of the network scan is "inadequate communication quality" or "network congestion", the estimated result of the network scan may become "normal" by rescanning, and the result of the network scan can be accurately estimated.

[0670] In an embodiment of the present invention, the inference result of a network scan consisting of “normal” constitutes the “first inference result,” the inference result of a network scan consisting of “poor communication quality” constitutes the “second inference result,” and the inference result of a network scan consisting of “network congestion” constitutes the “third inference result.” And the first inference result, the second inference result, and the third inference result constitute the “multiple known inference results.”

[0671] In addition, in this embodiment of the present invention, the standard sample data D_standard_1 constitutes the "first standard sample data", the standard sample data D_standard_2 constitutes the "second standard sample data", and the standard sample data D_standard_3 constitutes the "third standard sample data".

[0672] Furthermore, in an embodiment of the present invention, the first estimation result may be a result estimated by the estimation algorithm ALG using the standard sample data D_standard_1, the second estimation result may be a result estimated by the estimation algorithm ALG using the standard sample data D_standard_2, and the third estimation result may be a result estimated by the estimation algorithm ALG using the standard sample data D_standard_3. Also, each of the first estimation result, the second estimation result, and the third estimation result may be a result estimated by an algorithm other than the estimation algorithm ALG.

[0673] In the embodiment of the present invention, the threshold value Corr r constitutes the "first threshold" and the threshold S r constitutes the "second threshold" and the threshold (N terminal ×D_thres) constitutes the "third threshold value."

[0674] In the embodiment of the present invention, the communication characteristic CMbest constitutes a "first communication characteristic" and the correlation coefficient r ij is the threshold Corr r The communication characteristics extracted in descending order of score Q constitute "D (D is a positive integer) second communication characteristics", and the communication characteristics extracted in descending order of score Q constitute "K (K is a positive integer) second communication characteristics".

[0675] Furthermore, in this embodiment of the invention, the D-score or count value DS_count constitutes a "discrimination score."

[0676] Furthermore, in this embodiment of the present invention, the D score D_2 constitutes a "first discriminant score", the D score D_3 constitutes a "second discriminant score", and the D score D_1 constitutes a "third discriminant score".

[0677] Furthermore, in the embodiment of the present invention, the packet loss rate PLR ​​shown in FIG. m , Jitter JT m, maximum scan delay DLmax m and scan minimum delay DLmin m is the correlation coefficient r ij is the threshold Corr r The first communication characteristic is a second communication characteristic having a number D (D is a positive integer) greater than the first communication characteristic.

[0678] Furthermore, in the embodiment of the present invention, the above-mentioned K communication characteristics constitute "K second communication characteristics."

[0679] Furthermore, in this embodiment of the present invention, group CLS1 constitutes a "first group" in which the network scan is normal, and groups CLS2 and CLS3 constitute a "second group" and a "third group," respectively, in which the network scan fails due to reasons arising from the communication channel with the terminal device.

[0680] Furthermore, in this embodiment of the present invention, Model 2 constitutes a "first model" and Model 3 constitutes a "second model."

[0681] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not by the description of the embodiments described above, and is intended to include all modifications within the meaning and scope of the claims. [Industrial Applicability]

[0682] The present invention is applicable to a network scanning device, a program to be executed by a computer, and a computer-readable recording medium on which the program is recorded. [Explanation of symbols]

[0683] 1 network scanning device, 10 communication system, 11 network scanner, 12 scan analysis manager, 13 host information database, 14 extraction means, 15 calculation means, 16 estimation means, 17 scan scheduler, 20 scanned network, 30 user.

Claims

1. a scanning means for executing a network scan in a first communication environment in which the results of a network scan for M (M is an integer equal to or greater than 2) terminal devices are unknown, and acquiring a first scan response delay of the network scan for each of the M terminal devices; a detection means for executing a detection process for detecting, as a similar scan response delay, M' second scan response delays that are most similar to the M first scan response delays among a plurality of sets of M' (M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are a plurality of known estimated results; a generating means for acquiring, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating the known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generating similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays; an update means for training an estimation algorithm for estimating the result of the network scan executed in the first communication environment based on the similar standard sample data, and updating the estimation algorithm using an evaluation index so as to approach an optimal algorithm; an estimation means for estimating a result of a network scan executed in the first communication environment by using the estimation algorithm updated by the update means, the optimization algorithm comprises an algorithm for estimating a result of a network scan such that the estimated result of the network scan matches the known estimated result; A network scanning device, wherein the evaluation index is an index for evaluating the accuracy of an estimation result when the result of a network scan is estimated using the trained algorithm.

2. The generating means further generates training data and test data from the similar standard sample data, and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of each of a plurality of standard sample data corresponding to the plurality of known estimation results and each of which is composed of the M′ second scan response delays and the M′ optimal communication characteristics, and generates provisional sample data by replacing a portion of the initial sample data composed of standard sample data of the same type as the training set with a portion of the training set; 2. The network scanning device according to claim 1, wherein the update means updates the estimation algorithm with the provisional sample data when a first evaluation index for evaluating an estimation result when the estimation algorithm is trained using the initial sample data and the network scan result is estimated by the trained estimation algorithm using the validation set is equal to or less than a second evaluation index for evaluating an estimation result when the estimation algorithm is trained using the provisional sample data and the network scan result is estimated by the trained estimation algorithm using the validation set.

3. the plurality of known inference results include a first inference result indicating that the result of the network scan is normal, a second inference result indicating that the network scan has failed due to insufficient communication quality, and a third inference result indicating that the network scan has failed due to network congestion; The plurality of standard sample data are first standard sample data, which is standard sample data for estimating that the result of the network scan is the first estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is normal; second standard sample data, which is standard sample data for estimating that the result of the network scan is the second estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which network communication quality is insufficient; and third standard sample data, which is standard sample data for estimating that the result of the network scan is the third estimation result, the third standard sample data being composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is congested, the first estimation result is a result estimated by the estimation algorithm using the first standard sample data; the second estimation result is a result estimated by the estimation algorithm using the second standard sample data, 3. The network scanning device according to claim 2, wherein the third estimation result is a result estimated by the estimation algorithm using the third standard sample data.

4. the generating means replaces a part of the sample data, which is made of standard sample data of the same type as the training set, among the initial sample data with one of a plurality of subset data of the training set, for all of the plurality of subset data, to generate a plurality of provisional sample data constituting the provisional sample data; The update means includes a first process for training the estimation algorithm using the initial sample data, and generating a first confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the trained estimation algorithm using the validation set; a second process for training the estimation algorithm using one of a plurality of provisional sample data constituting the provisional sample data, and generating a second confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the trained estimation algorithm using the validation set, for all of the plurality of provisional sample data; and a first evaluation index calculated based on the first confusion matrix and a second confusion matrix calculated based on the plurality of second confusion matrices.

4. The network scanning device according to claim 2, further comprising: a third process for updating the estimation algorithm with optimal sample data consisting of one provisional sample data among the plurality of provisional sample data when a maximum evaluation index of the plurality of second evaluation indexes is obtained, the optimal sample data being greater than the first evaluation index and being based on a plurality of second evaluation indexes obtained by the estimation; a fourth process for generating a third confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the updated estimation algorithm using the test data after the third process; and a fifth process for approving the updated estimation algorithm and the optimal sample data when a third evaluation index calculated based on the third confusion matrix is ​​equal to or greater than the maximum evaluation index.

5. 5. The network scanning device according to claim 4, wherein when the third evaluation index is smaller than the maximum evaluation index, the update means repeatedly executes the first process to the fifth process using the estimation algorithm updated with the optimal sample data until the third evaluation index becomes equal to or greater than the maximum evaluation index.

6. Further comprising a calculation means for calculating the evaluation index, Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur; 6. The network scanning device according to claim 4, wherein the calculation means calculates Precision, which is a ratio of a ratio actually classified into the first element among a ratio classified into the first element or the fourth element, and Recall, which is a ratio of a ratio actually classified into the first element among a ratio classified into the first element or the third element, and calculates 2×Precision / (Precision+Recall) based on the Precision and the Recall, based on the first confusion matrix to calculate the first evaluation index, executes the third calculation process based on the plurality of second confusion matrices to calculate the plurality of second evaluation indexes, and executes the third calculation process based on the third confusion matrix to calculate the third evaluation index.

7. Further comprising a calculation means for calculating the evaluation index, Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur; 6. The network scanning device according to claim 4, wherein the calculation means calculates the first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates the plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates the third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix.

8. The network scanning device according to any one of claims 1 to 7, wherein, in the detection process, the detection means detects, from the multiple sets of M' second scan response delays, the M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays based on a plurality of distributions of the multiple sets of M' second scan response delays and a distribution of the M first scan response delays, as the similar scan response delay.

9. 9. The network scanning device of claim 8, wherein the detection means, in the detection process, performs a first calculation process of calculating a first mode indicating a value of the first scan response delay most likely to occur based on the M first scan response delays and calculating a first mean absolute deviation which is an average value of absolute values ​​of differences between the M first scan response delays and the first mode, and after the first calculation process, calculates a second mode indicating a value of the second scan response delay most likely to occur based on the M' second scan response delays, calculates a second mean absolute deviation which is an average value of absolute values ​​of differences between the M' second scan response delays and the second mode, calculates a difference in mean absolute deviation consisting of an absolute value of a difference between the first mean absolute deviation and the second mean absolute deviation, and subtracts the difference in mean absolute deviation from 1 to calculate a similarity, for all of the plurality of known estimation results to obtain a plurality of similarities, and detects the M' second scan response delays which were used to calculate the highest similarity among the obtained plurality of similarities as the similar scan response delay.

10. A first step in which a scanning means executes a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquires a first scan response delay of the network scan for each of the M terminal devices; a second step of executing a detection process in which the detection means detects, as similar scan response delays, M' second scan response delays that are most similar to the M first scan response delays among a plurality of sets of M' (M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are a plurality of known estimated results; a third step in which a generating means acquires, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating the known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays; a fourth step in which an update means trains an estimation algorithm for estimating a result of a network scan executed in the first communication environment based on the similar standard sample data, and updates the estimation algorithm using an evaluation index so as to approach an optimal algorithm; a fifth step in which an estimation means estimates a result of a network scan performed in the first communication environment by using the estimation algorithm updated in the fourth step; the optimization algorithm comprises an algorithm for estimating a result of a network scan such that the estimated result of the network scan matches the known estimated result; A program to be executed by a computer, wherein the evaluation index is an index for evaluating the accuracy of an estimation result when the result of a network scan is estimated by the trained algorithm.

11. In the third step, the generating means further generates training data and test data from the similar standard sample data, and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of a plurality of standard sample data that correspond to the plurality of known estimation results and each of which is composed of the M′ second scan response delays and the M′ optimal communication characteristics, and generates temporary sample data by replacing a portion of the initial sample data composed of standard sample data of the same type as the training set with a portion of the training set, 11. The program for causing a computer to execute the program according to claim 10, wherein in the fourth step, when a first evaluation index for evaluating an estimation result obtained when the estimation algorithm is trained using the initial sample data and the network scan result is estimated by the trained estimation algorithm using the validation set is equal to or lower than a second evaluation index for evaluating an estimation result obtained when the estimation algorithm is trained using the provisional sample data and the network scan result is estimated by the trained estimation algorithm using the validation set, the update means updates the estimation algorithm with the provisional sample data.

12. the plurality of known inference results include a first inference result indicating that the result of the network scan is normal, a second inference result indicating that the network scan has failed due to insufficient communication quality, and a third inference result indicating that the network scan has failed due to network congestion; The plurality of standard sample data are first standard sample data, which is standard sample data for estimating that the result of the network scan is the first estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is normal; second standard sample data, which is standard sample data for estimating that the result of the network scan is the second estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which network communication quality is insufficient; and third standard sample data, which is standard sample data for estimating that the result of the network scan is the third estimation result, the third standard sample data being composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is congested, the first estimation result is a result estimated by the estimation algorithm using the first standard sample data; the second estimation result is a result estimated by the estimation algorithm using the second standard sample data, The program for causing a computer to execute the program according to claim 11 , wherein the third estimation result is a result estimated by the estimation algorithm using the third standard sample data.

13. In the third step, the generating means replaces a part of the sample data of the initial sample data, which is made of standard sample data of the same type as the training set, with one of a plurality of subset data of the training set, for all of the plurality of subset data, to generate a plurality of provisional sample data constituting the provisional sample data; The fourth step includes: a first sub-step in which the update means trains the estimation algorithm using the initial sample data, and generates a first confusion matrix based on an estimation result when the trained estimation algorithm estimates a result of a network scan using the validation set; a second sub-step in which the update means trains the estimation algorithm using a first set of provisional sample data among a plurality of provisional sample data constituting the provisional sample data, and generates a second confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the trained estimation algorithm using the validation set; a third sub-step of the update means setting the second evaluation index to the first evaluation index when the second evaluation index calculated based on the second confusion matrix is ​​greater than the first evaluation index calculated based on the first confusion matrix, and updating the estimation algorithm with the first provisional sample data; a fourth sub-step in which the update means does not set the second evaluation index to the first evaluation index when the second evaluation index is not greater than the first evaluation index, and selects second provisional sample data different from the first provisional sample data from a plurality of provisional sample data; a fifth substep in which the updating means repeatedly executes the second substep to the fourth substep for all of the plurality of virtual sample data; a sixth sub-step of generating a third confusion matrix based on an estimation result when the update means estimates a result of a network scan by the estimation algorithm finally updated in the fifth sub-step using the test data; 13. The program for causing a computer to execute the program according to claim 11 or 12, further comprising a seventh substep in which the updating means approves the finally updated estimation algorithm when a third evaluation index calculated based on the third confusion matrix is ​​equal to or greater than the first evaluation index finally obtained in the fifth substep, and the first provisional sample data when the second evaluation index constituting the finally obtained first evaluation index is obtained.

14. The fourth step includes:

14. The program for causing a computer to execute the program according to claim 13, further comprising an eighth substep in which, when the third evaluation index is smaller than the first evaluation index finally obtained in the fifth substep, the updating means repeatedly executes the second substep to the seventh substep using the finally updated estimation algorithm until the third evaluation index becomes equal to or larger than the first evaluation index finally obtained in the fifth substep.

15. The calculation means further causes the computer to execute a sixth step of calculating the evaluation index; Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur; The calculation means, in the sixth step, calculates Precision, which is a ratio of the ratio of the ratio classified into the first element or the fourth element to the ratio actually classified into the first element, and Recall, which is a ratio of the ratio of the ratio classified into the first element or the third element to the ratio actually classified into the first element, and calculates 2×Precision / (Precision+Recall) based on the Precision and the Recall based on the first confusion matrix to calculate the first evaluation index, and calculates the multiple second confusion matrices to calculate the multiple second evaluation indexes, and calculates the third evaluation index based on the third confusion matrix. The program for causing a computer to execute the program according to claim 13 or 14.

16. The calculation means further causes the computer to execute a sixth step of calculating the evaluation index; Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur; 15. The program for causing a computer to execute the program according to claim 13 or 14, wherein, in the sixth step, the calculation means calculates the first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates the plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates the third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix.

17. 17. A program for causing a computer to execute the program described in any one of claims 10 to 16, wherein the detection means, in the detection process of the second step, detects the M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays from the multiple sets of M' second scan response delays as the similar scan response delay based on multiple distributions of the multiple sets of M' second scan response delays and the distribution of the M first scan response delays.

18. The detection means, in the detection process of the second step, performs a first calculation process of calculating a first mode indicating a value of the first scan response delay most likely to occur based on the M first scan response delays, and calculating a first mean absolute deviation that is an average value of absolute values ​​of differences between the M first scan response delays and the first mode, and after the first calculation process, calculates a second mode indicating a value of the second scan response delay most likely to occur based on the M' second scan response delays, and calculates a first mean absolute deviation that is an average value of absolute values ​​of differences between the M first scan response delays and the first mode. a second mean absolute deviation that is an average of absolute values ​​of differences between the first mean absolute deviation and the second mean absolute deviation, a difference in mean absolute deviation that is an absolute value of the difference between the first mean absolute deviation and the second mean absolute deviation, and a similarity calculated by subtracting the difference in mean absolute deviation from 1.

19. A computer-readable recording medium having the program according to any one of claims 10 to 18 recorded thereon.

Images