Network scanning device, program for causing a computer to execute the program, computer-readable recording medium having the program recorded thereon, and data production method

Abstract

To provide a network scanning device capable of suppressing misidentification of the result of network scanning executed in a communication environment in which the result of network scanning is unknown.SOLUTION: Calculation means 16 calculates a plurality of sets of M' baseline scan response delays on the basis of a plurality of sets of M' scan response delays obtained in a communication environment in which network scan results are known, calculates M baseline scan response delays on the basis of the M scan response delays obtained in the communication environment in which the network scan result is unknown, and calculates a plurality of similarity ratios with the plurality of sets of M' baseline scan response delays and M baseline scan response delays. Updating means 14 aggregates the M' similar scan response delays and the M scan response delays when the highest similarity rate among the plurality of similarity rates is obtained to update the M' similar scan response delays.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a network scanning device, a program to be executed by a computer, a computer-readable recording medium on which the program is recorded, and a data production method. [Background technology]

[0002] With the practical application of the fifth-generation mobile communication (5G) system, many devices, such as mobile terminals, computers, sensors, and webcams, will be connected to the Internet. Such mobile terminals and computers connected to the Internet are called IoT (Internet of Things) devices.

[0003] In the IoT era, the number of IoT devices connected to communication networks is expected to increase rapidly. IoT devices that communicate wirelessly are expected to be used illegally without sufficient security, leading to an increased risk of being exploited in cyberattacks such as DDoS (Distributed Denial of Service) attacks.

[0004] To avoid such situations, it is necessary to conduct network scans of IoT devices connected to communication networks and investigate whether security measures are in place for each IoT device.

[0005] Conventionally, there is known a network scan described in Non-Patent Document 1. The network scan described in Non-Patent Document 1 scans applications in the IPv4 space, and is performed by the following method.

[0006] First, a port scan is performed on the IPv4 space to check for responses. Then, for hosts that respond, a handshake is performed using various protocols (such as HTTP (Hypertext Transfer Protocol), HTTPS (Hypertext Transfer Protocol Secure), and POP3 (Post Office Protocol version 3)) to obtain messages. Keywords useful for vulnerability assessment are then extracted from the obtained messages. Metadata and tag information are then assigned to the keywords. The keywords with the assigned metadata and tag information are then registered in a database as structured data. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Z. Durumeric, et al., “A Search Engine Backed by Internet-Wide Scanning”, CCS15, October 12-16, 2015. [Non-patent document 2] Katsunari Yoshioka, Kosuke Murakami, and Tsutomu Matsumoto, "Network Scanning Method for Detecting Malware-Infected Hosts and Automatic Generation of Detection Signatures," Transactions of the Information Processing Society of Japan, Vol. 51, No. 9, pp. 1633-1644 (Sep. 2010). [Non-patent document 3] GF Riley and TR Henderson, “The ns-3 Network Simulator,” In: K. Wehrle, M. Gunes M., and Gross J. (eds) Modeling and Tools for Network Simulation. Springer, Berlin, Heidelberg, 2010. [Non-patent document 4] 3GPP TS 36.814 v9.2.0-“A.2 System simulation scenarios”. [Non-patent document 5] B. Li, and L. Han, “Distance Weighted Cosine Similarity Measure for Text Classification,” In: Yin H. et al. (eds) Intelligent Data Engineering and Automated Learning - IDEAL 2013. IDEAL 2013. Lecture Notes in Computer Science, vol 8206. Springer, Berlin, Heidelberg. [Non-patent document 6] A. Heidarian and MJ Dinneen, “A Hybrid Geometric Approach for Measuring Similarity Level Among Documents and Document Clustering,” 2016 IEEE Second International Conference on Big Data Computing Service and Applications (BigDataService), Oxford, 2016, pp. 142-151, DOI: 10.1109 / BigDataService.2016.14. [Non-Patent Document 7] M. Mihajlovic and N. Xiong, “Finding the most similar textual documents using Case-Based Reasoning,” arXiv preprint arXiv:1911.00262, 2019. [Online]. https: / / arxiv.org / abs / 1911.00262. Summary of the Invention [Problem to be solved by the invention]

[0008] When a network scan is performed in a communication environment where the results of the network scan are unknown, misidentification of the network scan results may lead to incorrect estimation of the cause of the network scan failure. This misidentification of the network scan results may occur due to the following factors: (Factor 1) A similarity mismatch occurs because the magnitude, direction, and angle of a sample of a scan response, etc., obtained when a network scan is performed in a communication environment where the results of the network scan are unknown is very similar to a sample of a scan response, etc., obtained when a network scan is performed in a communication environment where the results of the network scan are known. (Factor 2) The difference in size between samples of scan responses, etc. obtained when a network scan is performed in a communication environment where the results of the network scan are unknown is similar to the difference in size between samples of scan responses, etc. obtained when a network scan is performed in a communication environment where the results of the network scan are known. (Factor 3) When compared with multiple known network scan results, the range of scan response delays obtained when performing a network scan in a communication environment where the network scan results are unknown overlap. (Factor 4) The sample size of scan responses, etc. obtained when a network scan is performed in a communication environment where the results of the network scan are unknown is different from the sample size of scan responses, etc. obtained when a network scan is performed in a communication environment where the results of the network scan are known.

[0009] Therefore, according to an embodiment of the present invention, a network scanning device is provided that can suppress misidentification of the results of a network scan executed in a communication environment where the results of the network scan are unknown.

[0010] Furthermore, according to an embodiment of the present invention, a program for causing a computer to execute suppression of misidentification of the results of a network scan executed in a communication environment where the results of the network scan are unknown is provided.

[0011] Furthermore, according to an embodiment of the present invention, a computer-readable recording medium is provided that stores a program for causing a computer to suppress misidentification of the results of a network scan performed in a communication environment where the results of the network scan are unknown.

[0012] Furthermore, according to an embodiment of the present invention, a data production method is provided for producing data used to suppress misidentification of the results of a network scan performed in a communication environment where the results of the network scan are unknown. [Means for solving the problem]

[0013] (Configuration 1) According to an embodiment of the present invention, a network scanning device includes a scanning means, a calculating means, a detecting means, and a determining means. The scanning means executes a network scan for M (M is an integer equal to or greater than 2) terminal devices in a first communication environment in which the results of the network scan are unknown, and obtains a first scan response delay of the network scan for all of the M terminal devices. The calculation means, when M first scan response delays are acquired by the scanning means, executes a first calculation process in which M' baseline scan response delays are calculated for each of multiple sets of M' (M' is an integer greater than or equal to 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are multiple known estimated results, and calculates M baseline scan response delays for the M first scan response delays, and a second calculation process in which, based on the multiple sets of M' baseline scan response delays and the M baseline scan response delays calculated in the first calculation process, the calculation means calculates similarity rates between the M' baseline scan response delays and the M baseline scan response delays for all of the multiple sets of M' baseline scan response delays to calculate multiple similarity rates. When the calculation means executes the second calculation process, the detection means detects the largest similarity rate among the multiple similarity rates calculated in the second calculation process. When the detection means detects the highest similarity rate, the determination means determines that the unknown estimated result of the network scan when the M first scan response delays are acquired is a known estimated result corresponding to the M' second scan response delays having the highest similarity rate. The baseline scan response delay is a scan response delay whose distribution is changed based on a lower limit and an upper limit so as to more easily produce similarities or differences in distribution between the M first scan response delays and the M' second scan response delays.

[0014] (Configuration 2) In configuration 1, the network scanning device further includes an updating means that performs an aggregation process to aggregate M' second scan response delays and M first scan response delays when the largest similarity rate is obtained, and updates the M' second scan response delays when the largest similarity rate is obtained by the aggregated M' second aggregated scan response delays in the aggregation process.

[0015] (Configuration 3) In configuration 2, when there are multiple sets of M first scan response delays when the highest similarity rate is obtained, the update means selects any one set of M first scan response delays from the multiple sets of M first scan response delays in the aggregation process, and aggregates the selected set of M first scan response delays with M' second scan response delays when the highest similarity rate is obtained.

[0016] (Configuration 4) In configuration 2 or 3, the update means, in the aggregation process, adds, to the M' second scan response delays when the highest similarity rate is obtained, first scan response delays among the M first scan response delays that are not included in the M' second scan response delays when the highest similarity rate is obtained, thereby aggregating the M' second scan response delays when the highest similarity rate is obtained and the M first scan response delays.

[0017] (Configuration 5) In any of configurations 1 to 4, the calculation means, in a first calculation process, calculates baseline scan response delays by subtracting a predetermined value from an average of the plurality of scan response delays and by adding a predetermined value to the average, for all of the plurality of scan response delays to calculate multiple baseline scan response delays, and performs this process on all of the plurality of scan response delays to calculate multiple sets of M' baseline scan response delays and M baseline scan response delays. The predetermined value is calculated based on relative variations in the plurality of scan response delays.

[0018] (Configuration 6) In configuration 5, in the first calculation process, the calculation means subtracts the inverse of the coefficient of variation in the distribution of the plurality of scan response delays from the scan response delay, multiplies the subtraction result by the standard deviation of the plurality of scan response delays, and divides the multiplication result by the number of the plurality of scan response delays to calculate a predetermined value.

[0019] (Configuration 7) In any one of configurations 1 to 6, the calculation means calculates a first mean absolute deviation that is the mean absolute deviation of M baseline scan response delays, and calculates a second mean absolute deviation that is the mean absolute deviation of M′ baseline scan response delays for all of the plurality of sets of M′ baseline scan response delays to calculate a plurality of second mean absolute deviations, and where X is the first mean absolute deviation and Y is any one of the plurality of second mean absolute deviations, the calculation means calculates 1 / (1+(YX) 2 The similarity rate is calculated by the formula (X / Y) for all of the second mean absolute deviations to calculate a plurality of similarity rates.

[0020] (Configuration 8) In any one of configurations 1 to 7, the network scanning device further includes a generating means, an algorithm updating means, and an estimating means. The generating means acquires, for all of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating a known estimation result from among a plurality of communication characteristics acquired together with one of the M' second scan response delays, and generates similar standard sample data including the acquired M' optimal communication characteristics and the M' second aggregate scan response delays. The algorithm updating means trains an estimation algorithm that estimates the results of a network scan performed in the first communication environment based on the similar standard sample data and updates the estimation algorithm using an evaluation index to approach the optimal algorithm. The estimation means estimates the results of a network scan performed in the first communication environment using the estimation algorithm updated by the algorithm updating means. The optimal algorithm comprises an algorithm that estimates the results of a network scan so that the estimated results match the known estimation results. The evaluation index is an index that evaluates the accuracy of the estimation results when the results of a network scan are estimated using the trained algorithm.

[0021] (Configuration 9) Furthermore, according to an embodiment of the present invention, the program includes a first step in which a scanning means executes a network scan in a first communication environment in which the results of the network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquires a first scan response delay of the network scan for all of the M terminal devices; a second step in which the calculation means, when M first scan response delays are acquired by the scanning means, executes a first calculation process in which the calculation means calculates M' baseline scan response delays for each of multiple sets of M' (M' is an integer of 2 or greater) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are multiple known estimated results, and calculates M baseline scan response delays for the M first scan response delays; and a second calculation process in which the calculation means calculates similarity rates between the M' baseline scan response delays and the M baseline scan response delays for all of the multiple sets of M' baseline scan response delays, based on the multiple sets of M' baseline scan response delays and the M baseline scan response delays calculated in the first calculation process, to calculate multiple similarity rates; a third step in which, when the second calculation process is executed by the calculation means, the detection means detects the largest similarity rate among the plurality of similarity rates calculated in the second calculation process; and a fourth step in which, when the highest similarity rate is detected in the third step, the determining means determines that the unknown estimation result of the network scan when the M first scan response delays are acquired is a known estimation result corresponding to the M' second scan response delays having the highest similarity rate. The baseline scan response delay is a scan response delay whose distribution has been changed based on a lower limit and an upper limit so as to more easily produce similarities or differences in distribution between the M first scan response delays and the M' second scan response delays.

[0022] (Configuration 10) In configuration 9, when the highest similarity rate is detected in the third step, the updating means causes the computer to further execute a fifth step of performing an aggregation process to aggregate M first scan response delays and M' second scan response delays when the highest similarity rate is obtained, and updating M' second scan response delays when the highest similarity rate is obtained using the M' second aggregated scan response delays aggregated in the aggregation process.

[0023] (Configuration 11) In configuration 10, when there are multiple sets of M first scan response delays when the highest similarity rate is obtained, the update means selects any one set of M first scan response delays from the multiple sets of M first scan response delays in the aggregation process, and aggregates the selected set of M first scan response delays with M' second scan response delays when the highest similarity rate is obtained.

[0024] (Configuration 12) In configuration 10 or 11, the update means, in the aggregation process, adds, to the M' second scan response delays when the highest similarity rate is obtained, first scan response delays among the M first scan response delays that are not included in the M' second scan response delays when the highest similarity rate is obtained, thereby aggregating the M' second scan response delays when the highest similarity rate is obtained and the M first scan response delays.

[0025] (Configuration 13) In any of configurations 9 to 12, the calculation means, in the first calculation process, performs a process of subtracting a predetermined value from an average of the plurality of scan response delays and calculating, as a baseline scan response delay, the result of subtracting a predetermined value from the average of the plurality of scan response delays and the result of adding a predetermined value to the average of the plurality of scan response delays, for all of the plurality of scan response delays to calculate multiple baseline scan response delays, and performs this process for all of M first scan response delays and multiple sets of M' second scan response delays to calculate multiple sets of M' baseline scan response delays and M baseline scan response delays, and the predetermined value is calculated based on the relative variation of the plurality of scan response delays.

[0026] (Configuration 14) In configuration 13, in the first calculation process, the calculation means subtracts the inverse of the coefficient of variation in the distribution of the plurality of scan response delays from the scan response delay, multiplies the subtraction result by the standard deviation of the plurality of scan response delays, and divides the multiplication result by the number of the plurality of scan response delays to calculate a predetermined value.

[0027] (Configuration 15) In any of configurations 9 to 14, the calculation means calculates a first mean absolute deviation that is the mean absolute deviation of M baseline scan response delays, and calculates a second mean absolute deviation that is the mean absolute deviation of M′ baseline scan response delays for all of the plurality of sets of M′ baseline scan response delays to calculate a plurality of second mean absolute deviations, and where X is the first mean absolute deviation and Y is one of the plurality of second mean absolute deviations, the calculation means calculates 1 / (1+(YX) 2 The similarity rate is calculated by the formula (X / Y) for all of the second mean absolute deviations to calculate a plurality of similarity rates.

[0028] (Configuration 16) In any of configurations 9 to 15, the program includes a sixth step in which the generating means acquires, for all of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating a known estimation result from among the plurality of communication characteristics acquired together with one of the M' second scan response delays, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second aggregated scan response delays; a seventh step in which the algorithm updating means trains an estimation algorithm that estimates the results of the network scan executed in the first communication environment based on the similar standard sample data, and updates the estimation algorithm using the evaluation index so as to approach an optimal algorithm; The method further causes the computer to execute an eighth step in which the estimation means estimates the results of a network scan performed in the first communication environment using the estimation algorithm updated by the algorithm update means. The optimal algorithm is an algorithm that estimates the results of a network scan so that the estimated results match known estimation results. The evaluation index is an index that evaluates the accuracy of the estimation results when the results of a network scan are estimated using the trained algorithm.

[0029] (Configuration 17) Furthermore, according to an embodiment of the present invention, a recording medium is a computer-readable recording medium on which the program according to any one of the ninth to sixteenth configurations is recorded.

[0030] (Configuration 18) Furthermore, according to an embodiment of the present invention, a data production method is a data production method for producing data used to estimate a result of a network scan, the method comprising: a first step of performing a network scan in a first communication environment in which the results of the network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquiring a first scan response delay of the network scan for all of the M terminal devices; a second step of executing a first calculation process, when M first scan response delays are acquired in the first step, to calculate M' baseline scan response delays for each of multiple sets of M' (M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are multiple known estimated results, and to calculate M baseline scan response delays for the M first scan response delays; and a second calculation process, based on the multiple sets of M' baseline scan response delays and the M baseline scan response delays calculated in the first calculation process, to calculate multiple similarity rates between the M' baseline scan response delays and the M baseline scan response delays for all of the multiple sets of M' baseline scan response delays. When the second calculation process is executed in the second step, a third step is performed to detect the largest similarity rate among the plurality of similarity rates calculated in the second calculation process; a fourth step of performing an aggregation process when the highest similarity rate is detected in the third step, aggregating the M' second scan response delays and the M first scan response delays when the highest similarity rate is obtained, and updating the M' second scan response delays when the highest similarity rate is obtained using the M' second aggregated scan response delays aggregated in the aggregation process; and a fifth step of acquiring, for all of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating a known estimation result from among the multiple communication characteristics acquired together with one of the M' second scan response delays updated in the fourth step, and generating similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays.The baseline scan response delay is a scan response delay whose distribution has been changed based on a lower limit and an upper limit so as to more easily produce a similarity or difference in distribution between the M first scan response delays and the M' second scan response delays.

[0031] (Configuration 19) In configuration 18, in the aggregation process of the fourth step, among the M first scan response delays, first scan response delays that are not included in the M' second scan response delays when the highest similarity rate is obtained are added to the M' second scan response delays when the highest similarity rate is obtained, and the M' second scan response delays when the highest similarity rate is obtained and the M first scan response delays are aggregated.

[0032] (Configuration 20) In configuration 18 or 19, in the first calculation process of the second step, a process of calculating a baseline scan response delay by subtracting a predetermined value from an average of the plurality of scan response delays and an addition result by adding a predetermined value to the average of the plurality of scan response delays is performed for all of the plurality of scan response delays to calculate a plurality of baseline scan response delays, and this process is performed for all of M first scan response delays and multiple sets of M' second scan response delays to calculate M' baseline scan response delays and multiple sets of M baseline scan response delays, and the predetermined value is calculated based on the relative variation of the plurality of scan response delays.

[0033] (Configuration 21) In configuration 20, in the first calculation process of the second step, the reciprocals of the coefficients of variation of the plurality of scan response delays are subtracted from the scan response delay, the subtraction result is multiplied by the standard deviation of the plurality of scan response delays, and the multiplication result is divided by the number of the plurality of scan response delays to calculate a predetermined value.

[0034] (Configuration 22) In any of configurations 18 to 21, in the second calculation process of the second step, a first mean absolute deviation that is the mean absolute deviation of M baseline scan response delays is calculated, and a second mean absolute deviation that is the mean absolute deviation of M' baseline scan response delays is calculated for all of the multiple sets of M' baseline scan response delays to calculate multiple second mean absolute deviations, and where X is the first mean absolute deviation and Y is one of the multiple second mean absolute deviations, 1 / (1+(YX) 2 The similarity rate is calculated by the formula (X / Y) for all of the second mean absolute deviations to calculate a plurality of similarity rates.

[0035] (Configuration 23) In any of configurations 18 to 22, the data production method includes a sixth step of training an estimation algorithm that estimates the results of the network scan using similar standard sample data; a seventh step of estimating known results of the network scan using the estimation algorithm trained in the sixth step; an eighth step of determining whether the estimation result estimated in the seventh step matches a known estimation result; The fifth step is executed when it is determined in the eighth step that the estimation result estimated in the seventh step matches a known estimation result. [Effects of the Invention]

[0036] This can reduce misidentification of the results of a network scan performed in a communication environment where the results of the network scan are unknown. [Brief explanation of the drawings]

[0037] [Figure 1] 1 is a schematic diagram of a communication system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram of the network scanning device shown in FIG. [Figure 3]3 is a conceptual diagram of a correspondence table TBL1 stored in the host information database shown in FIG. 2. FIG. [Figure 4] FIG. 3 is a conceptual diagram of another correspondence table TBL2 stored in the host information database shown in FIG. 2. [Figure 5] FIG. 4 is a conceptual diagram of yet another correspondence table TBL3 stored in the host information database shown in FIG. [Figure 6] FIG. 1 is a conceptual diagram of a scan schedule. [Figure 7] FIG. 1 illustrates the time dependency of traffic. [Figure 8] FIG. 10 illustrates an example of normalized scan response delay. [Figure 9] FIG. 10 is a diagram showing the score zscr and baseline scan response delay bRTT calculated for one terminal device. [Figure 10] FIG. 10 is a diagram illustrating an example of a baseline scan response delay bRTT calculated by randomly selecting one scan response delay RTT. [Figure 11] FIG. 10 is a diagram illustrating an example of a baseline scan response delay bRTT calculated using all of a plurality of scan response delays RTT corresponding to a plurality of ports in one terminal device. [Figure 12] FIG. 10 is a diagram showing a z table. [Figure 13] FIG. 10 is a diagram showing the baseline scan response delay bRTT calculated using the score zβ. [Figure 14] FIG. 10 illustrates an example of a baseline scan response delay (bRTT). [Figure 15] FIG. 15 is a diagram showing the results of counting the number of samples of each value of the baseline scan response delay bRTT shown in FIG. 14. [Figure 16] FIG. 16 is a diagram illustrating the distribution of the baseline scan response delay bRTT shown in FIG. 15. [Figure 17] FIG. 15 is a diagram showing the results of changing some of the values ​​of the baseline scan response delay bRTT shown in FIG. 14. [Figure 18] FIG. 18 is a diagram illustrating the distribution of the baseline scan response delay bRTT shown in FIG. 17. [Figure 19] FIG. 19 is a diagram showing the distribution of baseline scan response delays (bRTT) when the scale of the horizontal axis in FIG. 18 is changed to "0.005" units. [Figure 20] FIG. 10 shows the correspondence between the region of baseline scan response latency and the number of samples generated. [Figure 21] FIG. 10 illustrates an exemplary distribution of the probability density function of the baseline scan response delay. [Figure 22] FIG. 10 is a diagram illustrating another example of baseline scan response delay bRTT. [Figure 23] FIG. 23 is a diagram illustrating the distribution of the baseline scan response delay bRTT shown in FIG. 22. [Figure 24] FIG. 24 is a diagram showing the distribution of baseline scan response delays (bRTT) when the scale of the horizontal axis in FIG. 23 is changed to "0.005" units. [Figure 25] FIG. 10 is a diagram showing the relationship between the similarity rate bsim and the mean absolute deviation Wknown shown in equation (12). [Figure 26] FIG. 10 is a diagram for explaining a method for detecting the largest similarity rate bsim_max. [Figure 27] FIG. 10 is a diagram illustrating a method for aggregating two similar sets. [Figure 28] FIG. 10 is a diagram for explaining a method of updating known scan response delays {RTTm'_known}_z (m'=1' to M', z=1 to 3). [Figure 29] FIG. 10 is a diagram showing a similarity rate bsim calculated using three types of scores zscr_c, zscr_1 to zscr_R, and zβ. [Figure 30] FIG. 10 is a diagram for explaining the problem that occurs when determining sets that are similar to a set SG. [Figure 31] FIG. 10 is a diagram showing the distribution of scan response delays RTT and baseline scan response delays bRTT for sets SE, SF, and SG. [Figure 32] FIG. 1 shows the similarity rate of two sets. [Figure 33]FIG. 10 is a diagram illustrating an example of scan response delays {RTTm′_known}_3 (m′=1′ to M′) and {RTTm_unknown}_1 (m=1 to M) of sets SE and SF. [Figure 34] FIG. 10 is a diagram showing a set SH obtained by aggregating sets SE and SF. [Figure 35] 35 is a diagram showing an example of scan response delay {RTTm'_known}_3 (m'=1' to M') of set SE shown in FIG. 34 and an example of scan response delay RTT of set SF that is not common between set SE and SF. FIG. [Figure 36] FIG. 10 is a diagram showing the scan response delay {RTTm_unknown}_2 (m=1 to M) of the set SG. [Figure 37] FIG. 10 shows the baseline scan response delays of sets SG and SH. [Figure 38] FIG. 10 is a diagram showing the distribution of scan response delays RTT and baseline scan response delays bRTT for sets SG and SH. [Figure 39] FIG. 10 is a diagram showing the similarity rate bsim between sets SC, SD, SH and set SG. [Figure 40] FIG. 10 illustrates a comparison of aggregating scan response delays with a non-aggregating scan response delays. [Figure 41] FIG. 10 is a diagram illustrating a correlation coefficient of communication characteristics. [Figure 42] FIG. 10 is a diagram illustrating another correlation coefficient of communication characteristics. [Figure 43] FIG. 10 is a diagram illustrating yet another correlation coefficient of communication characteristics. [Figure 44] FIG. 10 is a diagram illustrating yet another correlation coefficient of communication characteristics. [Figure 45] FIG. 10 is a diagram illustrating an example of original data when selecting communication characteristics CMbest. [Figure 46] FIG. 10 is a diagram for explaining a method for selecting communication characteristics CMbest. [Figure 47] FIG. 10 is a diagram for explaining a method for calculating a score of a communication characteristic. [Figure 48] FIG. 10 is a diagram illustrating scores of communication characteristics. [Figure 49]FIG. 10 is a diagram showing the calculation results of rank weights Rk. [Figure 50] FIG. 10 is a diagram showing a specific distribution of scan response delays. [Figure 51] FIG. 10 is a diagram showing a specific distribution of scan response delays. [Figure 52] FIG. 10 is a diagram showing a specific distribution of scan response delays. [Figure 53] FIG. 10 is a diagram showing the correspondence between PDF regions and the number of samples generated. [Figure 54] FIG. 10 is a diagram illustrating a specific distribution of a probability density function of a scan response delay. [Figure 55] FIG. 10 illustrates another exemplary distribution of scan response delays. [Figure 56] FIG. 1 shows data when training the estimation algorithm ALG. [Figure 57] This is a conceptual diagram of sample data when training the estimation algorithm ALG. [Figure 58] FIG. 10 is a conceptual diagram illustrating updating of virtual sample data for modeling. [Figure 59] 1 is a conceptual diagram of training and evaluation of the estimation algorithm ALG. [Figure 60] FIG. 59 is a diagram showing a specific example of initial sample data D_sample_initial for modeling shown in FIG. 58. [Figure 61] FIG. 57 is a diagram showing a specific example of the training set shown in FIG. 56. [Figure 62] FIG. 57 is a diagram showing a specific example of the validation set shown in FIG. 56. [Figure 63] FIG. 57 is a diagram showing a specific example of the test data shown in FIG. 56. [Figure 64] FIG. 1 is a conceptual diagram of a confusion matrix. [Figure 65] FIG. 10 is a diagram illustrating a method for generating a confusion matrix. [Figure 66] 10 is a flowchart illustrating a network scan operation. [Figure 67] 67 is a first flowchart for explaining the detailed operation of step S10 in FIG. 66. [Figure 68] 67 is a second flowchart for explaining the detailed operation of step S10 in FIG. 66. [Figure 69] 67 is a first flowchart for explaining the detailed operation of step S11 in FIG. 66. [Figure 70] 67 is a second flowchart for explaining the detailed operation of step S11 in FIG. 66. [Figure 71] 67 is a third flowchart for explaining the detailed operation of step S11 in FIG. 66. [Figure 72] 70 is a flowchart for explaining the detailed operation of step S134 in FIG. 69. [Figure 73] 71 is a first flowchart illustrating the detailed operation of step S147 in FIG. 70. [Figure 74] 71 is a second flowchart illustrating the detailed operation of step S147 in FIG. 70. [Figure 75] 71 is a third flowchart for explaining the detailed operation of step S147 in FIG. 70. [Figure 76] 74 is a flowchart for explaining detailed operations of step S1472 in FIG. 73. [Figure 77] 74 is a flowchart for explaining the detailed operation of step S1476 in FIG. 73. [Figure 78] 71 is a flowchart for explaining the detailed operation of step S149 in FIG. 70. [Figure 79] 71 is a flowchart for explaining the detailed operation of step S149 in FIG. 70 when the Matthews correlation coefficient MCC is calculated in step S149. [Figure 80] 67 is a first flowchart illustrating the detailed operation of step S13 in FIG. 66. [Figure 81] 67 is a second flowchart for explaining the detailed operation of step S13 in FIG. 66. [Figure 82] 67 is a first flowchart illustrating the detailed operation of step S14 in FIG. 66. [Figure 83] 67 is a second flowchart for explaining the detailed operation of step S14 in FIG. 66. [Figure 84] FIG. 10 is a diagram illustrating an example of a confusion matrix. [Figure 85] FIG. 3 is a schematic diagram illustrating the operation of the network scanning device shown in FIG. [Figure 86] 10 is a flowchart illustrating a method for producing standard sample data D_similarity. [Figure 87] FIG. 10 is a diagram showing the relationship between the similarity rate bsim and the results of a network scan. [Figure 88] FIG. 10 is a diagram illustrating the relationship between the similarity rate bsim and the number of ports to be scanned. [Figure 89] FIG. 10 is a diagram showing a comparison of the similarity rate of the present method and the similarity rate of the conventional method. [Figure 90] FIG. 1 illustrates the accuracy of estimating the results of a network scan. [Figure 91] FIG. 10 is a diagram showing simulation results of D scores. [Figure 92] FIG. 10 is a diagram illustrating a probability density function of a scan response delay RTTm. DETAILED DESCRIPTION OF THE INVENTION

[0038] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals and description thereof will not be repeated.

[0039] Fig. 1 is a schematic diagram of a communication system according to an embodiment of the present invention. Referring to Fig. 1, a communication system 10 according to the embodiment of the present invention includes a network scanning device 1, base stations BS1 and BS2, an Internet service provider (ISP), and terminal devices TM1 to TM18.

[0040] A network scanning device 1, base stations BS1 and BS2, an internet provider ISP, and terminal devices TM1 to TM18 are located in a communication space.

[0041] Base station BS1 is a base station of wireless communication network CNW1, base station BS2 is a base station of wireless communication network CNW2, and internet provider ISP is a communications carrier of wired communication network CNW3.

[0042] The wireless communication networks CNW1 and CNW2 are communication networks that perform wireless communication in accordance with mutually different wireless communication standards.

[0043] In the embodiment of the present invention, each of the terminal devices TM1 to TM13 is a wireless terminal, and each of the terminal devices TM14 to TM18 is a wired terminal.

[0044] The terminal devices TM1 to TM6 are located outside the wireless communication network CNW2 and inside the wireless communication network CNW1.

[0045] The terminal device TM7 is located in the overlapping area of ​​the wireless communication network CNW1 and the wireless communication network CNW2.

[0046] The terminal devices TM8 to TM13 are located outside the wireless communication network CNW1 and inside the wireless communication network CNW2.

[0047] The terminal devices TM14 to TM18 are arranged within the wired communication network CNW3 and are connected to the internet provider ISP via wires.

[0048] Each of the terminal devices TM1 to TM13 includes two wireless communication modules capable of wireless communication with base station BS1, BS2, and therefore each of the terminal devices TM1 to TM13 is capable of wireless communication with base station BS1 or base station BS2.

[0049] Terminal devices TM1 to TM6 perform wireless communication with base station BS1, terminal device TM7 performs wireless communication with base station BS1 or base station BS2, and terminal devices TM8 to TM13 perform wireless communication with base station BS2.

[0050] The terminal devices TM14 to TM18 perform wired communication via an Internet provider ISP.

[0051] The network scanning device 1 performs a network scan on the terminal devices TM1 to TM18 using a method that will be described later.

[0052] Fig. 2 is a schematic diagram of the network scanning device 1 shown in Fig. 1. Referring to Fig. 2, the network scanning device 1 includes a network scanner 11, a scan analysis manager 12, a host information database 13, an updating means 14, an extracting means 15, a calculating means 16, an estimating means 17, and a scan scheduler 18.

[0053] The network scanner 11 receives scan timing from the scan scheduler 18, and based on the received scan timing, performs a network scan on the ports of terminal devices connected to the network 20 to be scanned, for example, using the well-known Nmap (see non-patent document 2).

[0054] The network scanner 11 then receives a scan response of the network scan from a terminal device connected to the network 20, and outputs the received scan response to a user (composed of network security, etc.) 30 and the scan analysis manager 12.

[0055] In addition, the network scanner 11 may detect a scan response delay (RTT) when the network scan is performed without monitoring background traffic. m (m is an integer that satisfies 1≦m≦M, and M is the total number of terminal devices TM1 to TM18, and is an integer of 2 or more), and the packet loss rate PLR m , maximum scan delay DLmaxm and scan minimum delay DLmin m Measure the scan response delay RTT m , packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m to the scan analysis manager 12. That is, when a network scan is performed without monitoring background traffic, the network scanner 11 outputs the scan response delay RTT m , packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m Communication characteristics CM1 = [scan response delay RTT m / Packet Loss Rate PLR m / Scan maximum delay DLmax m / Scan minimum delay DLmin m ] is measured, and the measured communication characteristic CM1 = [scan response delay RTT m / Packet Loss Rate PLR m / Scan maximum delay DLmax m / Scan minimum delay DLmin m ] is output to the scan analysis manager 12.

[0056] Furthermore, when the network scan is performed by monitoring background traffic, the network scanner 11 may m , packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m Measure the scan response delay RTT m , packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLminm to the scan analysis manager 12. That is, when the network scan is performed by monitoring background traffic, the network scanner 11 outputs the scan response delay RTT m , packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m Communication characteristics CM2 = [scan response delay RTT m / Packet Loss Rate PLR m / Delayed DL m / JitterJT m / Background Max Delay BGDLmax m / Background minimum delay BGDLmin m ] is measured, and the measured communication characteristic CM2 = [scan response delay RTT m / Packet Loss Rate PLR m / Delayed DL m / JitterJT m / Background Max Delay BGDLmax m / Background minimum delay BGDLmin m ] is output to the scan analysis manager 12.

[0057] The scan analysis manager 12 receives a scan response from the network scanner 11 and a communication log during the network scan. The scan analysis manager 12 also receives the communication characteristic CM1 or CM2 from the network scanner 11.

[0058] Then, when the scan analysis manager 12 receives the scan response and the communication characteristics CM1, it generates a scan result SC1 that associates the address of the terminal device to be scanned, the scan response, and the communication characteristics CM1 based on the scan response and the communication characteristics CM1, and creates or updates the host information database 13 by storing the generated scan result SC1 in the host information database 13.

[0059] Furthermore, when the scan analysis manager 12 receives the scan response and the communication characteristics CM2, it generates a scan result SC2 that associates the address of the terminal device to be scanned, the scan response, and the communication characteristics CM2 based on the scan response and the communication characteristics CM2, and creates or updates the host information database 13 by storing the generated scan result SC2 in the host information database 13.

[0060] The host information database 13 stores the scan response delay (RTT) obtained when a network scan is performed in a communication environment where the result of the network scan is "normal." m’ _known}_1(m'=1'~M') and the communication characteristics {CMbest} that are most suitable for estimating that the results of the network scan are "normal." m’ _known}_1 (m'=1' to M') are stored in advance in association with each other. Here, m' is an integer that satisfies 1≦m'≦M', and M' is the total number of terminal devices that are the target of network scans executed in a communication environment where the results of network scans are known. m' may be the same as m or may be different from m.

[0061] The host information database 13 also stores the scan response delay (RTT) obtained when a network scan is performed in a communication environment where the result of the network scan is unsuccessful due to "insufficient communication quality." m’ _known}_2(m'=1'~M') and the communication characteristics {CMbest} that are most suitable for estimating that the network scan result failed due to "insufficient communication quality." m’ _known}_2 (m'=1' to M') are stored in advance in association with each other.

[0062] Furthermore, the host information database 13 stores the scan response delay (RTT) obtained when a network scan is performed in a communication environment where the results of the network scan fail due to "network congestion." m’_known}_3(m'=1'~M') and the communication characteristics {CMbest} that are most suitable for estimating that the network scan result failed due to "network congestion" m’ _known}_3 (m'=1' to M') are stored in advance in association with each other.

[0063] Furthermore, the host information database 13 stores the scan result SC1 or the scan result SC2.

[0064] The update means 14 updates the scan response delay {RTT} acquired when the network scanner 11 performs a network scan in a communication environment where the results of the network scan are unknown, based on the communication characteristics CM1 (or communication characteristics CM2) stored in the host information database 13. m _unknown} (m=1 to M) from the host information database 13, and m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m'=1' to M') is read from the host information database 13.

[0065] The update means 14 then updates the scan response delay {RTT m _unknown}(m=1~M), scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3(m′=1′ to M′) is output to the calculation means 16.

[0066] Thereafter, the update means 14 updates the scan response delay {RTT m _unknown}(m=1~M) and scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’_known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3(m'=1'~M') and the three similarity rates b sim1 ~b sim3 When the calculation means 16 receives the similarity rate b sim1 ~b sim3 The largest similarity rate b sim_max Detect.

[0067] Then, the update means 14 updates the largest similarity b sim_max Scan response delay {RTT m’ _known}_similarity(m'=1'~M')(scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m'=1'~M')) and scan response delay {RTT m _unknown} (m=1 to M), and updates the correspondence table TBL3 stored in the host information database 13 based on the aggregated scan response delays.

[0068] The extraction means 15 extracts the communication characteristic {CMbest} best suited for estimating the results of the network scan based on the communication characteristic CM1 (or communication characteristic CM2) stored in the host information database 13 by a method to be described later. m _unknown} (m=1 to M). m _unknown}(m=1~M) and communication characteristics {CMbest m _unknown} (m=1 to M), and outputs the generated sample data D_unknown to the calculation means 16 and the estimation means 17.

[0069] The extraction means 15 also extracts from the updated correspondence table TBL3 of the host information database 13 the scan response delay {RTT m’ _known}_1(m'=1'~M') and communication characteristics {CMbest m’ _known}_1(m'=1'~M') and the detected scan response delay {RTT m’ _known}_1(m'=1'~M') and communication characteristics {CMbest m’ _known}_1 (m′=1′ to M′), and outputs the generated standard sample data D_standard_1 to the calculation means 16.

[0070] Furthermore, the extraction means 15 extracts from the updated correspondence table TBL3 of the host information database 13 the scan response delay {RTT m’ _known}_2(m'=1'~M') and communication characteristics {CMbest m’ _known}_2(m'=1'~M') and the detected scan response delay {RTT m’ _known}_2(m'=1'~M') and communication characteristics {CMbest m’ _known}_2 (m′=1′ to M′), and outputs the generated standard sample data D_standard_2 to the calculation means 16.

[0071] Furthermore, the extraction means 15 extracts from the updated correspondence table TBL3 of the host information database 13 the scan response delay {RTT m’ _known}_3(m'=1'~M') and communication characteristics {CMbest m’ _known}_3(m'=1'~M') and the detected scan response delay {RTT m’ _known}_3(m'=1'~M') and communication characteristics {CMbest m’_known}_3 (m′=1′ to M′), and outputs the generated standard sample data D_standard_3 to the calculation means 16.

[0072] Furthermore, the extraction means 15 extracts the scan response delay {RTT m’ _known}_1(m'=1'~M')~{RTT m’ _known}_3 (m'=1'~M') and the scan response delay RTT included in the sample data D_unknown m 3 similarities with _unknown P S1 ,P S2 ,P S3 are received from the calculation means 16, and the received three similarities P S1 ,P S2 ,P S3 The scan response delay {RTT m’ _known}_similarity(m'=1'~M')(=Scan response delay {RTT m’ _known}_1(m'=1'~M')~{RTT m’ _known}_3 (m'=1' to M')). Then, the extraction means 15 detects the scan response delay {RTT m’ _known}_similarity(m'=1'~M') and scan response delay {RTT m’ _known}_similarity (m'=1'~M') corresponding communication characteristics {CMbest m’ _known}similarity(=communication characteristics{CMbest m’ _known}_1(m'=1'~M')~{CMbest m’ _known}_3 (m′=1′ to M′)) to the estimation means 17.

[0073] The calculation means 16 calculates the scan response delay {RTT m_unknown}(m=1~M), scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m'=1' to M') from the update means 14. Then, the calculation means 16 calculates the scan response delay {RTT m _unknown}(m=1~M) and scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3(m'=1'~M') and the three similarity rates b sim1 ~b sim3 Calculate the similarity rate b sim1 ~b sim3 is output to the update means 14.

[0074] The calculation means 16 also receives the sample data D_unknown and the standard sample data D_standard_1 to D_standard_3 from the extraction means 15. The calculation means 16 then calculates the scan response delay {RTT m’ _known}_1(m'=1'~M')~{RTT m’ _known}_3 (m'=1'~M') and the scan response delay RTT included in the sample data D_unknown m 3 similarities P with unknown (m=1~M) S1 ,P S2 ,P S3 are calculated using the method described below, and the three calculated similarities P S1 ,P S2 ,P S3 is output to the extraction means 15.

[0075] Furthermore, the calculation means 16 receives from the estimation means 17 the estimation results (confusion matrices CFM, nCFM, tCFM) obtained by estimating the results of the network scan using the estimation algorithm ALG. Then, the calculation means 16 calculates an evaluation index EVD representing the accuracy of the estimation result based on the confusion matrix CFM, calculates an evaluation index nEVD representing the accuracy of the estimation result based on the confusion matrix nCFM, and calculates an evaluation index tEVD representing the accuracy of the estimation result based on the confusion matrix tCFM, using a method described below. Then, the calculation means 16 outputs the evaluation indexes EVD, nEVD, tEVD to the estimation means 17.

[0076] The estimation means 17 combines the standard sample data D_standard_1 to D_standard_3 to generate initial sample data D_sample_initial for modeling.

[0077] The estimation means 17 also estimates the standard sample data D_similarity (= scan response delay {RTT m’ _known}_similarity(m'=1'~M') and communication characteristics {CMbest m’ _known}_similarity (m'=1' to M').) from the extraction means 15. Then, the estimation means 17 divides the standard sample data D_similarity into training data and test data, and divides the training data into a training set and a validation set. Then, the estimation means 17 generates multiple tentative sample data D_sample_tentative based on the training set and initial sample data D_sample_initial by a method described later.

[0078] Then, the estimation means 17 trains the estimation algorithm ALG using the initial sample data D_sample_initial, estimates the results of the network scan by the estimation algorithm ALG (the trained estimation algorithm ALG) using the validation set, and generates a confusion matrix CFM based on the estimated results of the network scan and outputs it to the calculation means 16.

[0079] Next, the estimation means 17 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative, estimates the results of the network scan by the estimation algorithm ALG (the trained estimation algorithm ALG) using the validation set, and generates a confusion matrix nCFM based on the estimated results of the network scan and outputs it to the calculation means 16.

[0080] The estimation means 17 outputs the confusion matrix CFM to the calculation means 16, and then receives the evaluation index EVD from the calculation means 16, and outputs the confusion matrix nCFM to the calculation means 16, and then receives the evaluation index nEVD from the calculation means 16. When the evaluation index nEVD is greater than the evaluation index EVD, the estimation means 17 sets the evaluation index nEVD to the evaluation index EVD. On the other hand, when the evaluation index nEVD is not greater than the evaluation index EVD, the estimation means 17 does not set the evaluation index nEVD to the evaluation index EVD.

[0081] The estimation means 17 compares the evaluation index nEVD with the evaluation index EVD for all of the plurality of tentative sample data D_sample_tentative, and replaces the variables of the estimation algorithm ALG with the tentative sample data D_sample_tentative when the largest evaluation index nEVD is obtained.

[0082] Thereafter, the estimation means 17 estimates the results of the network scan by the estimation algorithm ALG (the estimation algorithm ALG in which the variables have been replaced) using the test data, generates a confusion matrix tCFM based on the estimation results of the network scan, and outputs it to the calculation means 16. Then, the estimation means 17 receives the evaluation index tEVD from the calculation means 16, and when the evaluation index tEVD is equal to or greater than the evaluation index EVD, approves the estimation algorithm ALG in which the variables have been replaced and the tentative sample data D_sample_tentative when the largest evaluation index nEVD is obtained.

[0083] In this way, the estimation means 17 trains the estimation algorithm ALG and evaluates the accuracy of the estimation result when the result of the network scan is estimated by the trained estimation algorithm ALG.

[0084] Furthermore, the estimation means 17 estimates the results of the network scan using the estimation algorithm ALG in the following manner: m Terminal devices having _unknown (m=1 to M) are classified into a group CLS1 in which the network scan is normal, and groups CLS2 and CLS3 in which the network scan is unsuccessful due to a cause attributable to the communication channel with the terminal device.

[0085] The estimation means 17 then calculates a D-score D_2 that distinguishes between groups CLS1 and CLS3, or a D-score D_3 that distinguishes between groups CLS2 and CLS3, and estimates whether the network scan is successful or unsuccessful based on the calculated D-score D_2 or D-score D_3. If the network scan is estimated to be unsuccessful, the estimation means 17 performs an estimation process to estimate the cause of the network scan failure. The estimation means 17 then outputs the estimation results obtained in the estimation process to the scan scheduler 18.

[0086] The scan scheduler 18 reads the scan results from the host information database 13 and receives scan requirements from the user 30. The scan scheduler 18 also stores in advance the global IP addresses and domain names of all terminal devices present in the area where the network scan is to be performed (for example, within Japan).

[0087] Then, based on the scan requirements and the scan results, the scan scheduler 18 generates a scan schedule SCH_scan that associates scan timings of network scans with terminal devices that are targets of network scans at each scan timing using a method described below.The scan scheduler 18 then outputs the scan schedule SCH_scan to the network scanner 11 and controls the network scanner 11 to execute a network scan in accordance with the scan schedule SCH_scan.

[0088] Furthermore, when the scan scheduler 18 receives the estimation result from the estimation means 17, it reschedules the scan schedule based on the cause of the network scan failure in the estimation result, outputs the rescheduled scan schedule RESCH_scan to the network scanner 11, and controls the network scanner 11 to execute the network scan in accordance with the scan schedule RESCH_scan.

[0089] Fig. 3 is a conceptual diagram of the correspondence table TBL1 stored in the host information database 13 shown in Fig. 2. Note that the correspondence table TBL1 shown in Fig. 3 is a correspondence table used when a network scan is performed without monitoring background traffic.

[0090] Referring to FIG. 3, the correspondence table TBL1 contains a time t and an IP address Add b_v (v=1 to V, where V is the total number of base stations.) and IP address Add m and the domain name DN m and port number PN r (r is a positive integer) and the scan response SR m and scan response delay RTT m and packet loss rate (PLR) m and the maximum scan delay DLmax m and the minimum scan delay DLmin m Includes:

[0091] Time t is expressed in YYYY / MM / DD / hh / mm / ss (year / month / day / hour / minute / second). Time t is a time within one scan timing, which has a fixed time length (described later), and represents the time when the test input in the network scan is sent to the terminal device that is the target of the network scan.

[0092] IP Address Add b_v is the address of the base station. IP Address Add m consists of the global IP address of the terminal device that is the target of the network scan.

[0093] Domain Name DN m Add the IP address m It was converted using the PTR (PolyTeR record) record based on the above.

[0094] Port number PN r is the port number that each terminal device uses for communication.

[0095] Scan Response SR m is "1" when a scan response is returned, and is "0" when no scan response is returned.

[0096] Scan Response Delay RTT m is the time from the time the test input was sent to the scan response SR m The time it takes for the call to be returned is the time it takes for the call to be returned.

[0097] Packet Loss Rate (PLR) m sends a packet consisting of test input and returns a scan response SR m This is the rate at which packets fail to be sent or received when receiving a packet.

[0098] Maximum scan delay DLmax m scan response delay RTT m is the maximum value of the minimum scan delay DLmin m scan response delay RTT mis the minimum value of

[0099] And the scan response SR m and scan response delay RTT m is the port number PN r Packet loss rate (PLR) m , maximum scan delay DLmax m and scan minimum delay DLmin m is the IP address of the terminal device to be scanned. m can be associated with

[0100] Addresses Add1 to Add3 are also b_1 is associated with,..., address Add M-1 ~Add M Add the address b_V As a result, it is possible to know the terminal devices that belong to each base station.

[0101] Fig. 4 is a conceptual diagram of another correspondence table TBL2 stored in the host information database 13 shown in Fig. 2. Note that the correspondence table TBL2 shown in Fig. 4 is a correspondence table used when a network scan is performed by monitoring background traffic.

[0102] 4, the correspondence table TBL2 is the maximum scan delay DLmax of the correspondence table TBL1 shown in FIG. m and scan minimum delay DLmin m Delay DL m , Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m The rest is the same as the correspondence table TBL1.

[0103] Delayed Download m , Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin mis the IP address of each terminal device. m can be associated with

[0104] Delayed Download m is the delay from the standard time in one-way communication from the terminal device.

[0105] Jitter JT m is the fluctuation in packet transmission time in one-way communication from the terminal device.

[0106] Background maximum delay BGDLmax m is the maximum transmission delay in one-way communication from the terminal device, and the background minimum delay BGDLmin m is the minimum transmission delay in one-way communication from the terminal device.

[0107] In the correspondence table TBL2, the scan response SR m and scan response delay RTT m , is the port number PN of each terminal device r can be associated with

[0108] FIG. 5 is a conceptual diagram of yet another correspondence table TBL3 stored in the host information database 13 shown in FIG.

[0109] Referring to FIG. 5, the correspondence table TBL3 lists the known estimation results, the time t', and the IP address Add b’_w (where w=1 to W, and W is the total number of base stations), and the IP address Add m’ and scan response delay RTT m’ _known and communication characteristics CMbest m’ _known. Known estimation result, time t', IP address Add b’_w , IP address Add m’ , scan response delay RTT m’ _known and communication characteristics CMbest m’ _known are associated with each other.

[0110] The known estimated results include "normal", "poor communication quality", and "network congestion" as estimated results of the network scan.

[0111] The time t' has the same format as the time t explained in FIG. b’_w is the address of the base station. IP Address Add m’ The IP address Add consists of the global IP address of the terminal device that is the target of the network scan. 1’ ~IP Address Add 3’ is the IP address of the base station. b’_1 Similarly, the IP address Add M’ is the IP address of the base station. b’_W can be associated with

[0112] Scan Response Delay {RTT 1’ _known}_1, Scan Response Delay {RTT 2’ _known}_1, Scan Response Delay {RTT 3’ _known}_1, ···, and scan response delay {RTT M’ _known}_1 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ ,...,IP addressAdd M’ , and is the scan response delay RTT obtained in a communication environment where the network scan result is "normal."

[0113] Communication characteristics {CMbest 1’ _known}_1, communication characteristics {CMbest 2’ _known}_1, communication characteristics {CMbest 3’ _known}_1, ···, and communication characteristics {CMbest M’ _known}_1 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ ,...,IP addressAddM’ The communication characteristic CMbest is associated with the communication characteristic CMbest obtained in a communication environment where the result of the network scan is "normal."

[0114] Scan Response Delay {RTT 1’ _known}_2, Scan Response Delay {RTT 2’ _known}_2, Scan Response Delay {RTT 3’ _known}_2, ···, and scan response delay {RTT M’ _known}_2 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ ,...,IP addressAdd M’ The scan response delay RTT is associated with the scan response delay RTT obtained in a communication environment where the network scan result is "insufficient communication quality."

[0115] Communication characteristics {CMbest 1’ _known}_2, communication characteristics {CMbest 2’ _known}_2, communication characteristics {CMbest 3’ _known}_2, ···, and communication characteristics {CMbest M’ _known}_2 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ ,...,IP addressAdd M’ The communication characteristic CMbest is associated with the communication characteristic CMbest obtained in a communication environment where the result of the network scan is "insufficient communication quality."

[0116] Scan Response Delay {RTT 1’ _known}_3, Scan Response Delay {RTT 2’ _known}_3, Scan Response Delay {RTT 3’ _known}_3, ···, and scan response delay {RTT M’ _known}_3 are the IP addresses Add 1’ , IP address Add 2’, IP address Add 3’ ,...,IP addressAdd M’ The scan response delay RTT is associated with the scan response delay RTT obtained in a communication environment where the network scan result is "network congestion."

[0117] Communication characteristics {CMbest 1’ _known}_3, communication characteristics {CMbest 2’ _known}_3, communication characteristics {CMbest 3’ _known}_3, ···, and communication characteristics {CMbest M’ _known}_3 are the IP addresses Add 1’ , IP address Add 2’ , IP address Add 3’ ,...,IP addressAdd M’ The communication characteristic CMbest is associated with the communication characteristic CMbest, which is acquired in a communication environment where the result of the network scan is "network congestion."

[0118] Scan Response Delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 constitutes the standard sample data D_standard_1.

[0119] Also, scan response delay (RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 constitutes the standard sample data D_standard_2.

[0120] Furthermore, the scan response delay (RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbestM’ _known}_3 constitutes the standard sample data D_standard_3.

[0121] In addition, the scan response delay (RTT) shown in the correspondence table TBL3 1’ _known}_1~{RTT M’ Each of the {_known}_1 actually represents one IP address Add m’ There are multiple ports in the network. 1’ _known}_1~{CMbest M’ Each of the {_known}_1 actually represents one IP address Add m’ There are multiple scan response delays (RTT 1’ _known}_2~{RTT M’ _known}_2;{RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2;{CMbest 1’ _known}_3~{CMbest M’ The same is true for _known}_3.

[0122] Also, scan response delay (RTT 1’ _known}_1~{RTT M’ _known}_1's "1" and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ The "1" in {_known}_1 represents the identification code D_code=1 assigned to the known estimated result of the network scan, "normal," and the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2's "2" and communication characteristics {CMbest 1’ _known}_2~{CMbest M’The "2" in _known}_2 represents the identification code D_code=2 assigned to the known estimated result of the network scan, "poor communication quality," and the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3's "3" and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ The "3" in {_known}_3 represents the identification code D_code=3 assigned to the known estimated result of the network scan, "Network is congested."

[0123] In this embodiment of the present invention, when a network scan is performed without monitoring background traffic, correspondence tables TBL1 and TBL3 are stored in the host information database 13, and when a network scan is performed while monitoring background traffic, correspondence tables TBL2 and TBL3 are stored in the host information database 13. Correspondence table TBL3 is stored in the host information database 13 before the network scanner 11 starts a network scan in a communication environment where the results of the network scan are unknown.

[0124] 6 is a conceptual diagram of a scan schedule. Referring to FIG. 6(a), the scan schedule SCH_scan is a scan schedule having a total scan time length T S _total, and T (T is an integer equal to or greater than 1) scan timings 1 S ~Scan timing T S Includes:

[0125] Total scan time length T S _total is set to 1 day, for example, and scan timing 1 S ~Scan timing T S Each of the scan timings t has a duration of, for example, one hour. S (t S =1 S ~T S ) is the start time t start and the end time tend and target addresses Add1 to Add M Includes:

[0126] start time t start is the scan timing t S represents the time when the network scan starts, and the end time t end is the scan timing t S t represents the time at which the network scan is to be completed. S At the end time t end When this message arrives, the network scan is terminated.

[0127] Target address Add1~Add M Each of the target addresses Add1 to Add M represents M terminal devices that are the targets of the network scan.

[0128] In addition, the scan timing t S Scan timing other than 1 S ~(t-1) S ,(t+1) S ~T S Each of the scan timings t S It has the same configuration as

[0129] Scan Timing 1 S ~T S are the same time length (=t end -t start ), and may have different time lengths (=t end -t start ) may also have scan timing 1 S ~T S may have the same number of IP addresses as each other, or may have different numbers of IP addresses from each other.

[0130] When performing a network scan for the first time, scan timing 1 shown in Figure 6 (a) is used.S ~T S is used.

[0131] In addition, when a network scan is performed during a period when communication is not busy, the scan timing 1' shown in (b) of FIG. S ~T' S (T' is an integer equal to or greater than 1) is used. In FIG. 6(b), scan timing 1' S ~T' S are set to be spaced apart in time, because multiple time periods with no communication activity are not adjacent to each other.

[0132] In addition, the scan timing t' S (t' S =1' S ~T' S ) is configured with the above-mentioned scan timing t S The configuration is the same as

[0133] Also, the scan timing t S ,t' S does not mean a single time, but a starting time t start From the end time t end It is a concept that has a predetermined time length.

[0134] Scan timing t' S A method for determining the above will be described below. Fig. 7 is a diagram showing the time dependency of traffic. In Fig. 7, the vertical axis represents traffic and the horizontal axis represents time. Furthermore, curve k1 represents the time dependency of traffic in, for example, terminal device A, and curve k2 represents the time dependency of traffic in, for example, terminal device B which is different from terminal device A.

[0135] Referring to (a) of FIG. 7, for terminal device A, traffic is heavier at night than during the day (see curve k1). That is, for terminal device A, communication is quiet during the day and congested at night. Referring to (b) of FIG. 7, for terminal device B, traffic is heavier during the day than at night (see curve k2). That is, for terminal device B, communication is congested during the day and congested at night.

[0136] Therefore, the scan scheduler 18 detects a time period when communication is not busy for the M terminal devices that are the targets of the network scan, and determines the scan timing t S (or scan timing t' S ) is determined.

[0137] The network scanner 11 receives the scan schedule SCH_scan from the scan scheduler 18, and executes a network scan on M terminal devices in accordance with the received scan schedule SCH_scan.

[0138] In this case, the network scan is performed without monitoring background traffic or with monitoring background traffic.

[0139] [Known Scan Response Delay {RTT m’ Update _known}_z] Known scan response delay {RTT m’ _known}_z will be explained. Here, z is an argument indicating the known estimated results of the network scan, and is an integer that satisfies 1≦z≦Z. Furthermore, Z is the total number of known estimated results of the network scan. In this embodiment of the present invention, there are three known estimated results of the network scan: "normal", "poor communication quality", and "network congestion", so z=1, 2, 3.

[0140] The update means 14 updates the scan response delay RTT from the correspondence table TBL1 or TBL2 stored in the host information database 13. 11 ~RTT M3 and obtains the scan response delay {RTT m’ _known}_1(m'=1'~M')~{RTT m’ _known}_3(m'=1'~M').

[0141] Scan Response Delay RTT 11 ~RTT M3 The first numbers "1" to "M" in the subscripts "11" to "M3" are arguments for the terminal devices, and the second numbers "1" to "3" in the subscripts "11" to "M3" are arguments for the port numbers in each terminal device.

[0142] On the other hand, scan response delay (RTT m’ As mentioned above, each of the {_known}_1 (m'=1' to M') actually represents one IP address Add m’ There are multiple scan response delays (RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ The same is true for _known}_3(m'=1'~M').

[0143] Therefore, the known scan response delay (RTT m’ In the explanation of updating _known}_z, the scan response delay RTT read from the correspondence table TBL1 or the correspondence table TBL2 is 11 ~RTT M3 Scan Response Delay {RTT m,PNr _unknown}(m=1~M), and the scan response delay {RTT m’ _known}_1(m'=1'~M') scan response delay {RTT m’,PNr _known}_1(m'=1'~M'), and the scan response delay {RTT m’ _known}_2(m'=1'~M') scan response delay {RTTm’,PNr _known}_2(m'=1'~M'), and the scan response delay {RTT m’ _known}_3(m'=1'~M') scan response delay {RTT m’,PNr _known}_3(m'=1'~M').

[0144] The update means 14 updates the scan response delay {RTT m,PNr _unknown}(m=1~M), scan response delay {RTT m’,PNr _known}_1(m'=1'~M'), scan response delay {RTT m’,PNr _known}_2(m'=1'~M') and scan response delay {RTT m’,PNr _known}_3(m'=1'~M'), each scan response delay {RTT m,PNr _unknown},{RTT m’,PNr _known}_1,{RTT m’,PNr _known}_2,{RTT m’,PNr _known}_3 is normalized to be greater than or equal to "0" to determine the scan response delay {RTT m,PNr _unknown}(m=1~M),{RTT m’,PNr _known}_1(m'=1'~M'),{RTT m’,PNr _known}_2(m'=1'~M'),{RTT m’,PNr _known}_3 (m'=1'~M').

[0145] Fig. 8 is a diagram showing an example of a normalized scan response delay RTT. Referring to Fig. 8, m / m' represents terminal device m or terminal device m'. Each of m and m' is, for example, 1 to 8. Also, PN1 to PN8 represent port numbers of each terminal device m (or each terminal device m').

[0146] Each row in FIG. 8 represents eight normalized scan response delays RTT when a network scan is performed on eight ports of one terminal device m (or one terminal device m').

[0147] In reality, a network scan was performed on 1,000 ports of each terminal device m (or terminal device m') to obtain 1,000 scan response delay RTTs. However, due to space limitations, FIG. 8 shows eight normalized scan response delays out of the 1,000 normalized scan response delay RTTs for each terminal device m (or each terminal device m').

[0148] Furthermore, in practice, network scans were performed on more than eight terminal devices m (or eight terminal devices m') to obtain scan response delay RTTs, but due to space limitations, Figure 8 shows scan response delay RTTs normalized for eight terminal devices m (or eight terminal devices m').

[0149] S1 to S4 are datasets of normalized scan response delay RTTs when the results of the network scan are different. In this embodiment of the present invention, as described above, the known estimated results of the network scan are three: "normal," "poor communication quality," and "network congestion." Therefore, three of datasets S1 to S4 are datasets of normalized scan response delay RTTs when the three known estimated results ("normal," "poor communication quality," and "network congestion") are obtained, and the remaining dataset is a dataset of normalized scan response delay RTTs when a network scan is performed in a communication environment where the results of the network scan are unknown.

[0150] The update means 14 updates each scan response delay {RTT m,PNr _unknown},{RTT m’,PNr _known}_1,{RTT m’,PNr _known}_2,{RTT m’,PNr _known}_3 is normalized to be greater than or equal to "0" to determine the scan response delay {RTT m,PNr _unknown}(m=1~M),{RTT m’,PNr _known}_1(m'=1'~M'),{RTT m’,PNr_known}_2(m'=1'~M'),{RTT m’,PNr _known}_3 (m'=1' to M') to create data sets S1 to S4 shown in FIG.

[0151] After creating the data sets S1 to S4, the update means 14 selects one data set (for example, data set S1) from the data sets S1 to S4.

[0152] Then, the update means 14 sequentially executes the following steps (i) to (iii) on the data set S1. (i) For each of the eight scan response delays RTT in one row of the data set S1, check whether the scan response delay RTT is smaller than "0". (ii) Delete scan response delay RTTs that are smaller than “0”. (iii) Execute (i) and (ii) for all eight rows (i.e., eight terminal devices).

[0153] The update means 14 sequentially executes the above steps (i) to (iii) for each of the data sets S2 to S4.

[0154] Thereafter, the update means 14 outputs the data sets S1 to S4 to the calculation means 16. The calculation means 16 receives the data sets S1 to S4 from the update means 14 and calculates the baseline scan response delay b based on the scan response delay RTT included in the received data sets S1 to S4. RTT Calculate.

[0155] Baseline scan response latency b RTT is a scan response delay RTT whose distribution has been changed based on an "upper limit" and a "lower limit" so as to more easily show similarities or differences between the scan response delay RTT_unknown obtained when a network scan is performed in a communication environment where the results of the network scan are unknown and the scan response delay RTT_known obtained when a network scan is performed in a communication environment where the results of the network scan are known.

[0156] Baseline scan response latency b RTT The calculation method for calculating the scan response delay RTT will be described below. Calculation means 16 selects data set S1 and extracts eight scan response delay RTTs in the first row (m / m'=1) from the selected data set S1. Then, calculation means 16 calculates the average μ and standard deviation σ of the eight scan response delay RTTs.

[0157] In this case, the calculation means 16 subtracts the average from one scan response delay RTT and calculates the square (square) of the subtraction result (RTT-μ) for all eight scan response delay RTTs, and calculates the eight squares (RTT-μ) 2 Calculate the sum of the 8 squares (RTT-μ) 2 The sum of the scan response delays is divided by the number of scan response delays (RTT) n (=8). The result is [(8 squares (RTT-μ) 2 square root of (8 squares (RTT-μ) 2 (sum of) / n] 1 / 2 The standard deviation σ is calculated by calculating

[0158] Thereafter, the calculation means 16 calculates a score z, which is a standard score of the scan response delay RTT, by the following formula: scr Calculate.

[0159]

number

[0160] In formula (1), x is one scan response delay RTT. μ / σ in formula (1) is the reciprocal of the coefficient of variation CV of the distribution of multiple scan response delay RTTs acquired by a network scan for one terminal device m (one terminal device m'). Therefore, calculation means 16 calculates score z by subtracting the reciprocal of the coefficient of variation CV from one scan response delay RTT. scr Calculate.

[0161] The score z is calculated by equation (1). scrThere are two ways to calculate this: (Method 1) Randomly select one scan response delay RTT from the eight scan response delay RTTs in the first row (m / m'=1), and substitute the selected scan response delay RTT for x in equation (1) to obtain one score z scr Calculate. (Method 2) Substituting each of the eight scan response delay RTTs in the first row (m / m'=1) into x in equation (1) yields eight scores z scr Calculate.

[0162] The calculation means 16 calculates the score z scr Calculating the score z scr , mean μ, standard deviation σ, and number of samples n (number of scan response delay RTTs in the first row of data set S1 (m / m'=1)) into equation (2) to obtain the lower limit L b Calculate the score z scr , mean μ, standard deviation σ, and number of samples n are substituted into equation (3) to obtain the upper limit U b Calculate.

[0163]

number

[0164]

number

[0165] The calculation means 16 calculates the score z by the above method 1. scr When calculating the lower bound L for one randomly selected scan response delay RTT, the lower bound L is calculated using Equation (2) and Equation (3), respectively. b and upper limit U b Calculate the set of {L b ,U b} one baseline scan response delay b RTT is calculated for one terminal device m (or one terminal device m').

[0166] On the other hand, the calculation means 16 calculates eight scores z scr When calculating the score, 8 scores z scr One score z out of scr Substituting the mean μ, standard deviation σ, and number of samples n into equations (2) and (3), we obtain the lower limit L b and upper limit U b Calculate the set of {L b ,U b} one baseline scan response delay b RTT For all of the first to eighth scan response delay RTTs in the first row (m / m'=1), the eight baseline scan response delays b RTT1 ~b RTT8 (={L b1 ,U b1}~L b8 ,U b8}) is calculated for one terminal device m (or one terminal device m').

[0167] The calculation means 16 calculates one baseline scan response delay b using the above-described method based on the eight scan response delay RTTs in each of the second row (m / m'=2) to the eighth row (m / m'=8) of the data set S1. RTT or 8 baseline scans response latency b RTT1 ~b RTT8 (={L b1 ,U b1}~L b8 ,U b8}) for one terminal device m (or one terminal device m') is calculated for all rows from the second row (m / m'=2) to the eighth row (m / m'=8) of the data set S1.

[0168] Thereafter, the calculation means 16 calculates one baseline scan response delay b for each of the data sets S2 to S4 by the above-described method. RTT or 8 baseline scans response latency b RTT1 ~b RTT8 (={L b_1 ,U b_1}~Lb_8 ,U b_8}) is calculated for one terminal device m (or one terminal device m').

[0169] FIG. 9 shows the score z calculated for one terminal device. scr and baseline scan response latency b RTT FIG.

[0170] In FIG. 9, the score z scr is the score z when calculated using method 1 or method 2 above. scr and baseline scan response latency b RTT Shows.

[0171] Referring to FIG. 9, the calculation means 16 calculates the score z scr When calculating, for example, a scan response delay RTT of "0.012949" corresponding to port number PN1 is randomly selected from the eight scan response delay RTTs corresponding to port numbers PN1 to PN8.

[0172] Then, the calculation means 16 substitutes the scan response delay RTT of “0.012949” into the formula (1) to obtain the score z scr Then, the calculation means 16 calculates the score z scr Substituting into equations (2) and (3), the lower limit L b and upper limit U b As a result, one baseline scan response latency b RTT ={L b ,U b} is a score z scr is calculated corresponding to

[0173] On the other hand, the calculation means 16 calculates the score z by the above method 2. scr When calculating the score, eight scan response delay RTTs corresponding to the port numbers PN1 to PN8 are selected sequentially and eight scores z are calculated using Equation (1).scr1 ~z scr8 Then, the calculation means 16 calculates the eight scores z scr1 ~z scr8 Substituting into equations (2) and (3), the lower limit L b and upper limit U b The resulting response latencies for the eight baseline scans are b RTT1 ~b RTT8 Each score is z scr1 ~z scr8 is calculated corresponding to

[0174] In this case, the score z is calculated for the scan response delay RTT of “0.012949”. scr1 is calculated and the score z scr1 The baseline scan response latency b corresponds to RTT1 ={L b1 ,U b1} is calculated. Also, the score z is calculated for the scan response delay RTT of “0.009000”. scr2 is calculated and the score z scr2 The baseline scan response latency b corresponds to RTT2 ={L b2 ,U b2 Similarly, the score z is calculated for a scan response delay RTT of "0.013000". scr8 is calculated and the score z scr8 The baseline scan response latency b corresponds to RTT8 ={L b8 ,U b8} is calculated.

[0175] Figure 10 shows the baseline scan response delay b calculated by randomly selecting one scan response delay RTT. RTT FIG.

[0176] Referring to Figure 10, baseline scan response latency b RTT1 ~b RTT4 are associated with the data sets S1 to S4, respectively.

[0177] Baseline scan response latency bRTT1 In this case, the lower limit L corresponding to m / m'=1 b =0.011943 and upper limit U b = 0.009294 is the lower limit L calculated by formulas (1) to (3) by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 1 in the data set S1 in Figure 8. b and upper limit U b is.

[0178] Also, baseline scan response latency b RTT1 In this case, the lower limit L corresponding to m / m'=2 b =0.011661 and upper limit U b = 0.009076 is the lower limit L calculated by formulas (1) to (3) by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 2 in the data set S1 in Figure 8. b and upper limit U b is.

[0179] Similarly, the baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=3 b =0.012224 and upper limit U b = 0.009513 is calculated by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 3 in the data set S1 in Figure 8 and using equations (1) to (3), and the baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=4 b =0.012922 and upper limit U b = 0.010065 is calculated by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 4 in the dataset S1 in Figure 8 and using equations (1) to (3). The baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=5 b =0.019816 and upper limit U b= 0.015421 is calculated by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 5 in the data set S1 in Figure 8 and using equations (1) to (3), and the baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=6 b =0.019956 and upper limit U b = 0.015532 is calculated by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 6 in the data set S1 in Figure 8 and using equations (1) to (3). The baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=7 b =0.020095 and upper limit U b = 0.015642 is calculated by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 7 in the data set S1 in Figure 8 and using equations (1) to (3). The baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=8 b =0.019395 and upper limit U b = 0.015092 is calculated by randomly selecting one scan response delay RTT from the eight scan response delay RTTs associated with m / m' = 8 in the data set S1 of FIG. 8 and using equations (1) to (3).

[0180] Similarly, the baseline scan response latency b corresponding to data set S2 RTT2 Lower limit L b and upper limit U b , baseline scan response latency b corresponding to Dataset S3 RTT3 Lower limit L b and upper limit U b , and the corresponding baseline scan response latency b in Dataset S4 RTT4 Lower limit L b and upper limit U bare calculated using equations (1) to (3) by randomly selecting one scan response delay RTT from each of the eight scan response delay RTTs for m / m'=1 to 8 in data sets S2 to S4 shown in FIG. 8.

[0181] As shown in Figure 10, we randomly select one scan response delay RTT to obtain the baseline scan response delay b RTT When calculating the baseline scan response delay b RTT is calculated.

[0182] Note that the baseline scan response delay b RTT1 ~b RTT4 In each of the b The value of is the lower limit L b This is because the score z scr This is because is a negative value.

[0183] FIG. 11 shows the baseline scan response delay b calculated using all of the multiple scan response delay RTTs corresponding to multiple ports in one terminal device. RTT 11 is a diagram showing an example of the number of ports in one terminal device.

[0184] Referring to Figure 11, baseline scan response latency b RTT1 ~b RTT8 are associated with the port numbers PN1 to PN8, respectively.

[0185] In dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN1 RTT1 Lower limit L b =0.011943 and upper limit U b = 0.009294 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.012949” associated with m / m’=1 and PN1 in the data set S1 in Figure 8. b and upper limit U b is.

[0186] Also, in dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN2 RTT2 Lower limit L b =0.011944 and upper limit U b = 0.009293 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.009000” associated with m / m’=1 and PN2 in the data set S1 in Figure 8. b and upper limit U b is.

[0187] Furthermore, in dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN3 RTT3 Lower limit L b =0.011944 and upper limit U b = 0.009293 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.010000” associated with m / m’=1 and PN3 in the data set S1 in Figure 8. b and upper limit U b is.

[0188] Furthermore, in dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN4 RTT4 Lower limit L b =0.011944 and upper limit U b = 0.009293 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.010000” associated with m / m’=1 and PN4 in the data set S1 in Figure 8. b and upper limit U b is.

[0189] Furthermore, in dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN5 RTT5 Lower limit L b =0.011944 and upper limit U b= 0.009293 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.010000” associated with m / m’=1 and PN5 in the data set S1 in Figure 8. b and upper limit U b is.

[0190] Furthermore, in dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN6 RTT6 Lower limit L b =0.011943 and upper limit U b = 0.009294 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.012000” associated with m / m’=1 and PN6 in the data set S1 in Figure 8. b and upper limit U b is.

[0191] Furthermore, in dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN7 RTT7 Lower limit L b =0.011944 and upper limit U b = 0.009293 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.008000” associated with m / m’=1 and PN7 in the data set S1 in Figure 8. b and upper limit U b is.

[0192] Furthermore, in dataset S1, the baseline scan response delay b corresponding to m / m'=1 and PN8 RTT8 Lower limit L b =0.011943 and upper limit U b = 0.009294 is the lower limit L calculated by Equations (1) to (3) by selecting the scan response delay RTT of “0.013000” associated with m / m’=1 and PN8 in the data set S1 in Figure 8. b and upper limit U b is.

[0193] In this way, the baseline scan response delays b associated with m / m'=1 and PN1 to PN8 in data set S1 are RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=1 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0194] Similarly, the baseline scan response delays b RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=2 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0195] In addition, the baseline scan response delays b associated with m / m'=3 and PN1 to PN8 in dataset S1 RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=3 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0196] Furthermore, the baseline scan response delays b associated with m / m'=4 and PN1 to PN8 in dataset S1 RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=4 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0197] Furthermore, the baseline scan response delays b associated with m / m'=5 and PN1 to PN8 in dataset S1 RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=5 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0198] Furthermore, the baseline scan response delays b associated with m / m'=6 and PN1 to PN8 in dataset S1 RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=6 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0199] Furthermore, the baseline scan response delays b associated with m / m'=7 and PN1 to PN8 in dataset S1 RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=7 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0200] Furthermore, the baseline scan response delays b associated with m / m'=8 and PN1 to PN8 in the data set S1 RTT1 ~b RTT8 are the lower bounds L calculated by Equations (1) to (3) by selecting eight scan response delay RTTs corresponding to m / m'=8 and PN1 to PN8 in the data set S1 in FIG. b and upper limit U b It consists of:

[0201] Baseline scan response latency in data sets S2–S4 b RTT1 ~b RTT8 The same is true for

[0202] As shown in FIG. 11, the baseline scan response delay b RTT When calculating the response delay b RTT1 ~b RTT8 is calculated.

[0203] Note that the baseline scan response delay b RTT1 ~b RTT8 In each of the scr1 ~z scr8 Since each of these consists of negative values, the upper limit U b The value of is the lower limit L b is smaller than the value of

[0204] In this embodiment of the present invention, the score z scr Score z instead β Baseline scan using response latency b RTT may be calculated.

[0205] In this case, the calculation means 16 first selects a confidence level, which is set to, for example, 95%.

[0206] When the confidence level is selected, the calculation means 16 calculates β by β=1-(confidence level / 100). When the confidence level is 95%, the calculation means 16 calculates β by β=1-(95 / 100)=0.05.

[0207] Then, the calculation means 16 calculates the critical probability p * p * = 1 - (β / 2). When β is 0.05, the calculation means 16 calculates p * = 1 - (0.05 / 2) = 0.975, so the critical probability p *Calculate.

[0208] 12 is a diagram showing a z table. The calculation means 16 calculates the critical probability p * Calculating the critical probability p * The column-wise z value (=1.9) corresponding to (=0.975) and the critical probability p * The z value (=0.06) in the row direction corresponding to the z value (=0.975) is found, and the two z values ​​("1.9" and "0.06") found are added together to get a score z of "1.96". β The calculation means 16 calculates the score z β When calculating, the z table is stored in advance.

[0209] The z table indicates the probability that the z value calculated by (RTT-μ) / σ is equal to or greater than a certain value when the scan response delay RTT follows a standard normal distribution. In the above example, the probability that the z value is equal to or greater than "1.96" is 97.5%. In this embodiment of the present invention, the critical probability p is calculated based on the reliability level (=95%) to be achieved. * (=0.975), and the critical probability p * Based on the z table, find the z value (=1.96) when the critical probability p * When the z value (=0.975) is obtained, the score z (=1.96) is used. β The critical probability p of “0.975” * is obtained for z values ​​greater than or equal to 1.96, so z values ​​greater than or equal to 1.96 are the confidence interval. Therefore, the score z β consists of the confidence interval when achieving the confidence level (=95%).

[0210] The calculation means 16 calculates the score z β Once calculated, the standard error SE is calculated using the following formula.

[0211]

number

[0212] Subsequently, the calculation means 16 calculates the degree of freedom df by the following equation.

[0213]

number

[0214] In equations (4) and (5), n is the number of scan response delay RTTs in one terminal device m (or one terminal device m').

[0215] The calculation means 16 calculates the standard error SE and the degree of freedom df, and then calculates the score z β , standard error SE, and degrees of freedom df are substituted into equations (6) and (7) to obtain the lower limit L b and upper limit U b Calculate.

[0216]

number

[0217]

number

[0218] Score Z β Using the lower bound L b and upper limit U b When calculating the RTTs, multiple scan response delays RTT for one terminal device m (or one terminal device m') are used only to calculate the mean μ and standard deviation σ. Therefore, one baseline scan response delay b RTT (Lower limit L b and upper limit U b is calculated.

[0219] In an embodiment of the present invention, z in formula (2) and formula (3) scr ×(σ / n) constitutes the "predetermined value" and is expressed by z β×(SE / (df) 1 / 2 ) constitutes a "predetermined value." And z scr As shown in equation (1), z is the result of subtracting the inverse (μ / σ) of the coefficient of variation (σ / μ) in the distribution of multiple scan response delays x (= RTT) from the scan response delay x (= RTT). The coefficient of variation (σ / μ) represents the relative variation of multiple scan response delays x (= RTT). Therefore, z scr ×(σ / n) is calculated based on the relative variation (i.e., the coefficient of variation (σ / μ)) of multiple scan response delays x (= RTT). As a result, the "predetermined value" is calculated based on the relative variation (i.e., the coefficient of variation (σ / μ)) of multiple scan response delays x (= RTT).

[0220] Figure 13 shows the score z β Baseline scan response latency b calculated using RTT FIG.

[0221] Referring to Figure 13, the baseline scan response latency b RTT1 ~b RTT4 are associated with the data sets S1 to S4, respectively.

[0222] Baseline scan response latency b RTT1 In this case, the lower limit L corresponding to m / m'=1 b =0.010168 and upper limit U b =0.011069 is the number n, mean μ, standard deviation σ, and score z of the eight scan response delay RTTs associated with m / m'=1 in the dataset S1 in Figure 8. β Based on this, the lower limit L calculated by equations (4) to (7) b and upper limit U b is.

[0223] Also, baseline scan response latency b RTT1 In this case, the lower limit L corresponding to m / m'=2 b =0.009832 and upper limit U b=0.010905 is the number n, mean μ, standard deviation σ, and score z of the eight scan response delay RTTs associated with m / m'=2 in the dataset S1 in Figure 8. β Based on this, the lower limit L calculated by equations (4) to (7) b and upper limit U b is.

[0224] Similarly, the baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=3 b =0.010339 and upper limit U b =0.011398 is the number n, mean μ, standard deviation σ, and score z of the eight scan response delay RTTs associated with m / m'=3 in the dataset S1 in Figure 8. β Based on this, the baseline scan response delay b is calculated using equations (4) to (7). RTT1 In this case, the lower limit L corresponding to m / m'=4 b =0.010609 and upper limit U b =0.012378 is the number n, mean μ, standard deviation σ, and score z of the eight scan response delay RTTs associated with m / m'=4 in the dataset S1 in Figure 8. β Based on this, the baseline scan response delay b is calculated using equations (4) to (7). RTT1 In this case, the lower limit L corresponding to m / m'=5 b =0.017133 and upper limit U b =0.018105 is the number n, mean μ, standard deviation σ, and score z of the eight scan response delay RTTs associated with m / m'=5 in the dataset S1 in Figure 8. β Based on this, the baseline scan response delay b is calculated using equations (4) to (7). RTT1 In this case, the lower limit L corresponding to m / m'=6 b =0.017165 and upper limit U b =0.018322 is the number n, mean μ, standard deviation σ, and score z of the eight scan response delay RTTs associated with m / m'=6 in the dataset S1 in Figure 8. βBased on this, the baseline scan response delay b is calculated using equations (4) to (7). RTT1 In this case, the lower limit L corresponding to m / m'=7 b =0.017225 and upper limit U b =0.018513 is the number n, mean μ, standard deviation σ, and score z for the eight scan response delay RTTs associated with m / m'=7 in dataset S1 in Figure 8. β Based on this, the baseline scan response delay b is calculated using equations (4) to (7). RTT1 In this case, the lower limit L corresponding to m / m'=8 b =0.016817 and upper limit U b =0.017670 is the number n, mean μ, standard deviation σ, and score z of the eight scan response delay RTTs associated with m / m'=8 in the dataset S1 in Figure 8. β Based on the above, it is calculated by the formulas (4) to (7).

[0225] Similarly, the baseline scan response latency b corresponding to data set S2 RTT2 Lower limit L b and upper limit U b , baseline scan response latency b corresponding to Dataset S3 RTT3 Lower limit L b and upper limit U b , and the corresponding baseline scan response latency b in Dataset S4 RTT4 Lower limit L b and upper limit U b are the number n, mean μ, standard deviation σ, and score z for each of the eight scan response delay RTTs for m / m'=1 to 8 in the data sets S2 to S4 shown in Figure 8. β Based on the above, it is calculated by the formulas (4) to (7).

[0226] As shown in Figure 13, the score z β Baseline scan using response latency b RTT When calculating the score z, the same score z is used for each of the data sets S1 to S4. βis used, and in each of the data sets S1 to S4, one baseline scan response delay b RTT is calculated.

[0227] Note that the baseline scan response delay b RTT1 ~b RTT4 , the baseline scan response latency shown in Figure 11 b RTT1 ~b RTT8 , and the baseline scan response latency b shown in Figure 13 RTT1 ~b RTT4 is actually calculated using the number n, mean μ, and standard deviation σ of 1000 scan response delay RTTs corresponding to 1000 ports in one terminal device m (or one terminal device m').

[0228] Baseline scan response latency as described above b RTT and the baseline scan response delay b per terminal device m (or terminal device m') RTT The number of these is shown in Table 1.

[0229] [Table 1]

[0230] As shown in Table 1, the score z scr is one z scr_c and R(R=PN r_MAX ) z scr_1 ~z scr_R and one z β It consists of either:

[0231] Score Z scr There is one z scr_c If it consists of, score z scr_c is a scan response delay RTT (= x) randomly selected from R scan response delay RTTs corresponding to R ports in one terminal device. c ) and the lower limit L b and upper limit U bEach of these is assigned a score z scr_c Therefore, the baseline scan response delay per terminal device b RTT is 1 piece ({L b ,U b}).

[0232] Score Z scr R z scr_1 ~z scr_R If it consists of z scr_1 ~z scr_R are R scan response delays RTT (=x1 to x2) corresponding to R ports in one terminal device, respectively. R ) and the lower limit L b_r and upper limit U b_r Each of the R scores z scr_1 ~z scr_R Therefore, the baseline scan response delay per terminal device b RTT R ({L b1 ,U b1},L b2 ,U b2},···,L bR ,U bR}).

[0233] Score Z scr There is one z β If it consists of, score z β is the critical probability p calculated based on the confidence level * and the z table. b and upper limit U b Each of these is assigned a score z β Therefore, the baseline scan response delay per terminal device b RTT is 1 piece ({L b ,U b}).

[0234] The calculation means 16 calculates the scan response delay {RTT m,PNr _unknown}(m=1~M), scan response delay {RTT m’,PNr_known}_1(m'=1'~M'), scan response delay {RTT m’,PNr _known}_2(m'=1'~M') and scan response delay {RTT m’,PNr _known}_3 (m'=1'~M') for all baseline scan response latencies b RTT Calculate.

[0235] Here, scan response delay {RTT m,PNr Baseline scan response latency b for {_unknown} (m=1~M) RTT the baseline scan response latency {b RTT _ m,PNr _unknown}(m=1~M), and the scan response delay {RTT m’,PNr Baseline scan response latency b for _known}_1 (m'=1'~M') RTT the baseline scan response latency {b RTT _ m’,PNr _known}_1(m'=1'~M'), and the scan response delay {RTT m’,PNr Baseline scan response latency b for _known}_2 (m'=1'~M') RTT the baseline scan response latency {b RTT _ m’,PNr _known}_2(m'=1'~M'), and the scan response delay {RTT m’,PNr Baseline scan response latency b for _known}_3 (m'=1'~M') RTT the baseline scan response latency {b RTT _ m’,PNr _known}_3(m'=1'~M').

[0236] Thereafter, the calculation means 16 calculates the baseline scan response delay {b RTT _ m,PNr _unknown}(m=1~M) based on the mean absolute deviation W unknown Calculate the baseline scan response latency {b RTT _ m’,PNr _known}_1(m'=1'~M') based on the mean absolute deviation W known_1Calculate the baseline scan response latency {b RTT _ m’,PNr _known}_2(m'=1'~M') based on the mean absolute deviation W known_2 Calculate the baseline scan response latency {b RTT _ m’,PNr _known}_3(m'=1'~M') based on the mean absolute deviation W known_3 Calculate.

[0237] Mean absolute deviation W unknown ,W known_1 ,W known_2 ,W known_3 is calculated by the following formula:

[0238]

number

[0239] In equation (8), W represents the mean absolute deviation, and M a represents the number of terminal devices m / m' consisting of M or M', and x i_bRTT is the baseline scan response latency b RTT where m(Mo) is the most likely baseline scan response latency b RTT Represents the value of

[0240] The method for determining m(Mo) is explained below. Figure 14 shows the baseline scan response delay b RTT 15 is a diagram showing an example of the baseline scan response delay b RTT 16 is a graph showing the results of counting the number of samples for each value of the baseline scan response delay b RTT FIG.

[0241] FIG. 14 shows the baseline scan response delay b shown in FIG. 10 to help understand how to determine m(Mo). RTT1 ~b RTT4 The results are rounded to four decimal places. As shown in Figure 14, the baseline scan response latency b RTT1 ~bRTT4 consists of values ​​in the range [0,0.021].

[0242] Baseline scan response latency shown in Figure 14b RTT1 ~b RTT4 The number of samples for each value was counted, resulting in a baseline scan response latency of 0.009. RTT is "7" and the baseline scan response latency b is 0.010 RTT is "6", and the baseline scan response latency b is 0.012 RTT is "9", and the baseline scan response latency b is 0.013 RTT is "7" and the baseline scan response latency b is 0.015 RTT is "12" and the baseline scan response latency b is 0.016 RTT is "5" and the baseline scan response latency b is 0.017 RTT is "2" and the baseline scan response latency b is 0.019 RTT is "4" and the baseline scan response latency b is 0.020 RTT is "10" and the baseline scan response latency b is 0.021 RTT There are two of them, and the other baseline scan response delays b RTT The number of items is "0" (see FIG. 15).

[0243] Baseline scan response latency b RTT The number of samples for each value can be plotted as shown in Figure 16.

[0244] Figure 17 shows the baseline scan response latency b RTT 18 shows the results of changing some of the values ​​of the baseline scan response delay b RTT FIG.

[0245] To better understand how m(Mo) is determined, with reference to FIG. 17, the baseline scan response delay b RTT4 The upper limit of U bIn the example, the value corresponding to m / m'=8 was changed from "0.015" to "0.020" (see circle). As a result, as shown in Figure 18, the number of samples for "0.015" became 11, and the number of samples for "0.020" became 11.

[0246] Figure 19 shows the baseline scan response delay b when the horizontal axis scale is changed to "0.005" units in Figure 18. RTT Figure 20 shows the distribution of baseline scan response latencies b RTT 21 shows a specific distribution of the probability density function of the baseline scan response delay.

[0247] Baseline scan response latency shown in Figure 18 b RTT Counting the number of samples for each specific value in the specific distribution of and displaying the number of samples for each specific value, we get the result shown in Figure 19. As a result, from Figure 19, we can see that the baseline scan response delay b RTT It can be seen that is most likely to occur at probability density values ​​of 0.015 and 0.020.

[0248] As shown in Figure 20, the number of samples generated in each of the PDF regions of 0.0 to 0.005, 0.0051 to 0.010, 0.011 to 0.015, 0.0151 to 0.020, 0.021 to 0.025, and 0.0251 to 0.030 are "0", "13", "27", "22", "2", and "0", respectively.

[0249] The correspondence between the PDF area and the number of samples occurring in each area is as shown in Figure 21. The number of samples (=22) occurring in the PDF area of ​​0.0151 to 0.020 is r, the interval of 0.0151 to 0.020 (=0.005) is h, 0.015 is L, the number of samples (=27) occurring in the PDF area of ​​0.011 to 0.015 is r1, and the number of samples (=2) occurring in the PDF area of ​​0.0211 to 0.025 is r2. Therefore, the mode m (Mo) can be calculated by substituting r (=22), r1 (=27), r2 (=2), h (=0.005), and L (=0.015) into the following equation.

[0250]

number

[0251] In equation (9), L is the most likely baseline scan response delay b when the network scan result is unknown or known. RTT The lower boundary of the region of the probability density function (= PDF (Probability Density Function) region) that includes these two values ​​is also shown. r is the number of samples in the PDF region that are most likely to occur when the results of the network scan are unknown or known (= baseline scan response delay b RTT Furthermore, r1 denotes the number of times the baseline scan response delay b is longer than the most likely PDF region when the network scan result is unknown or when the network scan result is known. RTT The number of samples in the previous PDF region in the direction where the value is smaller (= baseline scan response delay b RTT Furthermore, r2 denotes the number of times the baseline scan response delay b is longer than the most likely PDF region when the network scan result is unknown or when the network scan result is known. RTT The number of samples in the next PDF region in the direction where the value is larger (= baseline scan response delay b RTTFurthermore, h denotes the interval between each PDF region (i.e., the width of each PDF region).

[0252] As shown in Figure 21, r is the baseline scan response latency b RTT The PDF region REG bounded by the two most likely values ​​(=0.015 and 0.020) of and the curve showing the probability density function PDF Baseline scan response latency in b RTT L is the baseline scan response latency b RTT The lower value (=0.015) of the two most likely values ​​(=0.015 and 0.020) of h is the value of the PDF region REG. PDF than baseline scan response latency b RTT The smaller side of the PDF area REG PDF and r2 is the number of samples in the region of width h adjacent to the PDF region REG PDF than baseline scan response latency b RTT The larger side of the PDF area REG PDF It consists of the number of samples in a region of width h adjacent to

[0253] Figure 22 shows the baseline scan response latency b RTT 23 is a diagram showing another example of the baseline scan response delay b RTT 24 shows the distribution of the baseline scan response delay b when the horizontal axis in FIG. 23 is scaled in units of "0.005". RTT FIG.

[0254] Referring to FIG. 22, to help understand how m(Mo) is determined, the baseline scan response delay b RTT2 Lower limit L b In the above, the value corresponding to m / m'=1 was changed from "0.013" to "0.012" (see circle), and the baseline scan response delay b RTT4 Lower limit L bIn the above, the value corresponding to m / m'=2 was changed from "0.013" to "0.012" (see circle), and the baseline scan response delay b RTT4 The upper limit of U b In the above, the value corresponding to m / m'=8 was changed from "0.015" to "0.020" (see circle). As a result, as shown in Figure 23, the baseline scan response delay b RTT There are three possible values ​​of . In this case, the three most likely baseline scan response latencies b RTT Adjust the interval h to the width of the PDF region that contains the values ​​of . Then, adjust ... RTT Then, the two baseline scan response delays b that are most likely to produce the minimum and maximum values ​​are detected. RTT are used as the values ​​of r, r1, r2, L, and h are determined by the method described with reference to FIGS. 18 to 21, and the determined r, r1, r2, L, and h are substituted into equation (9) to calculate the mode m (Mo).

[0255] The most likely baseline scan response delay is b RTT If there are four or more values ​​of b, the most likely baseline scan response delay b RTT The mode m (Mo) is calculated in the same manner as when there are three values ​​of .

[0256] The calculation means 16 calculates the baseline scan response delay {b RTT _ m,PNr _unknown} (m=1~M), calculate the mode m(Mo)_unknown and the baseline scan response delay {b RTT _ m’,PNr _known}_1 (m'=1'~M'), calculate the mode m(Mo)_known_1, and calculate the baseline scan response delay {b RTT _ m’,PNr _known}_2 (m'=1'~M'), calculate the mode m(Mo)_known_2, and calculate the baseline scan response delay {b RTT _ m’,PNrCalculate the mode m(Mo)_known_3 based on {_known}_3 (m'=1' to M').

[0257] Then, the calculation means 16 calculates the baseline scan response delay {b RTT _ m,PNr _unknown}(m=1~M) and mode m(Mo)_unknown are substituted into equation (8B) to obtain C(x i_bRTT ) and calculate the calculated C(x i_bRTT ) and baseline scan response latency {b RTT _ m,PNr _unknown}(m=1~M) and the number of samples M into equation (8A) to obtain the mean absolute deviation W unknown Calculate.

[0258] The calculation means 16 also calculates the baseline scan response delay {b RTT _ m’,PNr _known}_1(m'=1'~M') and mode m(Mo)_known_1 are substituted into equation (8B) to obtain C(x i_bRTT ) and calculate the calculated C(x i_bRTT ) and baseline scan response latency {b RTT _ m’,PNr _known}_1 (m'=1'~M') and the number of samples M' are substituted into equation (8A) to obtain the mean absolute deviation W known_1 Calculate.

[0259] Furthermore, the calculation means 16 similarly calculates the baseline scan response delay {b RTT _ m’,PNr _known}_2 (m'=1'~M') and mode m(Mo)_known_2, the mean absolute deviation W is calculated by Eq. (8). known_2 Calculate the baseline scan response latency {b RTT _ m’,PNr _known}_3 (m'=1'~M') and mode m(Mo)_known_3, the mean absolute deviation W is calculated by Eq. (8). known_3 Calculate.

[0260] Note that C(x i_bRTT) is expressed by the following formula:

[0261]

number

[0262] In equation (10), the first line of the equation is the baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=1 b (=0.012) and upper limit U b (=0.009) respectively using C(a 1Lb ),C(a 1Ub ) and the second equation is the baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=2 b (=0.012) and upper limit U b (=0.009) respectively using C(a 2Lb ),C(a 2Ub ) and similarly, the equation in the last line is the baseline scan response delay b RTT1 In this case, the lower limit L corresponding to m / m'=8 b (=0.019) and upper limit U b (=0.015) and C(a ALb ),C(a AUb ) is calculated. In other words, equation (10) calculates the baseline scan response delay b RTT1 In this case, the baseline scan response delay {L b ,U b}, the baseline scan response delay corresponding to m / m'=2 {L b ,U b}, ···, the baseline scan response delay corresponding to m / m'=8 {L b ,U b} into equation (8B) to obtain C(x i_bRTTAs a result, A is calculated based on the baseline scan response delays b RTT Indicates the total number of

[0263] As shown in FIG. 11, multiple baseline scan response delays b RTT1 ~b RTT8 is calculated, the first line of equation (10) is used to calculate the baseline scan response delay b RTT1 The lower limit L corresponding to m / m'=1 b (=0.011943) and upper limit U b (=0.009294) respectively using C(a 1Lb ),C(a 1Ub ) and the second equation calculates the baseline scan response latency b RTT2 The lower limit L corresponding to m / m'=1 b (=0.011944) and upper limit U b (=0.009293) and C(a 2Lb ),C(a 2Ub ) and similarly, the final equation is used to calculate the baseline scan response latency b RTT8 The lower limit L corresponding to m / m'=8 b (=0.019396) and upper limit U b (=0.015091) and C(a ALb ),C(a AUb ) is calculated. In other words, equation (10) calculates the baseline scan response delay b RTT1 ~b RTT8 , the baseline scan response latency b corresponding to m / m'=2 RTT1 ~b RTT8 , ···, baseline scan response delay b corresponding to m / m'=8 RTT1 ~b RTT8Substituting these into equation (8B) to get C(x i_bRTT As a result, A consists of 8×8=64 (i.e., [total number M (or M') of terminal devices in each data set S1 to S4] × [number of ports in one terminal device m / m']).

[0264] The calculation means 16 calculates the mean absolute deviation W unknown ,W known_1 ,W known_2 ,W known_3 When the similarity rate b is calculated, sim Calculate.

[0265]

number

[0266] In this case, the calculation means 16 calculates the mean absolute deviation W unknown ,W known_1 Substituting into equation (11) the similarity rate b sim1 Calculate the mean absolute deviation W unknown ,W known_2 Substituting into equation (11) the similarity rate b sim2 Calculate the mean absolute deviation W unknown ,W known_3 Substituting into equation (11) the similarity rate b sim3 Calculate.

[0267] Then, the calculation means 16 calculates the three similarity rates b sim1 ,b sim2 ,b sim3 is output to the update means 14.

[0268] The update means 14 updates the three similarity rates b sim1 ,b sim2 ,b sim3 is received from the calculation means 16, and the three similarity rates b sim1 ,b sim2 ,b sim3 The largest similarity rate b sim_max (b sim1 ,b sim2 ,b sim3The largest similarity rate b sim_max If there are a plurality of similarities b sim_max Detect any one of the following:

[0269] By transforming equation (11), the following equation is obtained.

[0270]

number

[0271] Figure 25 shows the similarity rate b sim and mean absolute deviation W known In FIG. 25, the vertical axis represents the similarity rate b sim The horizontal axis represents the mean absolute deviation W known Also, the curve k3 represents the mean absolute deviation W unknown The similarity rate b when is a constant value (=3) sim and mean absolute deviation W known In Figure 25, the mean absolute deviation W known ,W unknown is the baseline scan response latency b shown in Figures 10, 11, 13, and 14. RTT The similarity rate b sim is the mean absolute deviation W known Therefore, in Figure 25, the mean absolute deviation W unknown is set to a constant value (e.g., "3"), and the mean absolute deviation W known was varied in the range of 1 to 11.

[0272] Referring to Figure 25, the similarity rate b sim is W known =W unknown When = 3, it is at its maximum and is 1.0. Also, the similarity rate b sim is the mean absolute deviation W known is the mean absolute deviation W unknown When the shift is smaller than the mean absolute deviation Wknown is the mean absolute deviation W unknown When the similarity rate b is shifted to a larger value, it becomes smaller than the maximum value (=1.0) (see curve k3). sim is the mean absolute deviation W known is the mean absolute deviation W unknown It gets larger as it approaches .

[0273] Figure 26 shows the largest similarity rate b sim_max 26 is a diagram for explaining a method for detecting three similarity rates b sim_1 ,b sim_2 ,b sim_3 Among them, the similarity rate b sim_1 When is the maximum, the maximum similarity b sim_max is the similarity rate b sim_1 and the similarity rate b sim_2 When is the maximum, the maximum similarity b sim_max is the similarity rate b sim_2 and the similarity rate b sim_3 When is the maximum, the maximum similarity b sim_max is the similarity rate b sim_3 It consists of:

[0274] Also, the three similarity rates b sim_1 ,b sim_2 ,b sim_3 In b sim_1 =b sim_2 >b sim_3 When this is the case, the maximum similarity b sim_max is the similarity rate b sim_1 or similarity rate b sim_2 It consists of b sim_1 =b sim_3 >b sim_2 When this is the case, the maximum similarity b sim_max is the similarity rate b sim_1 or similarity rate b sim_3 It consists of b sim_2 =b sim_3 >b sim_1 When this is the case, the maximum similarity b sim_max is the similarity rate b sim_2 or similarity rate b sim_3 It consists of b sim_2 =b sim_3 >b sim_1 When this is the case, the maximum similarity bsim_max is the similarity rate b sim_1 ,b sim_2 ,b sim_3 It consists of any one of the following:

[0275] The update means 14 updates the maximum similarity b sim_max When detecting, the calculation means 16 calculates three similarity rates b sim_1 ,b sim_2 ,b sim_3 received b sim_1 ,b sim_2 ,b sim_3 Based on this, the maximum similarity b sim_max Detect.

[0276] The update means 14 updates the largest similarity b sim_max When the largest similarity rate b sim_max The baseline scan response latency b when RTT (Baseline scan response latency {b RTT _ m’,PNr _known}_1(m'=1'~M'),{b RTT _ m’,PNr _known}_2(m'=1'~M'),{b RTT _ m’,PNr _known}_3 (m'=1'~M')) m’,PNr _known_b sim_max}(m'=1'~M')(Scan response delay {RTT m’,PNr _known}_1(m'=1'~M'),{RTT m’,PNr _known}_2(m'=1'~M'),{RTT m’,PNr _known}_3 (m'=1'~M')) and scan response delay {RTT m,PNr _unknown}(m=1~M) and the scan response delay {RTT m’,PNr _known_aggregation}(m'=1'~M').

[0277] Scan Response Delay {RTT m’,PNr _known_bsim_max}(m'=1'~M') and scan response delay {RTT m,PNr _unknown}(m=1 to M) will be explained.

[0278] 27 is a diagram illustrating a method for aggregating two similar sets. Referring to FIG. 27(a), a set S A ,S B is the largest similarity b sim_max are two sets when A ,S B Each of these includes 16 scan response delays RTT in one terminal device as elements.

[0279] The update means 14 updates the set S A and set S B The set S of scan response delay RTTs included in both A ∩S B (See (b) of Figure 27). In this case, the set S A contains four "0.019", six "0.021" and two "0.020999", but the set S A ∩S B When detecting, four "0.019", six "0.021", and two "0.020999" are grouped into a set S as one "0.019", one "0.021", and one "0.020999", respectively. A ∩S B Configure.

[0280] The update means 14 then updates the set S B From the elements of set S A ∩S B Remove the elements of S' to create a set S' B (See (c) of FIG. 27).

[0281] Then, the updating means 14 updates the set S' B Set S A to the elements of the aggregated set S aggregation (See (d) of FIG. 27). In (d) of FIG. 27, the set surrounded by the dotted line is the set S'B As a result, the set S aggregation consists of 22 scan response delay RTTs.

[0282] The set S shown in (d) of FIG. aggregation is a set S A The scan response delay RTT of one terminal device included in the set S B This is an aggregate of the scan response delay RTT of one terminal device included in the table.

[0283] The updating means 14 applies the aggregation method described in FIG. A ,S B Execute the scan response delay RTT of all the terminal devices included in the set S A The scan response delay RTT and the set S B The scan response delay RTT included in m’,PNr Generate a known aggregation {m'=1'~M'}. Scan response delay {RTT m’,PNr _known_aggregation} (m'=1' to M'), the number of scan response delay RTTs in one terminal device m' is the original set S A may be increased more than

[0284] Figure 28 shows the known scan response delay {RTT m’ 10 is a diagram for explaining a method for updating {_known}_z (m'=1' to M', z=1 to 3). FIG.

[0285] 28, in correspondence table TBL3, the scan response delay RTT is associated with a known estimation result as shown in FIG. 3 (see (a) of FIG. 28).

[0286] The update means 14 is configured to update the scan response delay (RTT) in a communication environment where the result of the network scan is unknown. m _unknown}(m=1~M) is obtained, and the scan response delay {RTT m’_known}_2(m'=1'~M') is the scan response delay {RTT m _unknown}(m=1~M) with the largest similarity b sim_max , the scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m _unknown}(m=1~M) and the scan response delay {RTT m’ _known_aggregation}_2 (m'=1' to M'). Then, the update means 14 generates the scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known_aggregation}_2 (m'=1' to M') is stored in correspondence table TBL3 in association with the known estimation result "insufficient communication quality" to update correspondence table TBL3 (see (a) and (b) of FIG. 28).

[0287] In addition, the update means 14 updates the scan response delay {RTT m _unknown}(m=1~M) is obtained, and the scan response delay {RTT m’ _known}_1(m'=1'~M') is the new scan response delay {RTT m _unknown}(m=1~M) with the largest similarity b sim_max , the scan response delay {RTT m’ _known}_1(m'=1'~M') and new scan response delay {RTT m _unknown}(m=1~M) and the scan response delay {RTT m’ _known_aggregation}_1 (m'=1' to M'). Then, the update means 14 generates the scan response delay {RTT m’ _known}_1(m'=1'~M') and scan response delay {RTT m’_known_aggregation}_1 (m'=1' to M') is stored in correspondence table TBL3 in association with the known estimation result "normal" (see (b) and (c) of FIG. 28), thereby updating correspondence table TBL3.

[0288] In this way, the update unit 14 can calculate the scan response delay (RTT) in a communication environment where the results of the network scan are unknown. m _unknown}(m=1~M) is acquired, the scan response delay {RTT m _unknown}(m=1~M) and the scan response delay {RTT m’ _known}_z (m'=1'~M', z=1~3) and the scan response delay {RTT m’ _known_aggregation}_z (m'=1'~M', z=1~3) and calculate the generated scan response delay {RTT m’ _known_aggregation}_z (m'=1'~M', z=1~3) m’ _known}_z (m'=1' to M', z=1 to 3) is updated (that is, the host information database 13 is updated).

[0289] Although not shown in FIG. 28, the updated scan response delay {RTT m’ _known_aggregation}_2(m'=1'~M') is the newly aggregated scan response delay {RTT m’ It may be further updated by _known_aggregation(new)}_2(m'=1'~M').

[0290] In this way, the scan response delay (RTT) associated with the known estimation result is m’ _known}_z (m'=1' to M', z=1 to 3), the scan response delay RTT obtained for one terminal device m' m’The number of RTTs increases due to aggregation. As a result, as shown in FIG. 92 described later, the probability density function of the scan response delay RTT when the known estimation result is "normal", the probability density function of the scan response delay RTT when the known estimation result is "insufficient communication quality", and the probability density function of the scan response delay RTT when the known estimation result is "network congestion" are all different from each other. Therefore, the scan response delay RTT m’ _known}_z (m'=1'~M', z=1~3), the scan response delay {RTT m _unknown}(m=1~M) similar to the scan response delay {RTT m’ _known}_z (m'=1'~M', z=1~3) is used as the similarity rate b sim This can reduce errors in detection.

[0291] In the embodiment of the present invention, the aggregate scan response delay {RTT m’ _known_aggregation}_1(m'=1'~M'),{RTT m’ _known_aggregation}_2 (m'=1' to M') constitutes "M' aggregate scan response delays."

[0292] 3 types of scores scr_c ,z scr_1 ~z scr_R ,z β The similarity rate b calculated using sim This article explains:

[0293] As described in Table 1, baseline scan response latencies b RTT are three types of scores z scr_c ,z scr_1 ~z scr_R ,z β The results are calculated using one of three scores: scr_c ,z scr_1 ~z scr_R ,z β Three different baseline scan response latencies depending on b RTTis calculated.

[0294] Figure 29 shows three types of scores z scr_c ,z scr_1 ~z scr_R ,z β The similarity rate b calculated using sim FIG.

[0295] In FIG. 29, (a) shows the score z scr_c The similarity rate b calculated using sim (b) shows the score z scr_1 ~z scr_R The similarity rate b calculated using sim is shown for known estimation results (normal, poor communication quality, and network congestion), and (c) is the score z β The similarity rate b calculated using sim is shown for known estimation results (normal, poor communication quality, and network congestion).

[0296] As shown in Figure 29(a), (b), and (c), there are three types of scores z scr_c ,z scr_1 ~z scr_R ,z β The similarity rate b calculated using sim is almost the same for the known estimation results (normal, poor communication quality, and network congestion). Therefore, the mean absolute deviation W known_1 ,W known_2 ,W known_3 are three types of scores z scr_c ,z scr_1 ~z scr_R ,z β The results are almost the same regardless of which method is used to calculate the maximum similarity b sim_max Even if the maximum similarity rate b sim_max will be the same.

[0297] The advantage of aggregating scan response delay RTT is explained below. The scan response delay (RTT) when the network scan result becomes a known estimated result of "normal" is calculated. m’ _known}_1(m'=1'~M') C and the scan response delay {RTT m’ Set S consisting of {_known}_2(m'=1'~M') D and the scan response delay {RTT} when the network scan result is a known estimated result of "network congestion." m’ Set S consisting of {_known}_3 (m'=1'~M') E and the scan response delay (RTT) obtained in a communication environment where the results of network scans are unknown. m _unknown}_1(m=1~M) F and the scan response delay (RTT) obtained in a communication environment where the results of network scans are unknown. m _unknown}_2(m=1~M) G Assume that there exists a set S C ,S D ,S E consists of multiple types of scan response delay RTTs that are dissimilar to each other. m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3(m'=1'~M') are mutually dissimilar.

[0298] Figure 30 shows the set S G 10 is a diagram for explaining a problem that occurs when determining sets that are similar to each other.

[0299] Referring to Figure 30, the set S E is a set S F Similarly, the set S E is a set S G is not similar to the set SF is a set S G Similar to.

[0300] In this situation, the set S F is a set S C ,S D ,S E Among them, set S E Similarity rate b with sim is 0.89, and the largest similarity rate b sim_max (See FIG. 30(a)).

[0301] On the other hand, set S G is a set S C ,S D ,S E Among them, set S D Similarity rate b with sim is 0.82, and the largest similarity rate b sim_max (See FIG. 30(b)).

[0302] As a result, the set S F is the set S E The probability of correctly guessing that the set S is similar to G The probability that the set S will erroneously estimate a similar set is 98% (see (d) in Figure 30). G is a set S F Since it is similar to E Similarity rate b with sim must be the largest, but in reality, the set S D Similarity rate b with sim As a result, the set S G There is a 98% chance that the similar sets will be misidentified.

[0303] Therefore, in the assumed situation shown in Figure 30, the set S G Scan Response Delay {RTT m _unknown}_2(m=1~M) may result in incorrect interpretation of the results of network scans.

[0304] Figure 31 shows the set S E,S F ,S G Scan response latency RTT and baseline scan response latency b RTT FIG.

[0305] In FIG. 31, (a) shows the set S E Scan response latency RTT and baseline scan response latency b RTT (b) shows the distribution of the set S F Scan response latency RTT and baseline scan response latency b RTT (c) shows the distribution of the set S G Scan response latency RTT and baseline scan response latency b RTT The distribution of

[0306] Referring to Figure 31, the set S E ,S F ,S G In the baseline scan, the response latency b RTT The distribution range of is narrower than the distribution range of scan response delay RTT.

[0307] Figure 32 shows the similarity rate between two sets. (a) of Figure 32 shows the similarity rate between set S C ,S D ,S E and set S F Similarity rate b with sim , and (b) of FIG. 32 shows the set S C ,S D ,S E and set S G Similarity rate b sim , and (c) of FIG. 32 shows the set S C ,S D ,S F and set S G Similarity rate b with sim Shows.

[0308] The same scan response delay {RTT} when a known estimated result is obtained m’_known}_z (m'=1'~M', z=1~3) but similar to each other, when the results of two network scans are unknown {RTT m _unknown}_1(m=1~M),{RTT m _unknown}_2(m=1~M), threshold b sim_th Similarity rate greater than b sim Scan response delay {RTT m’ _known}_z (m'=1' to M', z=1 to 3) and the scan response delay {RTT m _unknown}_1(m=1~M). Note that the threshold b sim_th is set to, for example, 0.85.

[0309] As mentioned above, the set S C scan response delay {RTT m’ _known}_1(m'=1'~M'), and the set S D scan response delay {RTT m’ _known}_2(m'=1'~M'), and the set S E scan response delay {RTT m’ _known}_3 (m'=1'~M'), and the set S F scan response delay {RTT m _unknown}_1 (m=1~M), and the set S G scan response delay {RTT m _unknown}_2 (m=1 to M), so (a) in Figure 32 shows the scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m'=1'~M') and the scan response delay {RTT m Similarity rate b with _unknown}_1(m=1~M)sim , and (b) in Figure 32 shows the scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m'=1'~M') and the scan response delay {RTT m Similarity rate b with _unknown}_2(m=1~M) sim , and (c) of Figure 32 shows the scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2 (m'=1'~M') and the scan response delay {RTT m _unknown}_1 (m=1~M) and the scan response delay {RTT m Similarity rate b with _unknown}_2(m=1~M) sim Shows.

[0310] In FIG. 32(a), the set S E and set S F Similarity rate b with sim is the threshold b sim_th The maximum similarity b is greater than (=0.85) sim_max (=“0.89”), so the set S E is a set S F Similar to.

[0311] In FIG. 32(b), the set S E and set S G Similarity rate b with sim is the threshold b sim_th Since it is “0.73” which is smaller than (=0.85), the set S E is a set S G Not similar to.

[0312] In FIG. 32(c), the set S F and set S G Similarity rate b with sim is the threshold b sim_th The maximum similarity b is greater than (=0.85) sim_max (=“0.94”), so the set S F is a set S G Similar to.

[0313] Therefore, the set S in the assumed situation in Figure 30 E ,S F ,S G A similarity / dissimilarity relationship exists between them.

[0314] Then, the set S described in the assumed situation in Figure 30 E ,S F ,S G As mentioned above, when the similarity / dissimilarity relationship between the sets S G There is a problem of incorrectly estimating sets similar to .

[0315] Therefore, the updating means 14 updates the threshold value b sim_th Similarity ratio b greater than (=0.85) sim Two sets S with (=0.89) E ,S F and the two detected sets S E ,S F In other words, the update unit 14 aggregates the scan response delay {RTT m’ _known}_3 (m'=1'~M') and the scan response delay {RTT m _unknown}_1(m=1~M) are aggregated using the method described above.

[0316] Figure 33 shows the set S E ,S F Scan Response Delay {RTT m’ _known}_3(m'=1'~M'),{RTT m33A is a diagram showing an example of the set S E Scan Response Delay {RTT m’ _known}_3(m'=1'~M'), and (b) of Figure 33 shows the set S F Scan Response Delay {RTT m _unknown}_1(m=1~M) E ,S F In each of the above, the number of ports is 16 and the number of terminal devices m / m' is 8.

[0317] The update means 14 updates two sets S E ,S F When aggregating, set S E 16 scan response delay RTTs corresponding to m / m'=1 E Detect and set S F 16 scan response delay RTTs corresponding to m / m'=1 F Detected 16 scan response delays RTT E and 16 scan response delay RTTs F and are aggregated using the method described above.

[0318] Thereafter, the updating means 14 updates the set S E 16 scan response delay RTTs corresponding to m / m'=2 E Detect and set S F 16 scan response delay RTTs corresponding to m / m'=2 F Detected 16 scan response delays RTT E and 16 scan response delay RTTs F and are aggregated using the method described above.

[0319] Similarly, the updating means 14 updates the set S E 16 scan response delays from RTT E Detect and set S F or 16 scan response delay RTT F Detected 16 scan response delays RTT Eand 16 scan response delay RTTs F The sum of the above results in 6 sets of 16 scan response delay RTTs corresponding to m / m' = 3 to 8. E and 16 scan response delay RTTs F , and two sets S E ,S F Aggregate.

[0320] Figure 34 shows the set S E and set S F The aggregate set S H 34, the set S H is a set S E Scan Response Delay {RTT m’ _known}_3(m'=1'~M') and the set S E ,S F Disjoint set S F The scan response delay is RTT.

[0321] set S E Scan Response Delay {RTT m’ _known}_3 (m'=1' to M') consists of 16 scan response delay RTTs corresponding to m / m'=1 to 8. E ,S F Disjoint set S F The scan response delay RTT is calculated by the set S corresponding to each of m / m'=1 to 8. E ,S F Disjoint set S F The scan response delay consists of one or more RTTs.

[0322] As a result, the set S H corresponds to m / m'=1 to 8, and the set S E ,S F Disjoint set S F and one or more scan response delays RTT.

[0323] FIG. 35 shows the set S shown in FIG. E Scan Response Delay {RTTm’ _known}_3(m'=1'~M') and the set S E ,S F Disjoint set S F 10A and 10B are diagrams illustrating examples of scan response delay RTT.

[0324] (a) of FIG. 35 shows the set S shown in FIG. E Scan Response Delay {RTT m’ _known}_3 (m'=1' to M'), and (b) of FIG. 35 shows the set S E ,S F Disjoint set S F An example of scan response delay RTT is shown below.

[0325] Referring to Figure 35, the set S E ,S F Disjoint set S F The six scan response delay RTTs corresponding to m / m'=1 of the scan response delay RTTs are included in the set S E Scan Response Delay {RTT m’ _known}_3 (m'=1'~M') is added to the 16 scan response delay RTTs corresponding to m / m'=1.

[0326] Also, the set S E ,S F Disjoint set S F The 16 scan response delay RTTs corresponding to m / m'=2 of the scan response delay RTTs are in the set S E Scan Response Delay {RTT m’ _known}_3 (m'=1' to M') are added to the 16 scan response delay RTTs corresponding to m / m'=2. The same applies to the scan response delay RTTs corresponding to m / m'=3 to 8.

[0327] Figure 36 shows the set S G Scan Response Delay {RTT m _unknown}_2 (m=1 to M). G ,S H FIG. 10 shows baseline scan response latency.

[0328] (a) of FIG. 37 shows the set S G Scan Response Delay {RTT m _unknown}_2 (m=1 to M), the baseline scan response delay b of eight terminal devices calculated by randomly selecting one scan response delay RTT from the 16 scan response delay RTTs corresponding to each terminal device m / m' RTT Shows.

[0329] 37(b) shows the set S shown in FIGS. 34 and 35. H The baseline scan response delay b of eight terminal devices is calculated by randomly selecting one scan response delay RTT from the multiple scan response delay RTTs corresponding to each terminal device m / m'. RTT Shows.

[0330] Figure 38 shows the set S G ,S H Scan response latency RTT and baseline scan response latency b RTT FIG.

[0331] In FIG. 38, (a) shows the set S G Scan response latency RTT and baseline scan response latency b RTT (b) shows the distribution of the set S H Scan response latency RTT and baseline scan response latency b RTT The distribution of

[0332] set S G ,S H In both cases, the baseline scan response latency b RTT The distribution range of is narrower than the distribution range of scan response delay RTT.

[0333] Figure 39 shows the set S C ,S D ,S H and set S G Similarity rate b with simThe figure shows the similarity rate b sim is the baseline scan response latency b shown in Figure 37 RTT 39, the set S C and set S G Similarity rate b with sim and set S D and set S G Similarity rate b with sim is the threshold b sim_th (=0.85). On the other hand, the set S H and set S G Similarity rate b with sim is the threshold b sim_th (=0.85) is "1.00". Therefore, the set S G is a set S C ,S D is not similar to the set S H Similar to.

[0334] FIG. 40 is a diagram illustrating a comparison between a method of aggregating scan response delays and a method of not aggregating scan response delays.

[0335] Referring to FIG. 40, when a method that does not aggregate scan response delays is used, the set S G The scan response delay RTT of is similar to the scan response delay RTT when the known estimated result is "poor communication quality (Low-SNR)" b sim is the largest (see Figure 40(a)).

[0336] As a result, the set S G The similarity of the sets is estimated incorrectly with a probability of 98% (see Figure 40(b)).

[0337] On the other hand, when the method of aggregating scan response delays is used, the set S G The scan response delay RTT of is the scan response delay RTT (= set S E Similarity rate b with scan response delay (RTT) sim is the largest (see Figure 40(c)).

[0338] As a result, the set S G It correctly estimates similar sets with a probability of 99% (see (d) in Figure 40).

[0339] set S H is the scan response delay (RTT) when the known estimated result is "Network Congestion". m’ Set S consisting of {_known}_3 (m'=1'~M') E and scan response delay {RTT} when the results of the network scan are unknown. m _unknown}_1(m=1~M) F (Set S F is a set S E It consists of a set of aggregates of (similar to

[0340] More specifically, the set S H is a set S E Scan Response Delay {RTT m’ _known}_3(m'=1'~M'), the set S F Scan Response Delay {RTT m _unknown}_1(m=1~M), set S E Scan response delay not included in RTT m _unknown_1 is added for each terminal device m / m' (see FIGS. 34 and 35).

[0341] As a result, the set S H In the set S G A set S similar to F Since it contains G is a set S of scan response delays when the known estimated result is "network congestion" (or similar). H is correctly estimated to be similar to

[0342] Therefore, by using the method of aggregating scan response delays, the “set S GThis solves the problem of "high probability of misestimating similar sets."

[0343] As described above, in the embodiment of the present invention, after estimating the results of the network scan, the scan response delay {RTT m’ _known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3 (m'=1'~M'), the scan response delay {RTT m _unknown}(m=1~M) m’ _known}_similarity (m'=1'~M') and the detected scan response delay {RTT m’ _known}_similarity(m'=1'~M') and scan response delay {RTT m _unknown}(m=1~M) and the aggregated scan response delay {RTT m’ _known_aggregation} (m'=1' to M') (i.e., the host information database 13 is updated). As a result, as described above, the scan response delay {RTT m’ _known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3 (m'=1'~M'), the scan response delay {RTT m _unknown}(m=1~M) similar to the scan response delay {RTT m’ _known_similarity}_z (m'=1'~M', z=1~3) can be erroneously estimated.

[0344] Next, extraction of the communication characteristic CMbest that is most suitable for estimating the results of a network scan will be described.

[0345] Fig. 41 is a diagram showing correlation coefficients of communication characteristics. The correlation coefficients shown in Fig. 41 are correlation coefficients of communication characteristics when communication quality is insufficient when a network scan is performed without monitoring background traffic. In addition, the scan response delay RTT shown in Fig. 41 m , packet loss rate PLR m , maximum scan delay DLmax m , and the minimum scan delay DLmin m are multiple communication characteristics of M terminal devices when a network scan is performed on M terminal devices. m indicates the scan response delay for one IP address targeted by a network scan, or the average of multiple scan response delays.

[0346] Scan Response Delay RTT m , packet loss rate PLR m , maximum scan delay DLmax m , and the minimum scan delay DLmin m The correlation coefficients between the above and below diagonal elements are shown in Figure 41. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in Figure 41 because they are the same as the correlation coefficients of the off-diagonal elements below the diagonal elements.

[0347] Referring to FIG. 41, when a network scan is performed without monitoring background traffic, if the communication quality is insufficient, the scan response delay RTT m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and the minimum scan delay DLmin m Correlation coefficient and maximum scan delay DLmax m and the minimum scan delay DLmin m The correlation coefficient with scan response delay RTT is greater than 0.5. m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and the minimum scan delay DLminm The correlation coefficient with the maximum scan delay DLmax m and the minimum scan delay DLmin m Therefore, the correlation coefficient is larger than that of the scan response delay RTT m is the maximum scan delay DLmax m and scan minimum delay DLmin m It has a large correlation with

[0348] Fig. 42 is a diagram showing another correlation coefficient of communication characteristics. The correlation coefficient shown in Fig. 42 is a correlation coefficient of communication characteristics when the network is congested (network congestion) in the case where a network scan is performed without monitoring background traffic. In addition, the scan response delay RTT m indicates the average value within one IP address that is the target of a network scan. Scan response delay RTT m , packet loss rate PLR m , maximum scan delay DLmax m , and the minimum scan delay DLmin m The correlation coefficients between the scan response delay RTTs shown in Fig. 42 are shown. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in Fig. 42 for the same reason as explained in Fig. 41. m , packet loss rate PLR m , maximum scan delay DLmax m , and the minimum scan delay DLmin m are a plurality of communication characteristics of M terminal devices when a network scan is executed on M terminal devices.

[0349] Referring to Figure 42, when a network scan is performed without monitoring background traffic, the scan response delay (RTT) is m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and the minimum scan delay DLmin m Correlation coefficient and maximum scan delay DLmaxm and the minimum scan delay DLmin m The correlation coefficient with scan response delay RTT is greater than 0.5. m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and the minimum scan delay DLmin m The correlation coefficient with scan response delay RTT is the same. m and maximum scan delay DLmax m Correlation coefficient with scan response delay RTT m and the minimum scan delay DLmin m The correlation coefficient with the maximum scan delay DLmax m and the minimum scan delay DLmin m Therefore, even when the network is congested, the scan response delay RTT m is the maximum scan delay DLmax m and scan minimum delay DLmin m It has a large correlation with

[0350] As shown in Figures 41 and 42, when a network scan is performed without monitoring background traffic, the scan response delay (RTT) is significantly longer when communication quality is poor and when the network is congested. m , packet loss rate PLR m , maximum scan delay DLmax m , and the minimum scan delay DLmin m The correlation coefficients of the two are different.

[0351] Fig. 43 is a diagram showing yet another correlation coefficient of communication characteristics, which is a correlation coefficient of communication characteristics when communication quality is insufficient when a network scan is performed by monitoring background traffic.

[0352] Scan Response Delay RTT m , packet loss rate PLR m , Delayed DL m , Jitter JT m, background maximum delay BGDLmax m , and background minimum delay BGDLmin m The correlation coefficients between the scan response delay RTTs shown in Fig. 43 are shown. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in Fig. 43 for the same reason as explained in Fig. 41. m , packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m are a plurality of communication characteristics of M terminal devices when a network scan is executed on M terminal devices.

[0353] Referring to FIG. 43, when a network scan is performed by monitoring background traffic, if the communication quality is insufficient, the scan response delay RTT m and packet loss rate (PLR) m , Delayed DL m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with packet loss rate PLR m And delayed download m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with delayed DL m and Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with jitter JT m and background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with, and background maximum delay BGDLmax m and background minimum delay BGDLmin mThe correlation coefficient with scan response delay RTT is greater than 0.5. m is the packet loss rate (PLR) m It has the largest correlation coefficient with

[0354] Fig. 44 is a diagram showing yet another correlation coefficient of communication characteristics. The correlation coefficient shown in Fig. 44 is the correlation coefficient of communication characteristics when the network is congested, in the case where a network scan is performed by monitoring background traffic.

[0355] Scan Response Delay RTT m , packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m The correlation coefficients between the scan response delay RTTs shown in FIG. 44 are shown. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in FIG. 44 for the same reason as explained in FIG. 41. m , packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m are a plurality of communication characteristics of M terminal devices when a network scan is executed on M terminal devices.

[0356] Referring to Figure 44, when a network scan is performed by monitoring background traffic, the scan response delay (RTT) m and packet loss rate (PLR) m , Delayed DL m , Jitter JT m and background maximum delay BGDLmax m Correlation coefficient with packet loss rate PLR m And delayed download m, background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with delayed DL m and Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with jitter JT m and background maximum delay BGDLmax m and background minimum delay BGDLmin m Correlation coefficient with, and background maximum delay BGDLmax m and background minimum delay BGDLmin m The correlation coefficient with scan response delay RTT is greater than 0.5. m is the background maximum delay BGDLmax m It has the largest correlation coefficient with

[0357] As shown in Figures 43 and 44, when network scans are performed by monitoring background traffic, the scan response delay (RTT) is significantly different between when communication quality is poor and when the network is congested. m , packet loss rate PLR m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m , and background minimum delay BGDLmin m The correlation coefficients of the two are different.

[0358] Note that the scan response delay RTT shown in Figures 41 to 44 m and packet loss rate (PLR) m , Delayed DL m , Jitter JT m , background maximum delay BGDLmax m and background minimum delay BGDLmin m The correlation coefficient is calculated by the scan response delay RTT stored in the correspondence table TBL1 shown in Figure 3 or the correspondence table TBL2 shown in Figure 4.m Scan response SR in one terminal device m The scan response delay is averaged over the number of ports that have RTT scans. m The correlation coefficient may be calculated using the scan response delay RTT stored in the correspondence tables TBL1 and TBL2. m and packet loss rate (PLR) m By using the communication characteristics such as scan response delay RTT m Regarding the scan response delay due to port differences, RTT m The correlation coefficient may be calculated by taking into consideration the difference in communication characteristics between the two devices.

[0359] When a network scan is performed, the results are classified as successful or failed. If the network scan fails, the following first, second, third, and fourth causes are assumed to be the causes of the network scan failure.

[0360] (First cause) The communication quality with the terminal device m that is the target of the network scan is insufficient.

[0361] (Second cause) The network to which the terminal device m, which is the target of the network scan, is connected is congested (that is, network congestion).

[0362] (Third cause) The terminal device m that is the target of the network scan is not connected to the network.

[0363] (Fourth cause) The port of the terminal device m that is the target of the network scan is blocked.

[0364] Therefore, in the embodiment of the present invention, the scan response delay RTT obtained when a network scan is executed is calculated by the method described below. m, packet loss rate PLR m and delayed DL m Using the communication characteristics such as the above, it is estimated whether the network scan is normal or unsuccessful, and if it is estimated that the network scan is unsuccessful, it estimates the cause of the failure (one of the first cause, second cause, third cause, and fourth cause).The first cause and second cause are causes attributable to the communication channel with the terminal device.

[0365] The method for estimating the results of a network scan is based on the scan response delay RTT shown in Figures 41 to 44. m , packet loss rate PLR m and delayed DL m and an estimation process for classifying terminal devices having the communication characteristics extracted in the extraction process into a group CLS1 in which the network scan is normal and groups CLS2 and CLS3 in which the network scan has failed due to a cause attributable to the communication channel with the terminal device, calculating a D-score D_2 for distinguishing between group CLS1 and group CLS3, or a D-score D_3 for distinguishing between group CLS2 and group CLS3, and estimating whether the network scan is normal or a failure based on the calculated D-score D_2 or D-score D_3, and, when it is estimated that the network scan has failed, estimating the cause of the failure of the network scan.

[0366] [Communication characteristics CMbest m Extracting _unknown] When a network scan is performed without monitoring background traffic according to the schedule SCH_scan shown in FIG. 6, the scan analysis manager 12 performs one scan timing t s Scan response delay in RTT m , packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin mis received from the network scanner 11, and the received scan response delay RTT m , packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m The host information database 13 is created or updated by storing the above in the correspondence table TBL1 of the host information database 13.

[0367] The extraction means 15 extracts M scan response delay RTTs m , M packet loss rates PLR m , M scans maximum delay DLmax m and M scan minimum delays DLmin m are read from the correspondence table TBL1 of the host information database 13, and the read M scan response delay RTTs are calculated. m , M packet loss rates PLR m , M scans maximum delay DLmax m and M scan minimum delays DLmin m Scan response delay based on RTT m , packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m The correlation coefficient r ij is calculated using the following formula:

[0368]

number

[0369] In equation (13), x ai is the correlation coefficient r ij The communication characteristic f i represents the sample of y aj is the correlation coefficient r ij The other communication characteristic f j represents a sample of i = 1~M, and a j =1~M.

[0370] The extraction means 15 extracts the communication characteristic f i ,f j The correlation coefficient r ij Among them, different communication characteristics f i ,f j Correlation coefficient r for (i≠j) ij (i≠j). For example, the calculated correlation coefficient r ij When the correlation coefficients shown in Figure 41 are used, the communication characteristic f i (i=1~4) are the scan response delays RTT in the column direction, respectively. m , packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m and the communication characteristic f j (j=1 to 4) are the row-wise scan response delay RTTs, respectively. m , packet loss rate PLR m , maximum scan delay DLmax m and scan minimum delay DLmin m is set to

[0371] Correlation coefficient r 11 ,r 22 ,r 33 ,r 44 are the diagonal elements, and the correlation coefficient r 21 ,r 31 ,r 41 ,r 32 ,r 42 ,r 43 are the off-diagonal elements.

[0372] The extraction means 15 extracts the correlation coefficient r of the off-diagonal elements. 21 ,r 31 ,r 41 ,r 32 ,r 42 ,r 43 are extracted sequentially, and each extracted correlation coefficient r 21 ,r 31 ,r 41 ,r 32 ,r 42 ,r 43 is the threshold Corr rDetermine whether the threshold Corr is greater than r is set to a real number in the range of 0.5 to 1.0, for example. r =0.5.

[0373] First, the extraction means 15 extracts the correlation coefficient r 21 is the threshold Corr r Then, the extraction means 15 determines whether the correlation coefficient r 21 is the threshold Corr r When the correlation coefficient r 21 The two communication characteristics f2 and f1 that were the basis for the calculation of 21 is the threshold Corr r When it is larger than 21 The two communication characteristics f2 and f1 used to calculate the above are saved.

[0374] In this case, the correlation coefficient r 21 (=0.25) is the threshold Corr r (=0.5) or less, the extraction means 15 extracts the correlation coefficient r 21 The two communication characteristics f2 (= packet loss rate PLR) that were used to calculate m ) and communication characteristic f1 (= scan response delay RTT m ) to delete it.

[0375] Next, the extraction means 15 extracts the correlation coefficient r 31 is the threshold Corr r Determine whether the correlation coefficient r is greater than 31 is 0.80, the extraction means 15 extracts the correlation coefficient r 31 is the threshold Corr r is determined to be greater than the correlation coefficient r 31 The two communication characteristics f3 (= maximum scan delay DLmax) that were used to calculate m ) and communication characteristic f1 (= scan response delay RTT m ) to save the file.

[0376] Subsequently, the extraction means 15 extracts the correlation coefficient r 41 is the threshold Corrr Determine whether the correlation coefficient r is greater than 41 is 0.97, the extraction means 15 extracts the correlation coefficient r 41 is the threshold Corr r is determined to be greater than the correlation coefficient r 41 The two communication characteristics f3 (= minimum scan delay DLmin) that were used to calculate m ) and communication characteristic f1 (= scan response delay RTT m ) to save.

[0377] Similarly, the extraction means 15 extracts the correlation coefficient r 32 ,r 42 ,r 43 is the threshold Corr r Then, the extraction means 15 sequentially determines whether the threshold value Corr is greater than r (=0.5) 43 The two communication characteristics f4 (= minimum scan delay DLmin) that were used to calculate m ), f3(= maximum scan delay DLmax m ) is stored. The extraction means 15 also stores the threshold value Corr r (=0.5) or less correlation coefficient r 32 ,r 42 The two communication characteristics f3 (= maximum scan delay DLmax) that were used to calculate m ), f2(= Packet loss rate PLR m ); Two communication characteristics f4 (= minimum scan delay DLmin m ), f2(= Packet loss rate PLR m ) to delete it.

[0378] As a result, two communication characteristics f3 (= maximum scan delay DLmax m ), f1(= scan response delay RTT m ); Two communication characteristics f3 (= minimum scan delay DLmin m ), f1(= scan response delay RTT m ); Two communication characteristics f4 (= minimum scan delay DLmin m ), f3(= maximum scan delay DLmax m) is stored, the extraction means 15 finally obtains the scan response delay RTT m , maximum scan delay DLmax m and scan minimum delay DLmin m The three communication characteristics are saved.

[0379] The extraction means 15 extracts the correlation coefficient r ij 42 to 44, the threshold Corr r A correlation coefficient greater than r ij The communication characteristics used to calculate the threshold Corr are saved. r A correlation coefficient r that is less than or equal to ij The communication characteristics used as the basis for the calculation are deleted.

[0380] Scan Response Delay RTT m is the RTT between sending a packet containing the test input and receiving a scan response when the network scan is normal. normal It is the best representation of the results of a network scan because it represents the delay relative to the

[0381] Therefore, the extraction means 15 extracts the finally stored communication characteristics (in the above example, the scan response delay RTT m , maximum scan delay DLmax m and scan minimum delay DLmin m ) to obtain the maximum scan delay DLmax m and scan minimum delay DLmin m is extracted as an explanatory variable, and the scan response delay RTT m is extracted as the dependent variable, and multiple regression analysis is performed to calculate the contribution rate (= R 2 ) is found.

[0382] Thereafter, the extraction means 15 extracts the contribution rate R 2 is the threshold S r It is determined whether the threshold value S is smaller than the threshold value S. r is set to a real number in the range of 0.5 to 1.0, for example.

[0383] The extraction means 15 extracts the contribution rate R 2 is the threshold S r When it is determined that the correlation coefficient is smaller than , the mode of the estimation algorithm ALG that estimates the results of the network scan is set to "uncorrelated" and the contribution rate R 2 is the threshold S r If it is determined that the difference is not smaller than , the mode of the estimation algorithm ALG is set to "correlation".

[0384] When the estimation algorithm ALG mode is set to "uncorrelated", it means that the communication characteristics finally saved are somewhat important. On the other hand, when the estimation algorithm ALG mode is set to "correlated", it means that the communication characteristics finally saved are very important.

[0385] Subsequently, the extraction means 15 uses the machine learning algorithm "SelectKbest" to select the communication characteristic CMbest that is most suitable for estimating the result of the network scan from the K communication characteristics.

[0386] A method for selecting communication characteristics CMbest will be described. Fig. 45 is a diagram showing an example of original data when selecting communication characteristics CMbest. Fig. 46 is a diagram for explaining a method for selecting communication characteristics CMbest.

[0387] The packet loss rate (PLR) is one of the communication characteristics obtained from the scan response of the network scan. m , Jitter JT m , maximum scan delay DLmax m , minimum scan delay DLmin m and delayed DL m is shown in Figure 45. In Figure 45, the packet loss rate PLR m , Jitter JT m , maximum scan delay DLmax m , minimum scan delay DLmin m and delayed DL m is shown for eight terminal devices.

[0388] Then, the extraction means 15 calculates the threshold value Corr by the above-mentioned method. r A correlation coefficient greater than r ij As shown in FIG. 46, for example, the packet loss rate (PLR) is m , Jitter JT m , maximum scan delay DLmax m and scan minimum delay DLmin m Select .

[0389] Thereafter, the extraction means 15 calculates the sum of each row and the sum of each column in FIG. 46, and calculates the grand total, which is the sum of the sum of each row and the sum of each column.

[0390] Subsequently, the extraction means 15 calculates the expected value E rc is calculated for every row and every column.

[0391]

number

[0392] In equation (14), e r represents the sum of the rth row, and e c represents the sum of the cth column. Also, N grand represents the grand total.

[0393] Fig. 47 is a diagram for explaining a method for calculating the score of the communication characteristics. Fig. 48 is a diagram showing the score of the communication characteristics. In Fig. 47, the maximum scan delay DLmax m We will explain how to calculate the score Q of

[0394] Referring to FIG. 47, the leftmost column is the sum and grand total of the rightmost column of FIG. 46, and the second column from the left is the maximum scan delay DLmax shown in FIG. m This is a sample.

[0395] The extraction means 15 calculates the sum of each row shown in FIG. 48 (=0.049, 0.034, 0.023, 0.038, 0.022, 0.022, 0.021, 0.023) by using e r Substitute the sum of the c-th column (=0.232) shown in Figure 47 into e in equation (14). c , and the total sum of Figure 47 is N in equation (14) grand Substituting into the expected value E rc Calculate.

[0396] Then, the extraction means 15 calculates the score Q by the following formula.

[0397]

number

[0398] In equation (15), O is the maximum scan delay DLmax shown in FIG. m and E is the expected value E calculated by equation (14). rc is.

[0399] The score Q is calculated based on one communication characteristic (e.g., maximum scan delay DLmax m ), the score Q represents the magnitude of deviation of each sample O from the expected value E. In equation (15), if each sample O is significantly deviated from the expected value E or if each sample O is significantly deviated from the expected value E, the score Q becomes large, and if each sample O is slightly deviated from the expected value E or if each sample O is slightly deviated from the expected value E, the score Q becomes small. Therefore, the score Q is calculated based on one communication characteristic (for example, the maximum scan delay DLmax m ) represents the characteristics of

[0400] After calculating the expected value E, the extraction means 15 calculates OE, and then calculates (OE) 2 Calculate the calculated (OE) 2 Divide by the expected value E (OE) 2 Calculate / E.

[0401] Then, the extraction means 15 extracts (OE) of each row. 2 Scan the sum of / E Maximum delay DLmax m That is, the extraction means 15 calculates the maximum scan delay DLmax by using the formula (15). m Calculate the score Q.

[0402] The extraction means 15 also extracts the packet loss rate PLR ​​shown in FIG. m , Jitter JT m and scan minimum delay DLmin m The score Q is calculated by the above-mentioned method for the packet loss rate PLR ​​as shown in FIG. m , Jitter JT m , maximum scan delay DLmax m and scan minimum delay DLmin m A score Q is obtained.

[0403] Then, the extraction means 15 extracts the packet loss rate PLR m , Jitter JT m , maximum scan delay DLmax m and scan minimum delay DLmin m From the scores Q, K (K is a positive integer) communication characteristics are selected in descending order of scores Q.

[0404] For example, when K=3, in the example shown in FIG. 48, the extraction means 15 extracts the jitter JT m , minimum scan delay DLmin m and maximum scan delay DLmax m Select .

[0405] After selecting K communication characteristics, the extraction means 15 selects a communication characteristic CMbest from the K communication characteristics by the following method.

[0406] First, the extraction means 15 uses classification algorithms such as LR (Linear Regression) and SVM (Support Vector Machine) to classify the K communication characteristics into a group GP1 of communication characteristics that are important for estimating the results of the network scan and a group GP2 of communication characteristics that are not important for estimating the results of the network scan.

[0407] Here, a support vector machine (SVM-train) is used as the classification algorithm.

[0408] SVM is a pattern recognition model that uses supervised learning, and the training data X0 = [x 01 ,x 02 ,···,x 0M ] T (T stands for transpose) to learn the boundary that most boldly separates it. To learn this boundary, a space of two or more dimensions consisting of two or more parameters is set.

[0409] Then, as a result of learning using training samples arranged in a two-dimensional or more space, the training data X0 = [x 01 ,x 02 ,···,x 0M ] T The boundary that most boldly separates the data is determined so that the distance (margin) between the boundary and the closest sample (data) is maximized. This determined boundary is called the maximum margin hyperplane. Note that the specific method for determining this boundary is well known and will not be described here.

[0410] The extraction means 15 extracts the training data X0=[x 01 ,x 02 ,···,x 0M ] T (T denotes transpose) to train a support vector machine (SVM-train), and use the boundaries determined by the training to train M sample vectors x m Each sample vector x (m=1 to M) satisfies the following equation: mis classified into either group GP1 or GP2. Here, the sample vector x m is x m =[CM 1m ,CM 2m ,···,CM km ,···,CM Km ]. That is, the sample vector x m is the K communication characteristics CM 1m ~CM Km As a result, the data to be classified into either group GP1 or GP2 is expressed by equation (17).

[0411]

number

[0412]

number

[0413] In equation (17), the communication characteristic CM km (k=1 to K, m=1 to M) represents the k-th communication characteristic of terminal device m.

[0414] In equation (16), z is a weight vector and b is a predetermined constant.

[0415] Group GP1 is a group of communication characteristics that are important for estimating the results of a network scan. Group GP2 is a group of communication characteristics that are not important for estimating the results of a network scan. m When y is classified into group GP1, m consists of "1" and the sample vector x m When y is classified into group GP2, m consists of "-1".

[0416] The extraction means 15 calculates the weight vector z by equation (16) to determine the classification index α of the following equation.

[0417]

number

[0418] In equation (18), X=[x1,x2,...,x M ] T (T stands for transpose), and y=[y1,y2,...,y M ] T (T stands for transpose).

[0419] The extraction means 15 extracts the sample vector x m When y is classified into group GP1 (i.e., y m consists of "1"), z x m The weight vector z is calculated by -b=1, and the calculated weight vector z is set as the classification index α. m When y is classified into group GP2 (i.e., y m consists of "-1"), z x m A weight vector z is calculated using -b=-1, and the calculated weight vector z is set as the classification index α.

[0420] The extraction means 15 extracts the k-th communication characteristic CM km When the classification index α for k ), sample vector x km , and y m (=“1” or “-1”) into the following equation to obtain the weight w, which indicates the importance of the communication characteristics for estimating the results of the network scan. k Calculate.

[0421]

number

[0422] In equation (19), x km is the sample vector x at terminal device m mThe kth communication characteristic sample CM km Therefore, the right side of equation (19) represents the k-th communication characteristic CM k1 ,CM k2 ,···,CM km ,···,CM kM Using the weight w k This means calculating

[0423] Thereafter, the extraction means 15 extracts the weight w calculated by the formula (19) k Substitute into the following equation and use the rank weight R k Calculate.

[0424]

number

[0425] The extraction means 15 extracts the rank weights R for all of the K communication characteristics by the above-mentioned method. k Calculate.

[0426] Figure 49 shows the rank weight R k 49 is a diagram showing the calculation result of the maximum scan delay DLmax m , Jitter JT m , minimum scan delay DLmin m and packet loss rate (PLR) m For rank weight R k and ranks are associated.

[0427] Then, the extraction means 15 selects the lowest ranked (i.e., the smallest rank weight R k ) communication characteristics (here, packet loss rate PLR m ) is determined as the communication characteristic to be deleted. After that, the extraction means 15 extracts the samples of the communication characteristic into the sample vectors x1 to x M The calculation of equations (16), (18) to (20) and the determination of communication characteristics to be deleted are repeated until all communication characteristics are deleted. kIf there are a plurality of communication characteristics that give the smallest rank weight R k If there are a plurality of communication characteristics that give the scan response delay RTT m Correlation coefficient r ij The communication characteristic with the smallest σ may be the communication characteristic to be deleted.

[0428] Then, the extraction means 15 extracts the last deleted communication characteristic as the communication characteristic CMbest m Detected as _unknown, and its detected communication characteristics CMbest m _unknown and scan response delay RTT m _unknown and output to the estimation means 17.

[0429] When a network scan is executed by monitoring background traffic according to the schedule SCH_scan shown in FIG. 6, the extraction unit 15 extracts the M scan response delay RTTs described above. m , M packet loss rates PLR m , M scans maximum delay DLmax m and M scan minimum delays DLmin m Instead, M scan response delay RTTs m , M packet loss rates PLR m , M delays DL m , M jitter JT m , M background maximum delays BGDLmax m and M background minimum delays BGDLmin m Based on this, the communication characteristic CMbest is calculated by the above-mentioned method. m Select _unknown.

[0430] Scan Response Delay RTT m Scan response delay RTT most similar to _unknown m Select _known] The extraction means 15 extracts the scan response delays RTT1 to RTT2 from the correspondence table TBL1 (or correspondence table TBL2) stored in the host information database 13. M Scan response delay RTT1_unknown~RTT M Detected as _unknown. In this case, scan response delay RTT1~RTT M Each of these addresses is an IP address. m It consists of one scan response delay or an average of multiple scan response delays within a

[0431] The extraction means 15 also extracts the scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3(m'=1'~M').

[0432] In Figure 5, the scan response delay (RTT m’ _known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3(m'=1'~M') is described, but the scan response delay {RTT m’ _known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3 (m'=1'~M') are the aggregated scan response delays {RTT m’ _known_aggregation}_z (m'=1' to M', z=1 to 3), the extraction means 15 updates the scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’_known}_3(m'=1'~M'), respectively, and scan response delay {RTT m’ _known_aggregation}_1(m'=1'~M'), scan response delay {RTT m’ _known_aggregation}_2(m'=1'~M') and scan response delay {RTT m’ _known_aggregation}_3(m'=1'~M') may also be detected.

[0433] Thereafter, the extraction means 15 extracts the scan response delay RTT1_unknown to RTT M _unknown, scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3(m′=1′ to M′) is output to the calculation means 16.

[0434] The calculation means 16 calculates the scan response delay RTT1_unknown to RTT M _unknown, scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTT m’ _known}_2(m'=1'~M') and scan response delay {RTT m’ _known}_3 (m′=1′ to M′) is received from extraction means 15.

[0435] The calculation means 16 calculates the scan response delay RTT mBased on this, mode m(Mo), which is the value of the scan response delay that is most likely to occur, is acquired. Note that if there are two values ​​of the scan response delay that are most likely to occur, calculation means 16 calculates mode m(Mo), which is the value of the scan response delay that is most likely to occur, using the above-mentioned equation (9). Also, if there are three or more values ​​of the scan response delay that are most likely to occur, calculation means 16 detects the minimum and maximum values ​​from the three or more scan response delays that are most likely to occur, and calculates mode m(Mo), which is the value of the scan response delay that is most likely to occur, using the above-mentioned equation (9) based on the detected minimum and maximum values.

[0436] In this case, in equation (9), L indicates the lower boundary of the probability density function region (=PDF (Probability Density Function) region) that includes two values ​​of the most likely scan response delay when the network scan result is unknown or when the network scan result is known. Also, r indicates the number of samples (=number of scan response delays) in the PDF region that is most likely to occur when the network scan result is unknown or when the network scan result is known. Furthermore, r1 indicates the region that is smaller than the scan response delay RTT than the PDF region that is most likely to occur when the network scan result is unknown or when the network scan result is known. m indicates the number of samples (= the number of scan response delays) in the previous PDF area in the direction of decreasing value. Furthermore, r2 indicates the scan response delay RTT (RTT) in the PDF area where the scan response delay is most likely to occur when the network scan result is unknown or when the network scan result is known. m indicates the number of samples in the next PDF region (= the number of scan response delays) in the direction in which h is larger. Furthermore, h indicates the interval between each PDF region (i.e., the width of each PDF region).

[0437] Figures 50 to 52 show specific distributions of scan response delays. Figure 53 shows the correspondence between PDF regions and the number of samples generated. Figure 54 shows specific distributions of probability density functions of scan response delays.

[0438] If the number of samples for each specific value in the specific distribution of scan response delay shown in Figure 50 is counted and the number of samples for each specific value is displayed, the result is shown in Figure 51. As a result, Figure 51 shows that scan response delay is most likely to occur at probability density values ​​of 0.41 and 0.6.

[0439] As shown in Figure 52, the number of samples generated in each of the PDF regions of 0.0 to 0.2, 0.21 to 0.4, 0.41 to 0.6, 0.61 to 0.8, and 0.81 to 1.0 is "5", "5", "14", "4", and "2", respectively.

[0440] The correspondence between the PDF area and the number of samples occurring in each area is as shown in Figure 53. The number of samples occurring in the PDF area from 0.41 to 0.6 (= 14) is r, the interval from 0.41 to 0.6 (= 0.2) is h, 0.41 is L, the number of samples occurring in the PDF area from 0.21 to 0.4 (= 5) is r1, and the number of samples occurring in the PDF area from 0.61 to 0.8 (= 4) is r2. Therefore, the mode m (Mo) can be calculated by substituting r (= 14), r1 (= 5), r2 (= 4), h (= 0.2), and L (= 0.41) into equation (9).

[0441] As shown in FIG. 54, r is the PDF region REG bounded by the two values ​​(=0.41 and 0.6) at which the scan response delay is most likely to occur and the curve showing the probability density function. PDF L is the lower value (=0.41) of the two values ​​(=0.41 and 0.6) that are most likely to cause scan response delay. h is the interval between the two values ​​(=0.41 and 0.6). r1 is the PDF region REG PDF The PDF area REG has a smaller scan response delay than PDF and r2 is the number of samples in the region of width h adjacent to the PDF region REG PDFThe PDF area REG on the side with a larger scan response delay than PDF It consists of the number of samples in a region of width h adjacent to

[0442] FIG. 55 shows another specific distribution of scan response delay. Referring to FIG. 55, there are three most likely scan response delay values. In this case, the interval h is adjusted to the width of the PDF region that includes the three most likely scan response delay values. Then, the minimum value (=0.35) and the maximum value (=0.6) are detected from the three most likely scan response delay values. Then, using the minimum value (=0.35) and the maximum value (=0.6) as the two most likely scan response delay values, r, r1, r2, L, and h are determined by the method described in FIGS. 50 to 54, and the determined r, r1, r2, L, and h are substituted into Equation (9) to calculate the mode m (Mo).

[0443] When there are four or more most likely scan response delay values, the mode m (Mo) is calculated in the same manner as when there are three most likely scan response delay values.

[0444] Below, scan response delay RTT1_unknown~RTT M Based on _unknown, the mode m(Mo) calculated by Equation (9) is denoted as "m(Mo)_unknown", and the scan response delay {RTT m’ _known}_1 (m'=1'~M'), the mode m(Mo) calculated by Equation (9) is denoted as "m(Mo)_known_1", and the scan response delay {RTT m’ _known}_2 (m'=1'~M'), the mode m(Mo) calculated by Equation (9) is denoted as "m(Mo)_known_2", and the scan response delay {RTT m’ The mode m(Mo) calculated by equation (9) based on {_known}_3 (m'=1' to M') is denoted as "m(Mo)_known_3".

[0445] The calculation means 16 calculates the modes m(Mo)_unknown, m(Mo)_known_1, m(Mo)_known_2, and m(Mo)_known_3 by the above-described method, and then calculates the scan response delays RTT1_unknown to RTT M Mean absolute deviation W' of _unknown unknown Calculate.

[0446]

number

[0447] C(x in Eq. (21A) i_unknown ) is represented by formula (21B), and x i_unknown Scan response delay RTT1_unknown~RTT M _unknown is assigned.

[0448] The calculation means 16 calculates the mean absolute deviation W' unknown , the scan response delay {RTT m’ _known}_z(m'=1'~M') mean absolute deviation W' known_z Calculate.

[0449]

number

[0450] In equation (22), z is an argument indicating the known estimated results of the network scan, and is an integer satisfying 1≦z≦Z. Furthermore, Z is the total number of known estimated results of the network scan. In this embodiment of the present invention, the known estimated results of the network scan are "normal", "poor communication quality", and "network congestion", so z=1, 2, 3 in equation (22).

[0451] The scan response delays corresponding to the known estimated network scan results of "normal," "poor communication quality," and "network congestion" may consist of one type of data or multiple types of data.

[0452] The calculation means 16 substitutes the mode m(Mo)_known_1 into m(Mo)_known_z in the formula (22B) to obtain the scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 into x in equation (22B) i_known_z Substituting into the equation, we obtain the mean absolute deviation W' by equation (22A). known_1 Calculate.

[0453] Furthermore, the calculation means 16 substitutes the mode m(Mo)_known_2 into m(Mo)_known_z in the formula (22B) to calculate the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 into x in equation (22B) i_known_z Substituting into the equation, we obtain the mean absolute deviation W' by equation (22A). known_2 Calculate.

[0454] Furthermore, the calculation means 16 substitutes the mode m(Mo)_known_3 into m(Mo)_known_z in equation (22B) to obtain the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 into x in equation (22B) i_known_z Substituting into the equation, we obtain the mean absolute deviation W' by equation (22A). known_3 Calculate.

[0455] That is, the calculation means 16 calculates the modes m(Mo)_known_1, m(Mo)_known_2, and m(Mo)_known_3 and the scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1,{RTT 1’ _known}_2~{RTT M’ _known}_2,{RTT 1’ _known}_3~{RTTM’ Based on the above, the mean absolute deviation W' is calculated by equation (22). known_1 ,W' known_2 ,W' known_3 Calculate.

[0456] Thereafter, the calculation means 16 calculates the mean absolute deviation W' unknown ,W' known_1 ,W' known_2 ,W' known_3 Substituting into the following equation, the difference in mean absolute deviation S S1 ,S S2 ,S S3 Calculate.

[0457]

number

[0458] In this case, the calculation means 16 calculates the mean absolute deviation W' known_1 W' in equation (23) known_z Substituting in, the mean absolute deviation W' unknown W' in equation (23) unknown Substituting into the difference in mean absolute deviation S S1 Calculate.

[0459] The calculation means 16 also calculates the mean absolute deviation W' known_2 W' in equation (23) known_z Substituting in, the mean absolute deviation W' unknown W' in equation (23) unknown Substituting into the difference in mean absolute deviation S S2 Calculate.

[0460] Furthermore, the calculation means 16 calculates the mean absolute deviation W' known_3 W' in equation (23) known_z Substituting in, the mean absolute deviation W' unknown W' in equation (23) unknown Substituting into the difference in mean absolute deviation S S3 Calculate.

[0461] The calculation means 16 calculates the difference S S1 ,S S2 ,S S3, the difference between the calculated mean absolute deviations S S1 ,S S2 ,S S3 Substituting into the following equation, the similarity P Sz Calculate.

[0462]

number

[0463] In this case, the calculation means 16 calculates the difference S S1 S in equation (24) Sz Substituting into the similarity P S1 Calculate the difference in mean absolute deviations, S S2 S in equation (24) Sz Substituting into the similarity P S2 Calculate the difference in mean absolute deviations, S S3 S in equation (24) Sz Substituting into the similarity P S3 Calculate.

[0464] The calculation means 16 calculates the similarity P S1 ,P S2 ,P S3 When the calculated similarity P S1 ,P S2 ,P S3 is output to the extraction means 15.

[0465] The extraction means 15 extracts the similarity P S1 ,P S2 ,P S3 is received from the calculation means 16, the similarity P S1 ,P S2 ,P S3 The highest similarity P S_highest Then, the extraction means 15 refers to the correspondence table TBL3 stored in the host information database 13 and extracts the host with the highest similarity P S_highest Scan response delay {RTT m’ _known}_similarity(m'=1'~M')(=Scan response delay {RTT m’ _known}_1(m'=1'~M'), scan response delay {RTTm’ _known}_2(m'=1'~M'), and scan response delay {RTT m’ _known}_3 (m'=1' to M')). The extraction means 15 also detects the scan response delay {RTT m’ _known}_similarity (m'=1'~M') and the communication characteristics {CMbest} used to estimate the known results of network scans. m’ _known}_similarity(m'=1'~M')(=communication characteristics {CMbest m’ _known}_1(m'=1'~M'), communication characteristics {CMbest m’ _known}_2(m'=1'~M'), and communication characteristics {CMbest m’ _known}_3 (m'=1'~M').

[0466] Then, the extraction means 15 extracts the scan response delay {RTT m’ _known}_similarity(m'=1'~M') and communication characteristics {CMbest m’ _known}_similarity (m′=1′ to M′), and outputs the generated standard sample data D_similarity to the estimation means 17.

[0467] As mentioned above, the scan response delay (RTT m’ _known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3(m'=1'~M') are the aggregated scan response delays {RTT m’ _known_aggregation}_z (m'=1' to M', z=1 to 3), the extraction means 15 extracts the aggregated scan response delay {RTT m’ _known_aggregation}_z (m'=1'~M', z=1~3) and the aggregated scan response delay {RTT m’_known_aggregation}_z (m'=1'~M', z=1~3) m’ _known}_z (m′=1′ to M′, z=1 to 3), and output the generated standard sample data D_similarity to the estimation means 17.

[0468] Furthermore, the extraction means 15 refers to the correspondence table TBL1 (or correspondence table TBL2) stored in the host information database 13 to extract the scan response delay RTT1_unknown to RTT M _unknown (= scan response delay RTT m (m=1 to M)) and the detected scan response delay RTT1_unknown to RTT M _unknown (= scan response delay RTT m The estimation means 17 generates sample data D_unknown consisting of the communication characteristics CMbest_unknown_m (m=1 to M) and the communication characteristics CMbest_unknown_m (m=1 to M).

[0469] The estimation means 17 receives the standard sample data D_similarity and the sample data D_unknown from the extraction means 15 .

[0470] Then, the estimation means 17 updates the estimation algorithm ALG to the optimum algorithm using the standard sample data D_similarity by the method described below.

[0471] Fig. 56 is a diagram showing data used when training the estimation algorithm ALG. Referring to Fig. 56, when training the estimation algorithm ALG using the standard sample data D_similarity, the estimation means 17 divides the standard sample data D_similarity into training data and test data. In this case, the estimation means 17 divides the standard sample data D_similarity into training data and test data so that the proportion of training data is 80% and the proportion of test data is 20%, for example. Note that when dividing the standard sample data D_similarity into training data and test data, the proportion of training data may be other than 80%, and the proportion of test data may be other than 20%.

[0472] Thereafter, the estimation means 17 divides the training data into a training set and a validation set. In this case, the estimation means 17 divides the training data into the training set and the validation set so that the proportion of the training set is 50% and the proportion of the validation set is 50%, for example. Note that the proportions of the training set and the validation set may be other than 50%.

[0473] The estimation means 17 divides the training data into a training set and a validation set, and then divides the training set into U subsets d1 to d U Divide into subsets d1~d U Each of the sets consists of data of size n, where n is, for example, 8. The estimation means 17 divides the validation set into U' subsets V 1’ ~V U’ Then, the data size of the training set is D S Then, U=INT(D S / n).

[0474] Fig. 57 is a conceptual diagram of sample data when training the estimation algorithm ALG. Referring to Fig. 57, when training the estimation algorithm ALG, the estimation means 17 creates initial sample data for modeling. More specifically, the estimation means 17 obtains the scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 and the read scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ Generate standard sample data D_standard_1 from {_known}_1.

[0475] The estimation means 17 also calculates the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 and the read scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ Generate standard sample data D_standard_2 from {_known}_2.

[0476] Furthermore, the estimation means 17 obtains the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ _known}_3 and read the scan response delay {RTT 1’_known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ Generate standard sample data D_standard_3 from {_known}_3.

[0477] Then, the estimation means 17 generates the initial sample data D_sample_initial as follows (see (a) of FIG. 57). In this case, the estimation means 17 generates a plurality of subsets d of the standard sample data D_standard_1. 1_1 ~d U_1 One subset d from 1_initial Select multiple subsets d of the standard sample data D_standard_2 1_2 ~d U_2 One subset d from 2_initial Select multiple subsets d of the standard sample data D_standard_3 1_3 ~d U_3 One subset d from 3_initial Then, "1", "2", and "3" in the standard sample data D_standard_1, D_standard_2, and D_standard_3 respectively represent the identification code ID_code=1 indicating that the result of the network scan is "normal", the identification code ID_code=2 indicating that the result of the network scan is "insufficient communication quality", and the identification code ID_code=3 indicating that the result of the network scan is "network congestion", so the estimation means 17 selects the subset d 1_initial ,d 2_initial ,d 3_initial When you select this, the subset d 1_initial ,d 2_initial ,d 3_initial are combined to generate the initial sample data D_sample_initial.

[0478] Note that multiple subsets d 1_1 ~d U_1The "1" in the standard sample data D_standard_1 corresponds to the "1" in the standard sample data D_standard_2, and multiple subsets d 1_2 ~d U_2 The "2" in the standard sample data D_standard_2 corresponds to the "2" in the standard sample data D_standard_2, and multiple subsets d 1_3 ~d U_3 The "3" in the standard sample data D_standard_3 corresponds to the "3" in the standard sample data D_standard_3. 1_1 ~d U_1 , multiple subsets d 1_2 ~d U_2 and multiple subsets d 1_3 ~d U_3 One subset d from each 1_initial ,d 2_initial ,d 3_initial may be selected randomly or according to any selection method.

[0479] As shown in FIG. 56, subsets d1 to d U is generated from the standard sample data D_similarity. The standard sample data D_similarity is the sample data that has the highest similarity to the sample data D_unknown among the standard sample data D_standard_1 to D_standard_3.

[0480] As a result, subsets d1 to d U is generated from the standard sample data D_standard_1 by the method described in FIG. 56 when the standard sample data D_standard_1 has the highest similarity to the sample data D_unknown. U is generated from the standard sample data D_standard_2 by the method described in FIG. 56 when the standard sample data D_standard_2 has the highest similarity to the sample data D_unknown. Uis generated from the standard sample data D_standard_3 by the method described in FIG. 56 when the standard sample data D_standard_3 has the highest similarity to the sample data D_unknown.

[0481] Subset d1~d U When generated from the standard sample data D_standard_1, the standard sample data D_standard_1 in the initial sample data D_sample_initial for modeling is subset d u By replacing the data with the standard sample data D_standard_1 in the initial sample data D_sample_initial, tentative sample data D_sample_tentative(D1) for modeling is generated (see (b) of FIG. 57). Note that "D1" in D_sample_tentative(D1) is a subset d u indicates that it has been replaced by

[0482] Also, subsets d1 to d U When generated from the standard sample data D_standard_2, the standard sample data D_standard_2 in the initial sample data D_sample_initial for modeling is subset d u By replacing the data with the data, tentative sample data D_sample_tentative(D2) for modeling is generated (see (c) of FIG. 57). Note that "D2" in D_sample_tentative(D2) means that the standard sample data D_standard_2 in the initial sample data D_sample_initial is a subset d u indicates that it has been replaced by

[0483] Furthermore, subsets d1 to d U When generated from the standard sample data D_standard_3, the standard sample data D_standard_3 in the initial sample data D_sample_initial for modeling is subset du By replacing the data with the standard sample data D_standard_3 in the initial sample data D_sample_initial, tentative sample data D_sample_tentative(D3) for modeling is generated (see (d) of FIG. 57). Note that "D3" in D_sample_tentative(D3) is a sub-set of the standard sample data D_standard_3 in the initial sample data D_sample_initial. u indicates that it has been replaced by

[0484] Fig. 58 is a conceptual diagram showing updating of temporary sample data for modeling. In Fig. 58, subsets d1 to d U A method for updating tentative sample data D_sample_tentative(D1) for modeling will be described below using an example in which the tentative sample data D_sample_tentative(D1) is generated from standard sample data D_standard_1.

[0485] 58, the estimation means 17 first creates initial sample data D_sample_initial for modeling by the method described above. Then, the estimation means 17 replaces the standard sample data D_standard_1 of the sample data D_sample_initial with the subset d1 to update the initial sample data D_sample_initial to tentative sample data D_sample_tentative_1 for modeling. Then, by updating the initial sample data D_sample_initial to the tentative sample data D_sample_tentative_1, the initial sample data D_sample_initial becomes the "old training sample" and the tentative sample data D_sample_tentative_1 becomes the "new training sample".

[0486] Then, if the evaluation index when estimating the results of a network scan using the tentative sample data D_sample_tentative_1 (= "new training sample") is better than the evaluation index when estimating the results of a network scan using the initial sample data D_sample_initial (= "old training sample"), the estimation means 17 determines that the tentative sample data D_sample_tentative_1 becomes the "old training sample."

[0487] Subsequently, the estimation means 17 updates the tentative sample data D_sample_tentative_1 to tentative sample data D_sample_tentative_2 by replacing the subset d1 of the tentative sample data D_sample_tentative_1 with the subset d2. Then, if the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_2 (="new training sample") is better than the evaluation index when the result of the network scan is estimated using the "old training sample", the tentative sample data D_sample_tentative_2 becomes the "old training sample". When comparing evaluation indices between tentative sample data D_sample_tentative_2 (= "new training sample") and an "old training sample," the "old training sample" consists of tentative sample data D_sample_tentative_1 if, in the previous comparison of evaluation indices, the evaluation index when the results of a network scan were estimated using tentative sample data D_sample_tentative_1 was better than the evaluation index when the results of a network scan were estimated using initial sample data D_sample_initial; and consists of initial sample data D_sample_initial if the evaluation index when the results of a network scan were estimated using initial sample data D_sample_initial was better than the evaluation index when the results of a network scan were estimated using tentative sample data D_sample_tentative_1. In other words, the sample data with the better evaluation index becomes the "old training sample" to be compared in the next comparison of evaluation indices.

[0488] Furthermore, the estimation means 17 continues by replacing the subset d2 of the tentative sample data D_sample_tentative_2 with the subset d3, thereby updating the tentative sample data D_sample_tentative_2 to tentative sample data D_sample_tentative_3. Then, if the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_3 is better than the evaluation index when the result of the network scan is estimated using the "old training sample", the tentative sample data D_sample_tentative_3 becomes the "old training sample".

[0489] Thereafter, in a similar manner, the estimation means 17 repeats the process of updating the tentative sample data D_sample_tentative_u to the tentative sample data D_sample_tentative_u+1, and finally updates it to the tentative sample data D_sample_tentative_U.

[0490] In this way, the estimation means 17 divides the standard sample data D_standard_1 of the initial sample data D_sample_initial into U subsets d1 to d U , and thereby the initial sample data D_sample_initial is updated to tentative sample data D_sample_tentative_1 to D_sample_tentative_U in turn.

[0491] Then, when the standard sample data D_standard_1 has the highest similarity to the sample data D_unknown, the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are used to train the estimation algorithm ALG.

[0492] Furthermore, when the standard sample data D_standard_2 has the highest similarity to the sample data D_unknown, the provisional sample data D_sample_tentative (D2) shown in (c) of Figure 57 constitutes the provisional sample data D_sample_tentative_1, and the estimation means 17 sequentially updates the initial sample data D_sample_initial to the provisional sample data D_sample_tentative_1 to D_sample_tentative_U using the method described in Figure 58.

[0493] Furthermore, when the standard sample data D_standard_3 has the highest similarity to the sample data D_unknown, the tentative sample data D_sample_tentative (D3) shown in (d) of Figure 57 constitutes the tentative sample data D_sample_tentative_1, and the estimation means 17 sequentially updates the initial sample data D_sample_initial to the tentative sample data D_sample_tentative_1 to D_sample_tentative_U using the method described in Figure 58.

[0494] As described above, the initial sample data D_sample_initial is composed of the standard sample data D_standard_1 to D_standard_3. The tentative sample data D_sample_tentative_1 to D_sample_tentative_U are each composed of one of the standard sample data D_standard_1 to D_standard_3 in the initial sample data D_sample_initial as a subset d1 to d2, depending on which of the standard sample data D_standard_1 to D_standard_3 the standard sample data D_similarity is composed of. U Then, the subsets d1 to d U is composed of any one of the standard sample data D_standard_1 to D_standard_3 that make up the standard sample data D_similarity.

[0495] Therefore, the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are basically configured by combining parts of the standard sample data D_standard_1 to D_standard_3, respectively.

[0496] The initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are configured by combining the standard sample data D_standard_1 to D_standard_3 for the following reason.

[0497] As will be described later, the estimation algorithm ALG handles group CLS1 where the network scan result is "normal," group CLS2 where the network scan result is "insufficient communication quality," and group CLS3 where the network scan result is "network congestion." Therefore, the initial sample data D_sample_initial and tentative sample data D_sample_tentative_1 to D_sample_tentative_U, which are the data used to train the estimation algorithm ALG, are composed of standard sample data D_standard_1 obtained in a communication environment where the network scan result is "normal," standard sample data D_standard_2 obtained in a communication environment where the network scan result is "insufficient communication quality," and standard sample data D_standard_3 obtained in a communication environment where the network scan result is "network congestion," so that the network scan result can be accurately estimated as any of "normal," "insufficient communication quality," and "network congestion."

[0498] 56 to 58, the scan response delay {RTT m’_known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3 (m'=1'~M'), the standard sample data D_standard_1, D_standard_2, D_standard_3 are generated respectively. m’ _known}_1(m'=1'~M') instead of scan response delay {RTT m’ _known_aggregation}_1(m'=1'~M') may be used to generate standard sample data D_standard_1, and the scan response delay {RTT m’ _known}_2(m'=1'~M') instead of scan response delay {RTT m’ Standard sample data D_standard_2 may be generated using _known_aggregation}_2 (m'=1'~M'), and the scan response delay {RTT m’ _known}_3(m'=1'~M') instead of scan response delay {RTT m’ _known_aggregation}_3 (m'=1' to M') may be used to generate standard sample data D_standard_3.

[0499] As a result, the aggregate scan response delay {RTT m’ _known_aggregation}_z (m'=1'~M', z=1~3) to find subsets d1~d of the training set. U , a subset V of the validation set 1’ ~V U’ and test data is generated (see Figure 56), the estimation algorithm ALG calculates the aggregated scan response delay {RTT m’ _known_aggregation}_z (m'=1' to M', z=1 to 3) and the training results are evaluated.

[0500] Fig. 59 is a conceptual diagram of training and evaluation of the estimation algorithm ALG. Referring to Fig. 59, the estimation means 17 trains the estimation algorithm ALG using initial sample data D_sample_initial. In this case, the estimation means 17 trains the estimation algorithm ALG by repeatedly estimating the results of a network scan by the estimation algorithm ALG using the initial sample data D_sample_initial.

[0501] Then, the estimation means 17 estimates the results of the network scan using the validation set, and then evaluates the estimated results of the network scan using an evaluation index EVD.

[0502] Subsequently, the estimation means 17 (I) trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, (II) selects a validation set (=subset V 1’ ~V U’(III) estimates the results of the network scan using the entire data set D_sample_tentative_1 to D_sample_tentative_U, and (III) evaluates the estimated results of the network scan using the evaluation index nEVD_u for all of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U. In this case, the estimation means 17 determines whether the evaluation index nEVD_u is greater than the evaluation index EVD each time it executes (I) to (III), and when the evaluation index nEVD_u is greater than the evaluation index EVD, it sets the evaluation index nEVD_u to the evaluation index EVD and updates the evaluation index EVD by the evaluation index nEVD_u, and when the evaluation index nEVD_u is not greater than the evaluation index EVD, it does not update the evaluation index EVD (i.e., it maintains the evaluation index EVD). Then, the estimation means 17 maintains the variables of the estimation algorithm ALG when the evaluation index EVD is best when estimating the results of the network scan using the initial sample data D_sample_initial, and replaces the variables of the estimation algorithm ALG with the hypothetical sample data (any of the hypothetical sample data D_sample_tentative_1 to D_sample_tentative_U) when the evaluation index nEVD_u is best when estimating the results of the network scan using hypothetical sample data.

[0503] Thereafter, the estimation means 17 estimates the results of the network scan using the test data and evaluates whether to approve the training model (the estimation algorithm ALG after training) and the training dataset based on the evaluation index tEVD. More specifically, when the evaluation index tEVD is equal to or greater than the evaluation index EVD (the final updated evaluation index EVD), the estimation means 17 approves the training model (the estimation algorithm ALG after training) and the training dataset, and when the evaluation index tEVD is not equal to or greater than the evaluation index EVD (the final updated evaluation index EVD), the estimation means 17 does not approve the training model (the estimation algorithm ALG after training) and the training dataset. Then, when the estimation means 17 does not approve the training model (the estimation algorithm ALG after training) and the training dataset, steps (I) to (IV) shown in FIG. 59 are repeatedly executed using the estimation algorithm ALG whose variables have been finally replaced.

[0504] The approved training dataset becomes the initial sample data D_sample_initial when the next training and evaluation are performed. The evaluation index EVD is composed of the F1 score F1 or the Matthews correlation coefficient MCC, which will be described later. The evaluation index nEVD_u is composed of the F1 score nF1 or the Matthews correlation coefficient nMCC, which will be described later. The evaluation index tEVD is composed of the F1 score tF1 or the Matthews correlation coefficient tMCC, which will be described later.

[0505] In this way, the estimation means 17 trains the estimation algorithm ALG using the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, and also trains the estimation algorithm ALG using the validation set (=subset V 1’ ~V U’ ) is used to evaluate the estimation results of the network scan results using the trained estimation algorithm ALG, using the evaluation indices EVD and nEVD. Finally, using the test data, the approval of the trained estimation algorithm ALG and the training dataset is evaluated using the evaluation index tEVD.

[0506] Fig. 60 is a diagram showing a specific example of the initial sample data D_sample_initial for modeling shown in Fig. 58. Fig. 61 is a diagram showing a specific example of the training set shown in Fig. 56. Fig. 62 is a diagram showing a specific example of the validation set shown in Fig. 56. Fig. 63 is a diagram showing a specific example of the test data shown in Fig. 56.

[0507] 60 to 63, the number of terminal devices targeted for network scanning is eight (i.e., M'=8), and the number of samples of the standard sample data D_similarity is one for the IP address Add m’ Specific examples of the initial sample data D_sample_initial, the training set, the validation set, and the test data are shown below, taking the case where there are 20 samples for each.

[0508] Referring to FIG. 60, the standard sample data D_standard_1 is a scan response delay RTT when a network scan is performed in a communication environment where the result of the network scan is "normal." m’ and communication characteristics CMbest m’ Therefore, in the standard sample data D_standard_1, there are eight scan response delay RTTs. m’ Specific values ​​of and eight communication characteristics CMbest m’ The specific value of is stored in the correspondence table TBL3 shown in Figure 5. 1’ _known}_1~{RTT M’ _known}_1, where M'=8.

[0509] In addition, the standard sample data D_standard_2 is the scan response delay RTT when a network scan is performed in a communication environment where the network scan result is "insufficient communication quality." m’ and communication characteristics CMbest m’Therefore, in the standard sample data D_standard_2, there are eight scan response delay RTTs. m’ Specific values ​​of and eight communication characteristics CMbest m’ The specific value of is the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2, where M'=8, is a subset of the standard sample data D_standard_2.

[0510] Furthermore, the standard sample data D_standard_3 is a scan response delay RTT when a network scan is performed in a communication environment where the network scan result is "network congestion." m’ and communication characteristics CMbest m’ Therefore, in the standard sample data D_standard_3, there are eight scan response delay RTTs. m’ Specific values ​​of and eight communication characteristics CMbest m’ The specific value of is the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3, where M'=8, is a subset of the standard sample data D_standard_3.

[0511] As a result, the initial sample data D_sample_initial for modeling is composed of the standard sample data D_standard_1 to D_standard_3 combined with the identification code D_code=1 assigned to the result of the network scan "normal," the identification code D_code=2 assigned to the result of the network scan "poor communication quality," and the identification code D_code=3 assigned to the result of the network scan "network congestion."

[0512] 61, the training set consists of eight subsets d1 to d8. Since M'=8 for each of the subsets d1 to d8, there are eight scan response delay RTTs. 1’~RTT 8’ and 8 communication characteristics CMbest 1’ ~CMbest 8’ Therefore, in each of the subsets d1 to d8, the vertical direction is made up of eight scan response delay RTTs corresponding to eight terminal devices. 1’ ~RTT 8’ and 8 communication characteristics CMbest 1’ ~CMbest 8’ Indicates that.

[0513] The number of subsets d1 to d8 is eight because the number of samples in the standard sample data D_similarity is 20, which is 20×0.8×0.5=8.

[0514] Furthermore, when the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the subsets d1 to d8 is made up of the scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 and the standard sample data D_similarity is the standard sample data D_standard_2, the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 and the standard sample data D_similarity is the standard sample data D_standard_3, the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ _known}_3.

[0515] Furthermore, each of the subsets d1 to d8 has a scan response delay {RTT 1’ _known}_1~{RTT 8’ _known}_1 and communication characteristics {CMbest 1’ _known}_1~{CMbest 8’ _known}_1, the standard sample data D_standard_1 of the initial sample data D_sample_initial shown in FIG. 60 is replaced with subsets d1 to d8 in turn, thereby sequentially generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 58.

[0516] Furthermore, each of the subsets d1 to d8 has a scan response delay {RTT 1’ _known}_2~{RTT 8’ _known}_2 and communication characteristics {CMbest 1’ _known}_22~{CMbest 8’ _known}_2, the standard sample data D_standard_2 of the initial sample data D_sample_initial shown in FIG. 60 is replaced with the subsets d1 to d8 in turn, thereby sequentially generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 58.

[0517] Furthermore, each of the subsets d1 to d8 has a scan response delay {RTT 1’ _known}_3~{RTT 8’ _known}_3 and communication characteristics {CMbest 1’ _known}_3~{CMbest 8’_known}_3, the standard sample data D_standard_3 of the initial sample data D_sample_initial shown in FIG. 60 is replaced with the subsets d1 to d8 in turn, thereby generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 58 in turn.

[0518] Referring to Figure 62, the validation set is made up of eight subsets V 1’ ~V 8’ Consists of the subset V 1’ ~V 8’ Since M'=8, each of the scan response delays is 8 RTTs. 1’ ~RTT 8’ and 8 communication characteristics CMbest 1’ ~CMbest 8’ Therefore, the subset V 1’ ~V 8’ In each of the above, the vertical direction is eight scan response delay RTTs corresponding to eight terminal devices. 1’ ~RTT 8’ and 8 communication characteristics CMbest 1’ ~CMbest 8’ Indicates that.

[0519] Also, the subset V 1’ ~V 8’ The reason why the number of subsets d1 to d8 is eight is the same as the reason why the number of subsets d1 to d8 is eight.

[0520] Furthermore, the subset V 1’ ~V 8’ When the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the scan response delays {RTT 1’ _known}_1~{RTT M’ _known}_1 part and communication characteristics {CMbest 1’ _known}_1~{CMbest M’_known}_1 and the standard sample data D_similarity is the standard sample data D_standard_2, the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 part and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 and the standard sample data D_similarity is the standard sample data D_standard_3, the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 part and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ _known}_3.

[0521] 63, the test data consists of four subsets TD1 to TD4. Since M'=8 for each of the subsets TD1 to TD4, there are eight scan response delay RTTs. 1’ ~RTT 8’ and 8 communication characteristics CMbest 1’ ~CMbest 8’ Therefore, in each of the subsets TD1 to TD4, the vertical direction is made up of eight scan response delay RTTs corresponding to eight terminal devices. 1’ ~RTT 8’ and 8 communication characteristics CMbest 1’ ~CMbest 8’ Indicates that.

[0522] The reason why the number of subsets TD1 to TD4 is four is because the number of samples in the standard sample data D_similarity is one IP address Add m’ This is because there are 20 samples for D_similarity, and the test data consists of 20% (20×0.2=4) of the number of samples (=20) of the standard sample data D_similarity.

[0523] Furthermore, when the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the subsets TD1 to TD4 has a scan response delay {RTT 1’ _known}_1~{RTT M’ _known}_1 part and communication characteristics {CMbest 1’ _known}_1~{CMbest M’ _known}_1 and the standard sample data D_similarity is the standard sample data D_standard_2, the scan response delay {RTT 1’ _known}_2~{RTT M’ _known}_2 part and communication characteristics {CMbest 1’ _known}_2~{CMbest M’ _known}_2 and the standard sample data D_similarity is the standard sample data D_standard_3, the scan response delay {RTT 1’ _known}_3~{RTT M’ _known}_3 part and communication characteristics {CMbest 1’ _known}_3~{CMbest M’ _known}_3.

[0524] The estimation means 17 trains the estimation algorithm ALG using the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U. When training the estimation algorithm ALG using the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, the estimation means 17 sequentially generates tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) including subsets d1 to d8, respectively, by the method described in Fig. 58 and Fig. 61, and trains the estimation algorithm ALG by repeatedly estimating the results of a network scan by the estimation algorithm ALG using the generated tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8).

[0525] The estimation means 17 trains the estimation algorithm ALG using the initial sample data D_sample_initial, and then estimates the results of the network scan by the estimation algorithm ALG using the validation set, and generates a confusion matrix CFM based on the estimated results of the network scan.

[0526] Furthermore, the estimation means 17 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, then estimates the results of the network scan by the estimation algorithm ALG using the validation set, and generates a confusion matrix nCFM based on the estimated results of the network scan. The estimation means 17 performs this operation for all of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U.

[0527] Fig. 64 is a conceptual diagram of a confusion matrix. When estimating the results of a network scan, there are three known estimated results: "normal," "poor communication quality," and "network congestion." Therefore, Fig. 64 shows a confusion matrix for the three known estimated results: "normal," "poor communication quality," and "network congestion."

[0528] 64, TP indicates the percentage of cases where the occurrence of an assumed event (the occurrence of the assumed event) was correctly estimated when the assumed event actually occurred, TN indicates the percentage of cases where the occurrence of an assumed event (the non-occurrence of the assumed event) was correctly estimated when the assumed event did not actually occur, FN indicates the percentage of cases where the occurrence of an assumed event (the occurrence of the assumed event) was incorrectly estimated as a different event when the assumed event did occur, and FP indicates the percentage of cases where the assumed event was incorrectly estimated to have occurred when the assumed event did not actually occur. Also, in (a), (b), and (c) of FIG. 64, "Normal," "Poor Communication Quality," and "Network Congestion" in the column direction (vertical direction) represent events that actually occurred, and "Normal," "Poor Communication Quality," and "Network Congestion" in the row direction (horizontal direction) represent estimated events. Furthermore, (a) in Figure 64 represents the confusion matrix when the result of the network scan is "normal," (b) in Figure 64 represents the confusion matrix when the result of the network scan is "insufficient communication quality," and (c) in Figure 64 represents the confusion matrix when the result of the network scan is "network congestion."

[0529] In (a) of Figure 64, the TP at the intersection of the "Normal" row and the "Normal" column indicates the percentage of times when an assumed "normal" event (the occurrence of an assumed "normal" event) was correctly estimated as a "normal" event, the FN at the intersection of the "Normal" row and the "Insufficient communication quality" column indicates the percentage of times when an assumed "normal" event was incorrectly estimated as a different "Insufficient communication quality" event, and the FN at the intersection of the "Normal" row and the "Network congestion" column indicates the percentage of times when an assumed "normal" event was incorrectly estimated as a different "Network congestion" event.

[0530] Furthermore, the FP at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases where an "inadequate communication quality" event, which is not an assumed "normal" event, occurred (i.e., the assumed "normal" event did not actually occur), was incorrectly assumed to be a "normal" event; the TN at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases where an "inadequate communication quality" event, which is not an assumed "normal" event, was incorrectly assumed to be a "inadequate communication quality" event (i.e., the percentage of cases where an assumed "normal" event did not actually occur, was correctly assumed to be a "normal" event); and the TN at the intersection of the row of "inadequate communication quality" and the column of "network congestion" indicates the percentage of cases where an "inadequate communication quality" event, which is not an assumed "normal" event, was incorrectly assumed to be a "network congestion" event (i.e., the percentage of cases where an assumed "normal" event did not actually occur, was correctly assumed to be a "normal" event).

[0531] Furthermore, the FP at the intersection of the "Network Congestion" row and the "Normal" column indicates the percentage of cases where a "Network Congestion" event that is not an assumed "Normal" event occurred (i.e., the assumed "Normal" event did not actually occur) was incorrectly estimated to have occurred, whereas the TN at the intersection of the "Network Congestion" row and the "Insufficient Communication Quality" column indicates the percentage of cases where a "Network Congestion" event that is not an assumed "Normal" event occurred was incorrectly estimated to have occurred (i.e., the assumed "Normal" event did not actually occur) was correctly estimated). The TN at the intersection of the "Network Congestion" row and the "Network Congestion" column indicates the percentage of cases where a "Network Congestion" event that is not an assumed "Normal" event occurred was incorrectly estimated to have occurred (i.e., the assumed "Normal" event did not actually occur) was correctly estimated).

[0532] In (b) of Figure 64, the TN at the intersection of the "normal" row and the "normal" column indicates the proportion of cases where a "normal" event other than the assumed "poor communication quality" occurred, and the proportion of cases where a "normal" event occurred (i.e., cases where the assumed "poor communication quality" event did not actually occur, and the proportion of cases where the assumed "poor communication quality" event did not ... The TN at the intersection of the "Normal" row and the "Network Congestion" column indicates the percentage of cases where a "Normal" event occurred that was not the assumed "Inadequate Communication Quality" and a "Network Congestion" event occurred (i.e., cases where the assumed "Inadequate Communication Quality" event did not actually occur and the percentage of cases where the TN was correctly estimated to be the case (that the assumed "Inadequate Communication Quality" event did not occur)).

[0533] Furthermore, the FN at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases where, when an assumed "inadequate communication quality" event occurs, the occurrence of the assumed "inadequate communication quality" event is mistakenly assumed to be a different "normal" event; the TP at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases where, when an assumed "inadequate communication quality" event occurs, the occurrence of the assumed "inadequate communication quality" event is correctly mistakenly assumed to be a "inadequate communication quality" event; and the FN at the intersection of the row of "inadequate communication quality" and the column of "network congestion" indicates the percentage of cases where, when an assumed "inadequate communication quality" event occurs, the occurrence of the assumed "inadequate communication quality" event is mistakenly assumed to be a different "network congestion" event.

[0534] Furthermore, the TN at the intersection of the "Network Congestion" row and the "Normal" column indicates the percentage of times when a "network congestion" event that was not the assumed "inadequate communication quality" occurred was estimated to have occurred (i.e., when the assumed "inadequate communication quality" event did not actually occur, the percentage of times when it was correctly estimated that the assumed "inadequate communication quality" event did not occur), the FP at the intersection of the "Network Congestion" row and the "inadequate communication quality" column indicates the percentage of times when a "network congestion" event that was not the assumed "inadequate communication quality" occurred was estimated to have occurred, and the TN at the intersection of the "Network Congestion" row and the "Network Congestion" column indicates the percentage of times when a "network congestion" event that was not the assumed "inadequate communication quality" occurred was estimated (i.e., when the assumed "inadequate communication quality" event did not actually occur, the percentage of times when it was correctly estimated that the assumed "inadequate communication quality" event did not occur).

[0535] In (c) of Figure 64, the TN at the intersection of the "Normal" row and the "Normal" column indicates the percentage of cases where a "normal" event that is not the assumed "network congestion" occurred and it was estimated that a "normal" event had occurred (i.e., when the assumed "network congestion" event did not actually occur, it was correctly estimated that this (the assumed "network congestion" event did not occur)), the TN at the intersection of the "Normal" row and the "Inadequate Communication Quality" column indicates the percentage of cases where a "normal" event that is not the assumed "network congestion" occurred and it was estimated that an "inadequate communication quality" event had occurred (i.e., when the assumed "network congestion" event did not actually occur, it was correctly estimated that this (the assumed "network congestion" event did not occur)), and the FP at the intersection of the "Normal" row and the "Network Congestion" column indicates the percentage of cases where a "normal" event that is not the assumed "network congestion" occurred and it was estimated that the assumed "network congestion" event had occurred (i.e., when the assumed "network congestion" event did not actually occur), it was incorrectly estimated that the assumed "network congestion" event had occurred.

[0536] Furthermore, the TN at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases where a "normal" event was estimated to have occurred when an "inadequate communication quality" event, which was not the assumed "network congestion", occurred (i.e., when the assumed "network congestion" event did not actually occur, the percentage of cases where the fact (the assumed "network congestion" event did not occur) was correctly estimated). The TN at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases where a "inadequate communication quality" event, which was not the assumed "network congestion" event, occurred (i.e., when the assumed "network congestion" event did not actually occur, the percentage of cases where the fact (the assumed "network congestion" event did not occur) was correctly estimated). The FP at the intersection of the row of "inadequate communication quality" and the column of "network congestion" indicates the percentage of cases where an "inadequate communication quality" event, which was not the assumed "network congestion" event, occurred (i.e., when the assumed "network congestion" event did not actually occur), the percentage of cases where the incorrectly estimated "network congestion" event occurred.

[0537] Furthermore, the FN at the intersection of the "Network Congestion" row and the "Normal" column indicates the percentage of cases where, when an assumed "Network Congestion" event occurs, the occurrence of the assumed "Network Congestion" event is mistakenly estimated as a different "Normal" event; the FN at the intersection of the "Network Congestion" row and the "Insufficient Communication Quality" column indicates the percentage of cases where, when an assumed "Network Congestion" event occurs, the occurrence of the assumed "Network Congestion" event is mistakenly estimated as a different "Insufficient Communication Quality" event; and the TP at the intersection of the "Network Congestion" row and the "Network Congestion" column indicates the percentage of cases where, when an assumed "Network Congestion" event occurs, the occurrence of the assumed "Network Congestion" event is correctly estimated as a "Network Congestion" event.

[0538] A method for generating a confusion matrix will be explained. Fig. 65 is a diagram for explaining a method for generating a confusion matrix. Note that Fig. 65 explains a method for generating a confusion matrix by taking as an example a case where a confusion matrix is ​​generated when the result of the network scan shown in Fig. 64(a) is "normal."

[0539] Referring to FIG. 65, since the result of the network scan is "normal", in the column direction, "normal" is an expected event, and "poor communication quality" and "network congestion" are unexpected events.

[0540] The estimation means 17 trains the estimation algorithm ALG using old training data consisting of the initial sample data D_sample_initial, and then uses the validation set (= the subset V shown in FIG. 62) 1’ ~V 8’ ) is used to estimate the results of a network scan by an estimation algorithm ALG, and based on the estimated results of the network scan, the process of counting A, which is the number of correctly estimated expected events (=“normal”) when an expected event (=“normal”) occurs, B, which is the number of incorrectly estimated expected events (=“normal”) as unexpected events (=“poor communication quality”) when an expected event (=“normal”) occurs, and C, which is the number of incorrectly estimated expected events (=“normal”) as unexpected events (=“network congestion”) when an expected event (=“normal”) occurs, is performed using a subset V shown in FIG. 1’ ~V 8’ Execute this for all lines of

[0541] Subset V shown in Figure 62 1’ ~V 8’ One line of the scan response delay RTT m’ and communication characteristics CMbest m’ RTT m’ ,CMbest m’}, so if the expected event (="normal") is correctly estimated, A=8, B=C=0 are obtained. As a result, the counts of A, B, and C are included in the subset V shown in Figure 62.1’ ~V 8’ When this is performed for all rows, if the expected event (= “normal”) is correctly estimated in all subsets, A=64, B=C=0 will be obtained.

[0542] The estimation means 17 estimates the validation set (= the subset V shown in FIG. 62) 1’ ~V 8’ The process of estimating the results of the network scan using the estimation algorithm ALG is performed using the subset V 1’ ~V 8’ After executing this for all rows and counting A, B, and C, the counted A, B, and C are assigned to the confusion matrix table as shown in FIG. 65.

[0543] Then, the estimation means 17 obtains A as OE, B+C as OIE, and A+B+C as ECE.

[0544] Here, OE represents the total number of events that occurred as expected but were correctly estimated, ECE represents the total number of events that occurred as expected, and OIE represents the total number of events that occurred as expected but were incorrectly estimated as unexpected.

[0545] Then, the estimation means 17 calculates TP and FN based on the acquired OE, OIE, and ECE using the following equations.

[0546]

number

[0547] By calculating TP and FN based on OE, OIE, and ECE using equation (25), the TP and FN in the first row of the confusion matrix CFM shown in Figure 64(a) are obtained. In this case, as shown in Figure 64(a), FN is obtained as the sum of the two elements of "insufficient communication quality" and "network congestion" in the confusion matrix. In this way, TP and FN are obtained based on the estimation results of the network scan.

[0548] Thereafter, the estimation means 17 calculates FP and TN based on the calculated TP and FN using the following equations.

[0549]

number

[0550] The TP at the intersection of the "Normal" row and the "Normal" column shown in Figure 64(a) is calculated by substituting the total number of correctly estimated "Normal" events that occurred as expected, OE, and the total number of "Normal" events that occurred as expected, ECE, into equation (25A).Then, the FP at the intersection of the "Insufficient Communication Quality" row and the "Normal" column shown in Figure 64(a) and the FP at the intersection of the "Network Congestion" row and the "Normal" column are calculated by substituting the TP calculated by equation (25A) into equation (26A).

[0551] Furthermore, FN (=FN1) at the intersection of the "Normal" row and the "Insufficient Communication Quality" column shown in Figure 64(a) is calculated by substituting the total number OIE of "Normal" events that occurred as expected but were mistakenly estimated as "Insufficient Communication Quality" events outside the expected range, and the total number ECE of "Normal" events that occurred as expected, into equation (25B). TN at the intersection of the "Insufficient Communication Quality" row and the "Insufficient Communication Quality" column shown in Figure 64(a) and TN at the intersection of the "Network Congestion" row and the "Insufficient Communication Quality" column are calculated by substituting FN (=FN1) calculated by equation (25B) into equation (26B).

[0552] Furthermore, FN (=FN2) at the intersection of the "Normal" row and the "Network Congestion" column shown in Figure 64(a) is calculated by substituting the total number OIE of "Normal" events that occurred as expected but were mistakenly estimated as unexpected "Network Congestion" events, and the total number ECE of "Normal" events that occurred as expected, into equation (25B).Then, TN at the intersection of the "Insufficient Communication Quality" row and the "Network Congestion" column shown in Figure 64(a) and TN at the intersection of the "Network Congestion" row and the "Network Congestion" column are calculated by substituting FN (=FN2) calculated by equation (25B) into equation (26B).

[0553] After calculating TP, FN, TN, and FP by the above-described method, the estimation means 17 generates a confusion matrix CFM shown in FIG. 64(a) that includes the calculated TP, FN, TN, and FP.

[0554] The estimation means 17 similarly generates the confusion matrices CFM shown in (b) and (c) of Figure 64. In this case, since the result of the network scan is "inadequate communication quality," the confusion matrix CFM shown in (b) of Figure 64 has "inadequate communication quality" as an expected event in the column direction, and "normal" and "network congestion" as unexpected events. As a result, by the above-mentioned method, TP and FN in the second row of the confusion matrix CFM shown in (b) of Figure 64 are calculated based on OE, OIE, and ECE, and then FP and TN in the first and third rows are calculated based on TP and FN.

[0555] In addition, in the confusion matrix CFM shown in (c) of Figure 64, since the result of the network scan is "network congestion," "network congestion" is an expected event in the column direction, and "normal" and "poor communication quality" are unexpected events. As a result, by the above-mentioned method, TP and FN in the third row of the confusion matrix CFM shown in (c) of Figure 64 are calculated based on OE, OIE, and ECE, and then FP and TN in the first and second rows are calculated based on TP and FN.

[0556] In this way, the estimation means 17 acquires TP and FN from the estimation results of the network scan, calculates FP and TN based on the acquired TP and FN, and generates a confusion matrix CFM consisting of TP, FN, FP, and TN. Furthermore, the estimation means 17 calculates FN, FP, and TN by adding up multiple elements in the confusion matrix.

[0557] This method of generating the confusion matrix CFM differs from a general method of generating a confusion matrix in which TP, FN, FP, and TN are obtained based on the observation of the subject for which the confusion matrix is ​​to be generated, and a confusion matrix is ​​generated.

[0558] As described above, the estimation means 17 trains the estimation algorithm ALG using old training data consisting of the initial sample data D_sample_initial, and then uses the validation set (= the subset V shown in FIG. 62) 1’ ~V 8’ When the results of the network scan are estimated using the estimation algorithm ALG using all rows of , a confusion matrix CFM is generated.

[0559] In addition, the estimation means 17 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, and then similarly calculates a validation set (= a subset V shown in FIG. 62) 1’ ~V 8’ ) is used to estimate the results of the network scan using the estimation algorithm ALG, and the confusion matrix nCFM when the results of the network scan are estimated is generated in the same way as the confusion matrix CFM.

[0560] Furthermore, after generating the confusion matrix nCFM, the estimation means 17 estimates the result of the network scan by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD1 to TD4 shown in FIG. 63), and generates the confusion matrix tCFM when the result of the network scan is estimated by the same method as the confusion matrix CFM.

[0561] In this case, the estimation means 17 estimates the results of the network scan using the estimation algorithm ALG using the test data (= consisting of one row of the subsets TD1 to TD4 shown in Figure 63), and performs the process of counting A, B, and C for all rows of the subsets TD1 to TD4 shown in Figure 63 based on the estimated results of the network scan.

[0562] One row of the subsets TD1 to TD4 shown in Figure 63 is the scan response delay RTT m’ and communication characteristics CMbest m’ RTT m’ ,CMbest m’}, there are four pairs, so if the expected event (="normal") is correctly estimated, A=4 and B=C=0 are obtained. As a result, when counting A, B, and C for all rows of subsets TD1 to TD4 shown in Figure 63, if the expected event (="normal") is correctly estimated in all subsets, A=32 and B=C=0 are obtained. The same applies when the expected event is "poor communication quality" or "network congestion."

[0563] After generating the confusion matrices CFM, nCFM, tCFM, the estimation means 17 outputs the generated confusion matrices CFM, nCFM, tCFM to the calculation means 16.

[0564] The calculation means 16 receives the confusion matrices CFM, nCFM, and tCFM from the estimation means 17. Then, the calculation means 16 uses the elements TP, TN, FP, and FN of the confusion matrices CFM, nCFM, and tCFM to calculate Precision, which is the ratio of the proportion actually classified as TP out of the proportion classified as TP or FP, and Recall, which is the ratio of the proportion actually classified as TP out of the proportion classified as TP or FN.

[0565] Precision is calculated by equation (27), and Recall is calculated by equation (28).

[0566]

number

[0567]

number

[0568] After calculating Precision and Recall, the calculation means 16 calculates the F1 score F1 by substituting the calculated Precision and Recall into the following equation.

[0569]

number

[0570] The calculation means 16 calculates an F1 score F1 using equations (27) to (29) based on TP, TN, FP, and FN of the confusion matrix CFM. Furthermore, the calculation means 16 calculates an F1 score nF1_u using equations (27) to (29) based on TP, TN, FP, and FN of the confusion matrix nCFM. Furthermore, the calculation means 16 calculates an F1 score tF1 using equations (27) to (29) based on TP, TN, FP, and FN of the confusion matrix tCFM. Then, the calculation means 16 outputs the F1 scores F1, nF1_u, and tF1 to the estimation means 17.

[0571] The F1 score F1 is calculated by training the estimation algorithm ALG using the initial sample data D_sample_initial and testing it on the validation set (= the subset V shown in Figure 62). 1’ ~V 8’ It is an evaluation index EVD that represents the accuracy of the estimation results when the results of a network scan are estimated using the estimation algorithm ALG (the estimation algorithm ALG after training) using all rows of .

[0572] In addition, the F1 score nF1_u is calculated by training the estimation algorithm ALG using the tentative sample data D_sample_tentative_u and then calculating the F1 score nF1_u using the validation set (= the subset V shown in Figure 62). 1’ ~V 8’It consists of all rows of . ) and is an evaluation index nEVD_u that represents the accuracy of the estimation results when the results of a network scan are estimated using the estimation algorithm ALG (trained estimation algorithm ALG).

[0573] Furthermore, the F1 score tF1 is an evaluation index tEVD that represents the accuracy of the estimation results when the results of a network scan are estimated by the estimation algorithm ALG (the estimation algorithm ALG after training) using the test data (= consisting of all rows of the subsets TD1 to TD4 shown in Figure 63).

[0574] In an embodiment of the present invention, Matthews correlation coefficients MCC, nMCC_u, and tMCC may be used as the evaluation indices EVD, nEVD_u, and tEVD, respectively, instead of the F1 scores F1, nF1_u, and tF1. In this case, the calculation means 16 calculates the Matthews correlation coefficient MCC according to equation (30) based on TP, TN, FP, and FN of the confusion matrix CFM, calculates the Matthews correlation coefficient nMCC_u according to equation (30) based on TP, TN, FP, and FN of the confusion matrix nCFM, and calculates the Matthews correlation coefficient tMCC according to equation (30) based on TP, TN, FP, and FN of the confusion matrix tCFM.

[0575]

number

[0576] The Matthews correlation coefficient (MCC) calculated by equation (30) is "+1" when the best training data is used, "-1" when the worst training data is used, and "0" when neutral (neither "best" nor "worst") training data is used.

[0577] After calculating the Matthews correlation coefficients MCC, nMCC_u, and tMCC, the calculation means 16 outputs the Matthews correlation coefficient MCC to the estimation means 17 as an evaluation index EVD, outputs the Matthews correlation coefficient nMCC_u to the estimation means 17 as an evaluation index nEVD_u, and outputs the Matthews correlation coefficient tMCC to the estimation means 17 as an evaluation index tEVD.

[0578] When the estimation means 17 receives the evaluation index nEVD_u consisting of the F1 score nF1_1 and the evaluation index EVD consisting of the F1 score F1 from the calculation means 16, it determines whether the F1 score nF1_1 is greater than the F1 score F1.

[0579] When the estimation means 17 determines that the F1 score nF1_1 is greater than the F1 score F1, it sets the F1 score nF1_1 to the F1 score F1, and sets tentative sample data D_sample_tentative_1 including the subset d1 when the F1 score nF1_1 was obtained as training data. Then, the estimation means 17 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 17 trains the estimation algorithm ALG using the next subset d2, and sets a validation set (= subset V 1’ ~V 8’ The estimation means 17 estimates the result of the network scan using the estimation algorithm ALG (the post-training estimation algorithm ALG) and generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 17 outputs the generated confusion matrix nCFM_2 to the calculation means 16.

[0580] On the other hand, when the estimation means 17 determines that the F1 score nF1_1 is not greater than the F1 score F1, it does not set the F1 score nF1_1 to the F1 score F1, and trains the estimation algorithm ALG using tentative sample data D_sample_tentative_2 including the next subset d2, and sets the F1 score nF1_1 to the validation set (= subset V 1’ ~V 8’The result of the network scan is estimated by the estimation algorithm ALG (the post-training estimation algorithm ALG) using the above. The confusion matrix nCFM_2 obtained when the result of the network scan is estimated is generated and output to the calculation means 16.

[0581] Thereafter, when the estimation means 17 receives the evaluation index nEVD_u consisting of the F1 score nF1_2 from the calculation means 16, it determines whether or not the F1 score nF1_2 is greater than the F1 score F1.

[0582] When the estimation means 17 determines that the F1 score nF1_2 is greater than the F1 score F1, it sets the F1 score nF1_2 to the F1 score F1, and sets tentative sample data D_sample_tentative_2 including the subset d2 when the F1 score nF1_2 is obtained as training data. Then, the estimation means 17 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 17 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d3, and sets a validation set (=subset V 1’ ~V 8’ The estimation means 17 estimates the result of the network scan using the estimation algorithm ALG (the post-training estimation algorithm ALG) and generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 17 outputs the generated confusion matrix nCFM_3 to the calculation means 16.

[0583] On the other hand, when the estimation means 17 determines that the F1 score nF1_2 is not greater than the F1 score F1, it does not set the F1 score nF1_2 to the F1 score F1, and trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d3, and sets the F1 score nF1_2 to the F1 score F1. 1’ ~V 8’The estimation means 17 estimates the result of the network scan using the estimation algorithm ALG (the post-training estimation algorithm ALG) and generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 17 outputs the generated confusion matrix nCFM_3 to the calculation means 16.

[0584] Thereafter, the estimation means 17 uses the F1 score F1,nF1_u to estimate the subsets d3 to d U The above-described operations are repeatedly performed for all of the above.

[0585] In addition, when the estimation means 17 receives an evaluation index EVD consisting of the Matthews correlation coefficient MCC and an evaluation index nEVD_1 consisting of the Matthews correlation coefficient nMCC_1 from the calculation means 16, it determines whether the Matthews correlation coefficient nMCC_1 is greater than the Matthews correlation coefficient MCC.

[0586] When the estimation means 17 determines that the Matthews correlation coefficient nMCC_1 is greater than the Matthews correlation coefficient MCC, it sets the Matthews correlation coefficient nMCC_1 to the Matthews correlation coefficient MCC, and sets tentative sample data D_sample_tentative_1 including the subset d1 when the Matthews correlation coefficient nMCC_1 is obtained as training data. Then, the estimation means 17 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 17 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_2 including the next subset d2, and sets a validation set (=subset V 1’ ~V 8’ The estimation means 17 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 17 generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 17 outputs the generated confusion matrix nCFM_2 to the calculation means 16.

[0587] On the other hand, when the estimation means 17 determines that the Matthews correlation coefficient nMCC_1 is not greater than the Matthews correlation coefficient MCC, it does not set the Matthews correlation coefficient nMCC_1 to the Matthews correlation coefficient MCC, and trains the estimation algorithm ALG using tentative sample data D_sample_tentative_2 including the next subset d2, and sets the validation set (=subset V 1’ ~V 8’ The estimation means 17 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 17 generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 17 outputs the generated confusion matrix nCFM_2 to the calculation means 16.

[0588] Thereafter, when the estimation means 17 receives the evaluation index nEVD_2 consisting of the Matthews correlation coefficient nMCC_2 from the calculation means 16, it determines whether or not the Matthews correlation coefficient nMCC_2 is greater than the Matthews correlation coefficient MCC.

[0589] When the estimation means 17 determines that the Matthews correlation coefficient nMCC_2 is greater than the Matthews correlation coefficient MCC, it sets the Matthews correlation coefficient nMCC_2 to the Matthews correlation coefficient MCC, and sets tentative sample data D_sample_tentative_2 including the subset d2 when the Matthews correlation coefficient nMCC_2 is obtained as training data. Then, the estimation means 17 replaces variables of the estimation algorithm ALG using the training data by a method described later. Thereafter, the estimation means 17 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d3, and sets a validation set (=subset V 1’ ~V 8’ The estimation means 17 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 17 generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 17 outputs the generated confusion matrix nCFM_3 to the calculation means 16.

[0590] On the other hand, when the estimation means 17 determines that the Matthews correlation coefficient nMCC_2 is not greater than the Matthews correlation coefficient MCC, it does not set the Matthews correlation coefficient nMCC_2 to the Matthews correlation coefficient MCC, and trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d3, and sets the validation set (=subset V 1’ ~V 8’ The estimation means 17 estimates the result of the network scan by the estimation algorithm ALG using all of the above. The estimation means 17 generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 17 outputs the generated confusion matrix nCFM_3 to the calculation means 16.

[0591] Thereafter, the estimation means 17 estimates the subsets d3 to d6 using the Matthews correlation coefficient MCC,nMCC_u. U The above-described operations are repeatedly performed for all of the above.

[0592] [Estimation of network scan results] In the training of the estimation algorithm ALG, the estimation means 17 calculates the subsets d1 to d2 generated from the initial sample data D_sample_initial and the standard sample data D_similarity. U Using tentative sample data D_sample_tentative_1 to D_sample_tentative_U each including the above, the results of the network scan are estimated by the following method.

[0593] In the explanation of the method for estimating the results of a network scan, the scan response delay (RTT) included in each of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U is used. 1’ _known}_z~{RTT M’ _known}_z as “Scan Response Delay RTT m " and the communication characteristics CMbest included in each of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U.m _unknown and the communication characteristics included in the standard sample data D_similarity {CMbest 1’ _known}_z~{CMbest M’ _known}_z is expressed as "communication characteristics CMbest".

[0594] The estimation means 17 estimates the scan response delay RTT m and communication characteristics CMbest from the extraction means 15. Then, the estimation means 17 classifies the terminal devices having the communication characteristics CMbest into G (G is an integer of 3 or more) groups CLS1 to CLS2 using an LDA (Linear Discriminant Analysis) algorithm. G Classify into:

[0595] The LDA algorithm divides terminal devices having communication characteristics CMbest into G groups CLS1 to CLS2 so as to maximize the ratio of inter-group variance to intra-group variance. G Therefore, the groups CLS1 to CLS G This ensures maximum isolation between the

[0596] The LDA algorithm spheres the data (terminal devices with communication characteristics CMbest) based on the common covariance, and classifies the data (terminal devices with communication characteristics CMbest) into the closest group centroid in the transformed space.

[0597] To achieve this, the separability between different groups (also called "inter-group covariance"), which is the distance between the means of different groups, is calculated. Then, the distance between the mean and the sample of each group (also called "intra-group covariance") is calculated. This constructs a low-dimensional space that maximizes the inter-group covariance and minimizes the intra-group covariance. Then, in the constructed low-dimensional space, data (terminal devices having communication characteristics CM best) are classified into the closest group centroid. The LDA algorithm is described in "T. Hastie, R. Tibshirani, and J.H. Friedman, "The elements of statistical learning: data mining, inference, and prediction," 2nd ed. Chapter 4, pp. 106-118, New York: Springer, 2009," and is publicly known. Therefore, a detailed description of the classification method using the LDA algorithm will be omitted.

[0598] Also, groups CLS1 to CLS G The number G of the groups CLS1 to CLS2 is set to the same number as the number of causes of the network scan results caused by the communication channel with the terminal device among the estimated results of the network scan. That is, the three results of "normal", "poor communication quality" and "network congestion" are causes caused by the communication channel with the terminal device, so the groups CLS1 to CLS2 are set to the same number as the number of causes of the network scan results caused by the communication channel with the terminal device among the estimated results of the network scan. G The number G of these is set to "3". As a result, the estimation means 17 classifies the terminal devices having the communication characteristic CMbest into three groups CLS1 to CLS3 using the LDA algorithm. The group CLS1 is a "normal" group, the group CLS2 is a "poor communication quality" group, and the group CLS3 is a "network congestion" group.

[0599] When classifying terminal devices having communication characteristics CMbest into groups CLS1 to CLS3, the estimation means 17 learns an LDA algorithm using scan data when the network scan is "network congested" (communication characteristics CMbest when "network is congested"), and classifies terminal devices having communication characteristics CMbest into three groups CLS1 to CLS3 using the learned LDA algorithm. This makes it possible to most accurately estimate that "network congestion" is the cause of a network scan failure.

[0600] The estimation means 17 uses Model1, Model2 and Model3 to estimate the results of the network scan. Model1 handles groups CLS1 and CLS2, Model2 handles groups CLS1 and CLS3, and Model3 handles groups CLS2 and CLS3.

[0601] Then, when using Model 1, the estimation means 17 calculates a D score D_1 that distinguishes between groups CLS1 and CLS2, when using Model 2, it calculates a D score D_2 that distinguishes between groups CLS1 and CLS3, and when using Model 3, it calculates a D score D_3 that distinguishes between groups CLS2 and CLS3.

[0602] When the D scores D_1 to D_3 are calculated, the discriminant function δ is expressed by the following equation: pq (x) is used.

[0603]

number

[0604] In equation (31A), x is the scan response delay RTT m and the communication characteristic CMbest are assigned. Then, x is the scan response delay RTT when the results of the network scan are known. mand communication characteristics CMbest (optimal training data), or scan response delay RTT when the results of network scans are unknown. m and the communication characteristic CMbest. Also, μ in Equation (31A) g When calculating the discriminant function for group p, the average μ p When calculating the discriminant function for group q, the average μ q Furthermore, the prior average Π in Eq. (31A) g When calculating the discriminant function for group p, the prior average Π of the communication characteristics shown in equation (31E) is used. p When calculating the discriminant function for group q, the prior average Π of the communication characteristics shown in equation (31F) is q is substituted. The joint covariance matrix C of Eq. (31A) pq is expressed by equation (31B). On the right side of equation (31B), the first term is the covariance matrix in group p, and the second term is the covariance matrix in group q. Then, x in equation (31B) mp The communication characteristics (scan response delay RTT m and the communication characteristic CMbest) is substituted into x mq The communication characteristics (scan response delay RTT m and the communication characteristic CMbest) is substituted.

[0605] In formulas (31B), (31C), and (31D), M p is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest), and M q is the communication characteristics (scan response delay RTT m and the communication characteristic CMbest), and m p is one communication characteristic (scan response delay RTT m and the communication characteristic CMbest) and m qis one communication characteristic (scan response delay RTT m and the communication characteristic CMbest). In equations (31E) and (31F), M is the communication characteristic (scan response delay RTT m and communication characteristics CMbestt). Equation (31C) is the total number of communication characteristics (scan response delay RTT m and the communication characteristic CMbest) p and equation (31D) expresses the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) q Equation (31E) expresses the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) p and equation (31F) expresses the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) q Represents.

[0606] Equation (31B) is the combined covariance matrix C, which is obtained by combining two covariance matrices in two groups (two groups CLS1 and CLS2; two groups CLS1 and CLS3; or two groups CLS2 and CLS3). pq The combined covariance matrix C is obtained by combining the two covariance matrices in the two groups CLS1 and CLS2. 12 In the calculation of Equation (31A), p represents the group CLS1, and q represents the group CLS2. In addition, the merged covariance matrix C 13 In formula (20A), p represents group CLS1 and q represents group CLS3. Furthermore, the two covariance matrices in the two groups CLS2 and CLS3 are combined to form a merged covariance matrix C 23 When calculating, in formula (31A), p represents group CLS2 and q represents group CLS3.

[0607] Joint covariance matrix C in Model 1 12 is expressed by the following equation:

[0608]

number

[0609] In formula (32), M1 is M in formula (31B). p M2 corresponds to M in formula (31B) q and m1 corresponds to m in formula (31B). p m2 corresponds to m in formula (31B) q Also, μ1 corresponds to M1 in equation (31C). p and the communication characteristics classified into group CLS1 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mp μ2 is calculated by substituting M2 into M in equation (31D) q and the communication characteristics classified into group CLS2 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x in equation (31D). mq The joint covariance matrix C 12 is represented by the sum of the covariance matrix in group CLS1 and the covariance matrix in group CLS2.

[0610] Joint covariance matrix C in Model 2 13 is expressed by the following equation:

[0611]

number

[0612] In formula (33), M1 is M in formula (31B). p M3 corresponds to M in formula (31B) q and m1 corresponds to m in formula (31B). pm3 corresponds to m in formula (31B) q Also, μ1 corresponds to M1 in equation (31C). p and the communication characteristics classified into group CLS1 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mp μ3 is calculated by substituting M3 into M in equation (31D) q and the communication characteristics classified into group CLS3 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x in equation (31D). mq The joint covariance matrix C 13 is represented by the sum of the covariance matrix in group CLS1 and the covariance matrix in group CLS3.

[0613] Joint covariance matrix C in Model 3 23 is expressed by the following equation:

[0614]

number

[0615] In formula (34), M2 is M in formula (31B). p M3 corresponds to M in formula (31B) q m2 corresponds to m in formula (31B) p m3 corresponds to m in formula (31B) q Also, μ2 corresponds to M2 in equation (31C). p and the communication characteristics classified into group CLS2 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x mp μ3 is calculated by substituting M3 into M in equation (31D) q and the communication characteristics classified into group CLS3 (scan response delay RTT m and the communication characteristic CMbest) is expressed as x in equation (31D). mqThe joint covariance matrix C 23 is expressed as the sum of the covariance matrix in group CLS2 and the covariance matrix in group CLS3.

[0616] When calculating the D score D_1 in Model1, the discriminant function f in group CLS1 is 1 and the discriminant function f in group CLS2 2 is expressed by the following equation:

[0617]

number

[0618] In equation (35A), the prior average Π1 is calculated by dividing M1 by M in equation (31E). p and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (31E). Also, in equation (35B), the prior average Π2 is calculated by substituting M2 into M in equation (31F). q and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (31F). Also, the average μ1 in equation (35A) and the average μ2 in equation (35B) are calculated by the method explained in equation (32).

[0619] In addition, when calculating the D score D_2 in Model 2, the discriminant function f 1 and the discriminant function f in group CLS3 2 is expressed by the following equation:

[0620]

number

[0621] In equation (36A), the prior average Π1 is calculated by dividing M1 by M in equation (31E). pand the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (31E). Also, in equation (36B), the prior average Π2 is calculated by substituting M3 into M in equation (31F). q and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (31F). Also, the average μ1 in equation (36A) and the average μ3 in equation (36B) are calculated by the method explained in equation (33).

[0622] Furthermore, when calculating the D score D_3 in Model 3, the discriminant function f 1 and the discriminant function f in group CLS3 2 is expressed by the following equation:

[0623]

number

[0624] In equation (37A), the prior average Π2 is calculated by dividing M2 by M in equation (31E). p and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (31E). Also, in equation (37B), the prior average Π3 is calculated by substituting M3 into M in equation (31F). q and the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) into M in equation (31F). Also, the average μ2 in equation (37A) and the average μ3 in equation (37B) are calculated by the method explained in equation (34).

[0625] The D score D_1 is calculated using the discriminant function f 1 and the discriminant function f shown in equation (35B) 2 difference from = f 1 -f 2 It is expressed by:

[0626] The D score D_2 is calculated using the discriminant function f 1 and the discriminant function f shown in equation (36B) 2 difference from = f 1 -f 2 It is expressed by:

[0627] Furthermore, the D score D_3 is calculated using the discriminant function f 1 and the discriminant function f shown in equation (37B) 2 difference from = f 1 -f 2 It is expressed by:

[0628] Therefore, the D scores D_1 to D_3 can be calculated by using equations (31) to (37).

[0629] The estimation means 17 uses one of Model1 to Model3 to calculate a D score (one of D_1 to D_3).

[0630] Then, the estimation means 17 determines whether the D score is less than zero (=0). When the estimation means 17 determines that the D score is less than zero (=0), it increments the count value DS_count, which indicates the number of D scores less than zero (=0), by "1".

[0631] On the other hand, when the estimation means 17 determines that the D score is equal to or greater than zero (=0), it keeps the count value DS_count unchanged.

[0632] The estimation means 17 executes this process for all terminal devices belonging to one base station.

[0633] The estimation means 17 counts the count value DS_count of the D score, and then sets the Model for estimating the results of the network scan to Model2 or Model3 based on the count value DS_count.

[0634] More specifically, the estimation means 17 first sets Model to "1" (i.e., sets Model1), and when the count value DS_count is zero (=0) or when the count value DS_count is (N terminal × D_thres), the model Model is set to "2" (i.e., set to Model2). terminal is the total number of terminal devices belonging to one base station. D_thres is set to a real number in the range of 0 to 1.

[0635] Furthermore, the estimation means 17 determines whether the count value DS_count is equal to or greater than zero (=0) and whether the count value DS_count is equal to or greater than (N terminal ×D_thres), the model Model is set to "3" (i.e., set to Model3).

[0636] Then, when the model Model is not "1" (that is, when the model Model is "2" or "3"), the estimation means 17 determines whether the count value DS_count is zero (=0).

[0637] When it is determined that the count value DS_count is zero (=0), the estimation means 17 determines the "normal" scan data for the terminal device.

[0638] On the other hand, when it is determined that the count value DS_count is not zero (=0), the estimation means 17 determines whether the count value DS_count is (N terminal × D_thres).

[0639] And the count value DS_count is (N terminal ×D_thres), the estimation means 17 estimates that the cause of the failure of the network scan for the terminal device having a D score smaller than "0" is "insufficient communication quality."

[0640] On the other hand, if the count value DS_count is (N terminal×D_thres) (i.e., when it is determined that the count value DS_count is not smaller than (N terminal ×D_thres), the estimation means 17 estimates that the cause of the failure of the network scan for all terminal devices belonging to the base station is "network congestion (i.e., network congestion)."

[0641] Furthermore, when it is determined that the count value DS_count is zero (=0) and further that a scan response has been received from all ports of the terminal device, the estimation means 17 estimates that the network scan for the terminal device is "normal."

[0642] On the other hand, when it is determined that the count value DS_count is zero (=0) and further it is determined that a response has been received from some of the ports of the terminal device, the estimation means 17 estimates that the cause of the failure of the network scan for the terminal device is "port closed."

[0643] Furthermore, when it is determined that the count value DS_count is zero (=0) and furthermore that there is no scan response from all ports of the terminal device, the estimation means 17 estimates that the cause of the failure of the network scan for the terminal device is "no network connection."

[0644] The estimation algorithm ALG is an algorithm that estimates the results of a network scan using the method described above.

[0645] [Variable substitution in estimation algorithm] The estimation means 17 generates a validation set (=subset V 1’ ~V U’ ) and the F1 score nF1 when the network scan results are estimated by the estimation algorithm ALG using the above algorithm is greater than the F1 score F1, or the validation set (= subset V 1’ ~V U’When the Matthews correlation coefficient nMCC_u is larger than the Matthews correlation coefficient MCC when the network scan results are estimated by the estimation algorithm ALG using the above, the subset d is obtained when the F1 score nF1 is determined to be larger than the F1 score F1. max (Subset d1~d U The variables of the estimation algorithm ALG are replaced with tentative sample data D_sample_tentative_max containing either (D_sample_tentative_max) or the subset d when the Matthews correlation coefficient nMCC_max determined to be larger than the Matthews correlation coefficient MCC is finally obtained. max (Subset d1~d U The variables of the estimation algorithm ALG are replaced with the tentative sample data D_sample_tentative_max, which includes the following:

[0646] As explained in the above equation (31), x in equation (31A) is the scan response delay RTT m and the communication characteristic CMbest. p is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest) p and μ in equation (31D) q is the communication characteristics (scan response delay RTT m and the communication characteristic CMbest) q Furthermore, the prior average Π p is the communication characteristic (scan response delay RTT m and the communication characteristic CMbest), and the prior average Π q is the communication characteristics (scan response delay RTT m and the communication characteristic CMbest).

[0647] Here, the n-size data (data d1 to d2) when the F1 score nF1 determined to be greater than the F1 score F1 is finally obtained. U ) or the subset d when the Matthews correlation coefficient nMCC_u is finally determined to be larger than the Matthews correlation coefficient MCC. max (Subset d1~d U The tentative sample data D_sample_tentative_max including the above (either one of the above) is referred to as "sample data D_sample_optimum".

[0648] The estimation means 17 substitutes the sample data D_sample_optimum into x in the formula (31A) to obtain the scan response delay RTT m and the scan response delay RTT of the communication characteristics CMbest classified into group p. m The average of the vector consisting of the communication characteristics CMbest is μ p The scan response delay RTT that constitutes the sample data D_sample_optimum is calculated as m and the scan response delay RTT of the communication characteristics CMbest classified into group q. m The average of the vector consisting of the communication characteristics CMbest is μ q The scan response delay RTT that constitutes the sample data D_sample_optimum is calculated as m and the scan response delay RTT of the communication characteristics CMbest classified into group p. m The prior mean of the vector consisting of the communication characteristic CMbest and the prior mean Π p The scan response delay RTT that constitutes the sample data D_sample_optimum is calculated as m and the scan response delay RTT of the communication characteristics CMbest classified into group q. m The prior mean of the vector consisting of the communication characteristic CMbest and the prior mean Π q The variables of the estimation algorithm ALG are replaced with the sample data D_sample_optimum by calculating as follows:

[0649] This improves the accuracy of estimation when the estimation algorithm ALG estimates the results of a network scan.

[0650] In addition, subsets d1 to d U As mentioned above, the aggregate scan response delay {RTT m’ _known_aggregation}_z (m'=1'~M', z=1~3), so the subsets d1~d U Scan response delay {RTT m’ When generated based on a known aggregation, the scan response delay is m’ Subsets d1 to d consisting of a part of {_known_aggregation}_z (m'=1' to M', z=1 to 3) U is used to replace the variables of the estimation algorithm ALG.

[0651] [Verification using test data] After the model variables are replaced by the optimal training data, the estimation means 17 uses the test data (=consisting of all of the subsets TD1 to TD4) to estimate the results of the network scan using the model (estimation algorithm ALG) in which the model variables have been replaced, and generates a confusion matrix tCFM based on the estimated results of the network scan and outputs it to the calculation means 16.

[0652] The calculation means 16 receives the confusion matrix tCFM from the estimation means 17, calculates the F1 score tF1 using equations (27) to (29) based on the elements TP, TN, FP, and FN of the received confusion matrix tCFM, and outputs the calculated F1 score tF1 to the estimation means 17.

[0653] The estimation means 17 receives the F1 score tF1 from the calculation means 16 and determines whether the F1 score tF1 is equal to or greater than the F1 score F1.

[0654] When the estimation means 17 determines that the F1 score tF1 is equal to or greater than the F1 score F1, it approves the model (estimation algorithm ALG) and training data (optimal training data) in which the model variables have been replaced.

[0655] When it is determined that the F1 score tF1 is not equal to or greater than the F1 score F1, the estimation means 17 does not approve the model (estimation algorithm ALG) and training data (optimal training data) in which the model variables have been replaced.

[0656] In this way, the estimation means 17 uses the test data (= consisting of all of the subsets TD1 to TD4) to estimate the results of a network scan using the model (estimation algorithm ALG) in which the model variables have been replaced, and verifies the model (estimation algorithm ALG) in which the model variables have been replaced and the training data (optimal training data) by confirming that the F1 score tF1 is equal to or greater than the F1 score F1. In other words, the estimation means 17 uses the test data (= consisting of all of the subsets TD1 to TD4) to verify that the model (estimation algorithm ALG) in which the model variables have been replaced can more accurately estimate the results of a network scan performed in a communication environment in which the results of the network scan are unknown, compared to the model (estimation algorithm ALG) before the model variables were replaced.

[0657] 66 is a flowchart for explaining the operation of the network scan. Referring to FIG. 66, when the operation of the network scan is started, network scanner 11 reads addresses Add1 to Add2 of the terminal devices to be the targets of the network scan. M and port number PN r The scan schedule and the scan list including the above are obtained from the scan scheduler 18 (step S1).

[0658] Then, the network scanner 11 sets m=1 (step S2) and r= 1 (step S3).

[0659] Thereafter, the network scanner 11 performs a network scan with or without monitoring background traffic (step S4), and then acquires a scan response and a waiting time (step S5).

[0660] Then, the network scanner 11 will r =PN r _MAX(Port Number PN r It is determined whether the value is the maximum value of the above (step S6).

[0661] In step S6, PN r =PN r If it is determined that the value is not _MAX, the network scanner 11 r =PN r +1 is set (step S7). After that, the series of operations proceeds to step S4, and in step S6, PN r =PN r Steps S4 to S7 are repeatedly executed until it is determined that the value is _MAX.

[0662] Then, in step S6, PN r =PN r If it is determined that m is MAX, the network scanner 11 determines whether m=M (step S8).

[0663] If it is determined in step S8 that m is not equal to M, the network scanner 11 sets m to m+1 (step S9). After that, the series of operations proceeds to step S3, and steps S3 to S9 are repeatedly executed until it is determined in step S8 that m is equal to M.

[0664] If it is determined in step S8 that m=M, the network scanner 11 obtains the communication characteristics (scan response delay RTTm The network scanner 11 acquires the scan response and communication characteristics (multiple communication characteristics such as the number of scans, ...

[0665] Thereafter, the estimation means 17 calculates the scan response delay RTT from the correspondence table TBL1 (or correspondence table TBL2). m and reads out multiple communication characteristics such as the scan response delay RTT m The communication characteristic CMbest that is most suitable for estimating the results of the network scan is extracted based on a plurality of communication characteristics such as (step S10).

[0666] Next, the estimation means 17 trains an estimation algorithm ALG that estimates the results of the network scan and selects training data (step S11). In this case, the estimation means 17 estimates the results of the network scan using standard sample data D_similarity and trains the estimation algorithm ALG by evaluating the accuracy of the estimated results. Furthermore, the estimation means 17 selects sample data (sample data D_sample_optimum) that results in the highest accuracy of the estimation results as training data.

[0667] Then, the estimation means 17 estimates the results of a network scan that has been performed in a communication environment in which the results of the network scan are unknown, using the trained estimation algorithm ALG (step S12).

[0668] Thereafter, the update means 14 updates the correspondence table TBL1 (or the correspondence table TBL2) with the scan response delay {RTT m_unknown}(m=1 to M) and calculate the scan response delay {RTT m’ _known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3 (m'=1' to M'). Then, the update means 14 and the calculation means 16 read out the scan response delay {RTT m _unknown}(m=1~M),{RTT m’ _known}_1(m'=1'~M'),{RTT m’ _known}_2(m'=1'~M'),{RTT m’ _known}_3 (m'=1'~M'), respectively, the baseline scan response delay {b RTT _ m _unknown}(m=1~M),{b RTT _ m’ _known}_1(m'=1'~M'),{b RTT _ m’ _known}_2(m'=1'~M'),{b RTT _ m’ _known}_3 (m'=1' to M') is generated (step S13).

[0669] The updating means 14 then updates the baseline scan response delay {b RTT _ m _unknown}(m=1~M),{b RTT _ m’ _known}_1(m'=1'~M'),{b RTT _ m’ _known}_2(m'=1'~M'),{b RTT _ m’ _known}_3 (m'=1' to M'), the scan response delays are aggregated and the known scan response database (=correspondence table TBL3) is updated (step S14).

[0670] Then, in step S11, the scan scheduler 18 receives from the estimation means 17 the estimated results of the network scan that will be most accurate, creates a rescan list based on the received estimated results of the network scan (step S15), and outputs the created rescan list to the network scanner 11.

[0671] The reason for creating a rescan list is that if the result of a network scan is "insufficient communication quality" or "network congestion," the result of the network scan may become "normal" if the network scan is performed at a different scan timing.

[0672] After step S15, the network scanner 11 determines whether the rescan list is empty (step S16).

[0673] If it is determined in step S16 that the rescan list is not empty, the series of operations proceeds to step S2, and steps S2 to S16 are repeatedly executed until it is determined in step S16 that the rescan list is empty.

[0674] Then, if it is determined in step S16 that the rescan list is empty, the network scan operation ends.

[0675] 67 and 68 are first and second flowcharts, respectively, for explaining the detailed operation of step S10 in FIG.

[0676] 67, when it is determined in step S8 of FIG. 66 that m=M, the extraction means 15 extracts the correlation coefficient r ij is calculated (step S101), where i is an argument in the column direction in FIGS. 41 to 44, and j is an argument in the row direction in FIGS.

[0677] Then, the extraction means 15 sets j=1 (step S102) and sets i=1 (step S103). After that, the extraction means 15 extracts the communication characteristic f i ,f j The correlation coefficient r ij (step S104), and the extracted correlation coefficient r ij is the threshold Corr r It is determined whether it is greater than (step S105).

[0678] In step S105, the correlation coefficient r ij is the threshold Corr r When it is determined that the communication characteristic f i ,f j (Step S106).

[0679] On the other hand, in step S105, the correlation coefficient r ij is the threshold Corr r When it is determined that the communication characteristic f i ,f j is saved (step S107).

[0680] Then, after step S106 or step S107, the extraction means 15 determines whether i=I (I is the maximum number of communication characteristics in the column direction in FIGS. 41 to 44) (step S108).

[0681] If it is determined in step S108 that i is not equal to I, the extraction means 15 sets i to i+1 (step S109). After that, the series of operations proceeds to step S104, and steps S104 to S109 are repeatedly executed until it is determined in step S108 that i is equal to I.

[0682] Then, when it is determined in step S108 that i=I, the extraction means 15 determines whether j=J (J is the maximum number of communication characteristics in the row direction in FIGS. 41 to 44) (step S110).

[0683] If it is determined in step S110 that j is not equal to J, the extraction means 15 sets j to j+1 (step S111). After that, the series of operations proceeds to step S103, and steps S103 to S111 are repeatedly executed until it is determined in step S110 that j is equal to J.

[0684] Then, in step S110, if it is determined that j=J, the extraction means 15 extracts the scan response delay RTT based on all the stored communication characteristics by the above-described method, using all the stored communication characteristics as explanatory variables. m A multiple linear regression is calculated using the objective variables (step S112), and the contribution rate R 2 Get.

[0685] Thereafter, the extraction means 15 extracts the contribution rate R 2 is the threshold S r It is determined whether the value is smaller than (step S113).

[0686] In step S113, the contribution rate R 2 is the threshold S r If it is determined that the difference is smaller than , the extraction means 15 sets the mode of the estimation algorithm ALG to "uncorrelated" (step S114).

[0687] On the other hand, in step S113, the contribution rate R 2 is the threshold S r If it is determined that this is the case, the extraction means 15 sets the mode of the estimation algorithm ALG to "correlation" (step S115).

[0688] After step S114 or step S115, the extraction means 15 stores the waiting time and communication characteristics in the network scan and the mode of the estimation algorithm ALG (step S116).

[0689] Referring to FIG. 68, after step S116 in FIG. 67, the extraction means 15 calculates the threshold value Corr rA correlation coefficient greater than r ij Then, the computer calculates a plurality of scores Q of a plurality of communication characteristics having the above-mentioned characteristic, and selects K communication characteristics from the plurality of communication characteristics in descending order of score Q (step S117).

[0690] Then, the extraction means 15 uses the training data to learn a classification algorithm (for example, SVM-train) (step S118).

[0691] Thereafter, the extraction means 15 sets k=K (step S119). Subsequently, the extraction means 15 extracts (k×M) sample vectors x of the communication characteristics using the trained classification algorithm. 11 ~x kM are classified into two groups GP1 and GP2 (step S120).

[0692] Then, the extraction means 15 calculates k weights w1 to w2 of the k communication characteristics by using equations (16), (18), and (19). k is calculated (step S121).

[0693] Then, the extraction means 15 extracts k weights w1 to w k Based on this, k rank weights R1 to R2 are calculated by Equation (20). k is calculated (step S122).

[0694] Then, the extraction means 15 extracts k rank weights R1 to R k Based on the smallest rank weight R k_min The extraction means 15 then determines the sample of the communication characteristic to which the smallest rank weight R is assigned as the communication characteristic to be deleted (step S123). k_min The sample of the communication characteristics with 11 ~x kM (step S124).

[0695] Then, the extraction means 15 determines whether the subset of communication characteristics is empty or not (step S125).

[0696] When it is determined in step S125 that the subset of communication characteristics is not empty, the extracting means 15 sets k=k−1 (step S126). That is, the extracting means 15 decreases the number of samples in the subset of communication characteristics by 1. Thereafter, the series of operations proceeds to step S120, and steps S120 to S126 are repeatedly executed until it is determined in step S125 that the subset of communication characteristics is empty.

[0697] Then, in step S125, if it is determined that the subset of communication characteristics is empty, the extraction means 15 extracts the last deleted communication characteristic as the communication characteristic CMbest. m (Step S127) Then, the series of operations proceeds to Step S11 in FIG.

[0698] After the first execution of steps S120 to S126, steps S120 to S126 are repeatedly executed until the subset of communication characteristics is empty, thereby sequentially deleting samples of communication characteristics that are least important for estimating the results of the network scan. At this time, after deleting one communication characteristic sample, the remaining samples are reclassified into two groups GP1 and GP2 using the trained classification algorithm (see step S120). As described above, group GP1 is a group into which communication characteristics that are important for estimating the results of the network scan are classified, and group GP2 is a group into which communication characteristics that are not important for e...

Claims

1. a scanning means for executing a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquiring a first scan response delay of the network scan for each of the M terminal devices; a calculation means for executing a first calculation process, when the M first scan response delays are acquired by the scanning means, for calculating the M' baseline scan response delays for each of a plurality of sets of M' (M' is an integer of 2 or more) second scan response delays obtained when results of a network scan executed in a second communication environment in which results of a network scan are known are a plurality of known estimated results, and for calculating the M baseline scan response delays for the M first scan response delays; and a second calculation process, based on the plurality of sets of the M' baseline scan response delays and the M baseline scan response delays calculated in the first calculation process, for calculating similarity rates between the M' baseline scan response delays and the M baseline scan response delays for all of the plurality of sets of M' baseline scan response delays to calculate a plurality of similarity rates. a detection means for detecting a largest similarity rate among the plurality of similarity rates calculated in the second calculation process when the second calculation process is executed by the calculation means; a determination means for determining, when the highest similarity rate is detected by the detection means, that an unknown estimation result of a network scan when the M first scan response delays are acquired is the known estimation result corresponding to the M′ second scan response delays having the highest similarity rate; The baseline scan response delay is a scan response delay whose distribution is changed based on an "upper limit" and a "lower limit" so as to more easily obtain a similarity or difference in distribution between a scan response delay obtained when a network scan is performed in a communication environment where the results of the network scan are unknown and a scan response delay obtained when a network scan is performed in a communication environment where the results of the network scan are known.

2. 2. The network scanning device according to claim 1, further comprising an updating means for performing an aggregation process that aggregates the M' second scan response delays and the M first scan response delays when the largest similarity rate is obtained, and updating the M' second scan response delays when the largest similarity rate is obtained by the M' second aggregated scan response delays aggregated in the aggregation process.

3. 3. The network scanning device of claim 2, wherein, when there are multiple sets of the M first scan response delays when the highest similarity rate is obtained, the updating means selects, in the aggregation process, any set of M first scan response delays from the multiple sets of M first scan response delays, and aggregates the selected set of M first scan response delays with M' second scan response delays when the highest similarity rate is obtained.

4. 4. The network scanning device according to claim 2, wherein, in the aggregation process, the update means adds, among the M first scan response delays, the first scan response delays that are not included in the M' second scan response delays when the highest similarity rate is obtained to the M' second scan response delays when the highest similarity rate is obtained, thereby aggregating the M' second scan response delays when the highest similarity rate is obtained and the M first scan response delays.

5. the calculation means, in the first calculation process, executes a process of calculating, as the baseline scan response delay, a subtraction result obtained by subtracting a predetermined value from an average of a plurality of scan response delays and an addition result obtained by adding the predetermined value to the average, for all of the plurality of scan response delays to calculate a plurality of baseline scan response delays, the calculation means executes a process of calculating, for all of the M first scan response delays and the plurality of sets of M' second scan response delays, to calculate the plurality of sets of M' baseline scan response delays and the M baseline scan response delays; 5. The network scanning device according to claim 1, wherein the predetermined value is calculated based on relative variations in the plurality of scan response delays.

6. 6. The network scanning device according to claim 5, wherein in the first calculation process, the calculation means calculates the predetermined value by subtracting an inverse of a coefficient of variation in a distribution of the plurality of scan response delays from the scan response delay, multiplying the subtraction result by a standard deviation of the plurality of scan response delays, and dividing the multiplication result by a number of the plurality of scan response delays.

7. In the second calculation process, the calculation means calculates a first mean absolute deviation which is the mean absolute deviation of the M baseline scan response delays, and calculates a second mean absolute deviation which is the mean absolute deviation of the M′ baseline scan response delays for all of the multiple sets of M′ baseline scan response delays to calculate a plurality of second mean absolute deviations, and where the first mean absolute deviation is X and any one of the plurality of second mean absolute deviations is Y, the first mean absolute deviation is 1 / (1+(Y−X) ２ 7. The network scanning device according to claim 1, further comprising: a step of calculating the similarity rate by an equation of XY (where X is the number of similarities between the first and second mean absolute deviations) for all of the second mean absolute deviations to calculate the similarity rates.

8. a generating means for acquiring, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating the known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays, and generating similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second aggregate scan response delays; an algorithm updating means for training an estimation algorithm for estimating the result of the network scan executed in the first communication environment based on the similar standard sample data, and updating the estimation algorithm using an evaluation index so as to approach an optimal algorithm; an estimation means for estimating a result of a network scan executed in the first communication environment by using the estimation algorithm updated by the algorithm update means, the optimization algorithm comprises an algorithm for estimating a result of a network scan such that the estimated result of the network scan matches the known estimated result; 8. The network scanning device according to claim 1, wherein the evaluation index is an index for evaluating accuracy of an estimation result when a result of a network scan is estimated by the trained algorithm.

9. A first step in which a scanning means executes a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquires a first scan response delay of the network scan for each of the M terminal devices; a second step in which a calculation means, when the M first scan response delays are acquired by the scanning means, executes a first calculation process in which the calculation means calculates the M' baseline scan response delays for each of a plurality of sets of M' (M' is an integer of 2 or more) second scan response delays obtained when results of a network scan executed in a second communication environment in which results of a network scan are known are a plurality of known estimated results, and calculates the M baseline scan response delays for the M first scan response delays; and a second calculation process in which the calculation means calculates similarity rates between the M' baseline scan response delays and the M baseline scan response delays for all of the plurality of sets of M' baseline scan response delays based on the plurality of sets of the M' baseline scan response delays and the M baseline scan response delays calculated in the first calculation process, to calculate a plurality of similarity rates. a third step of detecting a largest similarity rate among the plurality of similarity rates calculated in the second calculation process by the detection means when the second calculation process is executed by the calculation means; a fourth step in which a determination means determines, when the highest similarity rate is detected in the third step, that an unknown estimation result of a network scan when the M first scan response delays are acquired is the known estimation result corresponding to the M′ second scan response delays having the highest similarity rate; A program for causing a computer to execute the program, wherein the baseline scan response delay is a scan response delay whose distribution has been changed based on an "upper limit" and a "lower limit" so as to more easily produce a similar or different distribution between a scan response delay obtained when a network scan is performed in a communication environment where the results of the network scan are unknown and a scan response delay obtained when a network scan is performed in a communication environment where the results of the network scan are known.

10. 10. The program for causing a computer to execute the program of claim 9, further comprising an update means for causing the computer to execute a fifth step of: when the highest similarity rate is detected in the third step, executing an aggregation process of aggregating the M first scan response delays and the M' second scan response delays when the highest similarity rate is obtained, and updating the M' second scan response delays when the highest similarity rate is obtained by the M' second aggregated scan response delays aggregated in the aggregation process.

11. 11. The program to be executed by a computer according to claim 10, wherein, when there are multiple sets of the M first scan response delays when the highest similarity rate is obtained, the updating means selects, in the aggregation process, any set of M first scan response delays from the multiple sets of M first scan response delays, and aggregates the selected set of M first scan response delays with M' second scan response delays when the highest similarity rate is obtained.

12. 12. The program for causing a computer to execute the program according to claim 10 or 11, wherein, in the aggregation process, the updating means adds the first scan response delays among the M first scan response delays that are not included in the M' second scan response delays when the highest similarity rate is obtained to the M' second scan response delays when the highest similarity rate is obtained, thereby aggregating the M' second scan response delays when the highest similarity rate is obtained and the M first scan response delays.

13. the calculation means, in the first calculation process, executes a process of calculating a baseline scan response delay by subtracting a predetermined value from an average of a plurality of scan response delays and an addition result by adding the predetermined value to the average for all of the plurality of scan response delays to calculate a plurality of baseline scan response delays, the calculation means ... plurality of baseline scan response delays by subtracting a predetermined value from an average of a plurality of scan response delays and an addition result by adding the predetermined value to the average for all of the plurality of scan response delays to calculate the plurality of sets of M' second scan response delays, 13. The program executed by a computer according to claim 9, wherein the predetermined value is calculated based on relative variations of the plurality of scan response delays.

14. 14. The program for causing a computer to execute the program according to claim 13, wherein in the first calculation process, the calculation means calculates the predetermined value by multiplying a subtraction result obtained by subtracting an inverse of a coefficient of variation in a distribution of the plurality of scan response delays from the scan response delay by a standard deviation of the plurality of scan response delays, and dividing the multiplication result by the number of the plurality of scan response delays.

15. In the second calculation process, the calculation means calculates a first mean absolute deviation which is the mean absolute deviation of the M baseline scan response delays, and calculates a second mean absolute deviation which is the mean absolute deviation of the M′ baseline scan response delays for all of the multiple sets of M′ baseline scan response delays to calculate a plurality of second mean absolute deviations, and where the first mean absolute deviation is X and any one of the plurality of second mean absolute deviations is Y, the first mean absolute deviation is 1 / (1+(Y−X) ２ 15. The program for causing a computer to execute the program according to claim 9, wherein the program calculates the similarity rate by the formula: X=XY / XY for all of the plurality of second mean absolute deviations to calculate the plurality of similarity rates.

16. a sixth step in which a generating means acquires, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating the known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second aggregate scan response delays; a seventh step in which an algorithm update means trains an estimation algorithm for estimating a result of a network scan executed in the first communication environment based on the similar standard sample data, and updates the estimation algorithm using an evaluation index so as to approach an optimal algorithm; and an eighth step in which an estimation means estimates a result of a network scan performed in the first communication environment using the estimation algorithm updated by the algorithm update means, the optimization algorithm comprises an algorithm for estimating a result of a network scan such that the estimated result of the network scan matches the known estimated result; 16. The program for causing a computer to execute the program according to claim 9, wherein the evaluation index is an index for evaluating the accuracy of an estimation result when a result of a network scan is estimated by the trained algorithm.

17. A computer-readable recording medium having the program according to any one of claims 9 to 16 recorded thereon.

18. 1. A data production method for producing data used to estimate results of a network scan, comprising: A first step in which a scanning means executes a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquires a first scan response delay of the network scan for each of the M terminal devices; a second step in which a calculation means executes a first calculation process, when the M first scan response delays are acquired in the first step, to calculate the M' baseline scan response delays for each of a plurality of sets of M' (M' is an integer of 2 or more) second scan response delays obtained when results of a network scan executed in a second communication environment in which results of a network scan are known are a plurality of known estimated results, and to calculate the M baseline scan response delays for the M first scan response delays; and a second calculation process, based on the plurality of sets of the M' baseline scan response delays and the M baseline scan response delays calculated in the first calculation process, to calculate similarity rates between the M' baseline scan response delays and the M baseline scan response delays for all of the plurality of sets of M' baseline scan response delays to calculate a plurality of similarity rates; a third step of detecting a largest similarity rate among the plurality of similarity rates calculated in the second calculation process by a detection means when the second calculation process is executed in the second step; a fourth step in which an updating means executes an aggregation process of aggregating the M' second scan response delays and the M first scan response delays when the highest similarity rate is obtained, when the highest similarity rate is detected in the third step, and updates the M' second scan response delays when the highest similarity rate is obtained by the M' second aggregated scan response delays aggregated in the aggregation process; a fifth step in which a generating means acquires, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating the known estimation result among the multiple communication characteristics acquired together with one of the M' second scan response delays updated in the fourth step, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays; A data production method in which the baseline scan response delay is a scan response delay whose distribution is changed based on an "upper limit" and a "lower limit" so as to more easily produce similar or different distributions between a scan response delay obtained when a network scan is performed in a communication environment where the results of the network scan are unknown and a scan response delay obtained when a network scan is performed in a communication environment where the results of the network scan are known.

19. The data production method described in claim 18, wherein the updating means, in the aggregation process of the fourth step, adds the first scan response delays among the M first scan response delays that are not included in the M' second scan response delays when the highest similarity rate is obtained to the M' second scan response delays when the highest similarity rate is obtained, thereby aggregating the M' second scan response delays when the highest similarity rate is obtained and the M first scan response delays.

20. The calculation means, in the first calculation process of the second step, executes a process of calculating a baseline scan response delay by subtracting a predetermined value from an average of a plurality of scan response delays and an addition result by adding the predetermined value to the average for all of the plurality of scan response delays to calculate a plurality of baseline scan response delays, the process executing the process of calculating a baseline scan response delay by subtracting a predetermined value from an average of a plurality of scan response delays and an addition result by adding the predetermined value to the average for all of the plurality of scan response delays to calculate a plurality of baseline scan response delays, 20. The data manufacturing method according to claim 18, wherein the predetermined value is calculated based on relative variations of the plurality of scan response delays.

21. The data manufacturing method described in Claim 20, wherein the calculation means, in the first calculation process of the second step, multiplies the subtraction result obtained by subtracting the inverse of the coefficient of variation of the multiple scan response delays from the scan response delay by the standard deviation of the multiple scan response delays, and divides the multiplication result by the number of the multiple scan response delays to calculate the specified value.

22. In the second calculation process of the second step, the calculation means calculates a first mean absolute deviation which is a mean absolute deviation of the M baseline scan response delays, and calculates a second mean absolute deviation which is a mean absolute deviation of the M' baseline scan response delays for all of the multiple sets of M' baseline scan response delays to calculate a plurality of second mean absolute deviations, wherein when the first mean absolute deviation is X and any one of the plurality of second mean absolute deviations is Y, 1 / (1+(Y-X) ２ 22. The data production method according to claim 18, further comprising the step of calculating the similarity rate by the formula: XY / X for all of the plurality of second mean absolute deviations to calculate the plurality of similarity rates.

Images