Web application program vulnerability detection method and device and storage medium

Abstract

The invention relates to the technical field of network security, in particular to a Web application program vulnerability detection method and device and a storage medium, and the method comprises the steps: obtaining Web application program information, and determining a search state according to a link number average value and a maximum link difference value; determining a search mode as reference threshold analysis or traversal page analysis according to the search state; determining the page type of the to-be-detected page according to the page updating frequency and the sub-page frequency reference value; determining a clustering method according to the page type, namely performing URL clustering according to the page difference degree or performing primary clustering according to the image similarity and performing secondary clustering according to the comprehensive similarity to obtain a plurality of similar page combinations; one URL is correspondingly extracted for each similar page combination for vulnerability detection, and after an attack vector is input, whether a vulnerability exists or not is judged according to the execution condition of the Web application program. According to the method, similar pages can be clustered, and the vulnerability detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a Web application vulnerability detection method, device and storage medium. Background Art

[0002] In the existing web application vulnerability detection, when crawling pages, there is too much page crawling, resulting in poor vulnerability detection efficiency, and when clustering similar pages, the clustering method is single, resulting in poor vulnerability detection effect. Therefore, how to select pages and cluster similar pages to improve vulnerability detection efficiency is a technical problem that needs to be solved urgently by those skilled in the art.

[0003] Chinese patent publication number CN112507341A discloses a vulnerability scanning method, device, equipment and storage medium based on a web crawler, including: obtaining the corresponding web pages to be crawled and the valid URLs of the web pages to be crawled according to a preset list to be crawled; sorting and filtering the valid URLs, and updating the current crawler depth; converting the valid URLs containing preset keywords into standard URLs; checking the standard URLs for duplicates in a preset crawling set, and updating the crawling set according to the results of the duplicate checking; when the list to be crawled is empty or the crawler depth exceeds the preset depth limit, terminating the crawler scan. It can be seen that the above technical solution has the following problems: only based on the list to be crawled to obtain URLs, it is impossible to change the URL acquisition method according to the actual scenario, the URL selection method is single, and similar pages are not clustered, and the URLs of similar pages are repeatedly detected, resulting in poor vulnerability detection efficiency. Summary of the invention

[0004] To this end, the present invention provides a Web application vulnerability detection method to overcome the problem that the prior art only obtains URLs based on a list to be crawled, cannot change the URL acquisition method according to the actual scenario, has a single URL selection method, and does not cluster similar pages. The URLs of similar pages are repeatedly detected, resulting in poor vulnerability detection efficiency.

[0005] To achieve the above object, the present invention provides a Web application vulnerability detection method, comprising:

[0006] Obtain Web application information and determine the search status based on the average number of links and the maximum link difference;

[0007] Determine the search mode as reference threshold analysis or traversal page analysis according to the search state;

[0008] Determine the page type of the page to be detected according to the page update frequency and the sub-page frequency reference value;

[0009] The clustering method is determined according to the page type, which is to cluster the URLs according to the page difference or to cluster them once according to the image similarity and then cluster them again according to the comprehensive similarity to obtain several similar page combinations;

[0010] For each similar page combination, a URL is extracted for vulnerability detection. After the attack vector is input, it is determined whether there is a vulnerability based on the execution of the Web application.

[0011] Furthermore, when the search state is that the average value of the number of links is greater than or equal to the preset average value of the number of links or the maximum link difference is greater than or equal to the preset maximum link difference, the search method is reference threshold analysis, and the reference threshold analysis includes:

[0012] A reference threshold analysis is performed on each initial sub-page corresponding to the initial page. When a reference threshold analysis is performed on a single initial sub-page, the initial sub-page is recorded as a target sub-page, and a reference threshold corresponding to the target sub-page is detected. If the reference threshold corresponding to the target sub-page is greater than a preset reference threshold, the target initial sub-page is recorded as a sub-page to be detected, and each sub-page to be detected and the initial page are recorded as a page to be detected;

[0013] The reference threshold is determined according to the sum of the page threshold corresponding to the target subpage and the page threshold corresponding to each associated subpage.

[0014] Furthermore, when the search state is that the average value of the number of links is less than the preset average value of the number of links and the maximum link difference is less than the preset maximum link difference, the search method is traversal page analysis, and the traversal page analysis includes:

[0015] The initial page and each initial sub-page corresponding to the initial page are recorded as pages to be detected.

[0016] Furthermore, the page type of the page to be detected is determined according to the page update frequency and the sub-page frequency reference value, and the page types include:

[0017] A type of page whose page update frequency is less than a preset page update frequency and whose sub-page frequency reference value is less than a preset sub-page frequency reference value;

[0018] The second type of page has a page update frequency greater than or equal to a preset page update frequency or a sub-page frequency reference value greater than or equal to a preset sub-page frequency reference value.

[0019] Further, a clustering method is determined according to the page type;

[0020] For a type of page, URL clustering is performed based on page differences;

[0021] For the second category pages, a clustering is performed based on image similarity, and a secondary clustering is performed based on comprehensive similarity.

[0022] Furthermore, URL clustering is performed for each category of pages according to the page differences, including:

[0023] Similar page detection is performed on a type of page corresponding to each URL. When similar page detection is performed on a type of page corresponding to a single URL, the type of page corresponding to the URL is recorded as the target page, and the type of pages corresponding to other URLs are recorded as reference pages. The page difference between the target page and each reference page is detected, and the set of all reference pages and the target page whose page difference with the target page is less than a preset page difference is recorded as a similar page combination. Similar page detection is continued for reference pages that are not recorded as similar page combinations until all type of pages corresponding to all URLs are recorded as similar page combinations.

[0024] Further, clustering is performed on each of the two types of pages according to the image difference to obtain a number of primary clustering combinations;

[0025] The image difference is determined based on the feature area difference value and the site mean difference value;

[0026] The relationship between the characteristic region difference value and the image difference is a positive correlation, and the relationship between the site mean difference value and the image difference is a positive correlation.

[0027] Furthermore, secondary clustering is performed on each primary clustering combination according to the comprehensive similarity to obtain several similar page combinations:

[0028] The comprehensive similarity is determined based on the apparent similarity and the structural difference;

[0029] The relationship between the apparent similarity and the comprehensive similarity is a positive correlation, and the relationship between the structural difference and the comprehensive similarity is a negative correlation.

[0030] The present invention also provides a Web application vulnerability detection device, comprising:

[0031] A state analysis module is used to obtain Web application information, determine the search state according to the average value of the number of links and the maximum link difference, and determine the search method as reference threshold analysis or traversal page analysis according to the search state;

[0032] A page clustering module, which is connected to the state analysis module, is used to determine the page type of the page to be detected according to the page update frequency and the sub-page frequency reference value, and to determine the clustering method according to the page type, which is to perform URL clustering according to page difference or to perform a primary clustering according to image similarity and a secondary clustering according to comprehensive similarity, to obtain a plurality of similar page combinations;

[0033] The vulnerability detection module is connected to the page clustering module and is used to extract a URL for vulnerability detection for each similar page combination. After inputting the attack vector, it determines whether there is a vulnerability based on the execution status of the Web application.

[0034] The present invention also provides a storage medium, in which a computer program is stored. When the computer program is run on a computer, the Web application vulnerability detection method can be executed.

[0035] Compared with the prior art, the beneficial effect of the present invention lies in that, in the technical scheme of the present invention, the search state is determined according to the average value of the number of links and the maximum link difference, the number of URLs to be searched is effectively predicted by the average value of the number of links and the maximum link difference, and then different search methods are adaptively selected according to the search state, so that the selection of the search method is more in line with the actual application scenario, avoiding the problem in the prior art that URLs are only obtained based on the list to be crawled and the URL acquisition method cannot be changed according to the actual scenario, and also avoiding the problem of poor search effect caused by too many page crawls when crawling pages, thereby improving the efficiency of searching URLs and thus improving the efficiency of vulnerability detection.

[0036] Furthermore, the present invention determines the page type of the page to be detected according to the page update frequency and the sub-page frequency reference value, and effectively reflects the update status of the page to be detected through the page update frequency and the sub-page frequency reference value, and then selects different clustering methods adaptively according to the page type, so that the selection of clustering method is more in line with the actual application scenario, avoiding the problem in the prior art that similar pages cannot be clustered, resulting in the inability to discover more vulnerabilities when vulnerability detection is performed on similar pages, and can perform a single detection on the URL of similar pages, which is beneficial to improving the efficiency of vulnerability detection, and also avoids the problem of inaccurate vulnerability detection due to a single clustering method, thereby improving the accuracy of vulnerability detection.

[0037] Furthermore, the present invention performs reference threshold analysis, which can filter URLs with fewer new pages when the number of URLs is large, thereby avoiding the problem of poor page crawling rate, improving the efficiency of searching URLs, and thereby improving the efficiency of vulnerability detection. It performs traversal page analysis, which can ensure the page crawling rate when the number of URLs is small, avoiding the problem of inaccurate vulnerability detection, and thereby improving the accuracy of vulnerability detection.

[0038] Furthermore, the present invention effectively reflects the differences in a category of pages through page differences, and then performs URL clustering according to the page differences, thereby avoiding the problem in the prior art that similar pages cannot be clustered, resulting in the inability to discover more vulnerabilities when performing vulnerability detection on similar pages, and also avoiding the problem of poor accuracy of the similarities of each page, and then performs URL clustering according to the page differences, thereby improving the efficiency of vulnerability detection.

[0039] Furthermore, the present invention performs a clustering according to image similarity, which can preliminarily classify pages that may have potential connections, thereby improving the similarity of each second category of pages in the similar page combination, avoiding the problem of poor clustering accuracy. A secondary clustering is performed according to the comprehensive similarity, which can more accurately classify pages, avoiding the problem in the prior art that similar pages cannot be clustered, resulting in the inability to discover more vulnerabilities when vulnerability detection is performed on similar pages. While clustering, the accuracy of clustering can also be improved, thereby improving the efficiency of vulnerability detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a Web application vulnerability detection method of the present invention;

[0041] Figure 2 A flow chart of determining a search mode according to a search state in the present invention;

[0042] Figure 3 A flow chart of a method for determining clustering of page types according to the present invention;

[0043] Figure 4 Schematic diagram of a Web application vulnerability detection device of the present invention. DETAILED DESCRIPTION

[0044] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0046] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0047] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] See also Figures 1 to 3 As shown, the present invention provides a Web application vulnerability detection method, comprising:

[0049] Obtain web application information and determine the search status based on the average number of links and the maximum link difference;

[0050] Determine the search mode as reference threshold analysis or traversal page analysis according to the search state;

[0051] Determine the page type of the page to be detected according to the page update frequency and the sub-page frequency reference value;

[0052] The clustering method is determined according to the page type, which is to cluster the URLs according to the page difference or to cluster them once according to the image similarity and then cluster them again according to the comprehensive similarity to obtain several similar page combinations;

[0053] For each similar page combination, a URL is extracted for vulnerability detection. After the attack vector is input, it is determined whether there is a vulnerability based on the execution of the Web application.

[0054] The Web application information in the present invention includes but is not limited to the initial page, URL and page links. URL is a uniform resource locator. Each page corresponds to a URL. A single page contains several links. Each link corresponds to a subpage. The initial page is the first page visited by the web crawler when performing data crawling operations. After performing a preset number of crawling operations on the URL of the initial page, the pages corresponding to each URL obtained are recorded as screening pages. This is content that is easy for technical personnel in this field to understand and will not be elaborated on in detail.

[0055] The value of the preset number of times can be determined by the user according to the actual application scenario. The higher the user's demand for improving the accuracy of URL quantity prediction, the larger the value of the preset number of times will be. A value of the preset number of times is provided, and the predicted number of times is 5.

[0056] The search status includes: the average value of the number of links is greater than or equal to the preset average value of the number of links or the maximum link difference is greater than or equal to the preset maximum link difference, and the average value of the number of links is less than the preset average value of the number of links and the maximum link difference is less than the preset maximum link difference;

[0057] The average number of links is the average number of links corresponding to each filtering page. The method for confirming the maximum link difference is to detect the number of links corresponding to each filtering page, and record the difference between the maximum number of links and the minimum number of links as the maximum link difference; the number of links is the number of links contained in a single filtering page.

[0058] The values ​​of the preset average number of links and the preset maximum link difference can be determined by the user according to the actual application scenario. The greater the user's demand for improving the efficiency of searching URLs, the smaller the values ​​of the preset average number of links and the preset maximum link difference. A value of the preset average number of links and the preset maximum link difference is provided, and each historical vulnerability detection record is detected. The average of the average numbers of links corresponding to the records that can meet the user's demand for searching URL efficiency is recorded as the preset average number of links, and the average of the maximum link differences corresponding to the records that can meet the user's demand for searching URL efficiency is recorded as the preset maximum link difference.

[0059] Specifically, when the search status is that the average value of the number of links is greater than or equal to the preset average value of the number of links or the maximum link difference is greater than or equal to the preset maximum link difference, the search method is reference threshold analysis, and the reference threshold analysis includes:

[0060] A reference threshold analysis is performed on each initial sub-page corresponding to the initial page. When a reference threshold analysis is performed on a single initial sub-page, the initial sub-page is recorded as a target sub-page, and a reference threshold corresponding to the target sub-page is detected. If the reference threshold corresponding to the target sub-page is greater than a preset reference threshold, the target initial sub-page is recorded as a sub-page to be detected, and each sub-page to be detected and the initial page are recorded as a page to be detected;

[0061] The reference threshold is determined according to the sum of the page threshold corresponding to the target subpage and the page threshold corresponding to each associated subpage.

[0062] Among them, a tree structure diagram is established based on the initial page, node analysis is performed on the initial page, all sub-pages corresponding to the initial page are extracted, several new sub-nodes are created for each sub-page, each sub-node is connected to the root node, and node analysis is performed on each sub-page, and each sub-node is connected to the corresponding sub-node until the page threshold of the sub-page is 0, and the node analysis is stopped. Finally, a tree structure diagram with the initial page as the root node and each sub-page as a sub-node is established. This is content that is easy for people in this field to understand, and it will not be described in detail. The sub-page corresponding to each sub-node in the tree structure diagram is recorded as the initial sub-page.

[0063] The value of the preset reference threshold can be determined by the user according to the actual application scenario. The greater the user's demand for the value of crawling sub-pages, the greater the value of the preset reference threshold. A value of the preset reference threshold is provided, and the preset reference threshold is 5.

[0064] The confirmation method of associated sub-pages is that, for a single initial sub-page, the initial sub-page is recorded as the target sub-page, and the associated sub-pages of the target sub-page include the initial sub-page corresponding to the node adjacent to the left of the node corresponding to the target sub-page in the tree structure diagram and the initial sub-page corresponding to the node adjacent to the right of the node corresponding to the target sub-page in the tree structure diagram. It can be understood that if the target sub-page is located at the leftmost position or the rightmost position of the tree structure diagram, there is only one associated sub-page.

[0065] The page threshold is the number of links a single subpage contains.

[0066] Specifically, when the search status is that the average value of the number of links is less than the preset average value of the number of links and the maximum link difference is less than the preset maximum link difference, the search method is traversal page analysis, and the traversal page analysis includes:

[0067] The initial page and each initial sub-page corresponding to the initial page are recorded as pages to be detected.

[0068] Specifically, the page type of the page to be detected is determined according to the page update frequency and the sub-page frequency reference value, and the page types include:

[0069] A type of page whose page update frequency is less than a preset page update frequency and whose sub-page frequency reference value is less than a preset sub-page frequency reference value;

[0070] The second type of page has a page update frequency greater than or equal to a preset page update frequency or a sub-page frequency reference value greater than or equal to a preset sub-page frequency reference value.

[0071] The page update frequency is the shortest time for updating the content of a single page to be detected, and the subpage frequency reference value is the average value of the shortest time for updating the content of each subpage corresponding to the single page to be detected.

[0072] The values ​​of the preset page update frequency and the preset sub-page frequency reference value can be determined by the user according to the actual application scenario. The larger the values ​​of the preset page update frequency and the preset sub-page frequency reference value are, the smaller the degree of change of the page content is. The values ​​of the preset page update frequency and the preset sub-page frequency reference value are provided. The preset page update frequency is 2h. The method for confirming the preset sub-page frequency reference value is to detect the records with the same update frequency as the current page to be detected during the historical clustering process, and record the average value of the sub-page frequency reference values ​​corresponding to the records that can meet the user's vulnerability detection needs as the preset sub-page frequency reference value.

[0073] Specifically, the clustering method is determined based on the page type;

[0074] For a type of page, URL clustering is performed based on page differences;

[0075] For the second category pages, a clustering is performed based on image similarity, and a secondary clustering is performed based on comprehensive similarity.

[0076] Specifically, URL clustering is performed for each category of pages based on page differences, including:

[0077] Similar page detection is performed on a type of page corresponding to each URL. When similar page detection is performed on a type of page corresponding to a single URL, the type of page corresponding to the URL is recorded as the target page, and the type of pages corresponding to other URLs are recorded as reference pages. The page difference between the target page and each reference page is detected, and the set of all reference pages and the target page whose page difference with the target page is less than a preset page difference is recorded as a similar page combination. Similar page detection is continued for reference pages that are not recorded as similar page combinations until all type of pages corresponding to all URLs are recorded as similar page combinations.

[0078] Among them, for two Class A pages, the method for confirming the page difference is to use a hash algorithm to convert the text content corresponding to the Class A page into a 64-bit binary string. This is easy for technical personnel in this field to understand and will not be described in detail. The two binary strings are compared bit by bit, and the number of different bit bits is recorded as the page difference.

[0079] The value of the preset page difference can be determined by the user according to the actual application scenario. It can be understood that the smaller the value of the preset page difference, the greater the similarity of each type of page in the similar page combination, and the greater the accuracy of the user's vulnerability detection. A value of the preset page difference is provided, and the records of URL clustering during the historical vulnerability detection process are detected. The average value of the page difference corresponding to the records that can meet the user's vulnerability detection needs is recorded as the preset page difference.

[0080] Specifically, each of the two types of pages is clustered according to the image difference to obtain a number of primary clustering combinations;

[0081] The image difference is determined based on the feature area difference value and the site mean difference value;

[0082] The relationship between the characteristic region difference value and the image difference is a positive correlation, and the relationship between the site mean difference value and the image difference is a positive correlation.

[0083] Among them, a clustering is performed for each of the two types of pages according to the image difference to obtain several primary clustering combinations, including:

[0084] A cluster detection is performed on each second-category page. When a cluster detection is performed on a single second-category page, the second-category page is recorded as a target second-category page, and other second-category pages excluding the target second-category page are recorded as reference second-category pages. The image difference between the target second-category page and each reference second-category page is detected, and all reference second-category pages and the target second-category page whose image difference is less than a preset image difference are recorded as a primary cluster combination. Cluster detection is continued for the reference second-category pages that are not recorded as primary cluster combinations until all second-category pages are recorded as primary cluster combinations.

[0085] The value of the preset image difference can be determined by the user according to the actual application scenario. The smaller the value of the preset image difference, the greater the similarity between the two types of pages in a clustering combination, and the greater the accuracy of the user's vulnerability detection. A value of the preset image difference is provided, and a clustering record is performed during the historical vulnerability detection process. The average value of the image difference corresponding to the records that can meet the user's vulnerability detection needs is recorded as the preset image difference.

[0086] Image difference = (feature region difference value / preset feature region difference value) × difference coefficient + (site mean difference value / preset site mean difference value) × site coefficient;

[0087] For a single Category II page, the method for confirming the feature area is to divide the area according to the page functional area, and record each divided area as a feature area. For a feature area in a page, the page functional area corresponding to the feature area may be the same as or different from the page functional areas corresponding to other feature areas of the page; the feature areas corresponding to the same page functional areas are recorded as feature areas of the same category, and the number of feature areas of the same category corresponding to each page may be different. The page functional area is the area with different functions in a single page. The page functional area includes a navigation area, a header area, a content display area, a sidebar, a bottom area, an interactive area and a promotion area. The segmentation of feature areas can be achieved through machine vision and a deep learning network. This is content that is easy for technicians in this field to understand and will not be elaborated here.

[0088] Feature area difference value = same feature area ratio coefficient + difference coefficient, the same feature area ratio coefficient is positively correlated with the same category quantity ratio, and the difference coefficient is positively correlated with the maximum feature area difference; the same category quantity ratio is confirmed by, for two second-category pages, recording the number of same-category feature areas corresponding to one second-category page as A, and recording the number of same-category feature areas corresponding to the other second-category page as B, and the same category quantity ratio = same category quantity / the larger value of A and B, and the same category quantity is the number of same-category feature areas possessed by both second-category pages;

[0089] For the first and second category pages, each feature area with the same page function area in the second category page is recorded as a feature area combination, and regional difference detection is performed on each feature area combination. When regional difference detection is performed on a feature area combination in the first and second category pages, the feature area combination is recorded as the target feature area combination, and the feature area combination with the same page function area corresponding to the target feature area combination in another second category page is recorded as the reference area combination. If there is no reference area combination in the other second category page, the regional difference is the number of feature areas corresponding to the target feature area combination. If there is a reference area combination in the other second category page, the absolute value of the difference between the number of feature areas corresponding to the target feature area combination and the number of feature areas corresponding to the reference area combination is recorded as the regional difference. The method for confirming the maximum feature area difference is that, for two second category pages, the maximum value of the regional differences corresponding to the two second category pages is recorded as the maximum feature area difference.

[0090] It can be understood that each of the second-category pages is a rectangular area, and the method for confirming the site mean difference value is to establish a rectangular coordinate system for a single second-category page, with the upper left corner of the second-category page as the origin, a straight line extending horizontally to the right through the origin as the x-axis, and a straight line extending downward perpendicular to the x-axis as the y-axis;

[0091] For the first and second category pages, the average value of the coordinates of the center position of each feature area corresponding to the second category page is recorded as the site mean. The method for confirming the site mean difference value is that for two second category pages, the site mean corresponding to the first and second category pages is recorded as (x1, y1), and the site mean corresponding to the other second category page is recorded as (x2, y2). The method for confirming the site mean difference value z is that the square of the difference between x1 and x2 is recorded as the first parameter, the square of the difference between y1 and y2 is recorded as the second parameter, and the square root of the sum of the first parameter and the second parameter is recorded as the site mean difference value; for a single feature area, the center position is the center of the minimum circumscribed circle of the feature area.

[0092] The values ​​of the preset feature region difference value and the preset site mean difference value can be determined by the user according to the actual application scenario. It can be understood that the smaller the values ​​of the preset feature region difference value and the preset site mean difference value are, the greater the similarity of the two types of pages in a clustering combination is, and the greater the accuracy of the user vulnerability detection is. A value of a preset feature region difference value and a preset site mean difference value is provided, and a clustering record is performed during the detection of historical vulnerability detection. The mean of the feature region difference values ​​corresponding to the records that can meet the user's vulnerability detection needs is recorded as the preset feature region difference value, and the mean of the site mean difference values ​​corresponding to the records that can meet the user's vulnerability detection needs is recorded as the preset site mean difference value;

[0093] The values ​​of the difference coefficient and the site coefficient can be obtained by the user through deep learning convolutional neural network learning on historical records. It can be understood that the present invention reflects the degree of difference between the two types of pages through image difference. The user can use deep learning through historical user data to obtain the influence of the feature area difference value and the site mean difference value on the degree of difference between the two types of pages, and then select the corresponding values ​​of the difference coefficient and the site coefficient, wherein the difference coefficient + site coefficient = 1, and one value of the difference coefficient and the site coefficient is provided, the difference coefficient is 0.5, and the site coefficient is 0.5.

[0094] Specifically, secondary clustering is performed on each primary clustering combination according to the comprehensive similarity to obtain several similar page combinations:

[0095] The comprehensive similarity is determined based on the apparent similarity and the structural difference;

[0096] The relationship between the apparent similarity and the comprehensive similarity is a positive correlation, and the relationship between the structural difference and the comprehensive similarity is a negative correlation.

[0097] Among them, secondary clustering is performed on each primary clustering combination according to the comprehensive similarity to obtain several similar page combinations, including:

[0098] A secondary clustering detection is performed on each second-category page corresponding to a single primary clustering combination. When the secondary clustering detection is performed on a single second-category page, the second-category page is recorded as a target clustering page, and other second-category pages excluding the target clustering page in the primary clustering combination are recorded as reference clustering pages. The comprehensive similarity between the target clustering page and each reference clustering page is detected, and all reference clustering pages and the target clustering page whose comprehensive similarity to the target clustering page is greater than a preset comprehensive similarity are recorded as a similar page combination, and the secondary clustering detection is continued for the reference clustering pages that are not recorded as similar page combinations, until all the second-category pages corresponding to the primary clustering combination are recorded as similar page combinations.

[0099] The value of the preset comprehensive similarity can be determined by the user according to the actual application scenario. The larger the value of the preset comprehensive similarity, the greater the similarity of each second category of pages in the similar page combination, and the greater the accuracy of the user's vulnerability detection. A value of the preset comprehensive similarity is provided, and the records of secondary clustering during the historical vulnerability detection process are detected. The average value of the comprehensive similarities corresponding to the records that can meet the user's vulnerability detection needs is recorded as the preset comprehensive similarity.

[0100] Comprehensive similarity = (apparent similarity / preset apparent similarity) × appearance coefficient - (structural difference / preset structural difference) × structural coefficient;

[0101] The present invention is provided with a monitoring cycle of continuous circulation, and the state of the characteristic area is determined once at the end of each monitoring cycle. The duration of the monitoring cycle can be set according to the needs of the user. The greater the user's demand for the accuracy of monitoring the characteristic area, the shorter the monitoring cycle duration. For a characteristic area, the preset duration is continuously monitored. If the characteristic areas of each monitoring cycle corresponding to the preset duration do not change, the characteristic area is recorded as a fixed characteristic area. The value of the preset duration can be set according to the actual application scenario. The higher the accuracy of the user's determination of the characteristic area state, the larger the value of the preset duration. A value of the preset duration is provided, and the preset duration is 30 monitoring cycles.

[0102] Apparent similarity = text similarity coefficient - distribution difference coefficient, the text similarity coefficient is positively correlated with the text similarity of the fixed feature area, and the distribution difference coefficient is positively correlated with the distribution difference degree;

[0103] For two second-category pages, a single fixed feature region in one of the second-category pages is recorded as a target fixed feature region, a fixed feature region in the other second-category page with the smallest position difference with the target fixed feature region and the same page functional area as the target fixed feature region is recorded as a reference fixed feature region, and the target fixed feature region and the reference fixed feature region are recorded as a detection combination;

[0104] For two second-category pages, the text similarity of the fixed feature area is the average of the text similarities of each detection combination. The text similarity is confirmed by converting the text into a measurable numerical form, using TF-IDF technology for text conversion, and using cosine similarity to calculate the cosine value of the angle between the two text vectors to measure their directional similarity. The cosine value of the angle between the two text vectors is recorded as the text similarity, and the value range of text similarity is [0,1].

[0105] Distribution difference = area difference coefficient + site difference coefficient. The relationship between the area difference coefficient and the mean of the area difference is positively correlated, and the relationship between the site difference coefficient and the mean of the site difference is positively correlated.

[0106] For two second-class pages, a single feature region in one of the second-class pages is recorded as a target region, and a feature region in the other second-class page with the smallest site difference with the target region and the same page functional region as the target region is recorded as a reference region, and the target region and the reference region are recorded as a region combination;

[0107] The mean area difference is the average of the area difference values ​​of each area combination. The area difference value is confirmed by recording the absolute value of the difference between the area of ​​one feature area and the area of ​​another feature area for the feature area corresponding to a region combination as the area difference value. The site difference coefficient is the average of the site difference values ​​corresponding to each region combination. The site difference value is confirmed by recording the center position of one feature area as (X1, Y1) and the center position of the other feature area as (X2, Y2) for two feature areas. The site difference value Z is confirmed by recording the square of the difference between X1 and X2 as the first difference parameter, recording the square of the difference between Y1 and Y2 as the second difference parameter, and recording the square root of the sum of the first difference parameter and the second difference parameter as the site difference value.

[0108] The method for confirming the structural difference is to convert the two second-category pages into a DOM tree structure, and record the minimum number of operations required to convert the DOM tree structure corresponding to one second-category page into the DOM tree structure corresponding to the other second-category page as the structural difference. The operations include definition addition, deletion and replacement.

[0109] The values ​​of the preset apparent similarity and the preset structural difference can be determined by the user according to the actual application scenario. The larger the value of the preset apparent similarity and the smaller the value of the preset structural difference, the greater the similarity between the two types of pages in the similar page combination, and the greater the accuracy of the user's vulnerability detection. A value of a preset apparent similarity and a preset structural difference is provided, and the records of secondary clustering during the historical vulnerability detection process are detected. The average value of the apparent similarities corresponding to the records that can meet the user's vulnerability detection needs is recorded as the preset apparent similarity, and the average value of the structural differences corresponding to the records that can meet the user's vulnerability detection needs is recorded as the preset structural difference.

[0110] The values ​​of the apparent coefficient and the structural coefficient can be obtained by the user through deep learning convolutional neural network learning on historical records. It can be understood that the present invention reflects the similarity of two types of pages through comprehensive similarity. The user can use deep learning through historical user data to obtain the influence of apparent similarity and structural difference on the similarity of two types of pages, and then select the values ​​of the apparent coefficient and the structural coefficient accordingly, wherein the apparent coefficient + the structural coefficient = 1, and one value of the apparent coefficient and the structural coefficient is provided, the apparent coefficient is 0.4, and the structural coefficient is 0.6.

[0111] In the present invention, a URL is extracted for each similar page combination for vulnerability detection, each extracted URT is used as a vulnerability detection target, an attack vector is input into each vulnerability detection target, and whether there is a vulnerability is determined according to the execution status of the Web application. If the Web application is executed incorrectly, there is a vulnerability, and if the Web application is executed correctly, there is no vulnerability. The attack vector is generated by constructing an attack vector factor library and based on rule constraints. This is content that is easy to understand for those skilled in the art and will not be described in detail.

[0112] See also Figure 4 As shown, it is a schematic diagram of a Web application vulnerability detection device of the present invention. The present invention also provides a Web application vulnerability detection device, including:

[0113] A state analysis module is used to obtain Web application information, determine the search state according to the average value of the number of links and the maximum link difference, and determine the search method as reference threshold analysis or traversal page analysis according to the search state;

[0114] A page clustering module, which is connected to the state analysis module, is used to determine the page type of the page to be detected according to the page update frequency and the sub-page frequency reference value, and to determine the clustering method according to the page type, which is to perform URL clustering according to page difference or to perform a primary clustering according to image similarity and a secondary clustering according to comprehensive similarity, to obtain a plurality of similar page combinations;

[0115] The vulnerability detection module is connected to the page clustering module and is used to extract a URL for vulnerability detection for each similar page combination. After inputting the attack vector, it determines whether there is a vulnerability based on the execution status of the Web application.

[0116] The present invention also provides a storage medium, in which a computer program is stored. When the computer program is run on a computer, the Web application vulnerability detection method can be executed.

[0117] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A Web application vulnerability detection method, characterized in that: include: Obtain web application information and determine the search status based on the average number of links and the maximum link difference; Determine the search mode as reference threshold analysis or traversal page analysis according to the search state; Determine the page type of the page to be detected according to the page update frequency and the sub-page frequency reference value; The clustering method is determined according to the page type, which is to cluster the URLs according to the page difference or to cluster them once according to the image similarity and then cluster them again according to the comprehensive similarity to obtain several similar page combinations; For each similar page combination, a URL is extracted for vulnerability detection. After the attack vector is input, it is determined whether there is a vulnerability based on the execution of the Web application.

2. The Web application vulnerability detection method according to claim 1, characterized in that: When the search status is that the average value of the number of links is greater than or equal to the preset average value of the number of links or the maximum link difference is greater than or equal to the preset maximum link difference, the search method is reference threshold analysis, which includes: A reference threshold analysis is performed on each initial sub-page corresponding to the initial page. When a reference threshold analysis is performed on a single initial sub-page, the initial sub-page is recorded as a target sub-page, and a reference threshold corresponding to the target sub-page is detected. If the reference threshold corresponding to the target sub-page is greater than a preset reference threshold, the target initial sub-page is recorded as a sub-page to be detected, and each sub-page to be detected and the initial page are recorded as a page to be detected; The reference threshold is determined according to the sum of the page threshold corresponding to the target subpage and the page threshold corresponding to each associated subpage.

3. The Web application vulnerability detection method according to claim 2, characterized in that: When the search status is that the average link quantity is less than the preset link quantity average and the maximum link difference is less than the preset maximum link difference, the search method is traversal page analysis, which includes: The initial page and each initial sub-page corresponding to the initial page are recorded as pages to be detected.

4. The Web application vulnerability detection method according to claim 3, characterized in that: The page type of the page to be detected is determined according to the page update frequency and the sub-page frequency reference value. The page types include: A type of page whose page update frequency is less than a preset page update frequency and whose sub-page frequency reference value is less than a preset sub-page frequency reference value; The second category of pages has a page update frequency greater than or equal to a preset page update frequency or a sub-page frequency reference value greater than or equal to a preset sub-page frequency reference value.

5. The Web application vulnerability detection method according to claim 4, characterized in that: Determine the clustering method based on page type; For a type of page, URL clustering is performed based on page differences; For the second category pages, a clustering is performed based on image similarity, and a secondary clustering is performed based on comprehensive similarity.

6. The Web application vulnerability detection method according to claim 5, characterized in that: URL clustering is performed for each category of pages based on page differences, including: Similar page detection is performed on a type of page corresponding to each URL. When similar page detection is performed on a type of page corresponding to a single URL, the type of page corresponding to the URL is recorded as the target page, and the type of pages corresponding to other URLs are recorded as reference pages. The page difference between the target page and each reference page is detected, and the set of all reference pages and the target page whose page difference with the target page is less than a preset page difference is recorded as a similar page combination. Similar page detection is continued for reference pages that are not recorded as similar page combinations until all type of pages corresponding to all URLs are recorded as similar page combinations.

7. The Web application vulnerability detection method according to claim 6, characterized in that: Performing a clustering operation on each of the two types of pages according to the image difference to obtain a number of primary clustering combinations; The image difference is determined based on the feature area difference value and the site mean difference value; The relationship between the characteristic region difference value and the image difference is a positive correlation, and the relationship between the site mean difference value and the image difference is a positive correlation.

8. The Web application vulnerability detection method according to claim 7, characterized in that: Perform secondary clustering on each primary clustering combination according to the comprehensive similarity to obtain several similar page combinations: The comprehensive similarity is determined based on the apparent similarity and the structural difference; The relationship between the apparent similarity and the comprehensive similarity is a positive correlation, and the relationship between the structural difference and the comprehensive similarity is a negative correlation.

9. A Web application vulnerability detection device using the method described in any one of claims 1 to 8, characterized in that: include: A state analysis module is used to obtain Web application information, determine the search state according to the average value of the number of links and the maximum link difference, and determine the search method as reference threshold analysis or traversal page analysis according to the search state; A page clustering module, which is connected to the state analysis module, is used to determine the page type of the page to be detected according to the page update frequency and the sub-page frequency reference value, and to determine the clustering method according to the page type, which is to perform URL clustering according to page difference or to perform a primary clustering according to image similarity and a secondary clustering according to comprehensive similarity, to obtain a plurality of similar page combinations; The vulnerability detection module is connected to the page clustering module and is used to extract a URL for vulnerability detection for each similar page combination. After inputting the attack vector, it determines whether there is a vulnerability based on the execution status of the Web application.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program runs on a computer, the Web application vulnerability detection method according to any one of claims 1 to 8 can be executed.

Images