Web page health detection method and device, equipment and storage medium

Abstract

The invention discloses a Web page health detection method and device, equipment and a storage medium. The method comprises the steps of determining a to-be-detected target page through link crawling based on an obtained full-amount source code; respectively performing page operation detection and data security detection on the target page, and respectively determining a page operation detection result and a data security detection result corresponding to the target page; and determining the target detection result based on the page operation detection result and the data security detection result corresponding to the target page by adopting the pre-configured large language model, so that the detection effect of the Web page is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to a Web page health detection method and device, equipment and storage medium. BACKGROUND

[0002] With the deep popularization of Internet technology and the diversified development of Web application, Web page has become the core carrier of enterprises to carry business and serve users, and widely covers key scenes such as e-commerce transaction, government service and online office. Whether the page is healthy will directly affect the user experience and business stability.

[0003] In the prior art, most of the detection methods of Web page focus more on function verification, ignoring the evaluation of security and performance efficiency. The monitoring of Web page abnormalities relies on passive reporting triggered by users, which has obvious lag, and the data analysis only stays in error log recording, which has single data analysis dimension, thereby leading to poor health detection effect of Web page. SUMMARY

[0004] Based on the above problems, the present application provides a Web page health detection method, device, equipment and storage medium, which aims to improve the health detection effect of Web page.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] In a first aspect, the present application provides a Web page health detection method, comprising:

[0007] Based on the obtained full source code, the target page to be detected is determined through link crawling;

[0008] The target page is subjected to page running detection and data security detection respectively, and the page running detection result and the data security detection result corresponding to the target page are determined respectively;

[0009] A preconfigured large language model is used to determine the target detection result based on the page running detection result and the data security detection result corresponding to the target page.

[0010] Optionally, as described above, if the page running detection result includes the rendering performance detection result, the resource access detection result, the network link detection result and the resource loading configuration detection result, the page running detection of the target page to determine the page running detection result corresponding to the target page comprises:

[0011] Based on the collected multiple rendering performance indicators of the target page, the abnormal score of the target page is determined by using a dynamic weight algorithm;

[0012] If the exception score of the target page is greater than a first preset threshold, it is determined that the rendering performance detection result is rendering performance exception.

[0013] Obtaining network request data of the target page; the network request data includes HTTP response status code, TCP connection time consumption and SSL handshake time consumption;

[0014] If the HTTP response status code is 404 or 500, it is determined that the resource access detection result is service response exception.

[0015] If the TCP connection time consumption is greater than a second preset threshold or the SSL handshake time consumption is greater than a third preset threshold, it is determined that the network link detection result is network link exception.

[0016] Extracting configuration data of static resources in the target page; the configuration data of static resources includes HTTP cache policy and file compression rate;

[0017] Based on the fact that the static resources are not configured with Cache-Control field or the file compression rate is less than a fourth preset threshold, it is determined that the resource loading configuration detection result is resource loading configuration exception.

[0018] Optionally, the method described above, the data security detection result includes data compliance detection result, variant sensitive word and potential risk point, the method of performing security detection on the target page data to determine the data security detection result corresponding to the target page includes:

[0019] Using abstract syntax tree to parse the source code corresponding to the target page in the full source code to determine the hard-coded key and sensitive information;

[0020] Using regular expression to perform data compliance verification on the hard-coded key and sensitive information to determine the data compliance verification result;

[0021] Based on a preset sensitive word library, using a natural language processing model to perform semantic association analysis on the content of the target page to identify the variant sensitive word in the target page;

[0022] Through static code feature analysis, the potential risk point in the source code corresponding to the target page is determined.

[0023] Optionally, the method described above, the method of using a preconfigured large language model to determine the target detection result based on the page running detection result and the data security detection result corresponding to the target page includes:

[0024] Using abstract syntax tree to perform structured parsing on the HTML and JavaScript source code of the target page to obtain source code structure data;

[0025] generate a multi-dimensional analysis context based on the source code structure data, the page running detection result, and the data security detection result;

[0026] input the multi-dimensional analysis context into a pre-configured large language model to output a model analysis result; the model analysis result includes a problem type, a specific code location, and a preliminary root cause;

[0027] determine the target detection result based on the model analysis result.

[0028] Optionally, the method described above, the full source code includes static source code and dynamic source code, based on the obtained full source code, the target page to be detected is determined by link crawling, including:

[0029] extract a hard-coded link based on the static source code;

[0030] construct a dynamic interaction link based on script source code that triggers Ajax and XHR requests in the dynamic source code;

[0031] generate potential routing paths based on SPA framework routing configuration source code in the dynamic source code;

[0032] determine the target page to be detected based on a preset user domain name whitelist, using regular expression and semantic similarity matching algorithms, according to the hard-coded link, dynamic interaction link, and potential routing paths.

[0033] Optionally, the method described above, the target page to be detected is determined based on a preset user domain name whitelist, using regular expression and semantic similarity matching algorithms, according to the hard-coded link, dynamic interaction link, and potential routing paths, including:

[0034] preprocess the hard-coded link, dynamic interaction link, and potential routing paths to obtain a first link;

[0035] filter out candidate links belonging to the internal of the Web application to be detected by matching link domain names through regular expressions based on a preset user domain name whitelist;

[0036] determine the target page to be detected from the candidate links using a semantic similarity matching algorithm.

[0037] Optionally, the method described above, the target page is multiple, and the method further includes:

[0038] Based on the initial detection process upper limit preset by the user, the real-time collected memory usage and CPU load, the detection tasks corresponding to the plurality of target pages are fragmented and allocated to obtain a plurality of fragments;

[0039] A sandbox environment is allocated for the target page detection task of each fragment in the plurality of fragments by using BrowserContext isolation technology.

[0040] During the detection process of the plurality of target pages, when the memory usage exceeds a first threshold or the CPU load exceeds a second threshold, based on the FIFO principle, the target page detection task corresponding to a low priority in the plurality of target pages is suspended.

[0041] When the memory usage is less than or equal to the first threshold, and the CPU load is less than or equal to the second threshold, the suspended target page detection task is resumed for execution.

[0042] In a second aspect, the present application provides a web page health detection device, comprising:

[0043] A target page determination module is configured to determine the target pages to be detected based on the obtained full source code through link crawling.

[0044] A page health detection module is configured to perform page running detection and data security detection on the target pages respectively, and determine the page running detection result and the data security detection result corresponding to the target pages respectively.

[0045] A detection result determination module is configured to determine the target detection result based on the page running detection result and the data security detection result corresponding to the target pages by using a preconfigured large language model.

[0046] In a third aspect, the present application provides an electronic device, comprising a processor and a memory in communication with the processor.

[0047] The memory stores computer execution instructions.

[0048] The processor executes the computer execution instructions stored in the memory to implement the web page health detection method described in any of the above embodiments.

[0049] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by the processor, the computer execution instructions are used to implement the web page health detection method described in any of the above embodiments.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] The method of the present application first determines the target page to be detected through link crawling based on the obtained full source code, avoids missing key pages, and improves detection coverage; respectively performs page running detection and data security detection on the target page, respectively determines the page running detection result and the data security detection result corresponding to the target page, synchronously investigates performance / resource abnormalities and sensitive information leakage / vulnerabilities, solves the problem of single detection dimension of the traditional detection, and comprehensively discovers page hidden dangers; finally, a preconfigured large language model is used to determine the target detection result based on the page running detection result and the data security detection result corresponding to the target page, which can accurately locate the problem root cause instead of only listing data, thereby improving the health detection effect of the Web page. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 A flowchart of a Web page health detection method provided by an embodiment of the present application;

[0054] Figure 2 A structural diagram of a Web page health detection device provided by an embodiment of the present application;

[0055] Figure 3 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with specific embodiments and with reference to the drawings. It should be particularly noted that the embodiments described in the embodiments of the present application are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application shall be understood as having the common meaning to those skilled in the art to which the embodiments of the present application belong. The terms "first", "second", and similar terms used in the embodiments of the present application do not denote any order, quantity, or importance, but are only used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms do not mean physical or mechanical connection, but can include electrical connection, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like only represent relative positional relationships, which can change accordingly when the absolute positions of the described objects change.

[0058] As described above, the health detection of the Web page is usually performed by using artificial test tools such as Selenium and Cypress at present, but these artificial test tools are mainly used for function verification, have single dimension, and lack health evaluation dimension. In addition, in terms of abnormal monitoring, passive monitoring is adopted, which depends on the abnormal report triggered by the user, has hysteresis, and thus leads to poor effect of health detection of the Web page.

[0059] The inventors have proposed a Web page health detection method, device, equipment, and storage medium, which improves the health detection effect of the Web page.

[0060] To enable personnel in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0061] Referring to Figure 1 , the figure is a flowchart of a Web page health detection method provided by the embodiments of the present application. As Figure 1 shown, the method comprises the following steps.

[0062] S101: Based on the obtained full-source code, the target page to be detected is determined through link crawling.

[0063] The full-source code includes static source code and dynamic source code. The target page to be detected is determined through link crawling by code analysis on the obtained full-source code of the Web application to be detected.

[0064] As an implementable manner, based on the obtained full source code, a specific implementation manner of determining the target page to be detected through link crawling includes the following steps:

[0065] Based on the static source code, the hard-coded link is extracted.

[0066] In the embodiment, the static source code in the full source code, including the HTML file, the basic JavaScript file without dynamic interaction logic and the CSS file, is subjected to structural analysis by using a DOM analysis tool, and the <ahref>The jump link defined in the label, Resource reference links such as jump links in the window. location, location. href, etc. are retrieved by regular expressions; finally, all the extracted links are de-duplicated to obtain the hard-coded links.

[0067] Based on the script source code that triggers Ajax and XHR requests in dynamic source code, a dynamic interaction link is constructed.

[0068] In this embodiment, the script source code containing Ajax and XHR request logic is screened out from the dynamic source code of the full-amount source code; the script source code is parsed using an abstract syntax tree parsing tool to locate the request triggering node, such as the request function bound to the button click event or the XHR call automatically executed when the page is loaded; a page running environment is simulated, such as a browser global object simulated by Node.js, the script is executed and the request URL and triggering condition are intercepted, and the dynamic interaction link is constructed according to the correspondence between the triggering condition and the request URL.

[0069] Based on the SPA framework routing configuration source code in the dynamic source code, potential routing paths are generated.

[0070] In this embodiment, the routing configuration source code of the SPA framework in the dynamic source code of the full-amount source code is identified: for a Vue project, the vue-router configuration file is located, and the routing path corresponding to the path attribute in the array is parsed; for a React project, the react-router related components are retrieved, and the path parameter value is extracted; for the route containing dynamic parameters, the potential parameter combination is generated combined with the common business parameter rules, and the potential routing paths are obtained by integrating all the parsed and generated paths.

[0071] According to the hard-coded links, dynamic interaction links and potential routing paths, based on the preset user domain name whitelist, regular expressions and semantic similarity matching algorithms are used to determine the target pages to be detected.

[0072] In this embodiment, the hard-coded links, dynamic interaction links and potential routing paths are integrated to form a full-amount candidate link set and de-duplicated; based on the preset user domain name whitelist, the link domain name is accurately matched again by regular expressions to retain the internal links belonging to the Web application to be detected; then, the semantic similarity matching algorithm is used to further filter non-core links such as test pages and abandoned routes; finally, the effective links after screening are sorted to obtain the target page list to be detected. Through the above embodiments, the target pages to be detected can be accurately identified, ensuring that no key pages are missed and improving the coverage integrity of Web page health detection.

[0073] Further, as a realizable manner, according to the hard-coded link, the dynamic interaction link and the potential routing path, based on the preset user domain name whitelist, a specific implementation of determining the target page to be detected includes the following steps:

[0074] The hard-coded link, the dynamic interaction link and the potential routing path are preprocessed to obtain a first link; based on the preset user domain name whitelist, the link domain name is matched by a regular expression to screen out candidate links belonging to the internal Web application to be detected; and a semantic similarity matching algorithm is used to determine the target page to be detected from the candidate links.

[0075] S102: The target page is subjected to page running detection and data security detection respectively to determine the page running detection result and the data security detection result corresponding to the target page.

[0076] In this embodiment, the target page to be detected is subjected to page running detection to determine the page running detection result corresponding to the target page. The page running detection is used to detect network abnormalities, rendering performance abnormalities and resource configuration abnormalities in the page running process. The target page to be detected is also subjected to data security detection to determine the data security detection result corresponding to the target page. The data security detection is used to detect whether the data of the current page is safe.

[0077] As a realizable manner, when the target page is subjected to page running detection, if the page running detection result includes the rendering performance detection result, the resource access detection result, the network link detection result and the resource loading configuration detection result, a specific implementation of the page running detection of the target page includes the following steps:

[0078] Based on the collected multiple rendering performance indicators of the target page, a dynamic weight algorithm is used to determine the abnormal score of the target page; if the abnormal score of the target page is greater than a first preset threshold, it is determined that the rendering performance detection result is rendering performance abnormality.

[0079] In the embodiment, first, the core rendering performance indicators of the target page are collected by the Puppeteer tool, including the largest contentful paint (LCP), first contentful paint (FCP), element quantity statistics, content analysis, skeleton screen identification, error log determination, and web screenshot color analysis; the dynamic weights are configured based on the business attributes of the target page, and the abnormal deviation degrees of the indicators are calculated. Then, the abnormal score of the rendering performance of the target page is obtained by multiplying the deviation degrees of the indicators by the corresponding weights and summing them up; if the first preset threshold is 30 points, when the abnormal score of the rendering performance of the target page is greater than 30 points, it is determined that the rendering performance detection result is abnormal, and the abnormal indicator with the highest contribution degree is marked, for example, "LCP deviation degree 48%, weight 40%, main abnormal reason".

[0080] The network request data of the target page is obtained; the network request data includes the HTTP response status code, TCP connection time consumption, and SSL handshake time consumption; if the HTTP response status code is 404 or 500, it is determined that the resource access detection result is service response abnormal; if the TCP connection time consumption is greater than the second preset threshold or the SSL handshake time consumption is greater than the third preset threshold, it is determined that the network link detection result is network link abnormal.

[0081] The TCP connection time consumption is the time from the request initiation to the TCP handshake completion; the SSL handshake time consumption is the time of SSL protocol negotiation in the HTTPS request.

[0082] In the embodiment, the page.on('response') event of Puppeteer is used to listen to all network requests of the target page, and the network request data is collected synchronously: the HTTP response status is extracted from the response object, for example, 20, 404, 500), the TCP connection time consumption and the SSL handshake time consumption of the corresponding request are obtained through the browser performance.getEntriesByType('resource'); if the second preset threshold is 300 ms and the third threshold is 200 ms; when the HTTP response status code is 404 (resource does not exist) or 500 (server error), it is directly determined that the resource access detection result is service response abnormal, and the URL and request method of the abnormal resource are recorded; when the TCP connection time consumption is greater than 300 ms or the SSL handshake time consumption is greater than 200 ms, it is determined that the network link detection result is network link abnormal, and the abnormal type is marked, for example, "TCP connection time consumption 380 ms, exceeds the threshold, suspected network congestion".

[0083] Extract the configuration data of the static resources in the target page; the configuration data of the static resources includes HTTP cache policy and file compression rate; based on the fact that the static resources are not configured with Cache-Control field or the file compression rate is less than the fourth preset threshold, it is determined that the resource loading configuration detection result is resource loading configuration exception.

[0084] In this embodiment, the network request records of static resources such as pictures, CSS and JavaScript in the target page are traversed, the HTTP cache policy configuration is extracted from the response header, and the original size and the size after transmission compression of the resource file are obtained through response.buffer(), and the file compression rate is calculated. Among them, the file compression rate = (1-compressed size / original size) x 100%; the fourth preset threshold is 60%, if the Cache-Control field is not configured in the response header of the static resource, or is configured as forbidden cache, it is determined that the resource has cache configuration exception; if the file compression rate is less than 60%, it is determined that the resource has compression configuration exception; the types, URLs and specific exception parameters of all abnormal resources are integrated, and the resource loading configuration detection result is determined as resource loading configuration exception.

[0085] In this embodiment, a full-stack health assessment model is used, which covers three-dimensional health degree analysis of network layer, rendering layer and resource layer, can accurately identify abnormal points affecting the health of Web page from all aspects, avoid abnormal point omission and misjudgment caused by single dimension detection, and improve the comprehensiveness and accuracy of Web page health detection.

[0086] When the target page is detected for data security, if the data security detection result includes the data compliance detection result, the variant sensitive word and the potential risk point, the target page is detected for data security, and one specific implementation of the data security detection result corresponding to the target page includes the following steps:

[0087] The source code corresponding to the target page in the abstract syntax tree is parsed, and the hard-coded key and sensitive information are determined; the data compliance of the hard-coded key and sensitive information is verified by using regular expression, and the data compliance verification result is determined; based on the preset sensitive word library, the semantic association analysis of the content of the target page is performed by using natural language processing model, and the variant sensitive word in the target page is identified; the potential risk point in the source code corresponding to the target page is determined through static code feature analysis.

[0088] In this embodiment, first, the abstract syntax tree tool is used to structurally parse the HTML and JavaScript source code corresponding to the target page in the full source code, traverse the variable declaration, string constant node of the abstract syntax tree, locate the hard-coded key prefixed with "API_KEY" and "Token", and the sensitive information conforming to the format of "name + ID number" and "mobile phone number", form an initial information set containing key content, sensitive information type and source code line number; then, based on the compliance standards of the Personal Information Protection Law, a regular expression library is constructed to verify the desensitization state of the ID number, at the same time, the key complexity is checked, the un-desensitized information and weak keys are marked, and the data compliance verification result is determined to be compliant or non-compliant; then, a preset sensitive word library is imported, a Natural Language Processing (NLP) model is called to perform semantic association analysis on the text content and popup prompt of the target page, and identify variant sensitive words such as "shoujihao" (mobile phone number pinyin) and "sfz" (ID card abbreviation), output the variant type and semantic matching degree; finally, through a static code feature analysis tool, the abstract syntax tree is screened for un-escaped innerHTML assignment, document.write() direct insertion of user input, Cross Site Scripting (XSS) vulnerability features, and missing cross-domain resource sharing CORS configuration permissions, to determine the type of potential risk points, such as XSS vulnerabilities or abnormal permission configurations, and determine the specific code location of the risk points, finally integrate the data compliance verification result, variant sensitive word list and potential risk point details to form a data security detection result, which realizes rapid positioning of known security vulnerability features through pattern matching, mining of deep logic risks through code semantic analysis, and reduction of missed and false positives through the cooperation of the two engines to compensate for the limitations of single detection, thereby improving the accuracy of security detection.

[0089] S103: A pre-configured large language model is used to determine the target detection result based on the page running detection result and the data security detection result corresponding to the target page.

[0090] In this embodiment, the pre-configured large language model is used to analyze and process the page running detection result and the data security detection result corresponding to the target page to obtain the final output target detection result.

[0091] As an implementable way, the specific implementation of determining the target detection result based on the page running detection result and the data security detection result corresponding to the target page using a pre-configured large language model can include the following steps:

[0092] The abstract syntax tree is used to perform structured parsing of the HTML and JavaScript source code of the target page to obtain source code structure data. Based on the source code structure data, page execution detection results, and data security detection results, a multi-dimensional analysis context is generated. The multi-dimensional analysis context is input into a pre-configured large language model, and the model analysis results are output. The model analysis results include the problem type, specific code location, and preliminary root cause. Based on the model analysis results, the target detection results are determined.

[0093] In this embodiment, an abstract syntax tree parsing tool is first used to perform structured parsing on the HTML source code and associated JavaScript source code of the target page. The parsed syntax tree nodes are traversed to extract key information such as variable declarations, function calls, DOM element definitions, and resource reference paths. This information is then organized into source code structure data containing code hierarchy, node types, and content descriptions, clarifying the functional attributes of each code segment. Based on this source code structure data, page execution detection results and data security detection results are integrated to supplement the target page's HTTP request timing data and resource dependency graph. This generates a multi-dimensional analysis context covering source code structure, runtime anomalies, security risks, and runtime sequence. This multi-dimensional analysis context is then input into a pre-configured large language model. Through the model's semantic understanding and code logic reasoning capabilities, the associated code nodes for each problem are located. For example, for LCP timeout, the source code is identified as being related to... The issue stemmed from a lack of compression and caching configuration. Regarding the XSS vulnerability, the root cause was identified as unfiltered user input in line 28 of the source code. The final output includes the problem type (performance anomaly / security vulnerability), specific code location (line number, code snippet), and a preliminary root cause analysis. Finally, the model analysis results and raw detection data are integrated, organized according to the structure of "Overall Health Rating - Problem Classification List - Root Cause Analysis - Data Basis." Specifically, a health rating conclusion of "qualified / needs optimization / high-risk" is given based on the severity of the problem. Then, based on "security vulnerability > performance anomaly," the problems are sorted, with each problem labeled with the root cause analyzed by the model and the corresponding raw data from page operation / data security testing. Finally, industry best practice optimization suggestions are added, ultimately forming a target detection result that combines quantitative evidence, traceability, and practical guidance.

[0094] In the embodiment, based on the obtained full source code, the target page to be detected is determined through link crawling, key pages are not missed, and the detection coverage is improved; the target page is subjected to page running detection and data security detection respectively, the page running detection result and the data security detection result corresponding to the target page are determined respectively, performance / resource abnormalities and sensitive information leakage / vulnerabilities are synchronously investigated, the problem of single detection dimension in the traditional detection is solved, and page hidden dangers are comprehensively found; finally, a preconfigured large language model is used, the target detection result is determined based on the page running detection result and the data security detection result corresponding to the target page, the problem root cause can be accurately located instead of only listing data, and the health detection effect of the Web page is improved.

[0095] It can be understood that the target page to be detected can be multiple, and the method can further include:

[0096] Based on the initial detection process upper limit preset by the user, the memory usage rate and the CPU load collected in real time, the detection tasks corresponding to the multiple target pages are fragmented and allocated to obtain multiple fragments; a sandbox environment is allocated for the target page detection task of each fragment in the multiple fragments by using BrowserContext isolation technology; during the detection process of the multiple target pages, when the memory usage rate exceeds a first threshold or the CPU load exceeds a second threshold, the target page detection task corresponding to a low-priority target page in the multiple target pages is suspended based on the FIFO principle; when the memory usage rate is less than or equal to the first threshold and the CPU load is less than or equal to the second threshold, the suspended target page detection task is resumed for execution.

[0097] In the embodiment, first, an initial detection process upper limit preset by a user is acquired, system resource data such as host memory usage and CPU load is collected in real time by an os module of Node.js at a period of 1 second, and the number of all target pages to be detected and the business attributes of each page are counted, for example, a core transaction page is marked as high priority, and a test page is marked as low priority; and the resource data and the total number of tasks are combined for sharding calculation. If the current memory usage is 35% lower than the first preset threshold of 70% and the CPU load is 40% lower than the second preset threshold of 80%, the detection task of 12 target pages of the process upper limit is split into 3 shards, for example, the first two shards each have 5 tasks, and the third shard has 2 tasks, to obtain a plurality of shards. Subsequently, a Puppeteer browser instance is started, an independent sandbox environment is created for each shard based on the BrowserContext isolation technology, memory leakage or abnormal processes of a certain shard are avoided to affect other shard tasks, and the stability of multi-task parallelism is ensured. During the execution of all shard tasks, the system resource state is continuously monitored and synchronized, when the memory usage is monitored to rise to 75% and the CPU load is 82% at a certain moment, the threshold condition is triggered, the task scheduling module is immediately called to extract all executing task queues, low-priority tasks are selected based on the FIFO principle, and the corresponding sandbox environment is temporarily terminated by context.close(), and the CPU and memory resources occupied by the task are released. When the memory usage is monitored to fall to 62% and the CPU load is reduced to 72% at a certain moment, the suspended tasks are executed in the order of suspension, and an independent sandbox is created for the suspended tasks to continue the detection process. Thus, while ensuring the detection efficiency, the stability of long task running is improved, the system crash rate is reduced, and the balance between high-concurrency detection and resource consumption is achieved.

[0098] Referring to Figure 2 The figure is a structural schematic diagram of a web page health detection device provided by the embodiment of the application. As Figure 2 shown, the device 20 comprises a target page determination module 21, a page health detection module 22, and a detection result determination module 23.

[0099] The target page determination module 21 is configured to determine the target page to be detected based on the acquired full source code through link crawling. The page health detection module 22 is configured to perform page running detection and data security detection on the target page respectively, and determine the page running detection result and the data security detection result corresponding to the target page respectively. The detection result determination module 23 is configured to determine the target detection result based on the page running detection result and the data security detection result corresponding to the target page by using a preconfigured large language model.

[0100] The Web page health detection device provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and has similar implementation principles and beneficial effects, which will not be repeated here.

[0101] Further, based on the above-mentioned embodiments, if the page running detection result includes the rendering performance detection result, the resource access detection result, the network link detection result, and the resource loading configuration detection result, the page health detection module 22 is specifically configured to determine an abnormal score of the target page based on the collected multiple rendering performance indicators of the target page by using a dynamic weight algorithm; if the abnormal score of the target page is greater than a first preset threshold, it is determined that the rendering performance detection result is rendering performance abnormality; network request data of the target page is obtained; the network request data includes HTTP response status code, TCP connection time consumption, and SSL handshake time consumption; if the HTTP response status code is 404 or 500, it is determined that the resource access detection result is service response abnormality; if the TCP connection time consumption is greater than a second preset threshold or the SSL handshake time consumption is greater than a third preset threshold, it is determined that the network link detection result is network link abnormality; configuration data of static resources in the target page is extracted; the configuration data of static resources includes HTTP cache policy and file compression rate; based on the fact that the static resources are not configured with a Cache-Control field or the file compression rate is less than a fourth preset threshold, it is determined that the resource loading configuration detection result is resource loading configuration abnormality.

[0102] The Web page health detection device provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and has similar implementation principles and beneficial effects, which will not be repeated here.

[0103] Further, based on the above-mentioned embodiments, if the data security detection result includes the data compliance detection result, the variant sensitive word, and the potential risk point, the page health detection module 22 is specifically configured to parse the source code corresponding to the target page in the full-amount source code by using an abstract syntax tree, to determine the hard-coded key and the sensitive information; perform data compliance verification on the hard-coded key and the sensitive information by using a regular expression, to determine a data compliance verification result; perform semantic correlation analysis on the content of the target page by using a natural language processing model based on a preset sensitive word library, to identify the variant sensitive word in the target page; and determine the potential risk point in the source code corresponding to the target page by static code feature analysis.

[0104] The Web page health detection device provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and has similar implementation principles and beneficial effects, which will not be repeated here.

[0105] Further, based on the above-mentioned embodiments, the detection result determination module 23 is specifically configured to perform structural parsing on the HTML and JavaScript source code of the target page by using an abstract syntax tree to obtain source code structure data; generate a multi-dimensional analysis context based on the source code structure data, the page running detection result, and the data security detection result; input the multi-dimensional analysis context into a pre-configured large language model to output a model analysis result; the model analysis result includes a problem type, a specific code location, and a preliminary root cause; and determine the target detection result based on the model analysis result.

[0106] The Web page health detection device provided in the embodiments of the present application can execute the technical solutions shown in the above-mentioned method embodiments, and has similar implementation principles and beneficial effects, which will not be described here in detail.

[0107] Further, based on the above-mentioned embodiments, the full-amount source code includes static source code and dynamic source code, the target page determination module 21 is specifically configured to extract hard-coded links based on the static source code based on the obtained full-amount source code; construct dynamic interaction links based on script source code triggering Ajax and XHR requests in the dynamic source code; generate potential routing paths based on SPA framework routing configuration source code in the dynamic source code; and determine the target page to be detected based on a preset user domain name whitelist by using a regular expression and a semantic similarity matching algorithm according to the hard-coded links, the dynamic interaction links, and the potential routing paths.

[0108] The Web page health detection device provided in the embodiments of the present application can execute the technical solutions shown in the above-mentioned method embodiments, and has similar implementation principles and beneficial effects, which will not be described here in detail.

[0109] Further, based on the above-mentioned embodiments, when the target page to be detected is determined based on a preset user domain name whitelist by using a regular expression and a semantic similarity matching algorithm according to the hard-coded links, the dynamic interaction links, and the potential routing paths, the target page determination module 21 is specifically configured to preprocess the hard-coded links, the dynamic interaction links, and the potential routing paths to obtain first links; filter out candidate links belonging to the internal of the Web application to be detected by matching link domain names by using a regular expression based on the preset user domain name whitelist; and determine the target page to be detected from the candidate links by using a semantic similarity matching algorithm.

[0110] The Web page health detection device provided in the embodiments of the present application can execute the technical solutions shown in the above-mentioned method embodiments, and has similar implementation principles and beneficial effects, which will not be described here in detail.

[0111] Further, based on the above-mentioned embodiments, the target page is multiple, and the target page determination module 21 can be further configured to perform the following operations. Based on the initial detection process upper limit preset by the user, the memory usage and the CPU load collected in real time, the target page determination module 21 performs sharding and distribution on detection tasks corresponding to the multiple target pages to obtain multiple shards. The target page determination module 21 uses the BrowserContext isolation technology to assign a sandbox environment for the target page detection task of each shard in the multiple shards. During the detection process of the multiple target pages, when the memory usage exceeds a first threshold or the CPU load exceeds a second threshold, the target page determination module 21 suspends the target page detection task corresponding to a low-priority target page in the multiple target pages based on the FIFO principle. When the memory usage is less than or equal to the first threshold and the CPU load is less than or equal to the second threshold, the target page determination module 21 resumes the suspended target page detection task.

[0112] The Web page health detection apparatus provided by the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0113] Referring to FIG. 1, Figure 3 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application, which includes:

[0114] The memory 11 is configured to store a computer program.

[0115] The processor 12 is configured to execute the computer program to implement the steps of the Web page health detection method according to any of the above-mentioned method embodiments.

[0116] In the embodiment, the device can be a vehicle-mounted computer, a PC (Personal Computer), a smart phone, a tablet computer, a palm computer, a portable computer, or other terminal devices.

[0117] The device can include a memory 11, a processor 12, and a bus 13.

[0118] The memory 11 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 11 can be an internal storage unit of the device in some embodiments, such as a hard disk of the device. The memory 11 can also be an external storage device of the device in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 can include both an internal storage unit and an external storage device. The memory 11 can be used to store application software installed in the device and various data, such as program codes for performing the web page health detection method, etc., and to temporarily store data that has been output or will be output. The processor 12 can be a central processing unit (CPU) in some embodiments.

[0119] The processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments, and is used to run program codes stored in the memory 11 or process data, such as program codes for performing the web page health detection method, etc.

[0120] The bus 13 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 3 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0121] Further, the device can also include a network interface 14, which can optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually used to establish a communication connection between the device and other electronic devices.

[0122] Optionally, the device can further comprise a user interface 15, which can include a display, an input unit such as a keyboard, and optionally a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, for displaying information processed in the device and for displaying a visualized user interface.

[0123] Figure 3 Only the device with components 11-15 is shown, and those skilled in the art can understand that, Figure 3 The structure shown does not constitute a limitation on the device, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0124] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the embodiments of the present application also provide a computer readable storage medium, which stores computer instructions for causing the computer to execute the method of any of the above embodiments.

[0125] The computer readable medium of the embodiments of the present application includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0126] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method of any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which are not described here.

[0127] It should be noted that each of the embodiments of the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be mutually referred to, and each of the embodiments focuses on the differences from other embodiments. In particular, for the method, device, electronic device and medium, since they are basically similar to the method embodiment, they are described more simply, and the relevant parts can refer to the part of the description of the method embodiment. The above-described method, device, electronic device and medium are only illustrative, and the units described as separate components can or can not be physically separated, and the components prompted as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to the actual needs. Those skilled in the art can understand and implement it without creative labor.

[0128] The above describes only one specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present application without creative labor, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / ahref>

Claims

1. A method for detecting the health of a web page, characterized in that, The method comprises the following steps: Based on the obtained full source code, the target page to be detected is determined by link crawling; The target page is respectively subjected to page running detection and data security detection to determine the page running detection result and the data security detection result corresponding to the target page; A pre-configured large language model is used to determine the target detection result based on the page running detection result and the data security detection result corresponding to the target page.

2. The method of claim 1, wherein, If the page running detection result includes rendering performance detection result, resource access detection result, network link detection result and resource loading configuration detection result, the page running detection of the target page is performed to determine the page running detection result corresponding to the target page, which comprises: Based on the collected multiple rendering performance indicators of the target page, a dynamic weight algorithm is used to determine the abnormal score of the target page; If the abnormal score of the target page is greater than a first preset threshold, it is determined that the rendering performance detection result is rendering performance abnormality; The network request data of the target page is obtained; the network request data includes HTTP response status code, TCP connection time consumption and SSL handshake time consumption; If the HTTP response status code is 404 or 500, it is determined that the resource access detection result is service response abnormality; If the TCP connection time consumption is greater than a second preset threshold or the SSL handshake time consumption is greater than a third preset threshold, it is determined that the network link detection result is network link abnormality; The configuration data of the static resource in the target page is extracted; the configuration data of the static resource includes HTTP cache policy and file compression rate; Based on the fact that the static resource is not configured with Cache-Control field or the file compression rate is less than a fourth preset threshold, it is determined that the resource loading configuration detection result is resource loading configuration abnormality.

3. The method of claim 1, wherein, The data security detection result includes data compliance detection result, variant sensitive word and potential risk point, and the data security detection of the target page to determine the data security detection result corresponding to the target page comprises: The source code corresponding to the target page in the full source code is parsed by using abstract syntax tree to determine the hard-coded key and sensitive information; The hard-coded key and sensitive information are subjected to data compliance verification by using regular expression to determine the data compliance verification result; Based on a preset sensitive word library, the content of the target page is subjected to semantic association analysis by using a natural language processing model to identify the variant sensitive word in the target page; Through static code feature analysis, the potential risk point in the source code corresponding to the target page is determined.

4. The method according to any one of claims 1 to 3, characterized in that, The method comprises the following steps: The HTML and JavaScript source code of the target page are subjected to structured parsing by using abstract syntax tree to obtain source code structure data; Based on the source code structure data, the page running detection result and the data security detection result, a multi-dimensional analysis context is generated; inputting the multi-dimensional analysis context into a pre-configured large language model, and outputting a model analysis result; the model analysis result includes a problem type, a specific code location, and a preliminary root cause; determining the target detection result based on the model analysis result.

5. The method of claim 1, wherein, The full-amount source code includes static source code and dynamic source code. Based on the obtained full-amount source code, a target page to be detected is determined through link crawling, including: extracting a hard-coded link based on the static source code; constructing a dynamic interaction link based on a script source code triggering an Ajax and XHR request in the dynamic source code; generating a potential routing path based on a SPA framework routing configuration source code in the dynamic source code; determining the target page to be detected based on a preset user domain name whitelist, a regular expression, and a semantic similarity matching algorithm, according to the hard-coded link, the dynamic interaction link, and the potential routing path.

6. The method of claim 5, wherein, The method further includes: preprocessing the hard-coded link, the dynamic interaction link, and the potential routing path to obtain a first link; filtering out candidate links belonging to the internal of the Web application to be detected by matching the link domain name through the regular expression based on the preset user domain name whitelist; determining the target page to be detected from the candidate links by using the semantic similarity matching algorithm.

7. The method of claim 1, wherein, If the target page is multiple, the method further includes: based on a user preset initial detection process upper limit, a real-time collected memory usage rate, and a CPU load, performing sharding and distribution on detection tasks corresponding to the multiple target pages to obtain multiple shards; allocating a sandbox environment for the target page detection task of each shard in the multiple shards by using a BrowserContext isolation technology; based on a FIFO principle, pausing a target page detection task corresponding to a low priority in the multiple target pages when the memory usage rate exceeds a first threshold or the CPU load exceeds a second threshold; resuming the execution of the paused target page detection task when the memory usage rate is less than or equal to the first threshold and the CPU load is less than or equal to the second threshold.

8. A web page health detection apparatus, characterized by comprising: The device includes: a processor, and a memory connected in communication with the processor; the memory stores computer execution instructions; ​ 9. An electronic device, comprising: ​ ​ The processor executes computer-executable instructions stored in the memory to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method of any one of claims 1-7.

Images