Webpage anomaly detection method and device, equipment, storage medium and program product

Abstract

The invention discloses a webpage anomaly detection method and device, equipment, a storage medium and a program product, and the method comprises the steps: generating a semantic vector based on response data in a webpage, and inputting the semantic vector into a first anomaly detection model corresponding to the webpage, the first probability density of the semantic vector in the normal semantic space of the webpage is estimated through a first anomaly detection model, the first anomaly detection model is a model obtained through training based on historical normal semantic vectors of the webpage, the historical normal semantic vectors are generated based on historical normal response data of the webpage, and the first probability density of the semantic vector in the normal semantic space of the webpage is obtained based on the first probability density; and determining whether the webpage is abnormal or not. According to the embodiment of the invention, the abnormity of the semantic level in the webpage can be identified, and the accuracy of webpage abnormity detection is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network data processing, and particularly relates to a webpage anomaly detection method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] With the continuous popularity of the Internet, web pages have become one of the main channels for people to obtain information. However, events such as webpage tampering that lead to webpage anomalies also occur frequently, affecting the accuracy and safety of people's information intake. Therefore, in order to improve the accuracy and safety of data, it is necessary to detect and maintain the abnormal situation of the webpage.

[0003] In related technologies, the processing method for webpage anomalies mainly includes analyzing user behavior data based on rules to determine whether there is an anomaly, or matching the file path of the rewriting function based on rules to determine whether data tampering has occurred. The above processing method relies on the collection of user behavior data or the monitoring of file system changes, and it is difficult to detect anomalies at the content level, has a detection blind area, and has a low detection accuracy. SUMMARY

[0004] The embodiments of the application provide a webpage anomaly detection method, device, equipment, storage medium and program product, which can identify semantic-level anomalies in web pages and improve the accuracy of webpage anomaly detection.

[0005] In a first aspect, the embodiments of the application provide a webpage anomaly detection method, comprising:

[0006] generating a semantic vector based on response data in the webpage;

[0007] inputting the semantic vector into a first anomaly detection model corresponding to the webpage, estimating a first probability density of the semantic vector in a normal semantic space of the webpage through the first anomaly detection model, the first anomaly detection model being a model trained based on historical normal semantic vectors of the webpage, and the historical normal semantic vectors being generated based on historical normal response data of the webpage;

[0008] determining whether the webpage has an anomaly based on the first probability density.

[0009] In a second aspect, the embodiments of the application provide a webpage anomaly detection device, characterized in that it comprises:

[0010] a semantic vector generation module configured to generate a semantic vector based on response data in the webpage;

[0011] The semantic anomaly detection module is configured to input the semantic vector into a first anomaly detection model corresponding to the webpage, estimate a first probability density of the semantic vector in a normal semantic space of the webpage by the first anomaly detection model, and determine whether the webpage has an anomaly based on the first probability density.

[0012] The anomaly judgment module is configured to determine whether the webpage has an anomaly based on the first probability density.

[0013] In a third aspect, an electronic device is provided, which includes a processor and a memory storing computer program instructions.

[0014] The processor executes the computer program instructions to implement the webpage anomaly detection method of the first aspect.

[0015] In a fourth aspect, a computer readable storage medium is provided, which stores computer program instructions. When the computer program instructions are executed by a processor, the webpage anomaly detection method of the first aspect is implemented.

[0016] In a fifth aspect, a computer program product is provided. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device performs the webpage anomaly detection method of the first aspect.

[0017] The webpage anomaly detection method, device, electronic device, storage medium and product provided in the embodiments of the present application can effectively identify the semantic anomalies in the webpage content by the first anomaly detection model, thereby improving the detection capability of the webpage content tampering and improving the accuracy of the webpage anomaly detection. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. Those skilled in the art can obtain other drawings according to these drawings without any creative labor.

[0019] Figure 1 is a flowchart of the webpage anomaly detection method provided by some embodiments of the present application;

[0020] Figure 2is a flowchart of a web page anomaly detection method provided by some embodiments of the present application.

[0021] Figure 3 is a structural diagram of a web page anomaly detection apparatus provided by some embodiments of the present application.

[0022] Figure 4 is a structural diagram of an electronic device provided by some embodiments of the present application. DETAILED DESCRIPTION

[0023] The features and exemplary embodiments of various aspects of the present application will be described below in detail, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of these specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0024] It should be noted that, in this paper, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0025] Before the present application embodiments are further described, the terms and names involved in the present application embodiments are explained, which are applicable to the following explanations.

[0026] Web application refers to an application program running on a server accessed through a browser. Web application includes multiple web pages.

[0027] Hypertext Markup Language page is a Web content carrier based on HTML (HyperText Markup Language), which is the technical basis of web pages in Web application.

[0028] An Application Programming Interface (API) is a predefined set of functions designed to provide applications and developers with the ability to access a set of routines based on certain software or hardware, without needing access to the source code or understanding the details of the internal workings. Web pages in web applications retrieve dynamic data through APIs.

[0029] Before providing a more detailed description of the embodiments of the present invention, the methods for handling webpage anomalies in related technologies are introduced. As mentioned above, the methods for handling webpage anomalies in related technologies mainly include analyzing user behavior data based on rules to determine whether anomalies exist, or matching the file path of the rewriting function based on rules to determine whether data tampering has occurred. For example, setting up tracking points in a web application to collect and report user behavior data, analyzing the user behavior data according to preset alarm rules, and determining whether an anomaly alarm needs to be triggered based on the analysis results. Alternatively, when a target file is rewritten, obtaining the rewriting function corresponding to the target file, determining the file path of the rewriting function, performing rule matching on the file path, determining whether the rewriting function is a hijacking operation, or matching user information in a whitelist, and performing corresponding anti-tampering operations based on the determination or matching results.

[0030] However, the aforementioned methods primarily rely on data collection of user behavior or monitoring of file system changes, making it difficult to detect anomalies in webpage content. For example, they struggle to identify instances where webpage content has been tampered with due to unauthorized modification, injection, or deletion. Furthermore, these technologies mainly rely on keyword matching, path detection, or format verification for detection, lacking effective monitoring mechanisms for dynamically generated content and a deep understanding of content semantics. They are unable to detect semantic anomalies, such as missing key information or logical errors—subtle but crucial issues. In particular, they cannot detect content with correct formatting but semantic inconsistencies. Moreover, they struggle to distinguish between reasonable and abnormal changes in content, easily leading to false positives or false negatives.

[0031] Furthermore, the relevant technologies generally rely on predefined rules for anomaly detection and anti-tampering. The comprehensiveness and accuracy of the rules directly affect the monitoring effect. Usually, rule updates are mainly based on the problems that have been discovered, lacking proactive learning and predictive capabilities. Even if a whitelist mechanism is introduced, the rules and whitelist still need to be maintained manually. Therefore, it is difficult to adapt to rapidly changing page content, resulting in poor monitoring accuracy.

[0032] In view of this, in order to improve the accuracy of webpage anomaly detection, this application provides a webpage anomaly detection method, apparatus, device, storage medium and program product, which captures the semantic features of response data in web applications, constructs an adaptive anomaly detection model, and realizes accurate monitoring of webpage functional anomalies and content integrity.

[0033] Referring to Figure 1 A flowchart of a webpage anomaly detection method provided for some embodiments of the present application is shown in FIG. 1, which includes the following steps S110-S130. Figure 1

[0034] S110. Generating a semantic vector based on response data in the webpage.

[0035] The webpage in step S110 above refers to a webpage currently subjected to anomaly detection, hereinafter referred to as the current webpage. The current webpage can be a webpage in a web application, wherein the response data includes data returned by the server after receiving a request of the web application.

[0036] The current webpage can be a static webpage or a dynamic webpage. The static webpage refers to a webpage with fixed content, which returns the same content each time it is accessed, such as a company introduction page, etc. The static webpage is composed of pure HTML and CSS and does not depend on backend data. The dynamic webpage refers to a webpage with content dynamically generated according to user requests or data changes, such as a user personal center page, etc. The dynamic webpage is supported by a backend API.

[0037] The response data of the static webpage can include HTML documents, CSS style sheets, JavaScript scripts, static resource files (pictures, fonts, etc.), and other content.

[0038] The response data of the dynamic webpage can include the response content of the server to the API request, i.e., API response data, which can include content in structured formats such as JSON and XML.

[0039] In anomaly detection of the current webpage, the text in the response data of the current webpage is mapped to a high-dimensional vector space, thereby generating a semantic vector. The semantic vector can represent the semantic information of the response data.

[0040] S120. Inputting the semantic vector into a first anomaly detection model corresponding to the webpage, and determining a first probability density of the webpage based on the distribution difference between the semantic vector and a normal semantic space by the first anomaly detection model.

[0041] The first anomaly detection model is a model for identifying semantic anomalies of the webpage, which detects whether the input semantic vector is abnormal by analyzing the distribution of the input semantic vector in the normal semantic vector space.

[0042] Considering that the contents of different webpages are different to some extent, in order to improve the accuracy of semantic anomaly recognition, different first anomaly detection models are trained for different webpages. In anomaly detection of the current webpage, the first anomaly detection model corresponding to the current webpage is used for anomaly detection, which can improve the accuracy of anomaly detection.​

[0043] For each webpage, the corresponding first anomaly detection model can be trained based on the historical normal semantic vector of the webpage. By training the model based on the historical normal semantic vector, the trained first anomaly detection model can capture the normal semantic variation range of the corresponding webpage, i.e., the normal semantic space, so as to estimate the probability density of the input semantic vector in the normal semantic space, thereby judging whether the input semantic vector is abnormal. The historical normal semantic vector refers to a semantic vector generated based on the historical normal response data of the current webpage. The normal response data refers to the response data when the current webpage is normal. Based on this, after obtaining the semantic vector of the current webpage through the above step S110, the semantic vector is input into the first anomaly detection model corresponding to the current webpage, and the first probability density of the semantic vector in the normal semantic space of the current webpage is estimated by the first anomaly detection model. The first probability density is used to represent the probability that the input semantic vector belongs to the normal semantic space. The greater the first probability density, the more the input semantic vector conforms to the distribution space of the normal semantic vector, and the more likely it belongs to the normal semantic vector. The smaller the first probability density, the more the input semantic vector deviates from the distribution space of the normal semantic vector, and the more likely it is an abnormal semantic vector.

[0044] In some embodiments, for each webpage, the corresponding first anomaly detection model can be updated as new normal response data of the webpage is collected, thereby realizing the self-adaptation of the first anomaly detection model to different types of content change patterns.

[0045] S130. Determine whether the webpage has an anomaly based on the first probability density.

[0046] As mentioned above, the first probability density is used to represent the probability that the semantic vector belongs to the normal semantic space. Based on this, it can be determined whether the semantic vector generated based on the response data of the current webpage is an abnormal semantic vector outside the normal semantic space based on the first probability density. If the semantic vector is an abnormal semantic vector, it means that the response data of the current webpage has a semantic anomaly, thereby determining that the current webpage has an anomaly.

[0047] In the embodiments of the present application, when detecting the anomaly of a webpage, a semantic vector is generated based on the response data in the webpage, the semantic vector is input into the first anomaly detection model corresponding to the webpage, and the first probability density of the webpage is determined by the first anomaly detection model based on the distribution difference between the semantic vector and the normal semantic space of the webpage. The first probability density can represent the probability that the webpage has a semantic anomaly. Based on the first probability density, it is determined whether the webpage has an anomaly. In this way, the first anomaly detection model can effectively identify the semantic anomalies in the content of the webpage, thereby improving the detection capability of the content tampering of the webpage and improving the accuracy of the anomaly detection of the webpage.

[0048] In some embodiments, to improve the accuracy of the semantic vector, in step S110, the semantic vector can be generated by steps S1101-S1102 as follows.

[0049] Step S1101. Extract the core text content of the webpage from the response data of the webpage.

[0050] The core text content refers to the part of the text that can convey the main theme and key information of the webpage. The core text content can include but is not limited to the title, the main text, the key explanatory text, etc.

[0051] Different types of webpages can extract core text content in different ways.

[0052] In the case of a static webpage including an HTML document, the core text content can be extracted by steps a1-a3 as follows.

[0053] Step a1. Extract the key content area from the HTML document.

[0054] The key content area in the HTML document can be extracted using a DOM selector. The key content area refers to the layout block in the HTML document that directly serves the theme of the webpage, distinguished from auxiliary areas such as navigation bar, footer, etc. The key content area can contain the core target information for users to access the webpage, which can be defined by HTML semantic tags, CSS / JS styles. Taking the HTML document of a shopping website as an example, the key content area therein can include content such as product details, search result list, account information, etc.

[0055] Step a2. Filter out the first elements in the key content area to obtain the target content area, the first elements including elements that change dynamically and do not affect the core function of the static webpage.

[0056] The core function of the static webpage refers to the main, valuable service or operational capability provided by the webpage to the user. Taking the static webpage of a shopping website as an example, its core function can include functions such as product search, purchase, online payment, order management, etc.

[0057] The first elements are elements in the HTML document that change dynamically and do not affect the core function of the static webpage. For example, time stamp, random recommended content, etc. The first elements are independent of the semantics of the HTML document, and by filtering out the first elements, the interference of irrelevant elements can be reduced, thereby improving the accuracy of the final generated semantic vector.

[0058] Step a3. Extract the text content and key attributes from the target content area as the core text content.

[0059] After obtaining the target content region, the HTML structure of the target content region is normalized to retain the text content and key attributes therein, and the retained text content and key attributes are taken as the core text content of the current static web page. The key attributes can include link addresses, resource paths, picture descriptions, form / element identifiers, style and function identifiers, and other attributes that are helpful for semantic understanding.

[0060] In the case where the web page is a dynamic web page including application program service interface response data, the core text content can be extracted through steps b1-step b2 as follows.

[0061] Step b1. Extract the response body data from the application program service interface response data.

[0062] For API response data in JSON, XML, or other formats, the response body data is extracted therefrom, and metadata such as pagination information and request IDs is ignored.

[0063] Step b2. Extract the key business fields and the values of the key business fields from the response body data as the core text content of the dynamic web page.

[0064] The key business fields refer to fields in the API return result that play a key role in the implementation of business logic, data processing, and satisfaction of business requirements. These fields are directly related to business scenarios and can reflect the core information of the business. Exemplarily, the key business fields can include fields for uniquely identifying business objects, status class fields, data content class fields, relationship class fields, and the like.

[0065] The values of the key business fields are specific data contents stored in the key business fields. The data types of the values of the key business fields can include, but are not limited to, numerical types, string types, Boolean types, array types, and the like. For array type data, sorting information can be optionally retained or only the content set of the key business fields can be extracted.

[0066] After the core text content is extracted, the extracted core text content can be further standardized, for example, to remove redundant white spaces, unify encodings, standardize numerical and date formats, and the like, so as to reduce the interference of non-semantic factors.

[0067] Step S1102. Generate a semantic vector based on the core text content.

[0068] After obtaining the core text content, the core text content is converted into a numerical vector in a high-dimensional space, and the numerical vector is taken as the semantic vector. The dimension of the semantic vector can be between 768 and 1536, so that the semantic information of the text can be fully captured.

[0069] In some embodiments, the obtained core text content can be input into a semantic vector generator, and the core text content is converted into a semantic vector by the semantic vector generator. In some examples, a large language model such as text-embedding-ada-002 of OpenAI or an open source model can be used as the semantic vector generator.

[0070] In some embodiments, the core text content can be converted into a semantic vector using a pre-trained model corresponding to the business field to which the content in the webpage belongs, according to the business field to which the content in the webpage belongs. For example, a semantic vector is generated using the hidden layer representation of a pre-trained model such as BERT, RoBERTa, etc. In this way, the semantic understanding of specific business content can be improved.

[0071] In some embodiments, when generating a semantic vector based on core text content, TF-IDF (Term Frequency-Inverse Document Frequency) features and deep semantic features of the core text content can be extracted, and a semantic vector is generated based on the mixed representation of the TF-IDF features and the deep semantic features. In this way, the semantic vector can combine the advantages of statistical characteristics and semantic understanding, thereby improving the accuracy and richness of semantic representation.

[0072] For longer core text content, the core text content can be first split into multiple subtexts, and a semantic vector is generated for each subtext. The semantic vectors of the multiple subtexts are spliced to serve as the semantic vector of the core text content. In this way, the strategy of segmented encoding and merging can ensure that key semantic information is not lost.

[0073] After generating the semantic vector, the generated semantic vector can be stored together with the corresponding core text content, URL / API identifier, timestamp, and other metadata for subsequent model learning and anomaly detection.

[0074] In some embodiments, for each webpage, the first anomaly detection model corresponding to the webpage can be trained through the following steps c1-step c4.

[0075] Step c1. Determine the response data distribution characteristics of the webpage.

[0076] The response data distribution characteristics of the webpage are used to indicate the state type and number of the webpage, the data distribution complexity, the parameterization of the webpage, and other information that can reflect the frequency and pattern of changes in the content of the webpage.

[0077] The response data distribution characteristics of the webpage can be determined by obtaining the historical response data of the webpage and analyzing the historical response data of the webpage through clustering analysis or other algorithms.

[0078] Step c2. Take the model matching the response data distribution characteristics as the initial model corresponding to the webpage.

[0079] The model matching the response data distribution characteristics refers to a model that can adapt to the content change frequency and mode of the webpage. Different webpages have different response data distribution characteristics, and the matching models are also different.

[0080] For example, for a webpage that may have multiple normal states, such as the homepage of a website containing a login state and a non-login state, a Gaussian Mixture Model (GMM) can be used as the model matching it.

[0081] For example, for a webpage with complex or non-parametric data distribution, a Kernel Density Estimation (KDE) model or an Autoencoder can be used as the model matching it.

[0082] For example, for a webpage with relatively simple data distribution, a simpler model such as a Single Gaussian Model (SGM), One-Class SVM, or Isolation Forest can be used as the model matching it.

[0083] In order to more conveniently determine the initial model, a correspondence between response data distribution characteristics and models can be pre-set, and the model matching the response data distribution characteristics of the webpage is determined based on the correspondence, so that the matching model is determined as the initial model corresponding to the webpage.

[0084] In the above manner, for a webpage with frequent content changes, a more "dispersed" model can be used as the model matching it, and for a relatively stable webpage, a more "compact" model can be used as the model matching it. In this way, the first anomaly detection model finally trained can adapt to the content change frequency and mode of the corresponding webpage, thereby more accurately identifying semantic anomalies in the webpage content.

[0085] Step c3. Obtain the historical normal semantic vector of the webpage.

[0086] As described above, the historical normal semantic vector is a semantic vector generated based on the historical normal response data of the webpage. Based on this, the historical normal response data of the webpage can be obtained, and then the historical normal semantic vector can be generated based on the historical normal response data using the steps S1101-S1102 as described above.

[0087] Step c4. Train the initial model based on the historical normal semantic vector to obtain the first anomaly detection model corresponding to the webpage.

[0088] The historical normal semantic vector of the webpage is used as training data to train the initial model, and the parameters in the initial model are optimized, thereby obtaining the first anomaly detection model corresponding to the webpage.

[0089] As mentioned above, different initial models can be used for different webpages, and different initial models can employ different parameter optimization methods.

[0090] For example, for GMM or SGM, the optimal number of components can be determined by Bayesian Information Criterion (BIC) or cross-validation when performing parameter optimization.

[0091] For example, for KDE, the bandwidth parameter can be optimized by cross-validation.

[0092] For example, for other models other than GMM, SGM and KDE, parameter optimization can be performed by regularization processing to prevent overfitting.

[0093] In some embodiments, after obtaining the first anomaly detection model corresponding to the webpage through training, in order to adapt the first anomaly detection model to the normal changes of the webpage, the first anomaly detection model can be updated based on a model updating strategy. Illustratively, the model updating strategy can include one or more of the following three strategies:

[0094] (1) Regular update: set a fixed update period, such as every week or every month, and periodically update the model based on the update period;

[0095] (2) Triggered update: trigger model update when a significant change in the distribution of normal response data of the webpage is detected;

[0096] (3) Incremental update: adjust the model parameters based on the new normal response data of the webpage without complete retraining to adapt to the new data.

[0097] In some embodiments, since the value range of the first probability density is usually very small, the probability density of the normal semantic vector and the first probability density of the abnormal semantic vector are not much different, and it is difficult to distinguish them. If the abnormality is directly judged based on the first probability density, it may be misjudged. In view of this, in the above step S130, the webpage can be determined to be abnormal or normal through the following steps S1301-S1303.

[0098] S1301. Determine the anomaly score of the webpage based on the first probability density.

[0099] For the first anomaly detection model trained by using different initial models, different methods can be used to calculate the anomaly score.

[0100] For example, if the first anomaly detection model is trained by using GMM or SGM, the negative log-likelihood of the first probability density can be used as the anomaly score of the webpage.

[0101] For example, if the first anomaly detection model is trained by using a KDE model, the negative logarithm density of the first probability density or the reciprocal of the density can be taken as the anomaly score.

[0102] For example, if the first anomaly detection model is trained by using a model other than GMM, SGM, and KDE, the distance between the input semantic vector and the normal semantic space can be calculated based on the first probability density, or the probability that the input semantic vector is an abnormal semantic vector can be calculated based on the first probability density, and the calculated distance or probability can be taken as the anomaly score.

[0103] The anomaly score is positively correlated with the abnormality degree of the semantic vector, that is, the higher the anomaly score of the semantic vector, the greater the possibility that the semantic vector is an abnormal semantic vector, and the lower the anomaly score of the semantic vector, the smaller the possibility that the semantic vector is an abnormal semantic vector.

[0104] To compare the abnormality degrees of different web pages, the anomaly score can be standardized. In some examples, the anomaly score can be standardized by using a z-score or min-max standardization method.

[0105] S1302. Compare the anomaly score with the abnormality threshold corresponding to the web page.

[0106] The abnormality threshold is a threshold for determining whether the web page is abnormal. Different web pages can correspond to different abnormality thresholds.

[0107] S1303. In the case where the anomaly score is greater than the abnormality threshold, it is determined that the web page is abnormal.

[0108] In the conventional anomaly detection method, a fixed threshold is usually used, which is difficult to adapt to the dynamic change of the content of the web page, resulting in low accuracy of anomaly detection and easy false positives. In view of this, in order to reduce the false positive rate and improve the accuracy of anomaly detection, in some embodiments, before the above step S1302, the abnormality threshold of the web page can be dynamically determined based on the historical anomaly scores of the web page by the following steps d1-d4.

[0109] Step d1. Obtain the historical anomaly score of the web page. The historical anomaly score is the anomaly score of the historical normal response data of the web page based on the first anomaly detection model.

[0110] Step d2. Determine the initial threshold of the web page based on the historical anomaly score.

[0111] Based on the historical anomaly score of the web page, the score distribution of the historical normal response data of the web page can be determined, and based on this, after obtaining the historical anomaly score of the web page, the score distribution can be determined based on the historical normal response data of the web page, and the initial threshold of the web page can be determined based on the score distribution.

[0112] In some examples, a high percentile (e.g., 99.5%) of the history normal response data score distribution of the webpage can be taken as the initial threshold. The core logic is that the abnormal scores of the normal response data should be concentrated in the low interval, and the scores at the high percentile can be taken as the boundary between the normal response data and the abnormal response data, so that taking the high percentile as the initial threshold can facilitate the differentiation between the normal response data and the abnormal response data. In order to balance the false positives and false negatives, a high percentile in the interval of [95.5%-99.5%] can be selected as the initial threshold, because a too low percentile (e.g., 90%) will result in a too low initial threshold, causing a large number of normal response data to be misjudged as abnormal, and a high false positive rate, while a too high percentile (e.g., 99.99%) will result in a too high initial threshold, causing abnormal response data to be missed due to the scores not exceeding the initial threshold, and a high false negative rate. Taking the high percentile 99.5% as the initial threshold as an example, it means that 99.5% of the normal response data scores are lower than the initial threshold, and only 0.5% of the normal response data can be misjudged as abnormal response data, i.e., the "false positive rate" is 0.5%.

[0113] Step d3. Track the history abnormal scores by the exponentially weighted moving average control chart to obtain the abnormal score trend of the webpage.

[0114] The exponentially weighted moving average (EWMA) control chart is a statistical tool combining time series data weight allocation and process control theory, and the core is to assign different weights to historical data through the "exponential weighting" mechanism, and then identify abnormal fluctuations in the data sequence. Based on this, tracking the history abnormal scores of the webpage by the exponentially weighted moving average control chart can timely discover the trend changes (e.g., continuous rise or fall) of the abnormal scores, and avoid the accumulation of small deviations into major faults.

[0115] Step d4. In the case where it is determined that the abnormal score distribution of the webpage has changed based on the abnormal score trend, adjust the initial threshold to obtain the abnormal threshold corresponding to the webpage.

[0116] In the case where it is determined that the abnormal scores of the webpage have abnormal fluctuations by tracking through the exponentially weighted moving average control chart, it means that the abnormal score distribution of the normal response data of the webpage has changed. In order to more accurately identify the abnormalities of the webpage based on the abnormal threshold, the initial threshold is adjusted based on the abnormal fluctuations, so that the adjusted threshold is taken as the abnormal threshold corresponding to the webpage for corresponding abnormal judgment.

[0117] In some embodiments, in the above step d4, the initial threshold can be adjusted by the following steps e1-e3.

[0118] Step e1. Obtain N scores in the front N positions in a score sequence, the score sequence being a sequence obtained by arranging the historical anomaly scores of the webpage in a time sequence from late to early, and N being a positive integer.

[0119] The historical anomaly scores of the webpage are arranged in a time sequence from late to early to obtain a score sequence, and the sliding window technique is combined to obtain N scores in the front N positions in the score sequence, thereby obtaining N scores closest to the current time, wherein N represents the length of the sliding window, and N is a positive integer greater than 1.

[0120] Step e2. Calculate the statistical indicator value of the N scores.

[0121] The statistical indicator value of the N scores is calculated based on the statistical indicator. The statistical indicator is an indicator that can reflect the statistical characteristics of the N scores, and the statistical indicator can include but is not limited to the median, the average, etc.

[0122] Step e3. Adjust the initial threshold value based on the statistical indicator value to obtain the anomaly threshold value corresponding to the webpage.

[0123] The fluctuation degree of the anomaly score can be determined based on the statistical indicator value of the N scores, and the initial threshold value is adjusted based on the fluctuation degree.

[0124] For example, assuming that the fluctuation degree of the anomaly score is increased by 5%, the initial threshold value is increased by 5% on the basis of the initial threshold value as the adjusted threshold value, that is, as the anomaly threshold value corresponding to the webpage.

[0125] For another example, assuming that the fluctuation degree of the anomaly score is decreased by 10%, the initial threshold value is decreased by 10% on the basis of the initial threshold value as the adjusted threshold value, that is, as the anomaly threshold value corresponding to the webpage.

[0126] In addition, if it is determined based on the anomaly score change trend that the anomaly score distribution of the webpage has not changed, the initial threshold value does not need to be adjusted, and the initial threshold value can be directly determined as the anomaly threshold value corresponding to the webpage.

[0127] In the above manner, the inherent variation law of the webpage response data is learned by statistical analysis and time sequence analysis of the historical anomaly scores of the webpage, and the anomaly threshold value of the webpage is dynamically set based on this. In this way, compared with the fixed threshold value, the anomaly threshold value set dynamically can more accurately identify the normal content fluctuation and abnormal change of the webpage, greatly reduce the false positive rate, and at the same time maintain a high detection rate.

[0128] In some embodiments, different webpages can correspond to different importance levels, and different importance levels of the webpages can adopt different anomaly threshold value setting strategies. The importance level of the webpage can be determined according to the business importance of the webpage, and the more important the business, the higher the importance level.

[0129] For some important level lower web page, the above steps d1-d4 can be used to determine the corresponding abnormal threshold, and for some important level higher web page, that is, for the web page with the preset level, in order to reduce the false alarm caused by the instantaneous abnormal score fluctuation, a more conservative threshold setting method can be used, specifically, on the basis of the above steps d1-d4, the abnormal threshold corresponding to the web page is further limited to be greater than or equal to a preset boundary value. The preset boundary value can be set according to actual needs, which is not limited in the embodiment. The preset level can be set according to the importance level of the web page, for example, the importance level is divided into three levels of unimportant, general important and very important, and the preset level can be the very important level.

[0130] In addition, when determining the abnormal threshold, a multiple sampling confirmation mechanism can be selectively combined to improve the accuracy of the abnormal threshold, thereby reducing the false alarm caused by the instantaneous fluctuation.

[0131] In some embodiments, a model-threshold linkage mechanism can be used. Before performing the above step S1302, it can be judged whether the first abnormal detection model of the web page is updated. In response to the update of the first abnormal detection model, the abnormal threshold corresponding to the web page is updated based on the updated first abnormal detection model. In this way, when the first abnormal detection model is updated, the abnormal threshold can be recalculated by using the above steps d1-d4, the correspondence between the model version and the threshold is maintained, and consistency is ensured.

[0132] In some embodiments, before the above step S110, the following steps can be performed:

[0133] Context information of the response data is obtained.

[0134] Based on this, in the above step S110, the semantic vector can be generated based on the response data and the context information of the web page.

[0135] The context information of the response data can include but is not limited to user session state, time factor and other context information.

[0136] By increasing the perception detection of the context of the response data, considering the user session state, time factor and other context information, the semantic vector is more accurate, thereby further improving the accuracy of semantic anomaly detection.

[0137] In some embodiments, before step S110, a low-cost detection method can be used to perform coarse-grained and rapid abnormal screening on the webpage. If it is determined through screening that the webpage may have an abnormality, step S110 is performed to analyze the webpage more finely and accurately determine whether the webpage has an abnormality. In some examples, before step S110, historical normal response data of the webpage can be obtained, the current response data of the webpage is compared with the historical normal response data to obtain difference data, and if the difference data meets a preset condition, step S110 is performed. The preset condition can be set according to actual needs, for example, the preset condition can include that the proportion of the difference data is greater than a proportion threshold.

[0138] By comparing the current response data and the historical normal response data, if the difference data of the two meets a preset condition, it is determined that the webpage may have an abnormality. At this time, the response data of the webpage is analyzed finely.

[0139] Through the above hierarchical detection method, the detection range can be quickly narrowed by using a low-cost detection method, and the webpage abnormality can be accurately identified by using a high-cost detection method, which not only avoids the waste of full data calculation, but also improves the identification ability of complex abnormalities.

[0140] In some embodiments, when detecting the abnormality of the webpage, in addition to detecting based on the semantic vector of the text in the response data, the visual elements (such as layout, image) in the response data can also be combined for comprehensive detection. Specifically, before step S130, the following steps can also be performed:

[0141] A visual feature vector is generated based on the visual elements in the response data, the visual feature vector is input into the second abnormality detection model corresponding to the webpage, and the second probability density of the webpage is determined based on the distribution difference between the visual feature vector and the normal visual feature space by the second abnormality detection model.

[0142] The second abnormality detection model is a model trained based on historical normal visual feature vectors of the webpage, and the historical normal visual feature vectors are generated based on historical normal response data of the webpage.

[0143] Correspondingly, in step S130, the first probability density and the second probability density can be used to determine whether the webpage has an abnormality.

[0144] Here, the training method of the second abnormality detection model is similar to that of the first abnormality detection model, and only the training data is replaced from the historical normal semantic vector to the historical normal visual feature vector. For specific training process, please refer to the description of the training process of the first abnormality detection model in the above embodiments. To avoid repetition, it will not be described here.

[0145] In the step of determining whether the webpage is abnormal based on the first probability density and the second probability density, the abnormal score of the webpage can be determined based on the first probability density and the second probability density respectively, and then the abnormal scores are weighted and summed, and the result of the weighted sum is taken as the final abnormal score of the webpage, and whether the webpage is abnormal is determined based on the final abnormal score.

[0146] It can be understood that, in the step of determining whether the webpage is abnormal based on the first probability density and the second probability density, the abnormal threshold value is also adjusted accordingly, and the abnormal threshold value is determined based on the score distribution of the historical normal semantic vector of the webpage and the score distribution of the historical normal visual feature vector of the webpage, and the logic of determining the abnormal threshold value is the same as that of steps d1-d4, and specific descriptions can be referred to the description of steps d1-d4 above, which will not be described in detail here.

[0147] In the above manner, the webpage is comprehensively detected by combining the text and visual elements of the response data, so as to further improve the accuracy of the abnormal detection.

[0148] In some embodiments, as shown in FIG. 13, in order to facilitate positioning of the abnormality in the webpage, the following steps S140-S160 can also be performed. Figure 2

[0149] S140. In the case where the webpage is abnormal, reference data samples are obtained from the historical normal response data of the webpage.

[0150] The reference data samples are representative normal response data in the historical normal response data of the webpage. Different ways can be used to obtain the reference data samples for different webpages.

[0151] For example, if the first abnormal detection model and / or the second abnormal detection model corresponding to the webpage is trained by using GMM or SGM, the normal response data corresponding to the center points of the Gaussian components can be selected as the reference data samples.

[0152] For another example, if the first abnormal detection model and / or the second abnormal detection model corresponding to the webpage is trained by using other models except GMM and SGM, the normal response data closest to the center of the current model can be selected as the reference data samples.

[0153] In order to more accurately select the reference data samples, in the case where the webpage is abnormal, the intermediate data generated in the inference process of the first abnormal detection model and / or the second abnormal detection model can be analyzed, so as to preliminarily analyze the abnormal type of the webpage, such as the score deviation degree, the affected vector space dimension, etc., and then the reference data samples can be more accurately screened based on the abnormal type.

[0154] ​S150. input the response data and the reference data sample into the large language model, identify the differences between the response data and the reference data sample through the large language model, and obtain difference information.

[0155] After obtaining the reference data sample, the response data of the webpage and the selected reference data sample are input into the large language model LLM, and the LLM is instructed to perform difference analysis, thereby obtaining difference information, which is used to indicate the problems existing in the response data of the webpage compared with the reference data sample.

[0156] The LLM can perform difference analysis on the response data and the reference data sample for one or more of the following problems a)-e):

[0157] a) functional loss: lack of key information or fields;

[0158] b) content error: data error, inconsistent format;

[0159] c) content tampering: injection or modification of content;

[0160] d) ordering problem: abnormal order of key content;

[0161] e) null value / default value problem: the content should be replaced with a null value or a default value.

[0162] S160. generate an abnormal diagnosis report of the response data based on the difference information.

[0163] After obtaining the difference information, a structured abnormal diagnosis report can be generated based on the difference information.

[0164] The abnormal diagnosis report can include one or more of the following a)-d):

[0165] a) problem description: summarize the detected abnormalities;

[0166] b) difference analysis: detailed list of main differences between the current response data and the reference data sample;

[0167] c) possible causes: speculate possible technical causes leading to the current problem;

[0168] d) suggested measures: provide preliminary repair or further diagnosis suggestions.

[0169] In this way, when detecting that the webpage has an abnormality, the difference between the current abnormal response data and the normal response data is compared using the LLM, and an understandable abnormal diagnosis report is automatically generated, which helps to quickly locate and repair the problem.

[0170] In some embodiments, a continuous learning and self-optimization mechanism can be employed to continuously monitor and optimize the models and / or thresholds used for anomaly detection. Specifically, the false positive rate, false negative rate, and the like of each anomaly detection model can be tracked, the trend of the anomaly score distribution of the web page is recorded, and the degree of fitting of the anomaly detection model to new data is monitored. In addition, feedback information of the response data can also be obtained, the feedback information being used to indicate whether the anomaly detection result of the response data is accurate, and the anomaly detection model and / or the anomaly threshold are adjusted based on the feedback information. The feedback information can include feedback of the anomaly alarm by an operation and maintenance personnel. In addition, the real normal response data of the web page can also be determined based on the feedback information, and the real normal response data is included in the normal sample set, and the anomaly detection model is optimized.

[0171] In addition, a detected anomaly pattern library can also be established, common problem types can be summarized, and characteristic knowledge of different web pages can be accumulated, so as to realize accumulation of the knowledge base, and thus optimize the detection strategy specific to the web page based on the accumulated knowledge.

[0172] To verify the effectiveness of the web anomaly detection method provided in the present application, a simulation test environment is constructed and systematic verification experiments are carried out.

[0173] The test environment constructed includes: a data set: a test data set constructed based on a real production environment, which contains normal samples and abnormal samples, the normal samples including 1000 HTML page and API response data, and the abnormal samples including 320 artificially constructed abnormal response data. The 320 abnormal response data contain abnormalities caused by the following reasons: malicious tampering (such as injection of scripts, dark links, and replacement of sensitive information), functional defects (such as missing of key fields, logical sequence errors, and abnormal default values), format compliance semantic errors (such as confusion of amount units, date format errors, and inconsistency of terms), and the like.

[0174] The web anomaly detection methods in the related art and the web anomaly detection method provided in the embodiments of the present application are respectively used to perform anomaly detection tests based on the above test data, and the test results obtained are as follows:

[0175]

[0176] Through verification of the dynamic adaptability of the anomaly detection model provided in the present application, it is determined that in the scenario of 20% content change of the simulation web page every week, the anomaly detection model provided in the present application only needs 4.2 seconds to update, and the detection accuracy rate remains above 95%.

[0177] Through verification of the self-adaptive ability of the anomaly threshold provided in the present application, it is determined that for high-frequency update pages (such as news headlines), the false positive rate of the dynamic threshold is reduced by 92% compared with the fixed threshold, and for static pages (such as policy announcements), the anomaly detection sensitivity is increased by 8 times.

[0178] Based on the webpage anomaly detection method provided in the above embodiments, the application also provides a specific implementation of a webpage anomaly detection device. Please refer to the following embodiments.

[0179] Referring to Figure 3 , a schematic diagram of the webpage anomaly detection device provided in the embodiments of the application is shown in Figure 3 , the device 300 includes the following modules:

[0180] The semantic vector generation module 301 is configured to generate a semantic vector based on the response data in the webpage.

[0181] The semantic anomaly detection module 302 is configured to input the semantic vector into a first anomaly detection model corresponding to the webpage, estimate a first probability density of the semantic vector in a normal semantic space of the webpage through the first anomaly detection model, and the first anomaly detection model is a model trained based on historical normal semantic vectors of the webpage, and the historical normal semantic vectors are generated based on historical normal response data of the webpage.

[0182] The anomaly judgment module 303 is configured to determine whether the webpage has an anomaly based on the first probability density.

[0183] In some embodiments, the device 300 can further include a model training module configured to:

[0184] determine the distribution characteristics of the response data of the webpage before inputting the semantic vector into the first anomaly detection model corresponding to the webpage and estimating the first probability density of the semantic vector in the normal semantic space of the webpage through the first anomaly detection model;

[0185] determine a model matching the distribution characteristics of the response data as an initial model corresponding to the webpage;

[0186] obtain historical normal semantic vectors of the webpage;

[0187] train the initial model based on the historical normal semantic vectors to obtain the first anomaly detection model corresponding to the webpage.

[0188] In some embodiments, the anomaly judgment module 303 is specifically configured to:

[0189] determine an anomaly score of the webpage based on the first probability density;

[0190] compare the anomaly score with an anomaly threshold value corresponding to the webpage;

[0191] determine that the webpage has an anomaly in a case where the anomaly score is greater than the anomaly threshold value.

[0192] In some embodiments, the anomaly judgment module 303 is further configured to:

[0193] Before comparing the anomaly score with the anomaly threshold corresponding to the webpage, a historical anomaly score of the webpage is obtained, the historical anomaly score being an anomaly score of historical normal response data of the webpage obtained based on the first anomaly detection model;

[0194] An initial threshold of the webpage is determined based on the historical anomaly score;

[0195] The historical anomaly score is tracked by an exponentially weighted moving average control chart to obtain a change trend of the anomaly score of the webpage;

[0196] In a case where it is determined that the anomaly score distribution of the webpage changes based on the change trend of the anomaly score, the initial threshold is adjusted to obtain the anomaly threshold corresponding to the webpage.

[0197] In some embodiments, the anomaly judgment module 303 is specifically configured to:

[0198] N scores located at the front N positions in the score sequence are obtained, the score sequence being a sequence obtained by arranging the historical anomaly scores in a sequence from late to early according to time, and N being a positive integer;

[0199] A statistical index value of the N scores is calculated;

[0200] The initial threshold is adjusted based on the statistical index value to obtain the anomaly threshold corresponding to the webpage.

[0201] In some embodiments, the importance level of the webpage is a preset level, and the anomaly threshold corresponding to the webpage is greater than or equal to a preset boundary value.

[0202] In some embodiments, the apparatus 300 further includes a threshold updating module configured to:

[0203] Before comparing the anomaly score with the anomaly threshold corresponding to the webpage, it is determined whether the first anomaly detection model of the webpage is updated;

[0204] In response to the first anomaly detection model being updated, the anomaly threshold corresponding to the webpage is updated based on the updated first anomaly detection model.

[0205] In some embodiments, the apparatus 300 further includes a report generation module configured to:

[0206] In a case where it is determined that the webpage has an anomaly, a reference data sample is obtained from the historical normal response data of the webpage;

[0207] The response data and the reference data sample are input into the large language model, and differences between the response data and the reference data sample are identified by the large language model to obtain difference information;

[0208] An anomaly diagnosis report of the response data is generated based on the difference information.

[0209] In some embodiments, the semantic vector generation module 301 comprises:

[0210] a text extraction submodule configured to extract core text content of the webpage from the response data;

[0211] a vector generation module configured to generate a semantic vector based on the core text content.

[0212] In some embodiments, the webpage is a static webpage comprising a hypertext markup language document, and the text extraction submodule is specifically configured to:

[0213] extract a key content area from the hypertext markup language document;

[0214] filter out first elements in the key content area to obtain a target content area, the first elements comprising elements that dynamically change and do not affect the core function of the static webpage;

[0215] extract text content and key attributes from the target content area as the core text content of the static webpage.

[0216] In some embodiments, the webpage is a dynamic webpage comprising application program interface response data, and the text extraction submodule is specifically configured to:

[0217] extract response body data from the application program interface response data;

[0218] extract key business fields and values of the key business fields from the response body data as the core text content of the dynamic webpage.

[0219] In some embodiments, the apparatus 300 further comprises an information acquisition module configured to:

[0220] acquire context information of the response data before generating the semantic vector based on the response data in the webpage;

[0221] the semantic vector generation module is specifically configured to:

[0222] generate the semantic vector based on the response data and the context information.

[0223] In some embodiments, the apparatus 300 further comprises an update module configured to:

[0224] acquire feedback information of the response data, the feedback information being used to indicate whether an anomaly detection result of the response data is accurate;

[0225] adjust the first anomaly detection model and / or the anomaly threshold based on the feedback information.

[0226] In some embodiments, the apparatus 300 further comprises a preliminary screening module configured to:

[0227] Before generating the semantic vector based on the response data of the webpage, historical normal response data of the webpage is acquired;

[0228] The response data is compared with the historical normal response data to obtain difference data;

[0229] The semantic vector generation module is specifically configured to:

[0230] In a case where the difference data meets a preset condition, the semantic vector is generated based on the response data.

[0231] In some embodiments, the apparatus 300 further includes:

[0232] The visual feature vector generation module is configured to, before determining whether the webpage is abnormal based on the first probability density, generate a visual feature vector based on visual elements in the response data.

[0233] The visual anomaly detection module is configured to input the visual feature vector into a second anomaly detection model corresponding to the webpage, and determine a second probability density of the webpage based on a distribution difference between the visual feature vector and a normal visual feature space by using the second anomaly detection model, the second anomaly detection model being a model trained based on historical normal visual feature vectors of the webpage, the historical normal visual feature vectors being generated based on historical normal response data of the webpage.

[0234] The anomaly judgment module is configured to:

[0235] Determine whether the webpage is abnormal based on the first probability density and the second probability density.

[0236] The webpage anomaly detection apparatus provided by the embodiments of the present application can implement the method embodiments, and each process is implemented by the webpage anomaly detection apparatus, so as to avoid repetition. Figures 1 to 2 The method embodiments implement each process, and to avoid repetition, the details are not repeated here.

[0237] Figure 4 A hardware structure schematic diagram of an electronic device provided by the embodiments of the present application is shown.

[0238] The electronic device 400 can include a processor 401 and a memory 402 storing computer program instructions.

[0239] Specifically, the processor 401 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits that implement the embodiments of the present application.

[0240] The memory 402 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 402 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 402 can include removable or non-removable (or fixed) media, where appropriate. The memory 402 can be internal or external to the integrated gateway disaster recovery device. In particular embodiments, the memory 402 is non-volatile, solid-state memory. The memory 402 can include read-only memory (ROM), random-access memory (RAM), a magnetic disk storage medium, an optical storage medium, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Accordingly, in general, the memory 402 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform any of the operations described above as being performed by any of the embodiments of the web anomaly detection method.

[0241] The processor 401 implements any of the web anomaly detection methods described above by reading and executing computer program instructions stored in the memory 402.

[0242] In one example, the electronic device 400 can also include a communication interface 403 and a bus 410. As shown, the processor 401, the memory 402, and the communication interface 403 are connected by the bus 410 and complete communication with each other. Figure 4

[0243] The communication interface 403 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0244] ​Bus 410 includes a hardware, software, or both that couples components of the online data traffic metering device to each other. As an example without limitation, bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or combination of busses, or two or more of these busses. Where appropriate, bus 410 can include one or more buses of the same type or as different types. Although the example embodiments described and illustrated herein relate to a particular bus configuration, the present application contemplates any suitable bus or interconnect.

[0245] In addition, in combination with the web page anomaly detection method in the above embodiments, the embodiments of the present application can provide a computer storage medium for implementation. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any of the web page anomaly detection methods in the above embodiments.

[0246] The embodiments of the present application also provide a computer program product, including a computer program, which is executed by a processor to implement any of the web page anomaly detection methods in the above embodiments.

[0247] It needs to be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0248] The functions noted in the structure block diagrams described above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, functional cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium, or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transport information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0249] It is also important to note that the examples mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the steps mentioned above, that is, the steps can be performed in the order mentioned in the examples, or in an order different from the examples, or several steps can be performed simultaneously.

[0250] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing devices to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing devices to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0251] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method for detecting webpage anomalies, characterized in that, include: Generate semantic vectors based on response data in web pages; The semantic vector is input into the first anomaly detection model corresponding to the webpage, and the first probability density of the semantic vector in the normal semantic space of the webpage is estimated by the first anomaly detection model. The first anomaly detection model is a model trained based on the historical normal semantic vector of the webpage, and the historical normal semantic vector is generated based on the historical normal response data of the webpage. Based on the first probability density, it is determined whether the webpage is abnormal.

2. The method according to claim 1, characterized in that, Before inputting the semantic vector into the first anomaly detection model corresponding to the webpage, and estimating the first probability density of the semantic vector in the normal semantic space of the webpage using the first anomaly detection model, the method further includes: Determine the response data distribution characteristics of the webpage; The model that matches the distribution characteristics of the response data is used as the initial model corresponding to the webpage; Obtain the historical normal semantic vector of the webpage; The initial model is trained based on the historical normal semantic vectors to obtain the first anomaly detection model corresponding to the webpage.

3. The method according to claim 1, characterized in that, The step of determining whether the webpage is abnormal based on the first probability density includes: Based on the first probability density, the anomaly score of the webpage is determined; Compare the anomaly score with the anomaly threshold corresponding to the webpage; If the anomaly score is greater than the anomaly threshold, the webpage is determined to be abnormal.

4. The method according to claim 3, characterized in that, Before comparing the anomaly score with the anomaly threshold corresponding to the webpage, the method further includes: Obtain the historical anomaly score of the webpage, wherein the historical anomaly score is the anomaly score of the historical normal response data of the webpage obtained based on the first anomaly detection model; The initial threshold for the webpage is determined based on the historical anomaly score; The historical anomaly scores are tracked using an exponentially weighted moving average control chart to obtain the anomaly score change trend of the webpage. If the distribution of abnormal scores of a webpage changes based on the abnormal score change trend, the initial threshold is adjusted to obtain the abnormal threshold corresponding to the webpage.

5. The method according to claim 4, characterized in that, The step of adjusting the initial threshold to obtain the abnormal threshold corresponding to the webpage includes: Obtain the N scores in the top N positions of the score sequence, where the score sequence is a sequence obtained by arranging the historical abnormal scores in chronological order from back to front, and N is a positive integer; Calculate the statistical index values ​​of the N scores; The initial threshold is adjusted based on the statistical indicator value to obtain the abnormal threshold corresponding to the webpage.

6. The method according to any one of claims 3-5, characterized in that, The importance level of the webpage is a preset level, and the abnormal threshold corresponding to the webpage is greater than or equal to a preset boundary value.

7. The method according to any one of claims 3-5, characterized in that, Before comparing the anomaly score with the anomaly threshold corresponding to the webpage, the method further includes: Determine whether the first anomaly detection model for the webpage has been updated; In response to an update of the first anomaly detection model, the anomaly threshold corresponding to the webpage is updated based on the updated first anomaly detection model.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: If it is determined that the webpage is abnormal, a reference data sample is obtained from the historical normal response data of the webpage. The response data and the reference data sample are input into a large language model, and the differences between the response data and the reference data sample are identified by the large language model to obtain difference information; An anomaly diagnosis report for the response data is generated based on the discrepancy information.

9. The method according to any one of claims 1-5, characterized in that, The generation of semantic vectors based on webpage response data includes: Extract the core text content of the webpage from the response data; Semantic vectors are generated based on the core text content.

10. The method according to claim 9, characterized in that, The webpage is a static webpage that includes a Hypertext Markup Language document; Extracting the core text content of the webpage from the response data includes: Extract key content regions from the hypertext markup language document; Filter out the first element in the key content area to obtain the target content area. The first element includes dynamically changing elements that do not affect the core functions of the static webpage. The text content and key attributes are extracted from the target content area to form the core text content of the static webpage.

11. The method according to claim 9, characterized in that, The webpage is a dynamic webpage that includes application service interface response data; Extracting the core text content of the webpage from the response data includes: Extract the response body data from the application service interface response data; The key business fields and their values ​​are extracted from the response body data and used as the core text content of the dynamic webpage.

12. The method according to any one of claims 1-5, characterized in that, Before generating semantic vectors based on response data in the webpage, the method further includes: Obtain the context information of the response data; The generation of semantic vectors based on response data in web pages includes: A semantic vector is generated based on the response data and the context information.

13. The method according to any one of claims 1-5, characterized in that, The method further includes: Obtain feedback information from the response data, the feedback information being used to indicate whether the anomaly detection result of the response data is accurate; The first anomaly detection model and / or anomaly threshold are adjusted based on the feedback information.

14. The method according to any one of claims 1-5, characterized in that, Before generating semantic vectors from the webpage-based response data, the method further includes: Obtain the historical normal response data of the webpage; By comparing the response data with the historical normal response data, the difference data is obtained; The generation of semantic vectors based on the response data of web pages includes: If the difference data meets the preset conditions, a semantic vector is generated based on the response data.

15. The method according to any one of claims 1-5, characterized in that, Before determining whether the webpage is abnormal based on the first probability density, the method further includes: Generate a visual feature vector based on the visual elements in the response data; The visual feature vector is input into the second anomaly detection model corresponding to the webpage. The second anomaly detection model determines the second probability density of the webpage based on the distribution difference between the visual feature vector and the normal visual feature space. The second anomaly detection model is a model trained based on the historical normal visual feature vector of the webpage. The historical normal visual feature vector is generated based on the historical normal response data of the webpage. The step of determining whether the webpage is abnormal based on the first probability density includes: Based on the first probability density and the second probability density, it is determined whether the webpage is abnormal.

16. A webpage anomaly detection device, characterized in that, include: The semantic vector generation module is used to generate semantic vectors based on response data in web pages. The semantic anomaly detection module is used to input the semantic vector into the first anomaly detection model corresponding to the webpage, and estimate the first probability density of the semantic vector in the normal semantic space of the webpage through the first anomaly detection model. The first anomaly detection model is a model trained based on the historical normal semantic vector of the webpage, and the historical normal semantic vector is generated based on the historical normal response data of the webpage. The anomaly detection module is used to determine whether the webpage has any anomalies based on the first probability density.

17. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the webpage anomaly detection method as described in any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the webpage anomaly detection method as described in any one of claims 1-15.

19. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the webpage anomaly detection method as described in any one of claims 1-15.

Images