Content anti-capturing method based on ISTIO

Abstract

The invention provides an ISTIO-based content anti-capture method, and relates to the technical field of data security, and the method comprises the steps: building an ANP model, carrying out the dynamic analysis on the basis of the relation of a request state, a duration and an associated request, carrying out the dynamic processing and real-time control through the early warning stage and the blocking recognition stage, and carrying out the real-time analysis of the state of the request, the duration and the associated request. On the basis of supporting blocked dynamic release, fine-grained connection control and release are added, data security is powerfully guaranteed, the security and performance of the system are effectively improved, the workload of personnel intervention is reduced, abnormal requests and data capturing behaviors are dynamically recognized, real-time blocking is carried out, and the data capturing efficiency is improved. And meanwhile, a dynamic release mechanism of blocking behaviors is supported, so that the normal access experience of a user is ensured while the network security is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to an ISTIO-based content anti-crawl method. Background Art

[0002] With the rapid development of information technology, the internet has not only greatly broadened the boundaries of information dissemination but has also profoundly transformed the global business ecosystem. In this wave, data, as a new era production factor, has become increasingly valuable, becoming a core element driving corporate innovation, optimizing decision-making, and enhancing competitiveness. Enterprises are building their own data ecosystems, leveraging technologies like big data and artificial intelligence to extract value from massive amounts of data to support diverse business scenarios such as mobile applications, web services, intelligent recommendations, and personalized marketing. However, while utilizing data resources, protecting these assets presents a challenge.

[0003] At the same time, the increasing value of data has led to the proliferation of data scraping, which often operates in a gray area. Data scraping not only infringes on the rights of data owners and disrupts normal market competition, but can also pose serious challenges to website operational stability, enterprise information security, and even personal privacy protection through improper use or leakage of sensitive information. Traditional WAF (Web Application Firewall) proxies and API (Application Programming Interface) application proxies can achieve control through IP addresses and sessions, but they lack real-time and dynamic management capabilities. In complex networking scenarios, this can lead to a high rate of ineffective control and operations. Traditional data scraping prevention strategies, such as feature matching, statistical identification, and IP filtering, can curb illegal scraping to a certain extent, but their limitations are also significant. Feature matching methods rely on predefined feature libraries and are unable to cope with rapidly changing data formats and unknown fields. Statistical identification and IP filtering can inadvertently harm legitimate users. Furthermore, data scrapers can easily circumvent these defenses through technical means (such as using proxy servers, frequently changing IP addresses, and mimicking normal user behavior) to continue their illegal scraping operations. Currently, when performing analysis based on access logs, there is usually a certain lag, and related data causes processing failure, or it is triggered only when data is lost, resulting in some data not appearing in the results, leading to inaccurate crawler identification. Summary of the Invention

[0004] The purpose of the present invention is to provide an ISTIO-based content anti-scraping method, which can dynamically identify abnormal requests and data scraping behaviors, and block them in real time, while supporting a dynamic release mechanism for blocking behaviors, ensuring network security while protecting users' normal access experience.

[0005] The technical solution of the present invention is:

[0006] In a first aspect, the present application provides an ISTIO-based content anti-scraping method, which includes the following steps:

[0007] S1. Collect client request data through the ISTIO gateway and save it in time periods.

[0008] S2. Configure the client according to the collected client request data;

[0009] S3 dynamically monitors configured client request data and issues an alarm when an anomaly is detected in the current user's request. It also collects the request content, response packet, and response status of the current URL and passes them to the APN model for network analysis to obtain data indicators.

[0010] S4. Build a decision matrix based on data indicators and solve the decision matrix to determine whether it is a data scraping request. If it is a data scraping request, block the illegal request through the ISTIO gateway and notify the client. At the same time, add the user's IP address to the warning zone for access restriction.

[0011] S5. Dynamically track users in the warning zone, record the user's access frequency and request interval, and determine through dynamic verification whether the user's access frequency and request interval meet the preset release conditions. If not, maintain the blocking state. If so, trigger the release condition to release the user and restore access rights.

[0012] Furthermore, in step S1, the client request data includes request URL, request time, request parameters, response status and data status.

[0013] Furthermore, in step S2, the process of configuring the client system includes: defining the name of each request of the client, the terminal user identification mode, the response time, the interval time and the associated URL.

[0014] Furthermore, in step S3, the data indicators include response time, request frequency, request frequency of the associated URL, and request success rate.

[0015] Furthermore, in step S4, the process of determining whether it is a data crawling request includes: calculating a weighted average per unit time based on the analyzed data, and calculating a consistency ratio based on the weighted average to perform consistency verification, and confirming it as a data crawling request when the consistency ratio is lower than 6%.

[0016] Furthermore, in step S5, the preset release condition includes that the user's request frequency decreases to a preset threshold, the user's request interval returns to normal, or the user's blocking time reaches a preset blocking threshold.

[0017] Furthermore, the preset blocking threshold is 1 hour.

[0018] In a second aspect, the present application provides an electronic device, comprising:

[0019] a memory for storing one or more programs;

[0020] processor;

[0021] When the above one or more programs are executed by the above processor, an ISTIO-based content anti-scraping method as described in any one of the above first aspects is implemented.

[0022] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an ISTIO-based content anti-scraping method as described in any one of the first aspects above.

[0023] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0024] The present invention provides an ISTIO-based content anti-scraping method. Without using a whitelist for control, the method establishes an ANP model, performs dynamic analysis based on the request status, duration and relationship of related requests, performs dynamic processing and real-time control through two stages of early warning and blocking identification, and adds fine-grained connection control and release on the basis of supporting dynamic release of blocking, which effectively guarantees data security, effectively improves the security and performance of the system, reduces the workload of personnel intervention, realizes dynamic identification of abnormal requests and data crawling behaviors, and performs real-time blocking, and supports a dynamic release mechanism of blocking behaviors, ensuring network security while protecting the user's normal access experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 This is a step diagram of a content anti-crawl method based on ISTIO of the present invention;

[0027] Figure 2To prevent crawling processing flow chart;

[0028] Figure 3 This is a schematic diagram of the network analysis (ANP) model;

[0029] Figure 4 The figure is a schematic structural block diagram of an electronic device.

[0030] Icon: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0031] Explanation of terms:

[0032] URL: (Uniform Resource Locator) Uniform Resource Locator;

[0033] ANP model: Analytic Network Process model.

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0036] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0037] It should be noted that, in this document, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprises..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the elements.

[0038] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features thereof may be combined with each other.

[0039] Example 1

[0040] See also Figure 1 , Figure 1 The figure shows a step diagram of an ISTIO-based content anti-scraping method provided in Example 1 of the present application.

[0041] In a first aspect, the present application provides an ISTIO-based content anti-scraping method, which includes the following steps:

[0042] S1. Collect client request data through the ISTIO gateway and save it in time periods.

[0043] S2. Configure the client according to the collected client request data;

[0044] S3 dynamically monitors configured client request data and issues an alarm when an anomaly is detected in the current user's request. It also collects the request content, response packet, and response status of the current URL and passes them to the APN model for network analysis to obtain data indicators.

[0045] S4. Build a decision matrix based on data indicators and solve the decision matrix to determine whether it is a data scraping request. If it is a data scraping request, block the illegal request through the ISTIO gateway and notify the client. At the same time, add the user's IP address to the warning zone for access restriction.

[0046] S5. Dynamically track users in the warning zone, record the user's access frequency and request interval, and determine through dynamic verification whether the user's access frequency and request interval meet the preset release conditions. If not, maintain the blocking state. If so, trigger the release condition to release the user and restore access rights.

[0047] In this embodiment, the client request data is saved in a time-division manner using a running log.

[0048] As a preferred implementation, in step S1, the client request data includes the request URL, request time, request parameters, response status and data status.

[0049] It's important to note that when collecting client request data through the ISTIO gateway, it's necessary to maintain URL requests within the same time period and their correlation, define relationships between them based on time correlations, distinguish the minimum time interval between URL requests, and mark the duration of a single URL request response. In the microservice model, the parameter names in the URL request are identified and assigned to specific requests, sub-item requests are distinguished, and response keys and values ​​are marked to distinguish their success status.

[0050] As a preferred implementation, in step S2, the process of configuring the client system includes: defining the name of each request of the client, the terminal user identification mode, the response time, the interval time and the associated URL.

[0051] It should be noted that when configuring the system, first define a name for each request, then define the terminal user differentiation mode, and define the response time and the interval time of each terminal for each interface according to the response time of the collected data; then select an associated URL of a URL and define the request time interval between the two; parameter configuration revolves around the response time and success status of the request, manages the request relationship to maintain the effectiveness data, and supports the analysis of abnormal requests according to the effectiveness of the request of the associated relationship.

[0052] As a preferred implementation, in step S3, the data indicators include response time, request frequency, request frequency of the associated URL, and request success rate.

[0053] It should be noted that in step S3, the situation where the current user's request is monitored to be abnormal mainly includes collecting URLs of a single user when the system is running, and finding that the current user's request is abnormal in the associated interval, and then starting the alarm mechanism.

[0054] As a preferred embodiment, in step S4, the process of determining whether it is a data crawling request includes: calculating the weighted average per unit time based on the analyzed data, and calculating the consistency ratio based on the weighted average to perform consistency verification, and confirming it as a data crawling request when the consistency ratio is lower than 6%.

[0055] It should be noted that in the embodiments of the present invention, based on network analysis methods, the network model can be divided into a decision-making layer and a warning layer. The decision-making layer includes a target layer and a basic indicator layer; the warning layer is a sub-indicator layer. The criterion layer is divided into fine-grained indicator processing based on four major indicators. Processing and refinement are performed based on response time, request frequency, associated request frequency, and request success rate. Secondary processing parameters for relevant indicators within a time period are constructed. Secondary processing of indicators is performed based on changes in effectiveness, thereby implementing blocking and notification.

[0056] As a preferred implementation, in step S5, the preset release conditions include the user's request frequency decreasing to a preset threshold, the user's request interval returning to normal, or the user's blocking time reaching a preset blocking threshold.

[0057] As a preferred implementation, the preset blocking threshold is 1 hour.

[0058] Example 2

[0059] like Figure 2 The figure shows the anti-scraping processing flow chart provided by Example 2 of the present invention, the main contents of which are as follows:

[0060] S110, Standardized Access and Baseline Data Collection: System functions are accessed and used in a standardized manner according to standard access methods, ensuring that all network data is collected under normal user usage patterns. This ensures accurate and reliable baseline data and related metadata is obtained. Baseline data is used to confirm the basic functionality of the system and establish foundational data for analysis. Furthermore, through the collection and definition of relevant data, a baseline for normal access behavior is established.

[0061] S120. Dynamic Access and Fluctuation Data Collection: Based on application baseline data, data from related dimensions is collected using business-specific models. Business relationships are dynamically collected and identified, focusing on the correlation of business interfaces. Intervals are defined and data captured, and fluctuating data, such as request frequency and request intervals, is recorded and marked. Furthermore, the basic format of request responses is identified and defined, supporting the identification of request message status, enabling timely detection of abnormal client collection behavior during the request process.

[0062] S130. Determine and maintain access baseline frequency and fluctuation range: By analyzing collected baseline and fluctuation data, as well as accurate data as a basic reference, and using weighted statistics of dynamic fluctuation data, we determine the pre- and post-process dependencies of associated requests and the access fluctuation range based on the necessity of business operations and the relevance of interfaces. This supports analysis of associated interfaces when abnormal access events are triggered. Defining the basic access frequency and associated fluctuation range is the main content of adjusting the model's weights. By setting reasonable interval thresholds, the system can distinguish between normal access and abnormal behavior analysis and identification.

[0063] S140. Trigger prevention events based on the interval and frequency of basic attributes: Based on the interval distribution and request frequency of the basic attributes of the request (such as IP address, user identity, etc.), dynamically track interface requests by client and user within the defined access event cycle, and identify them based on the defined benchmark standards such as access frequency, success rate, repeated access, and associated access interval. If the relevant parameters are not within the benchmark range, a prevention event is triggered. When a request or a series of requests exceeds the preset benchmark frequency or fluctuation range, the system regards it as a potential abnormal event and initiates a further analysis process to confirm whether there is crawling behavior.

[0064] S150. Collect and analyze interface-related indicator data, and use network analysis to determine data capture behavior: When confirming the triggered capture event, integrate and process the requests and related requests in the same time period around the request that triggered the event, obtain the relevant configuration benchmark and interval data, and calculate the access average, maximum, and minimum values ​​of the same time period by segment. At the same time, organize the request frequency, response time, request success rate, request repetition rate, and related request interval according to standard indicators, and process them based on the similarity and similarity of the request content and pattern. Use network analysis (ANP) technology to incorporate the user layer and network layer into the decision control indicators, the user access and response status into the indicator layer, and the event trigger blocking release into the solution control layer. Build an ANP analysis model, and bring the relevant parameters into the model in the form of indicators for analysis to confirm user data capture behavior.

[0065] S160: Restrict access to users, IP addresses, network segments, and terminals involved in data scraping: Upon confirming data scraping, ISTIO directly sends a notification to the client and requests a response. The client must fill in the image content and submit it. After confirming the scraping, the ISTIO gateway will not forward the client request to the application pod. If the client cannot effectively confirm, all client requests will be blocked, with no content response. Within the response range controlled by the configuration parameters, access restrictions will be implemented for the users, IP addresses, network segments, and terminals involved. Abnormal user access will be directly blocked, and the blocking level will be upgraded based on the characteristics of the access, including IP and network segment blocking. By restricting access and not responding on the ISTIO gateway side, access is blocked and illegal data scraping is managed.

[0066] S170: Dynamic Tracking and Dynamic Release of Restricted Behaviors: During access restriction blocking, blocking control is continued based on the number of access requests from the client and user. Dynamic tracking is also used to verify the blocking status and manage blocking behavior. Requests exceeding the configured interval are processed using a secondary network analysis model. Once the abnormal behavior is confirmed to have been eliminated or the user has resumed normal access, the system automatically releases the previous access restrictions, ensuring that legitimate users' normal access is not affected in the long term. This dynamic release mechanism enhances system flexibility, reduces system maintenance workload, and improves user experience.

[0067] Example 3

[0068] like Figure 3 The network analysis model diagram provided in Example 3 of the present invention mainly includes:

[0069] The network analysis model adopts a three-layer processing mode, and the model is defined and processed around indicators, rules and solutions. First, the access frequency, repetition rate, interval, success rate, response time and other indicators on the gateway side are defined to build benchmark indicators. Secondly, based on the collected benchmark data, dynamic indicators are defined around the interval and defined in the form of influence weights to form data items of the weighted super matrix. The waiting time between different requests is defined in the AnBm method, and the correlation of the associated requests is confirmed to form judgment matrix data items. Based on the matching of dynamic correlation indicators, the rationality of whether the current request is a data crawling behavior is confirmed.

[0070] By calculating the timeliness ranking vector of each pair of associated requests in the calculation matrix, and calculating the matching index of the request association, the abnormal access request is determined to be data scraping behavior. After determining the dynamic index of the effectiveness matching of the control layer associated access, the matching degree of the request associated access is confirmed based on the event interval of the cross-access. By comparing the interval with the benchmark interval, the data scraping behavior of the abnormal access is confirmed, and the dynamic confirmation index is screened.

[0071] Preferably, when confirming dynamic indicators, the indicators are sorted around the correlation of a single request. The dynamic indicators in the above processing process are sorted, where the request matching indicators are confirmed around the following parameters: request success rate p1, request repetition p2, correlation timeliness p3x, request interval p4 and request response time p5. Among them, p3x represents the correlation of multiple requests of the current request. The expression of the matching evaluation criterion set F for abnormal request access is:

[0072] F=(p1,p2,p3x,p4,p5)

[0073] Among them, the request association effectiveness indicator is reflected in the following sub-indicators: association request event interval, association request success status, association request ratio, and association request duration;

[0074] Secondly, based on the network analysis method, it is divided into two layers: the first layer is the control layer, and the second layer is the network layer. The first layer focuses on management around goals and criteria, and the second layer focuses on control around self-indicators and plans until the requirements are met.

[0075] Example 4

[0076] See also Figure 4 , Figure 4 This is a schematic structural block diagram of an electronic device provided in Example 4 of the present application.

[0077] An electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0078] Among them, the memory 101 can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0079] The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0080] It is understood that the structure shown in the figure is for illustration only, and an ISTIO-based content anti-scraping method may also include more or fewer components than shown in the figure, or have a different configuration than shown in the figure. Each component shown in the figure can be implemented using hardware, software, or a combination thereof.

[0081] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The embodiments described above are merely illustrative. For example, the flowcharts or block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the methods and computer program products according to multiple embodiments of the application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0082] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0083] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0084] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0085] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A content anti-scraping method based on ISTIO, characterized in that: The following steps are involved: S1. Collect client request data through the ISTIO gateway and save it in time periods. S2. Configure the client according to the collected client request data; S3 dynamically monitors configured client request data and issues an alarm when an anomaly is detected in the current user's request. It also collects the request content, response packet, and response status of the current URL and passes them to the APN model for network analysis to obtain data indicators. S4. Build a decision matrix based on data indicators and solve the decision matrix to determine whether it is a data scraping request. If it is a data scraping request, block the illegal request through the ISTIO gateway and notify the client. At the same time, add the user's IP address to the warning zone for access restriction. S5. Dynamically track users in the warning zone, record the user's access frequency and request interval, and determine through dynamic verification whether the user's access frequency and request interval meet the preset release conditions. If not, maintain the blocking state. If so, trigger the release condition to release the user and restore access rights.

2. The ISTIO-based content anti-crawl method according to claim 1, characterized in that: In step S1, the client request data includes request URL, request time, request parameters, response status and data status.

3. The ISTIO-based content anti-crawl method according to claim 1, characterized in that: In step S2, the process of configuring the client system includes: defining the name of each request of the client, the terminal user identification mode, the response time, the interval time and the associated URL.

4. The ISTIO-based content anti-crawl method according to claim 3, wherein: In step S3, the data indicators include response time, request frequency, request frequency of the associated URL, and request success rate.

5. The method for preventing content from being crawled based on ISTIO according to claim 4, wherein: In step S4, the process of determining whether it is a data crawling request includes: calculating the weighted average per unit time based on the analyzed data, and calculating the consistency ratio based on the weighted average to perform consistency verification, and confirming it as a data crawling request when the consistency ratio is lower than 6%.

6. The ISTIO-based content anti-scraping method according to claim 1, characterized in that: In step S5, the preset release conditions include the user's request frequency decreasing to a preset threshold, the user's request interval returning to normal, or the user's blocking time reaching a preset blocking threshold.

7. The ISTIO-based content anti-crawl method according to claim 6, characterized in that: The preset blocking threshold is 1 hour.

8. An electronic device, characterized in that: include: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, an ISTIO-based content anti-crawl method as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements an ISTIO-based content anti-scraping method as described in any one of claims 1 to 7.

Images