Method and apparatus for focused crawler data theft detection based on semantic context

Abstract

The application relates to a focus crawler data stealing detection method and device based on semantic context, which comprises the following steps: acquiring an initial training data set, analyzing the initial training data set to determine a plurality of first request-response pair data, dividing the first request-response pair data through a first fixed-length sliding window to determine initial sample data, and determining a sample feature matrix comprising first semantic correlation features and first access behavior features according to the initial sample data; and training based on the sample feature matrix to obtain a focus crawler detection model; acquiring target detection data, inputting the target detection data into the focus crawler detection model, and outputting a crawler detection judgment result, so that the technical problem of how to accurately and timely detect the collection process of focus crawlers on network data is solved, the accuracy and robustness of focus crawler detection are improved, and the network data security is improved.

Description

Technical Field

[0001] This application relates to the field of cybersecurity technology, and in particular to a method and apparatus for detecting web crawler data theft based on semantic context. Background Technology

[0002] With the rapid iteration and large-scale application of artificial intelligence (AI) technology, the industry's demand for high-quality data has exploded. Data, as the core foundation for AI model training, fine-tuning, and reinforcement learning (RL), directly determines model performance through its acquisition efficiency and quality. AI model optimization particularly relies on clean corpora from specific domains, and the targeted data collection characteristics of web crawlers make them a key technology for acquiring such high-quality training data, widely used in AI training data collection, industry data research, competitor analysis, and content crawling. However, not all data owners permit their data to be used for unauthorized AI model training. The conflict between data ownership protection and unauthorized collection is becoming increasingly prominent, and the abuse of web crawlers has become a major risk point for infringing on the legitimate rights and interests of data owners, making it a core target for website data security protection.

[0003] Currently, in related technologies, anti-crawler technology and crawler bypass technology are in constant conflict. Crawler programs evade detection through various disguise methods, causing the detection effectiveness of traditional anti-crawler methods to decline continuously. How to accurately identify highly disguised targeted crawlers has become a core research challenge in the current anti-crawler technology field. Existing anti-crawler technologies all focus on detecting the crawler's own disguise characteristics, such as IP and UA-based filtering rule detection, headless browser detection based on JS code injection, feature detection based on machine learning (ML) / deep learning (DL), online Turing test detection based on CAPTCHA, and fingerprint and fingerprint consistency verification based on TCP / IP / application layer. However, the detection in these technologies all target the crawler's own disguise characteristics and fail to grasp the essential behavioral need of crawler programs to "targetedly acquire domain-related data," let alone adapt to the new characteristics of targeted crawlers in the AI ​​era, such as enhanced targeted access and upgraded disguise methods in order to obtain training data.

[0004] The core of the battle between web crawlers and anti-crawler measures lies in the crawler's ability to hide and forge its own characteristics. Current technologies only design detection logic around these surface features, thus various detection methods have corresponding bypass techniques (such as using IP proxy pools to bypass IP filtering, modifying JavaScript to bypass headless browser detection, and using self-developed browsers to ensure fingerprint consistency). However, no matter how a crawler disguises its characteristics, its core objective as a focused crawler remains unchanged: starting from a specified seed URL / domain search term, it selectively acquires as much domain-related web page data as possible while minimizing access to irrelevant pages. This essential behavioral characteristic cannot be concealed by crawlers. Because related technologies have not explored and utilized the essential behavioral need of focused crawlers to "selectively acquire domain-related data," the detection accuracy against highly disguised focused crawlers is low, and their robustness is poor. They are unable to effectively resist unauthorized data collection by focused crawlers, and website data security still faces a serious threat. Therefore, this solution grasps the essential behavioral characteristics of focused crawlers acquiring domain-related data in a targeted manner, and proposes a data theft detection method for focused crawlers based on semantic context, in order to solve the problems that current anti-crawler technologies only target the surface camouflage features of crawlers, are easily bypassed, and have poor detection effects on focused crawlers.

[0005] Currently, there are no effective solutions to the technical problems of how to accurately and in real time detect the process of web crawlers collecting network data, so as to distinguish web crawler access behavior from normal human access behavior, and the fact that existing anti-crawler detection technologies are all based on the crawler's own disguise characteristics. However, highly disguised web crawlers can evade detection by forging the above characteristics, resulting in insufficient detection accuracy and robustness of related technologies. Summary of the Invention

[0006] This application provides a semantic context-based method, apparatus, electronic device, and storage medium for detecting web crawler data theft, to at least solve the problem in related technologies of how to accurately and in real time detect the process of web crawlers collecting network data, so as to distinguish web crawler access behavior from normal human access behavior on the network.

[0007] In a first aspect, embodiments of this application provide a method for detecting web crawler data theft based on semantic context.

[0008] In some embodiments, the semantic context-based focused crawler data theft detection method includes:

[0009] Obtain an initial training dataset and parse the initial training dataset to determine multiple first request-response pairs of data sorted by time;

[0010] The first request-response pair data is divided by a first fixed-length sliding window to determine initial sample data, and a first semantic relevance feature and a first access behavior feature are determined based on the initial sample data.

[0011] The sample feature matrix is ​​determined based on the first semantic relevance feature and the first access behavior feature, and the initial detection model is trained based on the sample feature matrix to obtain a trained focused crawler detection model.

[0012] Acquire the target data to be detected, input the target data to be detected into the focused crawler detection model, and output the crawler detection judgment result.

[0013] In some embodiments, the initial training dataset is a network packet capture file, and obtaining the initial training dataset and parsing the initial training dataset to determine multiple first request-response pairs of data ordered by time includes:

[0014] Obtain network packet capture files, which include focused crawler sample traffic and human access sample traffic;

[0015] The focused crawler sample traffic and the human access sample traffic are analyzed to determine the time-ordered first request-response pairs of data corresponding to each client address.

[0016] In some embodiments, the step of dividing the first request-response pair data through a first fixed-length sliding window to determine initial sample data, and determining the first semantic relevance feature and the first access behavior feature based on the initial sample data includes:

[0017] Determine the preset parameter value corresponding to the first fixed-length sliding window, and divide the first request-response pair data through the first fixed-length sliding window to determine the initial sample data corresponding to each first fixed-length sliding window, wherein each initial sample data includes the preset parameter value and the first request-response pair data;

[0018] The preset parameter values ​​are obtained for the first request-response pair data, including the user agent field, source page field, request Uniform Resource Locator, response status code, response packet length, and request-response pair time interval.

[0019] The hot-unique encoding feature value is determined based on the user agent field, the null value judgment feature value is determined based on the source page field, and the similarity feature value is determined based on the request Uniform Resource Locator. The hot-unique encoding feature value, the null value judgment feature value, and the similarity feature value are then determined as the first semantic relevance feature.

[0020] The response status code feature value is determined based on the response status code, the total length feature value of the response packet is determined based on the response packet length, and the time interval feature value is determined based on the request-response pair time interval. The response status code feature value, the total length feature value of the response packet, and the time interval feature value are then determined as the first access behavior feature.

[0021] In some embodiments, the step of determining a sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and training an initial detection model based on the sample feature matrix to obtain a trained focused crawler detection model includes:

[0022] Based on the first semantic relevance feature and the first access behavior feature, a sample feature matrix is ​​determined that includes the hot unique coding feature value, the null value judgment feature value, the similarity feature value, the response status code feature value, the total length feature value of the response packet, and the time interval feature value;

[0023] The sample feature matrix is ​​input into an initial detection model adapted to the temporal characteristics of the first request response data, and trained to obtain a trained focused crawler detection model.

[0024] In some embodiments, after determining the sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and training the initial detection model based on the sample feature matrix to obtain a trained focused crawler detection model, the following steps are included:

[0025] The focused crawler detection model is stored as a detection model file in a preset format, and the detection model file is deployed to the traffic detection server of the network gateway.

[0026] In some embodiments, acquiring the target data to be detected and inputting the target data to be detected into the focused crawler detection model to output a crawler detection judgment result includes:

[0027] The traffic detection server obtains initial data to be detected through the traffic mirroring function of the network gateway;

[0028] The initial data to be detected is parsed to determine multiple second request-response pairs sorted by time, and the second request-response pairs are divided by a second fixed-length sliding window to determine the preset parameter value of each second request-response pair corresponding to each second fixed-length sliding window;

[0029] Based on the preset parameter values ​​and the second request-response pair data, determine the second semantic relevance feature and the second access behavior feature;

[0030] Based on the second semantic relevance feature and the second access behavior feature, a detection data feature matrix is ​​determined, and the detection data feature matrix is ​​determined as the target data to be detected;

[0031] Each of the detection data feature matrices is input into the focused crawler detection model to output the corresponding crawler probability value for each of the second fixed-length sliding windows. The crawler detection judgment result is determined based on the crawler probability value and then output.

[0032] In some embodiments, the step of inputting each of the detection data feature matrices into the focused crawler detection model to output a corresponding crawler probability value for each of the second fixed-length sliding windows, determining a crawler detection judgment result based on the crawler probability value, and outputting the crawler detection judgment result includes:

[0033] Each of the detection data feature matrices is input into the focused crawler detection model to output the corresponding crawler probability value for each of the second fixed-length sliding windows;

[0034] Obtain M crawler probability values ​​corresponding to M consecutive second fixed-length sliding windows, and determine the average crawler probability value based on the M crawler probability values, where M is a preset integer value greater than 1;

[0035] Obtain a preset crawler probability threshold; if the average crawler probability value is greater than the preset crawler probability threshold, determine that the crawler detection judgment result is that a crawler has been detected, and output the crawler detection judgment result; if the average crawler probability value is less than or equal to the preset crawler probability threshold, determine that the crawler detection judgment result is that a crawler has not been detected, and output the crawler detection judgment result.

[0036] Secondly, embodiments of this application provide a semantic context-based focused crawler data theft detection device.

[0037] In some embodiments, the semantic context-based focused crawler data theft detection device includes a traffic parsing module, a sample feature extraction module, a detection model training module, and a detection result output module:

[0038] The traffic parsing module is used to obtain an initial training dataset and parse the initial training dataset to determine multiple first request-response pairs of data sorted by time.

[0039] The sample feature extraction module is used to divide the first request-response pair data through a first fixed-length sliding window to determine initial sample data, and to determine the first semantic relevance feature and the first access behavior feature based on the initial sample data;

[0040] The detection model training module is used to determine a sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and to train the initial detection model based on the sample feature matrix to obtain a trained focused crawler detection model.

[0041] The detection result output module is used to acquire the target data to be detected, input the target data to be detected into the focused crawler detection model, and output the crawler detection judgment result.

[0042] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the semantic context-based focused crawler data theft detection method as described in the first aspect above.

[0043] Fourthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the semantic context-based focused crawler data theft detection method described in the first aspect above.

[0044] Compared to related technologies, the semantic context-based focused crawler data theft detection method, apparatus, electronic device, and storage medium provided in this application, by acquiring an initial training dataset, parsing the initial training dataset to determine multiple first request-response pairs of data ordered by time, and dividing the first request-response pairs of data by a first fixed-length sliding window to determine initial sample data, and determining first semantic relevance features and first access behavior features based on the initial sample data; determining a sample feature matrix based on the first semantic relevance features and first access behavior features, and training the initial detection model based on the sample feature matrix to obtain a trained focused crawler detection model; acquiring target data to be detected, inputting the target data to be detected into the focused crawler detection model to output a crawler detection judgment result, solves the technical problem of how to accurately and in real time detect the focused crawler's collection process of network data, so as to distinguish between focused crawler access behavior and normal human access behavior in the network, improves the accuracy and robustness of focused crawler detection, effectively resists unauthorized data collection by focused crawlers, and enhances network data security.

[0045] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a hardware structure block diagram of a terminal for a semantic context-based focused crawler data theft detection method according to an embodiment of this application;

[0048] Figure 2 This is a flowchart of a semantic context-based focused crawler data theft detection method according to an embodiment of this application;

[0049] Figure 3 This is a flowchart of a semantic context-based focused crawler data theft detection method according to a preferred embodiment of this application;

[0050] Figure 4 This is a structural block diagram of a semantic context-based focused crawler data theft detection device according to an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0053] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the terminal for the semantic context-based focused crawler data theft detection method according to an embodiment of the present invention. Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0055] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the semantic context-based focused crawler data theft detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0057] This embodiment provides a semantic context-based method for detecting web crawler data theft. Figure 2 This is a flowchart of a semantic context-based focused crawler data theft detection method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0058] Step S201: Obtain the initial training dataset and parse the initial training dataset to determine multiple first request-response pairs of data sorted by time.

[0059] This application embodiment can include two main stages: a model training process and a real-time detection process. The model training process is executed offline, and the trained model file is deployed to a server used for detection, where it is used to calculate the crawler probability in the real-time detection process. Specifically, the main body executing the model training process in this application embodiment is a deep learning training server. The core objective is to build and train a temporal model, enabling the model to determine the crawler probability based on the sliding window feature matrix. After training is completed, the model file is saved and deployed to the gateway's traffic detection server.

[0060] In this embodiment of the application, the initial training dataset obtained during model training can be labeled network traffic packet capture (PCAP) files. The dataset needs to cover traffic in different scenarios and at different times to ensure the diversity of the dataset and improve the generalization ability of the model. Furthermore, the PCAP files are preprocessed to filter out invalid traffic (such as non-HTTP / HTTPS traffic and abnormal network traffic). After obtaining the PCAP file, the PCAP file is parsed to extract TCP / IP and HTTP protocol information from the traffic. The core processing steps are: 1. Parse each data packet in the PCAP file, identify Ethernet frames, IP packets, and TCP packets, and filter data packets that are not TCP / IP protocols; 2. Match HTTP request-response pairs based on the TCP four-tuple (source IP, source port, destination IP, destination port): (1) Define c2s_key (client → server) as the request identifier and s2c_key (server → client) as the response identifier, build a pending_requests queue to store requests that have not matched responses, and an active_responses dictionary to store responses that are being transmitted; (2) If an HTTP request packet is parsed, add it to the pending_requests queue; if an HTTP response packet is parsed, match it with the requests in the pending_requests queue using the four-tuple. If the match is successful, a request-response pair is formed; (3) Accumulate the payload length of the TCP fragmented response packets, and force settlement of the response packets that have not been transmitted after the PCAP file is parsed. Finally, build a completed_sessions dictionary and sort it by client address (Client IP) aggregates all request-response pairs to identify multiple first request-response pair data sorted by time.

[0061] Step S202: Divide the first request-response pair data by using a first fixed-length sliding window to determine the initial sample data, and determine the first semantic relevance feature and the first access behavior feature based on the initial sample data.

[0062] In this embodiment, the first request-response pair data is divided by a first fixed-length sliding window. A preset parameter value N can be set for the first fixed-length sliding window, such that after division, each sliding window includes N request-response pairs. These N request-response pairs are used as initial sample data, and features are extracted from each of the initial sample data. The features are then normalized / encoded to determine the first semantic relevance feature and the first access behavior feature. The first semantic relevance feature reflects the essential characteristics of URLs related to the domain targeted by the crawler, while the first access behavior feature distinguishes the differences between crawler and human access behaviors.

[0063] Step S203: Determine the sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and train the initial detection model based on the sample feature matrix to obtain the trained focused crawler detection model.

[0064] In this embodiment, the values ​​of the first semantic relevance feature and the first access behavior feature are arranged in chronological order of request-response pairs to construct a sample feature matrix, which serves as a single training sample for the initial detection model. The sample feature matrix is ​​input into the initial detection model, and training parameters, such as the number of training rounds, batch size, and validation set percentage, are set. After one training cycle, the model's performance is evaluated on the validation set. If the validation set accuracy meets expectations, the model training effect is deemed satisfactory; otherwise, the model parameters are adjusted and retrained. By training the initial detection model based on the sample feature matrix, a trained focused crawler detection model can be obtained.

[0065] Step S204: Obtain the target data to be detected, input the target data to be detected into the focused crawler detection model, and output the crawler detection judgment result.

[0066] This application embodiment also includes a real-time detection process, which is executed online in real time. The execution entity is a traffic detection server deployed on the network gateway. The core objective is to use the trained focused crawler detection model to detect the real-time access traffic (target data to be detected) of the input gateway mirror, output crawler detection judgment results to determine whether each Client IP is a focused crawler IP, and use the crawler detection judgment results for subsequent crawler management, such as blocking, rate limiting, etc.

[0067] Through the above steps, this application embodiment starts from the unmaskable nature of the targeted data collection of the web crawler, performs standardized feature preprocessing on the acquired initial training dataset, parses the initial training dataset to determine multiple first request-response pairs of data sorted by time, divides the first request-response pairs of data by a first fixed-length sliding window, performs semantic similarity analysis on the request URLs within the sliding window, determines the first semantic relevance features and the first access behavior features, and constructs a sample feature matrix. This transforms the inherent law of the web crawler "accessing URLs with high semantic relevance" into quantifiable detection features, and further trains a targeted web crawler detection model adapted to the traffic time sequence characteristics based on the sample feature matrix, thereby enabling real-time detection of targeted web crawlers. Therefore, this application's embodiments, on the one hand, utilize the semantic relevance of the access URL to characterize its unmaskable targeted collection features, fundamentally solving the problem that existing technologies are easily bypassed due to their reliance on masqueradeable features; on the other hand, it realizes a paradigm shift from "detecting masquerade" to "detecting intent," filling the technical gap in detection using essential behavioral features, achieving efficient collaborative processing throughout the entire process, enhancing the model's scenario generalization ability, improving the robustness and credibility of detection results, achieving real-time detection under high concurrency traffic, and supporting rapid model iteration and optimization to adapt to the iterative changes in crawler technology, thus improving overall operation and maintenance. It is lower in cost and can break through the traditional detection logic that only targets the surface camouflage features of crawlers. By mining the essential behavior of crawlers in acquiring domain data in a targeted manner, it extracts semantically relevant core features and combines them with access behavior features, thereby eliminating the interference of crawler camouflage methods from the root. It significantly improves the detection's anti-bypass capability, accurately distinguishes the differences between crawler and human access behavior, and effectively solves the core problems of traditional anti-crawler technology being easily bypassed and highly coupled with business. It can efficiently resist unauthorized data collection by focused crawlers, protect the website's core data assets, reduce the server's invalid load, and adapt to the detection needs of various network access scenarios.

[0068] In some embodiments, the initial training dataset is a network packet capture file, and step S201 includes:

[0069] Step S2011: Obtain network packet capture files, which include focused crawler sample traffic and human access sample traffic.

[0070] In this embodiment, the initial training dataset can be annotated network packet capture files, i.e., network traffic PCAP files, which contain focused crawler sample traffic and human access sample traffic. The focused crawler sample traffic includes the access traffic of focused crawlers to the target website collected from various fields, saved as a PCAP file, and labeled with a 1; the human access sample traffic includes the normal access traffic of real human users to the target website collected, saved as a PCAP file, and labeled with a 0.

[0071] Step S2012: Analyze the focused crawler sample traffic and human access sample traffic to determine the multiple first request-response pairs of data corresponding to each client address, sorted by time.

[0072] This application embodiment can use the dpkt library to parse PCAP files, or it can use network traffic parsing tools such as Scapy and Tshark. Based on the method of parsing PCAP files, the parsing focuses on crawler sample traffic and human access sample traffic to determine the multiple first request-response pairs of data corresponding to each Client IP, sorted by time.

[0073] Through the above steps, this application embodiment collects focused crawler sample traffic and human access sample traffic in different scenarios and time periods, ensuring the diversity of the dataset, improving the completeness of the samples and the generalization ability of the model, and further parses the PCAP file to determine the multiple first request response pairs corresponding to each Client IP in time order, which is highly feasible.

[0074] In some embodiments, step S202 includes:

[0075] Step S2021: Determine the preset parameter value corresponding to the first fixed-length sliding window, and divide the first request-response pair data through the first fixed-length sliding window to determine the initial sample data corresponding to each first fixed-length sliding window, wherein each initial sample data includes the preset parameter value of the first request-response pair data.

[0076] In this embodiment, a preset parameter value N for the first fixed-length sliding window is determined. Optionally, N=50, and the window sliding step size is 1. The first request-response pair data is divided by the first fixed-length sliding window to determine the initial sample data corresponding to each first fixed-length sliding window. This can be done by: for each Client IP, after sorting the request-response pairs, extracting the most recent N request-response pairs using the first fixed-length sliding window as initial sample data for a training sample; if the number of request-response pairs for a single IP is less than N, then the traffic data of that IP is discarded; if the number is greater than N, then multiple initial sample data are generated through the first fixed-length sliding window.

[0077] Step S2022: Obtain the preset parameter values ​​and the first request-response pair data including the user agent field, the source page field, the request Uniform Resource Locator, the response status code, the response packet length, and the request-response pair time interval.

[0078] In this embodiment, features are extracted one by one from the N request-response pairs within each sliding window, and the features are normalized / encoded to construct a sample feature matrix for the current window, which serves as input data for model training. The feature parameters used for feature extraction included in the first request-response pair data obtained in this embodiment include: user agent field, source page field, request Uniform Resource Locator (URL), response status code, response packet length, and request-response pair time interval.

[0079] Step S2023: Determine the hot unique encoding feature value based on the user agent field, determine the null value judgment feature value based on the source page field, determine the similarity feature value based on the request Uniform Resource Locator, and determine the hot unique encoding feature value, null value judgment feature value, and similarity feature value as the first semantic relevance feature.

[0080] Furthermore, the embodiments of this application can process feature parameters and extract semantic relevance features through the following methods: (1) Deduplicating the User Agent field (UA field) in the window, constructing a UA encoding mapping table, performing one-hot encoding on each UA field, generating a numerical encoding value, and determining the numerical encoding value as the hot-unique encoding feature value; (2) Binarizing the source page field (Referer field) based on whether it is empty: Referer field non-empty = 1, empty = 0, and determining the binarized value as the empty value judgment feature value; (3) Calculating the similarity of each request Uniform Resource Locator (request URL) in the window: a. URL cleaning: Parsing the original URL, extracting the `path` and `query` parts, removing the domain name and special symbols, and converting the URL into natural language text, as shown in the following example:

[0081] Original URL: https: / / www.example.com / products / detail-view?id=123&type=shoe

[0082] Text after cleaning: products detail view id 123 type shoe

[0083] b. URL Semantic Encoding: The SentenceTransformer model (all-MiniLM-L6-v2) is used to semantically encode the cleaned URL text, generating a vector of dimension D, which can be selected as D=384. Let the encoded vector of the i-th URL be:

[0084]

[0085] In addition, other semantic encoding models, such as BERT, SimCSE, and RoBERTa, can be used in the embodiments of this application. When using them, the code logic of URL encoding can be adjusted as needed without affecting the subsequent cosine similarity calculation.

[0086] c. Calculate the URL center vector: Calculate the average of the encoded vectors of the N URLs within the window to obtain the URL center vector Vc of the window, using the following formula:

[0087]

[0088] d. Calculate cosine similarity: Calculate the cosine similarity for each URL-encoded vector V. i The cosine similarity Sim(V) with the center vector Vc i (Vc), the formula is as follows:

[0089]

[0090] The similarity value is determined as the similarity feature value. Finally, the obtained hot unique encoding feature value, null value judgment feature value, and similarity feature value are determined as the first semantic relevance feature. The semantic relevance feature can characterize the essence of the crawler's large-scale targeted acquisition of specific domain data.

[0091] Step S2024: Determine the response status code feature value based on the response status code, determine the total length feature value of the response packet based on the response packet length, determine the time interval feature value based on the request-response pair time interval, and determine the response status code feature value, the total length feature value of the response packet, and the time interval feature value as the first access behavior feature.

[0092] Furthermore, the embodiments of this application can process feature parameters and extract access behavior related features through the following methods: (1) Determine the response status code of the request: directly extract the status code of the HTTP response (such as 200, 404), convert it into a numerical type, and determine the numerical status code as the response status code feature value; (2) Determine the response packet length: accumulate the length of all TCP fragment payloads of the HTTP response to obtain the total length of the response packet, perform normalization processing, and determine the normalized total length of the response packet as the response packet total length feature value; (3) Determine the time interval between the current request-response pair and the previous request-response pair: calculate the difference between the timestamp of the current request-response pair and the previous one to obtain the time interval, perform normalization processing, and determine the normalized time interval as the time interval feature value. Finally, the response status code feature value, the total length feature value of the response packet, and the time interval feature value are determined as the first access behavior feature, which can characterize the difference between the focused crawler access behavior and the normal human access behavior. It is worth mentioning that the normalization method in the embodiments of this application can be "min-max" normalization, Z-score normalization, decentralization, etc.

[0093] Through the above steps, this application embodiment aggregates the access traffic of a single IP and divides request-response pairs through a first fixed-length sliding window, extracting features that fuse semantic relevance and access behavior. The first semantic relevance feature reflects the semantic context features of the focused crawler (the essence of targeted access to domain-related URLs), and the first access behavior feature distinguishes between the access behavior features of crawlers and humans. It fundamentally uncovers the unmaskable features of crawlers, avoids the problem of focused crawlers being easily bypassed, and greatly improves the detection accuracy of highly disguised focused crawlers.

[0094] In some embodiments, step S203 includes:

[0095] Step S2031: Based on the first semantic relevance feature and the first access behavior feature, determine a sample feature matrix containing hot unique coding feature value, null value judgment feature value, similarity feature value, response status code feature value, response packet total length feature value and time interval feature value.

[0096] In this embodiment, the feature values ​​of the first semantic relevance feature and the first access behavior feature are arranged in chronological order of request-response pairs to construct a sample feature matrix, which serves as a single training sample for the initial detection model. Furthermore, each training sample is labeled with a binary label: if the IP corresponding to the sample is a focused crawler IP, the label is 1; if it is a normal human IP, the label is 0. The labeled training samples are divided into a training set and a validation set in a 9:1 ratio. The training set is used for model training, and the validation set is used for performance verification during model training. In addition, features in the sample feature matrix can be supplemented with features corresponding to parameters such as cookie consistency, request method distribution, and page dwell time. Existing features can also be replaced, such as replacing the hot-coded feature value of the User Agent (UA) with a word embedding encoded feature value.

[0097] Step S2032: Input the sample feature matrix into the initial detection model that adapts to the temporal characteristics of the first request response data, and train it to obtain the trained focused crawler detection model.

[0098] The sample feature matrix is ​​input into an initial detection model adapted to the temporal characteristics of the first request-response pair data for training. Optionally, this embodiment constructs a three-layer LSTM (Long Short-Term Memory) network model as the initial detection model to adapt to the temporal characteristics of the traffic data (request-response pairs are sorted by time and have temporal correlation). The model is a binary classification model, and the output is a crawler probability value of 0 to 1 (the closer the value is to 1, the higher the probability of being a crawler). The overall structure of the model is: input layer → first layer LSTM + Dropout → second layer LSTM + Dropout → fully connected layer → output layer. The input of the model is the sample feature matrix, and the output is the crawler probability value corresponding to a single window. The training set is input into the constructed LSTM model, and the training parameters are set as follows: optional, training epochs=20, batch size=32, validation set ratio 0.1. During model training, the effect is verified on the validation set after each training epoch, and the loss value and accuracy of the training set / validation set are monitored to prevent overfitting.

[0099] After a training cycle is completed, the model's performance is evaluated on the validation set. If the validation set accuracy meets expectations, the model training effect is considered satisfactory; if not, the model parameters (such as the number of LSTM units, learning rate, and number of training epochs) are adjusted and retrained until a well-trained, focused crawler detection model that meets the requirements is obtained. Furthermore, this application embodiment can also use other deep learning models adapted to time-series data for time sequence modeling, such as GRU (Gated Recurrent Unit), BiLSTM (Bidirectional LSTM), Transformer, etc., and the network structure and parameters of the model can be adjusted as needed.

[0100] Through the above steps, the detection model constructed in this application embodiment can be a three-layer LSTM long short-term memory network model, and the feature matrix within the sliding window is sequentially modeled. The trained focused crawler detection model is obtained through model training, so that it outputs the crawler probability value corresponding to a single window, adapts to the temporal characteristics of traffic, improves the accuracy of probability judgment, and further improves the accuracy of crawler detection results.

[0101] In some embodiments, the following is included after step S203:

[0102] Step S205: Store the focused crawler detection model as a detection model file in a preset format, and deploy the detection model file to the traffic detection server of the network gateway.

[0103] Furthermore, in this embodiment, the crawler detection model is stored as a detection model file in a preset format, such as saving a trained LSTM model as a model file (e.g., .h5 format), and deploying the detection model file to the traffic detection server of the network gateway for real-time crawler probability calculation during the detection process.

[0104] Through the above steps, this embodiment of the application deploys the detection node at the network gateway's mirror traffic location. The execution entity is a traffic detection server deployed on the network gateway. The detection server has hardware and software capabilities for traffic collection, data parsing, feature extraction, model inference, and result output (such as a server equipped with a GPU, a Python environment, and a deep learning framework). It collects and detects all access traffic to the server without intruding on the original website's business system, achieving complete decoupling between the detection and business systems. It does not affect the processing of normal access requests, ensuring the normal operation of the website's business and can quickly adapt to the anti-crawler detection needs of various websites.

[0105] In some embodiments, step S204 includes:

[0106] In step S2041, the traffic detection server obtains the initial data to be detected through the traffic mirroring function of the network gateway.

[0107] During the detection process in this embodiment of the application, the traffic detection server collects all access traffic to the target website server in real time through the traffic mirroring function of the network gateway to obtain initial data to be detected. The collected traffic (initial data to be detected) is the original TCP / IP data packet, which does not intrude into the original business system and does not affect the processing and response of normal HTTP requests.

[0108] Step S2042: parse the initial data to be detected to determine multiple second request-response pairs sorted by time, and divide the second request-response pairs into two parts by using a second fixed-length sliding window to determine the number of second request-response pairs corresponding to each second fixed-length sliding window using a preset parameter value.

[0109] This application embodiment performs parsing and preprocessing on the initial data to be detected collected in real time, corresponding to the training process: real-time parsing of TCP / IP packets to filter out invalid traffic that is not HTTP / HTTPS; real-time matching of HTTP request-response pairs based on TCP four-tuples to construct a session dictionary and aggregate all request-response pairs by Client IP; and sorting the request-response pairs of each Client IP in ascending order by request timestamp to ensure the timeliness of traffic.

[0110] Furthermore, based on the second fixed-length sliding window, using the same preset parameter value N as in the training process, for each Client IP after sorting the request-response pairs, the most recent N request-response pairs are extracted in real time using the second fixed-length sliding window; if the number of request-response pairs for a single IP is less than N, feature extraction and detection are temporarily not performed, and traffic data for that IP continues to be collected, and finally the preset parameter value and second request-response pair data corresponding to each second fixed-length sliding window are determined.

[0111] Step S2043: Determine the second semantic relevance feature and the second access behavior feature based on the preset parameter value and the second request-response pair data.

[0112] Furthermore, in this embodiment of the application, features are extracted from N request-response pairs within each second fixed-length sliding window using the same method as the training process, and then normalized / encoded to determine the second semantic relevance features and the second access behavior features.

[0113] Step S2044: Determine the detection data feature matrix based on the second semantic relevance feature and the second access behavior feature, and determine the detection data feature matrix as the target data to be detected.

[0114] Based on the second semantic relevance feature and the second access behavior feature, this application embodiment constructs a detection data feature matrix corresponding to the second fixed-length sliding window, which serves as the target data to be detected for focusing the real-time inference input of the crawler detection model.

[0115] Step S2045: Input each detection data feature matrix into the focused crawler detection model to output the corresponding crawler probability value for each second fixed-length sliding window, and determine the crawler detection judgment result based on the crawler probability value.

[0116] In this embodiment, the traffic detection server loads a pre-trained detection model file and constructs a real-time inference model (focused on crawler detection model). The constructed detection data feature matrix is ​​input into the inference model, and the model outputs the crawler probability value P (P∈[0,1]) of the current sliding window in real time. The crawler detection judgment result is determined based on the crawler probability value.

[0117] Through the above steps, this application embodiment constructs a real-time inference model, outputs the crawler probability value corresponding to each second fixed-length sliding window in real time, and determines the crawler detection judgment result based on the crawler probability value, thereby improving the accuracy and real-time performance of the detection result.

[0118] In some embodiments, step S2043 includes:

[0119] Step S2143: Input each detection data feature matrix into the focused crawler detection model to output the corresponding crawler probability value for each second fixed-length sliding window.

[0120] Step S2243: Obtain M crawler probability values ​​corresponding to M consecutive second fixed-length sliding windows, and determine the average crawler probability value based on the M crawler probability values, where M is a preset integer value greater than 1.

[0121] Step S2343: Obtain a preset crawler probability threshold; if the average crawler probability value is greater than the preset crawler probability threshold, determine that the crawler detection judgment result is that a crawler has been detected, and output the crawler detection judgment result; if the average crawler probability value is less than or equal to the preset crawler probability threshold, determine that the crawler detection judgment result is that a crawler has not been detected, and output the crawler detection judgment result.

[0122] After obtaining the crawler probability value corresponding to each second fixed-length sliding window, this embodiment of the application can reduce the false positives of single-window detection and improve the robustness of detection by using the probability sequence of continuous sliding windows. The method is as follows: Set the number of continuous windows M, where M is an integer greater than 1, optionally M=10; preset the crawler probability threshold T, optionally T=0.8; obtain the M crawler probability values ​​corresponding to M consecutive second fixed-length sliding windows; for each Client IP, record the crawler probability value of each sliding window in real time to form a continuous probability sequence P1, P2, ..., P M Calculate the average crawler probability value over M consecutive windows. Specifically, extract the M most recent crawler probability values ​​from the probability sequence and calculate their arithmetic mean. The average crawler probability value is obtained using the following formula:

[0123]

[0124] like >T: The server determines that the Client IP is a targeted crawler IP, outputs the crawler detection result, and pushes the IP to the gateway's management module for subsequent blocking, rate limiting, and other operations; if ≤T: Determine that the Client IP is a normal human access IP, and continue to perform sliding window detection and probability calculation on the real-time traffic of the IP.

[0125] Through the above steps, this embodiment of the application forms a crawler probability sequence through continuous sliding windows. Only when the average probability value of M consecutive windows exceeds a set threshold is the IP determined to be a crawler access, thus avoiding misjudgment in single-window detection and improving the robustness of detection.

[0126] Furthermore, in this embodiment, the detection server outputs the determination result of crawler / normal IP to the gateway's management system in real time. The management system processes the crawler IP according to preset rules, such as temporarily blocking the crawler IP's access permissions for a certain period of time; traffic limiting, restricting the crawler IP's access frequency / bandwidth; and sending alarm notifications, sending crawler access alarm information to the website administrator. Normal IPs are not processed in any way, ensuring their normal access.

[0127] The embodiments of this application will be described and illustrated below through preferred embodiments.

[0128] Figure 3 This is a flowchart of a semantic context-based focused crawler data theft detection method according to a preferred embodiment of this application. Figure 3 As shown, this semantic context-based focused crawler data theft detection method includes the following steps:

[0129] Step S301: Obtain the initial training dataset and parse the initial training dataset to determine multiple first request-response pairs of data sorted by time;

[0130] Step S302: Divide the first request-response pair data by using a first fixed-length sliding window to determine the initial sample data, and determine the first semantic relevance feature and the first access behavior feature based on the initial sample data;

[0131] Step S303: Based on the first semantic relevance feature and the first access behavior feature, determine a sample feature matrix containing hot unique coding feature value, null value judgment feature value, similarity feature value, response status code feature value, response packet total length feature value and time interval feature value;

[0132] Step S304: Input the sample feature matrix into the initial detection model that adapts to the temporal characteristics of the first request response data, and train it to obtain the trained focused crawler detection model.

[0133] Step S305: The focused crawler detection model is stored as a detection model file in a preset format, and the detection model file is deployed to the traffic detection server of the network gateway; the traffic detection server obtains the initial data to be detected through the traffic mirroring function of the network gateway.

[0134] Step S306: Based on the initial data to be detected, determine the detection data feature matrix corresponding to each second fixed-length sliding window, and determine the detection data feature matrix as the target data to be detected;

[0135] Step S307: Input each detection data feature matrix into the focused crawler detection model to output the corresponding crawler probability value for each second fixed-length sliding window, and determine the crawler detection judgment result based on the crawler probability value.

[0136] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0137] This embodiment also provides a semantic context-based focused crawler data theft detection device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0138] Figure 4 This is a structural block diagram of a semantic context-based focused crawler data theft detection device according to an embodiment of this application, such as... Figure 4 As shown, the device includes a flow analysis module 10, a sample feature extraction module 20, a detection model training module 30, and a detection result output module 40.

[0139] Traffic parsing module 10 is used to obtain an initial training dataset and parse the initial training dataset to determine multiple first request-response pairs of data sorted by time.

[0140] The sample feature extraction module 20 is used to divide the first request-response pair data through a first fixed-length sliding window to determine the initial sample data, and to determine the first semantic relevance feature and the first access behavior feature based on the initial sample data;

[0141] The detection model training module 30 is used to determine the sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and to train the initial detection model based on the sample feature matrix to obtain the trained focused crawler detection model.

[0142] The detection result output module 40 is used to acquire the target data to be detected, input the target data to be detected into the focused crawler detection model, and output the crawler detection judgment result.

[0143] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0144] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0145] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0146] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0147] Obtain the initial training dataset and parse it to determine multiple first request-response pairs of data sorted by time.

[0148] The first request-response pair data is divided by a first fixed-length sliding window to determine the initial sample data, and the first semantic relevance feature and the first access behavior feature are determined based on the initial sample data.

[0149] The sample feature matrix is ​​determined based on the first semantic relevance feature and the first access behavior feature, and the initial detection model is trained based on the sample feature matrix to obtain the trained focused crawler detection model.

[0150] Acquire the target data to be detected, input the target data to be detected into the focused crawler detection model, and output the crawler detection judgment result.

[0151] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0152] Furthermore, in conjunction with the semantic context-based focused crawler data theft detection method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the semantic context-based focused crawler data theft detection methods in the above embodiments.

[0153] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0155] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A semantic context-based focused method for detecting web crawler data theft, characterized in that, Includes the following steps: Obtain an initial training dataset and parse the initial training dataset to determine multiple first request-response pairs of data sorted by time; The first request-response pair data is divided by a first fixed-length sliding window to determine initial sample data, and a first semantic relevance feature and a first access behavior feature are determined based on the initial sample data. The sample feature matrix is ​​determined based on the first semantic relevance feature and the first access behavior feature, and the initial detection model is trained based on the sample feature matrix to obtain a trained focused crawler detection model. Acquire the target data to be detected, input the target data to be detected into the focused crawler detection model, and output the crawler detection judgment result.

2. The semantic context-based focused crawler data theft detection method according to claim 1, characterized in that, The initial training dataset is a network packet capture file. Obtaining the initial training dataset and parsing it to determine multiple first request-response pairs sorted by time includes: Obtain network packet capture files, which include focused crawler sample traffic and human access sample traffic; The focused crawler sample traffic and the human access sample traffic are analyzed to determine the time-ordered first request-response pairs of data corresponding to each client address.

3. The semantic context-based focused crawler data theft detection method according to claim 2, characterized in that, The step of dividing the first request-response pair data through a first fixed-length sliding window to determine initial sample data, and determining the first semantic relevance feature and the first access behavior feature based on the initial sample data, includes: Determine the preset parameter value corresponding to the first fixed-length sliding window, and divide the first request-response pair data through the first fixed-length sliding window to determine the initial sample data corresponding to each first fixed-length sliding window, wherein each initial sample data includes the preset parameter value and the first request-response pair data; The preset parameter values ​​are obtained for the first request-response pair data, including the user agent field, source page field, request Uniform Resource Locator, response status code, response packet length, and request-response pair time interval. The hot-unique encoding feature value is determined based on the user agent field, the null value judgment feature value is determined based on the source page field, and the similarity feature value is determined based on the request Uniform Resource Locator. The hot-unique encoding feature value, the null value judgment feature value, and the similarity feature value are then determined as the first semantic relevance feature. The response status code feature value is determined based on the response status code, the total length feature value of the response packet is determined based on the response packet length, and the time interval feature value is determined based on the request-response pair time interval. The response status code feature value, the total length feature value of the response packet, and the time interval feature value are then determined as the first access behavior feature.

4. The semantic context-based focused crawler data theft detection method according to claim 3, characterized in that, The step of determining a sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and training the initial detection model based on the sample feature matrix to obtain a trained focused crawler detection model includes: Based on the first semantic relevance feature and the first access behavior feature, a sample feature matrix is ​​determined that includes the hot unique coding feature value, the null value judgment feature value, the similarity feature value, the response status code feature value, the total length feature value of the response packet, and the time interval feature value; The sample feature matrix is ​​input into an initial detection model adapted to the temporal characteristics of the first request response data, and trained to obtain a trained focused crawler detection model.

5. The semantic context-based focused crawler data theft detection method according to claim 3 or claim 4, characterized in that, After determining the sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and training the initial detection model based on the sample feature matrix to obtain the trained focused crawler detection model, the process includes: The focused crawler detection model is stored as a detection model file in a preset format, and the detection model file is deployed to the traffic detection server of the network gateway.

6. The semantic context-based focused crawler data theft detection method according to claim 5, characterized in that, The step of acquiring the target data to be detected and inputting the target data to be detected into the focused crawler detection model to output the crawler detection judgment result includes: The traffic detection server obtains initial data to be detected through the traffic mirroring function of the network gateway; The initial data to be detected is parsed to determine multiple second request-response pairs sorted by time, and the second request-response pairs are divided by a second fixed-length sliding window to determine the preset parameter value of each second request-response pair corresponding to each second fixed-length sliding window; Based on the preset parameter values ​​and the second request-response pair data, determine the second semantic relevance feature and the second access behavior feature; Based on the second semantic relevance feature and the second access behavior feature, a detection data feature matrix is ​​determined, and the detection data feature matrix is ​​determined as the target data to be detected; Each of the detection data feature matrices is input into the focused crawler detection model to output the corresponding crawler probability value for each of the second fixed-length sliding windows. The crawler detection judgment result is determined based on the crawler probability value and then output.

7. The semantic context-based focused crawler data theft detection method according to claim 6, characterized in that, The step of inputting each of the detected data feature matrices into the focused crawler detection model to output the corresponding crawler probability value for each of the second fixed-length sliding windows, determining the crawler detection judgment result based on the crawler probability value, and outputting the crawler detection judgment result includes: Each of the detection data feature matrices is input into the focused crawler detection model to output the corresponding crawler probability value for each of the second fixed-length sliding windows; Obtain M crawler probability values ​​corresponding to M consecutive second fixed-length sliding windows, and determine the average crawler probability value based on the M crawler probability values, where M is a preset integer value greater than 1; Obtain a preset crawler probability threshold; if the average crawler probability value is greater than the preset crawler probability threshold, determine that the crawler detection judgment result is that a crawler has been detected, and output the crawler detection judgment result; if the average crawler probability value is less than or equal to the preset crawler probability threshold, determine that the crawler detection judgment result is that a crawler has not been detected, and output the crawler detection judgment result.

8. A semantic context-based focused crawler data theft detection device, used to implement the semantic context-based focused crawler data theft detection method as described in any one of claims 1 to 7, characterized in that, It includes a traffic parsing module, a sample feature extraction module, a detection model training module, and a detection result output module: The traffic parsing module is used to obtain an initial training dataset and parse the initial training dataset to determine multiple first request-response pairs of data sorted by time. The sample feature extraction module is used to divide the first request-response pair data by a fixed-length sliding window to determine the initial sample data, and to determine the first semantic relevance feature and the first access behavior feature based on the initial sample data. The detection model training module is used to determine a sample feature matrix based on the first semantic relevance feature and the first access behavior feature, and to train the initial detection model based on the sample feature matrix to obtain a trained focused crawler detection model. The detection result output module is used to acquire the target data to be detected, input the target data to be detected into the focused crawler detection model, and output the crawler detection judgment result.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the semantic context-based focused crawler data theft detection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the semantic context-based focused crawler data theft detection method according to any one of claims 1 to 7 at runtime.

Images