A method, system, equipment and medium for monitoring overseas data.

Abstract

This application provides a method, system, device, and medium for monitoring overseas data. The method includes the following steps: deploying a distributed proxy cluster; filtering target proxy nodes based on continuously collected node status data; dynamically scheduling collection requests containing access spoofing strategies to these nodes; obtaining raw monitoring data by spoofing access to the target platform through the target node and parsing the response data; performing compliance verification on the raw data in the local storage space and physically cleaning up non-compliant data; identifying attributes and performing matching de-identification processing on compliant data; fusing the de-identified data, node status, and compliance verification results using multi-dimensional features, inputting the data into an anomaly identification model to obtain anomaly event results; outputting early warnings based on the anomaly results and dynamically updating collection rules or spoofing strategies. Implementing the technical solution provided in this application constructs a secure closed loop for data collection and compliance flow, improving the system's concealment and adaptive evolution capabilities.

Description

Technical Field

[0001] This application relates to the field of data security monitoring technology, and in particular to a method, system, equipment and medium for monitoring overseas data. Background Technology

[0002] With the rapid development of the globalized digital economy and the overseas expansion of enterprises, the demand for data interaction and monitoring on overseas information platforms (such as TikTok, The New York Times, Twitter, Facebook, and Google) continues to rise. How to efficiently collect, process, and analyze multi-dimensional business data (such as webpage content, news headlines, authors, and update times) in these complex overseas network environments has become a core requirement for ensuring enterprise data security and improving the efficiency of global operations.

[0003] In existing technologies, the collection and monitoring of overseas data is typically implemented based on a single-node proxy and centralized storage architecture. For example, the system uses a proxy server with a fixed IP address to collect data (such as tweet content, news authors, etc.) from the aforementioned overseas target platforms, and then directly transmits the acquired raw data back to a local server for centralized storage and analysis. However, in practical applications, the existing technologies, which employ fixed strategies and centralized processing methods, are ill-suited to adapt to the complex anti-scraping mechanisms and fluctuating network environments overseas. Furthermore, the lack of localized compliance verification and de-identification mechanisms before data transmission poses risks in cross-border, multi-scenario data monitoring applications where data retention regulations vary from country to country. These risks include the potential for data collection tasks to be blocked, unauthorized transfer of sensitive privacy data, and triggering data compliance reviews in the target country. Summary of the Invention

[0004] In view of this, this application provides a method, system, device and medium for monitoring overseas data to solve the above problems.

[0005] Firstly, a method for monitoring overseas data is provided, which includes:

[0006] Deploy a distributed agent cluster in the target network area and continuously collect the network operation characteristics of each agent node in the distributed agent cluster to generate node status data.

[0007] Obtain data collection requests for the target platform within the target network area, and determine the availability status of each proxy node based on node status data. The data collection requests include access masquerading strategies and data collection rules that match the target platform.

[0008] Among all agent nodes, target agent nodes whose availability status meets preset conditions are selected, and data collection requests are dynamically scheduled to target agent nodes;

[0009] The target proxy node initiates access to the target platform based on the access masquerading policy and receives the response data returned by the target platform. The response data is then parsed using data collection rules to obtain the raw monitoring data.

[0010] The raw monitoring data is stored in the local storage space corresponding to the target network area. The raw monitoring data is then verified for compliance using a pre-defined compliance rule base that matches the target network area to generate compliance verification results. Based on the compliance verification results, the raw monitoring data that fails the compliance verification is cleaned up.

[0011] Identify the data attributes of the original monitoring data that have passed compliance verification, match the corresponding de-identification rules according to the data attributes, and perform de-identification processing on the original monitoring data that has passed compliance verification based on the de-identification rules to obtain de-identified monitoring data;

[0012] The desensitized monitoring data, node status data, and compliance verification results are fused to generate multi-dimensional fused features. The multi-dimensional fused features are then input into a preset anomaly identification model for anomaly analysis to obtain anomaly event results.

[0013] Based on the results of abnormal events, corresponding early warning information is generated, and data collection rules and / or access spoofing strategies are updated according to the results of abnormal events.

[0014] The above technical solution integrates underlying execution steps such as distributed agent dynamic scheduling, communication environment masquerading, local compliance verification, data hierarchical desensitization, and multi-dimensional feature fusion anomaly identification to construct a full-link closed-loop secure data monitoring architecture. It solves the technical problems of traditional crawler nodes being easily blocked, privacy leaks during the collection process, and the inability of the system to automatically iterate strategies based on the running status, and achieves efficient and covert collection and secure and compliant transfer of data from target platforms.

[0015] Optionally, network operation characteristics of each agent node within the distributed agent cluster can be continuously collected to generate node status data, specifically including:

[0016] Obtain network connection requests for each agent node, extract signaling protocol features from the network connection requests, and monitor the signaling protocol features in real time based on preset signaling protection rules;

[0017] When an abnormal routing identifier or unauthorized operation instruction is identified from the signaling protocol features based on the preset signaling protection rules, the network connection request containing the signaling protocol features is judged as an abnormal connection request, the abnormal connection request is blocked, and a security interception record corresponding to the abnormal connection request is generated.

[0018] The latency, stability, and survival parameters of each agent node are collected, and these parameters, along with the signaling security feature parameters calculated based on security interception records, are used to determine the network operation characteristics of each agent node.

[0019] The network operation characteristics are aggregated to obtain aggregated features, and the aggregated features are input into a preset machine learning evaluation algorithm to dynamically evaluate the node health and generate node status data.

[0020] The above technical solution improves the accuracy of dynamic health assessment of distributed agent clusters and the security of underlying data collection channels by real-time monitoring of connection requests and interception of abnormal commands at the network signaling protocol level, and by performing feature aggregation calculations on signaling security feature parameters and conventional physical network parameters such as latency and stability. In the machine learning evaluation model, risky agent nodes that are contaminated or have malicious routing are excluded.

[0021] Optionally, the target proxy node initiates access to the target platform based on an access masquerading policy and receives response data returned by the target platform, specifically including:

[0022] Based on the fingerprint simulation rules in the access spoofing strategy, the characteristics of the graphics rendering interface in the access environment of the target proxy node are modified to generate a fake browser fingerprint.

[0023] Based on the security protocol spoofing rules in the access spoofing policy, simulate the security communication handshake characteristics of a preset version to generate spoofed handshake characteristics;

[0024] The fake browser fingerprint and fake handshake features are encapsulated into the data collection request to obtain the encapsulated data collection request.

[0025] Obtain the anti-crawling frequency limit corresponding to the target platform in the data collection rules, and determine the sending frequency for the encapsulated data collection request based on the anti-crawling frequency limit;

[0026] According to the sending frequency, the encapsulated data collection request is sent to the target platform through the target proxy node, and the response data returned by the target platform is received.

[0027] The above technical solution generates a fake browser fingerprint by dynamically modifying the characteristics of the graphics rendering interface before initiating a data collection request, and simulates the secure communication handshake characteristics of a preset version. Combined with the anti-crawling frequency limit control of the target platform to control the packet sending frequency, it masks the identity characteristics of the real automated collection program at the two underlying dimensions of communication protocol and operating environment, effectively avoids the machine behavior detection mechanism of the target platform, and improves the success rate of monitoring data requests.

[0028] Optionally, the response data can be parsed using data collection rules to obtain the raw monitoring data, specifically including:

[0029] The system analyzes data collection rules, identifies the target monitoring scenario corresponding to the target platform, and matches the preset data extraction script corresponding to the target monitoring scenario from the preset script library.

[0030] By using the path language rules and preset character matching rules in the preset data extraction script, structured text content is extracted from the page source code of the response data;

[0031] The image elements in the location response data are identified, and the text in the image elements is identified using a preset optical character recognition algorithm to obtain the visual presentation content.

[0032] By associating and mapping structured text content with visual presentation content, raw monitoring data containing multi-dimensional business characteristics is generated.

[0033] The above technical solution, by calling a preset data extraction script that matches the target monitoring scenario, performs path and character matching on the page source code of the response data to extract structured text, and combines optical character recognition algorithm to locate and extract visual content in the image, and performs cross-modal association mapping between text content and visual content, realizes accurate extraction and structured reorganization of multi-dimensional business feature data in complex heterogeneous web page structures.

[0034] Optionally, the original monitoring data is subjected to compliance verification using a pre-defined compliance rule base that matches the target network area to generate compliance verification results. Based on these results, the original monitoring data that fails compliance verification is then cleaned up. Specifically, this includes:

[0035] Obtain a preset compliance template corresponding to the target network area. The preset compliance template contains multi-dimensional compliance logic verification rules for the original monitoring data. The multi-dimensional compliance logic verification rules include data collection location access rules, data transmission path encryption rules, and data retention duration limit rules. Build a preset compliance rule library based on the preset compliance template.

[0036] Real-time monitoring of the temporary storage duration of raw monitoring data in the local storage space, and extraction of the collection node location of the target agent node and the transmission path information of the raw monitoring data;

[0037] The temporary storage duration, data collection node location, and transmission path information are input into the preset compliance rule base and compared with the corresponding data retention duration limit rules, data collection location access rules, and data transmission path encryption rules to generate a compliance verification result containing risk assessment identifiers.

[0038] When the risk assessment indicator indicates a risk of violation, the communication session between the target agent node and the target platform is interrupted, and the storage destruction interface of the local storage space is invoked to perform a data overwrite and erase operation on the storage address corresponding to the original monitoring data.

[0039] The above technical solution introduces multi-dimensional compliance logic verification rules that include data collection location, transmission path encryption, and retention time. It monitors the life cycle and physical trajectory of the original monitoring data in the local storage space in real time. When a violation risk is determined by comparison, it directly calls the storage destruction interface to perform underlying data overwrite and erase operations and interrupts the communication session. This completely blocks the security risks of data leaving the country without authorization or being retained beyond the permitted period at the physical medium level.

[0040] Optionally, identify the data attributes of the original monitoring data that has passed compliance verification, match the corresponding de-identification rules according to the data attributes, and perform de-identification processing on the original monitoring data that has passed compliance verification based on the de-identification rules to obtain de-identified monitoring data, specifically including:

[0041] The system uses a pre-defined sensitive data classification dictionary to perform field traversal and comparison on the original monitoring data that has passed compliance verification. The sensitive data classification dictionary is configured with a first sensitivity label and a second sensitivity label based on different security protection levels.

[0042] Extract the field content that matches the first sensitivity tag into the first data fragment, extract the field content that matches the second sensitivity tag into the second data fragment, and determine the first sensitivity tag and the second sensitivity tag as data attributes;

[0043] Based on the tag mapping relationship in the preset desensitization rule library, the irreversible encryption rule bound to the first sensitivity tag and the character masking rule bound to the second sensitivity tag are extracted respectively, and the irreversible encryption rule and the character masking rule are combined into a desensitization rule;

[0044] The hash salting algorithm in the irreversible encryption rule is called to perform hash calculation on the first data fragment to generate an encrypted fragment, and the preset masking parameter in the character masking rule is called to perform character replacement operation on the second data fragment to generate a masked fragment;

[0045] Based on the original field layout structure of the original monitoring data, the encrypted fragments, masked fragments, and retained field contents of the missing tags are spliced ​​and assembled to generate de-identified monitoring data.

[0046] The above technical solution utilizes a pre-defined sensitive data classification dictionary to perform field-level traversal and comparison of compliant data, objectively extracting first and second data fragments with different security protection levels. It then strictly follows the label mapping relationship of the rule base to call hash salting algorithms and character replacement operations for irreversible and masked hierarchical desensitization. Finally, it reassembles the data according to the original field arrangement structure and the missing fields. Without destroying the original logical structure of the business data, it achieves precise anti-traceability protection for core privacy information.

[0047] Optionally, the de-identified monitoring data, node status data, and compliance verification results are fused to generate multi-dimensional fused features. These multi-dimensional fused features are then input into a pre-defined anomaly identification model for anomaly analysis to obtain anomaly event results, specifically including:

[0048] Business content features are extracted from de-identified monitoring data, node operation evaluation parameters are extracted from node status data, and compliance status identifiers are extracted from compliance verification results. The business content features, node operation evaluation parameters, and compliance status identifiers are then normalized to obtain a normalized feature vector.

[0049] Normalized feature vectors from multiple consecutive time windows are concatenated and combined to obtain a multidimensional temporal feature matrix as a multidimensional fusion feature, and the multidimensional temporal feature matrix is ​​input into the anomaly detection model.

[0050] The similarity score between the multidimensional temporal feature matrix and each benchmark feature sequence in the preset abnormal scene feature library is calculated using an anomaly detection model.

[0051] When the maximum similarity score in the similarity score is greater than or equal to a preset threshold, the target benchmark feature sequence corresponding to the maximum similarity score is determined, and the anomaly type identifier bound to the target benchmark feature sequence is extracted.

[0052] Using the anomaly type identifier as a mapping index, query and retrieve anomaly cause tracing information and handling suggestions associated with the anomaly type identifier from the anomaly scenario feature library;

[0053] The abnormality type identifier, abnormality cause tracing information, and handling suggestions are encapsulated to generate abnormal event results.

[0054] The above technical solution extracts three types of heterogeneous data: business content features, underlying node operation evaluation parameters, and compliance status identifiers. It then normalizes these data to eliminate differences in dimensions, concatenates them to construct a multi-dimensional time-series feature matrix, and inputs it into an anomaly identification model to calculate the similarity of the baseline sequence. The anomaly type identifier corresponding to the maximum similarity score is used as an index to retrieve specific traceability information and processing suggestions. This achieves a global anomaly root cause automatic localization mechanism that takes into account business fluctuations, underlying network health, and security compliance boundaries.

[0055] Secondly, an overseas data monitoring system is provided, the system comprising:

[0056] The status acquisition module is used to deploy a distributed agent cluster in the target network area and continuously collect the network operation characteristics of each agent node in the distributed agent cluster to generate node status data.

[0057] The request evaluation module is used to obtain data collection requests for the target platform within the target network area and determine the availability status of each proxy node based on node status data. The data collection request includes access masquerading strategies and data collection rules that match the target platform.

[0058] The dynamic scheduling module is used to filter out target agent nodes whose availability status meets preset conditions from among the agent nodes, and dynamically schedule data collection requests to the target agent nodes.

[0059] The data parsing module is used to initiate access to the target platform through the target proxy node based on the access masquerading policy and receive the response data returned by the target platform. It then uses data collection rules to parse the response data to obtain the raw monitoring data.

[0060] The compliance verification module is used to store the raw monitoring data in the local storage space corresponding to the target network area, use a preset compliance rule library that matches the target network area to perform compliance verification on the raw monitoring data to generate compliance verification results, and clean up the raw monitoring data that failed the compliance verification based on the compliance verification results.

[0061] The data anonymization module is used to identify the data attributes of the original monitoring data that has passed compliance verification, match the corresponding anonymization rules according to the data attributes, and perform anonymization processing on the original monitoring data that has passed compliance verification based on the anonymization rules to obtain anonymized monitoring data.

[0062] The fusion analysis module is used to fuse de-identified monitoring data, node status data, and compliance verification results to generate multi-dimensional fusion features. The multi-dimensional fusion features are then input into a preset anomaly identification model for anomaly analysis to obtain anomaly event results.

[0063] The early warning feedback module is used to generate corresponding early warning information based on the results of abnormal events, and to update data collection rules and / or access spoofing strategies according to the results of abnormal events.

[0064] Thirdly, an electronic device is provided, including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the above.

[0065] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing instructions that, when executed, perform the method as described in any of the preceding claims.

[0066] In summary, implementing one or more technical solutions provided in this application has at least the following technical effects or advantages:

[0067] By deeply coupling distributed node dynamic scheduling, multi-dimensional masquerading of communication links, localized physical storage control, and data field-level cleaning mechanisms, this application constructs an end-to-end data acquisition and protection architecture with adaptive evolution capabilities. On the one hand, this solution breaks down the information silos between business acquisition, network operation and maintenance, and legal compliance modules in traditional data monitoring systems, realizing the underlying collaboration and multi-dimensional fusion computation of cross-domain heterogeneous data (network underlying operating parameters, cross-modal business monitoring content, and compliance status identifiers). On the other hand, relying on the closed-loop feedback mechanism built on global anomaly root cause analysis, the system is endowed with self-repair and dynamic game-theoretic capabilities in complex and highly adversarial network environments, significantly reducing the manual intervention and operation and maintenance costs when dealing with anti-scraping strategy upgrades and node failures. While ensuring highly available and highly concealed data mining of the target platform, it transforms abstract legal compliance boundaries and privacy protection requirements into hard computer execution instructions, mitigating the risk of illegal auditing in large-scale data flow from the macro system architecture level, and effectively balancing the availability of business data extraction with the security and compliance of data flow. Attached Figure Description

[0068] Figure 1 This is an exemplary system architecture diagram of an overseas data monitoring method or an overseas data monitoring system applying this application;

[0069] Figure 2 This is a flowchart illustrating one of the overseas data monitoring methods disclosed in this application;

[0070] Figure 3 This is a schematic diagram of a module of an overseas data monitoring system disclosed in this application;

[0071] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in this application.

[0072] Figure reference numerals: 100, System architecture; 101, First terminal device; 102, Second terminal device; 103, Third terminal device; 104, Network; 105, Server; 301, Status acquisition module; 302, Request evaluation module; 303, Dynamic scheduling module; 304, Modular data parsing block; 305, Compliance verification module; 306, Data anonymization module; 307, Fusion analysis module; 308, Early warning feedback module; 401, Processor; 402, Communication bus; 403, User interface; 404, Network interface; 405, Memory. Detailed Implementation

[0073] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0074] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0075] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0076] Figure 1 An exemplary system architecture diagram is shown, illustrating an embodiment of an overseas data monitoring method or an overseas data monitoring system to which this application can be applied.

[0077] like Figure 1As shown, the system architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0078] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as model training applications, video recognition applications, web browser applications, social platform software, etc.

[0079] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0080] When terminals 101, 102, and 103 are hardware devices, video capture devices can also be installed on them. These video capture devices can be various devices capable of capturing video, such as cameras, sensors, etc. Users can use the video capture devices on terminals 101, 102, and 103 to capture video.

[0081] Server 105 can be a server that provides various services, such as a backend server for processing data displayed on terminal devices 101, 102, and 103. The backend server can analyze and process the received data and can feed back the processing results (such as recognition results) to the terminal devices.

[0082] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0083] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. In particular, if the target data does not need to be obtained remotely, the above system architecture may exclude the network and include only terminal devices or servers.

[0084] Figure 2 This is a flowchart illustrating a method for monitoring overseas data according to an embodiment of this application. This method can be implemented using a computer program, a microcontroller, or run on an overseas data monitoring system. The computer program can be integrated into an application or run as a standalone utility application. The specific steps of an overseas data monitoring method are described in detail below.

[0085] S201: Deploy a distributed agent cluster in the target network area and continuously collect the network operation characteristics of each agent node in the distributed agent cluster to generate node status data.

[0086] For example, in large-scale cross-regional network data acquisition scenarios targeting global multilingual information and social platforms (such as international news, Twitter, etc.), collection architectures relying on a single exit or static IP pool are prone to task interruption due to node failure or malicious exploitation. Therefore, this step constructs a distributed proxy node cluster in multiple target network regions such as North America, Europe, and Asia Pacific between the data monitoring center and the target platform, and establishes a continuous monitoring mechanism for node status. The system abandons the traditional static allocation mode, and through real-time detection of underlying communication signaling, assesses the availability and security status of all communication channels in advance. This eliminates nodes with high latency or security risks before executing substantive business data capture, providing a stable and reliable underlying network foundation for subsequent high-concurrency dynamic scheduling.

[0087] In one possible implementation, network operation characteristics of each agent node within the distributed agent cluster are continuously collected to generate node status data. Specifically, this includes: acquiring network connection requests for each agent node; extracting signaling protocol characteristics from the network connection requests; and monitoring the signaling protocol characteristics in real time based on preset signaling protection rules. When abnormal routing identifiers or unauthorized operation instructions are identified from the signaling protocol characteristics based on preset signaling protection rules, the network connection request containing the signaling protocol characteristics is determined as an abnormal connection request, the abnormal connection request is intercepted, and a security interception record corresponding to the abnormal connection request is generated. The latency parameters, stability parameters, and liveness parameters of each agent node are collected, and the latency parameters, stability parameters, liveness parameters, and signaling security feature parameters statistically derived from the security interception records are determined as the network operation characteristics of each agent node. The network operation characteristics are aggregated to obtain aggregated features, and the aggregated features are input into a preset machine learning evaluation algorithm for dynamic evaluation of node health, generating node status data.

[0088] In the embodiments of this application, node status data refers to a comprehensive output result used to fully characterize the current availability, physical transmission quality, and underlying anti-attack security level of each agent node in the distributed agent cluster. For example, the data can be a structured feature vector or comprehensive evaluation score that includes node connectivity (e.g., 99.9%), millisecond-level network latency (e.g., 50ms), and signaling anti-tampering security rating (e.g., healthy / abnormal).

[0089] Specifically, network connection requests for each proxy node are acquired. Through underlying packet capture or protocol stack parsing techniques, signaling protocol features (i.e., identifiers reflecting the underlying communication specifications such as packet header structure, handshake sequence, and routing labels during the connection establishment phase) are extracted from the network connection requests. Based on preset signaling protection rules (a pre-configured verification policy library containing standard patterns of legitimate communication protocols and signatures of known malicious network attacks), the extracted signaling protocol features are subjected to in-depth real-time monitoring. When abnormal routing identifiers (indicating forged redirect paths attempting to tamper with the real source IP or pollute the proxy pool) or unauthorized operation instructions (i.e., illegal port scanning, backdoor probing, or other payload code that does not conform to normal business logic) are identified from the signaling protocol features based on the preset signaling protection rules, the network connection request containing such risky signaling protocol features is judged as an abnormal connection request. The packet is directly dropped at the network gateway or firewall level to intercept the abnormal connection request, and the attack source, timestamp, and risk type are extracted to generate a security interception record corresponding to the abnormal connection request. While monitoring the underlying signaling, the system simultaneously collects latency parameters (reflecting network latency status, including round-trip time of data packets), stability parameters (representing packet loss rate or network jitter within a specific time window), and survival parameters (i.e., the online survival status of nodes maintained by heartbeat packet detection) of each agent node. The system then combines these physical dimension latency parameters, stability parameters, and survival parameters with signaling security feature parameters (i.e., the frequency of malicious requests intercepted per unit time and threat level scores) statistically derived from security interception records, across dimensions to determine the network operation characteristics of each agent node. Furthermore, the network operation characteristics, encompassing both physical network and security protection dimensions, are vectorized and normalized for feature aggregation, resulting in dimension-reduced and aligned aggregated features. These aggregated features are then input into a pre-defined machine learning evaluation algorithm (e.g., a random forest or gradient boosting tree model trained on massive node samples). The algorithm dynamically evaluates the node health based on the potential network vulnerabilities and performance degradation trends mapped by the aggregated features, outputting node status data that matches the current actual operating status of the node.

[0090] Specifically, in the process of aggregating network operational characteristics and assessing health, to accurately quantify the impact of each dimension parameter on node availability, the system can introduce a health scoring mechanism based on dynamic weights. For example, suppose a certain proxy node is in the current time window... The internal delay parameter is (Normalized to the [0, 1] interval; lower values ​​indicate lower latency), stability parameters (such as packet loss rate) are... Signaling security characteristic parameters (such as the frequency of interception alarms) are: Then the overall health score of the proxy node. The following formula can be used for calculation:

[0091]

[0092] in, , , These represent the latency weight, stability weight, and security penalty factor set by the system based on the current network environment of the monitoring task, respectively, and satisfy the following conditions: as well as ; It is a natural constant; This is an exponential penalty for the number of security interceptions, designed to exponentially lower the health score as the frequency of security alerts increases. For example, the delay decay function can be expressed as: or This algorithm is used to inversely map latency parameters to positive health gain values. Through this calculation, the system can map multi-dimensional underlying physical and security parameters into an intuitive, continuous health score. When the calculated... When the value falls below a preset circuit breaker threshold (e.g., 0.6), the system will immediately generate node status data indicating that the node is unavailable, thereby filtering out high-latency or risky nodes that have been contaminated by the platform's anti-crawler system at the underlying level.

[0093] S202: Obtain data collection requests for the target platform within the target network area, and determine the availability status of each proxy node based on node status data. The data collection requests include access masquerading strategies and data collection rules that match the target platform.

[0094] In this embodiment, the data acquisition request refers to the core driving instruction package that triggers the entire proxy scheduling and network monitoring process. It not only defines the endpoint network target for data acquisition, but also encapsulates a complete set of technical parameters and behavioral guidelines required to achieve secure and covert acquisition. For example, the request can be an automated information scraping task package targeting a specific overseas platform (such as international news websites, Twitter, Facebook, Google, etc.). It includes the URL path pointing to the overseas platform, dynamic request headers and browser fingerprint parameters configured to bypass the platform's anti-scraping mechanism, and a parsing script for accurately extracting business fields such as "title, content, author, and time" from the returned webpage HTML source code.

[0095] Specifically, the system obtains the data collection request by retrieving instructions from the task scheduling queue or upstream business modules targeting specific overseas social media or foreign media websites (i.e., the target platform) within a specific geographical location or logical network segment (i.e., the target network area). Simultaneously, by parsing the request body, two pre-configured core execution parameters are extracted: first, an access masquerading strategy specifically configured to modify underlying communication characteristics to achieve identity concealment against the target platform's existing machine traffic identification and interception mechanisms; and second, a parameter to guide subsequent processing of complex data. The system retrieves data collection rules that precisely extract the required business fields from the returned message; it then calls up node status data that reflects the underlying network physical quality and security indicators within the cluster, which is continuously collected and aggregated in the preceding steps. This data is used as a dynamic evaluation basis to perform real-time health checks and threshold filtering on all proxy nodes within the cluster, marking nodes that are alive and stable, without security anomalies, and whose latency meets the standards. This determines the availability status of each proxy node, thereby avoiding the risk of being blocked by the target platform and laying the foundation for subsequent state decision-making to accurately schedule requests containing spoofing and collection rules to target proxy nodes that meet the evaluation conditions.

[0096] S203: Among all agent nodes, select the target agent node whose availability status meets the preset conditions, and dynamically schedule the data collection request to the target agent node.

[0097] In this embodiment, a target proxy node refers to a network relay endpoint that meets preset requirements, which is precisely selected from a distributed cluster after a comprehensive network health and security assessment and is scheduled to carry out and actually execute a specific data scraping task. For example, in a resource pool containing hundreds or thousands of proxy IPs, a proxy server whose current network round-trip latency is less than 50 milliseconds, whose historical signaling has no records of being blocked or polluted by the platform, and whose public network exit IP's geographical location happens to be within the scope of the business regulations, is selected to represent a target proxy node that undertakes a data collection task for an international news website or Twitter.

[0098] Specifically, after completing the preliminary dynamic assessment of node health, the availability status of each proxy node (i.e., the basic traffic forwarding server distributed across different network topologies) within the underlying resource pool is obtained, reflecting its current physical connectivity quality and signaling security level. The characteristic parameters of each availability status are then compared and filtered against pre-defined conditions (representing the network performance baseline and security access threshold that must be crossed to execute the current high-coverage data collection task, such as requiring a network jitter rate of less than 1%, a survival time of more than 24 hours, and no abnormal routing interception records). After eliminating inferior nodes with high latency, frequent disconnections, or security risks, the process is further refined in each proxy... Among the nodes, target proxy nodes whose availability status fully meets the above preset conditions are accurately selected; after locking the optimal access channel, the data collection request containing complete parameter payloads such as specific access targets, underlying access masquerading strategies and business field parsing rules is extracted, and the data collection request is bypassed by the underlying load balancing algorithm or the real-time routing distribution mechanism based on the current node's concurrency margin, and dynamically scheduled across the network protocol stack (i.e., millisecond-level task distribution and traffic redirection allocation based on real-time network fluctuations) to the selected target proxy node, thereby laying a solid underlying link foundation for subsequent secure and covert access to the target platform.

[0099] S204: The target proxy node initiates access to the target platform based on the access masquerading policy and receives the response data returned by the target platform. The response data is then parsed using data collection rules to obtain the raw monitoring data.

[0100] For example, in complex network adversarial environments, target platforms typically deploy defense mechanisms such as machine traffic identification and sophisticated front-end rendering technologies. Traditional automated scripts are prone to triggering interception and struggle to extract unstructured content. Therefore, this step constructs a low-level feature simulation and multimodal data parsing mechanism at the data acquisition and execution layer. On one hand, the system modifies communication characteristics at the physical protocol and runtime environment levels to simulate normal user interaction behavior, thereby circumventing the target platform's automated detection mechanisms. On the other hand, the system combines text path addressing and image optical character recognition technologies to jointly parse complex response packets containing webpage source code and image elements. This collaborative mechanism ensures that the system can stably extract the required structured business data even in network environments with anti-crawler measures and data obfuscation.

[0101] In one possible implementation, the target proxy node initiates access to the target platform based on an access masquerading policy and receives response data returned by the target platform. Specifically, this includes: modifying the graphics rendering interface characteristics in the access environment of the target proxy node based on the fingerprint simulation rules in the access masquerading policy to generate a masquerading browser fingerprint; simulating a preset version of secure communication handshake characteristics based on the security protocol masquerading rules in the access masquerading policy to generate a masquerading handshake characteristic; encapsulating the masquerading browser fingerprint and the masquerading handshake characteristic into a data collection request to obtain a packaged data collection request; obtaining the anti-crawling frequency limit corresponding to the target platform in the data collection rules, and determining the sending frequency for the packaged data collection request based on the anti-crawling frequency limit; and sending the packaged data collection request to the target platform through the target proxy node according to the sending frequency, and receiving the response data returned by the target platform.

[0102] In this embodiment, a fake browser fingerprint refers to a set of false but logically consistent identifiers of the underlying hardware and software environment of a client browser or mobile app, created artificially through low-level code interception and parameter modification techniques without altering the actual hardware and operating system. For example, this fingerprint could be a hash string generated by modifying system-level parameters such as the Canvas drawing API return value, the WebGL renderer model (e.g., changing the virtual graphics card model of a real server cloud host to a common NVIDIA graphics card model for a regular personal computer), and the screen resolution. This hash string is used to deceive the target platform's anti-crawler system, making it mistakenly believe that a real ordinary user is using a mainstream Chrome browser to access Google or the New York Times, or to simulate a specific mobile phone model to access TikTok.

[0103] Specifically, based on the fingerprint simulation rules (strategy rules guiding how to replace and obfuscate client environment information) in the access masquerading strategy (i.e., a pre-configured set of underlying parameter tampering and behavior simulation logics used to bypass the target platform's machine traffic identification mechanism), the system dynamically switches the masquerading logic according to the front-end form of the target platform. For example, when collecting data from web websites such as Google or The New York Times, at the operating system or browser kernel level, the system modifies the graphics rendering interface characteristics in the access environment of the target proxy node (e.g., tampering with the underlying graphics card driver information and rendering hash value extracted from the WebGL or Canvas interface), thereby generating a fake browser fingerprint that can circumvent the target platform's detection mechanism at the underlying logic level. When initiating access to mobile applications such as TikTok, the fingerprint simulation rules specifically simulate a specific mobile device environment (e.g., forging underlying parameters of a specific version of iOS or Android system, device model identifier) ​​and dynamically generate security signature algorithm parameters (such as specific request tokens or X-Bogus anti-counterfeiting parameters) that match the mobile environment. At the same time, based on the security protocol masquerading rules in the above access masquerading strategy (i.e., feature modification strategies for encrypted communication protocol layers such as SSL / TLS), the system simulates a preset version (such as TLS 1.2 or TLS) at the network transport layer. 1.3 Secure communication handshake characteristics (including key JA3 fingerprint parameters such as Cipher Suites and supported Extensions) of specific mainstream browser client versions are used to generate fake handshake characteristics that appear as real and normal traffic at the network packet capture level. At the application layer, the original collection instructions are assembled, and the fake browser fingerprint constructed at the runtime environment level and the fake handshake characteristics constructed at the network protocol stack level are completely encapsulated as custom request headers or underlying communication parameters into the data collection request to be sent. Through this deep parameter injection and combination, an encapsulated data collection request that meets the requirements of covert access is obtained. The data collection rules used to guide the execution of the collection task are parsed to obtain the anti-scraping frequency limits corresponding to the target platform being collected (e.g., Twitter's restrictions on single IP addresses). The algorithm uses the GraphQL interface access frequency limit or Google search frequency threshold to determine the dynamic and human-like sending frequency for the encapsulated data collection request by introducing a time jitter algorithm or Poisson distribution model based on this anti-crawling frequency limit. Strictly following the calculated sending frequency, the algorithm establishes a low-level network connection through the selected target proxy node to securely and covertly send the encapsulated data collection request carrying the full set of disguised parameters to the target platform. This successfully bypasses the platform's defense mechanism and receives the response data returned by the target platform (i.e., real business message content that has not been intercepted or poisoned, including the target webpage's HTML source code, the valid business payload returned by the JSON data interface (such as tweet list, news text, etc.)).

[0104] In one possible implementation, the response data is parsed using data acquisition rules to obtain raw monitoring data. Specifically, this includes: parsing the data acquisition rules, identifying the target monitoring scenario corresponding to the target platform, and matching a preset data extraction script corresponding to the target monitoring scenario from a preset script library; extracting structured text content from the page source code of the response data using path language rules and preset character matching rules in the preset data extraction script; locating image elements in the response data, recognizing the text in the image elements using a preset optical character recognition algorithm to obtain visual presentation content; and associating and mapping the structured text content with the visual presentation content to generate raw monitoring data containing multi-dimensional business characteristics.

[0105] In this embodiment, the raw monitoring data refers to a basic data set with business analysis value initially formed from underlying network return messages containing various unstructured front-end format codes, after automated targeted extraction, data cleaning, and cross-modal (text and image) splicing and recombination. For example, this data can be a multidimensional business data packet in JSON format containing structured plaintext fields such as "news title," "author," "publication time," and "body content" extracted from the details page of a New York Times or Facebook webpage, and embedded text in images extracted from webpage posters using image recognition technology. The two are bound together by underlying page ID mapping.

[0106] Specifically, after receiving the complete message returned from the network layer, the system deeply analyzes the data collection rules issued from the front end (i.e., the set of configuration parameters and strategies that guide how to extract effective information from complex response data), accurately identifies the target monitoring scenario corresponding to the target platform to be processed (representing the collection task environment of a specific business dimension, such as "international news and public opinion monitoring scenario" or "overseas social media dynamic tracking scenario"). Based on this scenario identifier, the system automatically retrieves and matches the preset data extraction script (which refers to pre-written automated parsing code files specifically designed for a particular webpage's DOM tree or JSON interface structure) that corresponds to the target monitoring scenario and is adapted to the current webpage or interface structure from the preset script library that centrally stores various parsing template codes. Subsequently, the system calls different parsing logics according to the modality of the response data: for web-based news websites such as The New York Times, it calls the path language rules embedded in the preset data extraction script (e.g., XPath expressions or CSS selectors used to accurately locate specific node positions in XML or HTML document-level structures, such as accurately locating the content containing the article text). <article>Tags, including news headlines <h1>The system uses tags and preset character matching rules (i.e., regular expressions used to extract core values ​​or phrases from long string paragraphs according to specific rules) to remove redundant front-end rendering tags and decoration code from the page source code corresponding to the received response data, and extracts the required structured text content (representing clean character data that has been cleaned, extracted, and arranged according to preset fields, such as article content, title, author identifier, timestamp, etc.). For highly interface-based mobile or single-page application platforms such as TikTok and Twitter, the system directly uses preset data extraction scripts to parse the returned JSON data packets and extracts plaintext fields such as "blogger UID" and "tweet description" by key-value pair hierarchy. At the same time, to address the situation where key data is hidden as images due to anti-crawler strategies in web pages or tweets, the system further locates image elements in the response data (i.e., pointing to various price screenshots, detail long images, or verification codes containing business text) in the page document object model tree. The system links and loads the media file of the certificate code, and uses a preset optical character recognition algorithm (i.e., OCR algorithm model, used to binarize, detect contours and extract pixel features of images to convert them into machine-readable characters) to deeply scan and recognize the text in the image elements, obtaining the cross-modal parsed visual presentation content (representing the text sequence reconstructed from the pure visual pixel matrix); by extracting the unique business identifier common to the webpage context (such as the unique ID of the same page URL or tweet) as the mapping key, the structured text content obtained by parsing through the source code path or interface is bound and associated with the visual presentation content reconstructed by the image algorithm at the underlying level, thereby realizing the cross-dimensional fusion of different modal data sources, splicing and generating original monitoring data containing multi-dimensional business features (i.e., a set of business indicators that integrate explicit text dimensions and implicit image dimensions), providing a comprehensive and accurate data source base for subsequent compliance flow and de-identification analysis.

[0107] S205: Store the raw monitoring data in the local storage space corresponding to the target network area, use the preset compliance rule base that matches the target network area to perform compliance verification on the raw monitoring data to generate compliance verification results, and clean up the raw monitoring data that failed the compliance verification based on the compliance verification results.

[0108] For example, in cross-regional network data collection scenarios, different regions typically have different data localization and cross-border transmission compliance requirements. This step deploys an automated compliance verification mechanism at the physical edge node before the raw business data is centrally processed. The system translates relevant data protection compliance requirements into executable storage configurations and flow interception strategies, ensuring that monitoring data stored in the local space meets requirements such as location access and encrypted transmission, and dynamically verifies the data retention status at the front end. Once a violation risk is identified, the system directly executes physical-level connection blocking and data erasure operations, thereby cutting off the flow path of illegal data at the source and ensuring the compliance of the underlying data processing.

[0109] In one possible implementation, a pre-defined compliance rule base matching the target network region is used to perform compliance verification on the original monitoring data to generate a compliance verification result. Based on the compliance verification result, the original monitoring data that fails the compliance verification is cleaned up. Specifically, this includes: obtaining a pre-defined compliance template corresponding to the target network region. The pre-defined compliance template contains multi-dimensional compliance logic verification rules for the original monitoring data. These multi-dimensional compliance logic verification rules include data collection location access rules, data transmission path encryption rules, and data retention duration limit rules. A pre-defined compliance rule base is constructed based on the pre-defined compliance template. The temporary storage duration of the original monitoring data in the local storage space is monitored in real time, and the collection node location of the target proxy node and the transmission path information of the original monitoring data are extracted. The temporary storage duration, collection node location, and transmission path information are input into the pre-defined compliance rule base and compared with the corresponding data retention duration limit rules, data collection location access rules, and data transmission path encryption rules, respectively, to generate a compliance verification result containing a risk assessment indicator. When the risk assessment indicator indicates a violation risk, the communication session between the target proxy node and the target platform is interrupted, and the storage destruction interface of the local storage space is called to perform a data overwrite and erase operation on the storage address corresponding to the original monitoring data.

[0110] In this application embodiment, the compliance verification result refers to the comprehensive evaluation conclusion output after multi-dimensional legality and compliance testing of the acquired underlying business data. It is used to indicate whether the currently collected data meets the pre-set security legal boundaries in terms of physical location, transmission link, and lifecycle. For example, the result can be a structured log file containing a status code of "compliant" or "violation (there is a risk of cross-border data transmission)" and a corresponding set of violation feature fields.

[0111] Specifically, a pre-defined compliance template (a pre-compiled, standardized policy configuration file that transforms abstract legal and regulatory provisions into computer-executable instructions) corresponding to the target network region (referring to a logical network segment governed by a specific geographical location or specific data protection laws, such as a European data center governed by the EU GDPR, or a North American western node cluster governed by the California CCPA) is obtained. This pre-defined compliance template contains multi-dimensional compliance logic verification rules for the raw monitoring data (representing machine judgment logic covering multiple dimensions such as data lifecycle and flow links). These multi-dimensional compliance logic verification rules specifically include: data collection location access rules (used to limit the whitelist of IP regions where the physical servers performing the collection task are located, ensuring that the physical location of the nodes collecting Twitter or New York Times data is consistent with the target legal domain), data transmission path encryption rules (requiring that message network transmission must use a specific level of secure encryption links such as TLS, strictly preventing man-in-the-middle eavesdropping during cross-border public network transmission), and data retention time limit rules (specifying the maximum allowed number of seconds for sensitive data to reside in a temporary cache disk); based on the above-analyzed pre-defined compliance template, data is processed in memory or a security sandbox. The system dynamically builds a pre-defined compliance rule base for real-time comparison and interception within the container. Through the underlying storage monitoring component, it monitors in real time the temporary storage duration of the original monitoring data in the local storage space (i.e., the isolated disk area that meets the physical storage requirements of the target network area data localization) (representing the precise time span experienced by the data since it was written to the disk). It also deeply analyzes the underlying network connection session to extract the collection node location of the target proxy node that performed this collection task (i.e., the real geographical coordinates of the physical server or the location of the base station) and the transmission path information of the original monitoring data from the source to the destination (representing the number of network routing hops and the encryption protocol status of each hop node).

[0112] Furthermore, the collected temporary storage duration, collection node location, and transmission path information are used as input vectors and input into a preset compliance rule base. A comparison engine is then used to perform threshold comparisons against the corresponding data retention duration limit rules, data collection location access rules, and data transmission path encryption rules in the base. If any dimension fails to match the security whitelist or exceeds the threshold limit (for example, a message containing user device information scraped from international TikTok is found attempting to be transmitted back through an unencrypted cross-border routing node, or the disk residency time at a European territory node exceeds the temporary processing time limit stipulated by GDPR), a risk assessment identifier is generated (indicating unauthorized data transmission). The compliance verification result (Boolean value or alarm error code of the transfer or illegal retention behavior); when the risk judgment mark indicates that there is a risk of violation, a connection reset message (RST) is forcibly sent at the network protocol layer to directly interrupt the underlying communication session between the target agent node and the target platform, cut off the physical channel that may leak data, and immediately call the storage destruction interface of the underlying operating system of the local storage space (i.e., the disk underlying operation API with anti-data recovery mechanism), and use algorithms such as multiple random number filling to perform physical data overwrite and erase operations on the storage address corresponding to the original monitoring data on the disk medium, thereby realizing a closed loop of cross-border data security cleanup based on the compliance verification result.

[0113] S206: Identify the data attributes of the original monitoring data that have passed compliance verification, match the corresponding de-identification rules according to the data attributes, and perform de-identification processing on the original monitoring data that has passed compliance verification based on the de-identification rules to obtain de-identified monitoring data.

[0114] For example, in data flow and cross-domain analysis scenarios, applying a uniform global encryption strategy to monitoring data containing sensitive information can lead to the loss of the original logical structure of the data, affecting the availability of downstream business systems; without processing, there is a risk of privacy leakage. Therefore, this step establishes a desensitization mechanism based on dynamic hierarchical classification of data attributes. The system first identifies the fields with different security protection levels in the original data through traversal comparison, and then matches the corresponding hash or mask desensitization algorithm for sensitive data of different levels according to the preset mapping relationship. While eliminating the true identity pointing of sensitive data, it retains the original layout structure of non-sensitive data, taking into account both data security compliance and the business needs of downstream statistical analysis.

[0115] In one possible implementation, the data attributes of the original monitoring data that has passed compliance verification are identified, corresponding de-identification rules are matched according to the data attributes, and the original monitoring data that has passed compliance verification is de-identified based on the de-identification rules to obtain de-identified monitoring data. Specifically, this includes: using a preset sensitive data classification dictionary to perform field traversal comparison on the original monitoring data that has passed compliance verification, the sensitive data classification dictionary is configured with a first sensitivity label and a second sensitivity label based on different security protection levels; extracting the field content that matches the first sensitivity label as a first data fragment, extracting the field content that matches the second sensitivity label as a second data fragment, and matching the first sensitivity label with the second sensitivity label as a second data fragment. The two sensitivity labels are identified as data attributes. Based on the label mapping relationship in the preset desensitization rule library, the irreversible encryption rule bound to the first sensitivity label and the character masking rule bound to the second sensitivity label are extracted respectively, and the irreversible encryption rule and the character masking rule are combined into a desensitization rule. The hash salting algorithm in the irreversible encryption rule is called to perform hash calculation on the first data segment to generate an encrypted segment, and the preset masking parameter in the character masking rule is called to perform character replacement operation on the second data segment to generate a masked segment. According to the original field layout structure of the original monitoring data, the encrypted segment, the masked segment, and the reserved field content of the missing label are spliced ​​and assembled to generate desensitized monitoring data.

[0116] In this application embodiment, the de-identified monitoring data refers to a secure output data set that, after undergoing multi-level security cleaning and obfuscation, completely strips away the entity's real privacy identity while retaining the underlying basic business logic structure and contextual relationships. For example, this data could be a JSON business record file that replaces the real registered name (e.g., "John Doe") of an ordinary user on overseas social media (such as Facebook or Twitter) with partially obscured mask characters (e.g., "Je"), and converts the unique identity ID of the real user account into a random 256-bit hash value (i.e., irreversible encryption), while still retaining the original layout format of the user's "news title", "tweets or comments", "publication time (e.g., 2024-03-15 10:00:00)" and "publicly disclosed institutional author identifier".

[0117] Specifically, after ensuring the legality of physical storage and transfer of data, for the original monitoring data that has passed compliance verification (i.e., the pending business table that has eliminated the risk of physical violations such as unauthorized transmission), the field-by-field comparison of each key-value pair or data column within it is carried out using a preset sensitive data classification dictionary. The aforementioned sensitive data classification dictionary is used to represent a standard reference lexicon that pre-enters and maintains regular expressions and corresponding security levels for various overseas privacy fields (such as Personally Identifiable Information (PII)). Internally, it is configured with first and second sensitivity labels based on different security protection levels: for example, the first sensitivity label corresponds to a top-secret level identifier with extremely high privacy risk (such as the Social Security Number (SSN) of overseas users, PCI data from the payment card industry, or the underlying device hardware MAC address of international TikTok), while the second sensitivity label corresponds to a confidential level identifier with medium privacy risk (such as the international mobile phone number linked to a Twitter / Facebook account, the user's email prefix, or their real registered name). During the traversal process, the underlying field content that matches the first sensitivity label is independently extracted as the first data fragment, and the field content that matches the second sensitivity label is independently extracted as the second data fragment. The identified first and second sensitivity labels are then summarized and determined as data attributes reflecting the overall privacy risk level of this batch of data.

[0118] Furthermore, based on the hard-coded label mapping relationship (used to represent the unique binding pointer between sensitivity levels and corresponding security algorithms) in the preset desensitization rule library (referring to a policy repository that centrally stores configuration files for various data obfuscation, encryption, and desensitization algorithms), the irreversible encryption rules strongly bound to the first sensitivity label (i.e., one-way hashing algorithm strategies that cannot reverse plaintext from ciphertext) and the character masking rules bound to the second sensitivity label (i.e., replacement strategies that use asterisks or specific symbols to mask parts of plaintext) are precisely extracted. These irreversible encryption rules and character masking rules are then instantiated and concatenated in memory to construct a complete desensitization rule for the current data batch. Next, the substantive data cleaning stage begins, calling the hash salting algorithm embedded in the irreversible encryption rules (i.e., the underlying defense mechanism that inserts random strings before and after the original plaintext before performing SHA-256 hash calculations) to perform high-intensity hash calculations on the first data segment, generating an encrypted segment that completely eliminates the original characteristics (e.g., completely converting the obtained mobile device fingerprint or precise geographic coordinates into random hash values). The system calls the preset masking parameters set in the character masking rules (format configuration parameters that indicate retaining the first three and last four characters and replacing the middle with specific characters) to perform local character replacement operations on the second data segment, generating a partially visible masked segment (for example, masking the overseas user's registered email "john.doe@gmail.com" as "joh**@gmail.com"). In order not to disrupt the parsing logic of downstream business systems or data platforms, the system strictly follows the original field arrangement structure of the original monitoring data (i.e., the nesting level of the original JSON tree or the table column order of a relational database). The system splices and assembles the encrypted segment that has undergone high-strength processing, the masked segment that has undergone partial masking, and the retained field content that did not hit any sensitive tags in the first step of traversal (such as non-privacy public business data that is of core value for public opinion analysis, such as "news title", "tweet text content", "information release time" and "public author and organization name"). This process seamlessly generates desensitized monitoring data that can be securely transferred and analyzed by downstream systems while completely eliminating the risk of cross-border privacy leakage.

[0119] S207: The desensitized monitoring data, node status data and compliance verification results are fused to generate multi-dimensional fused features. The multi-dimensional fused features are then input into a preset anomaly identification model for anomaly analysis to obtain the anomaly event results.

[0120] For example, in real-world network data monitoring scenarios, single-dimensional data anomalies often fail to accurately pinpoint the root cause (e.g., a sudden drop in business data crawling volume could be due to excessive network latency at the underlying proxy nodes, request interception caused by an upgrade to the target platform's anti-crawler mechanism, or forced data destruction due to the triggering of local compliance policies). Therefore, this step cross-domain correlates anonymized monitoring data representing the quality of upper-layer business data extraction, node status data representing the operational status of the underlying physical links, and compliance verification results representing the security boundaries of data flow. The constructed multi-dimensional fusion features provide a global view of the system's operation for the pre-defined anomaly identification model, enabling the model to move beyond the limitations of traditional single-point monitoring. It comprehensively analyzes the interplay between network, business, and compliance layer features, thereby accurately inferring the global root cause of data collection obstruction or system fluctuations, outputting objective anomaly event results, and providing a reliable diagnostic basis for subsequent automated self-healing strategies.

[0121] In one possible implementation, anonymized monitoring data, node status data, and compliance verification results are fused to generate multidimensional fused features. These multidimensional fused features are then input into a pre-defined anomaly detection model for anomaly analysis to obtain anomaly event results. Specifically, this includes: extracting business content features from the anonymized monitoring data, extracting node operation evaluation parameters from the node status data, and extracting compliance status identifiers from the compliance verification results; normalizing the business content features, node operation evaluation parameters, and compliance status identifiers to obtain normalized feature vectors; and concatenating and combining the normalized feature vectors from multiple consecutive time windows to obtain a multidimensional time series as the multidimensional fused feature. The feature matrix is ​​generated and input into the anomaly recognition model. The anomaly recognition model calculates the similarity score between the multidimensional time-series feature matrix and each benchmark feature sequence in the preset anomaly scene feature library. When the maximum similarity score is greater than or equal to a preset threshold, the target benchmark feature sequence corresponding to the maximum similarity score is determined, and the anomaly type identifier bound to the target benchmark feature sequence is extracted. Using the anomaly type identifier as a mapping index, the anomaly cause tracing information and processing suggestions associated with the anomaly type identifier are queried and retrieved from the anomaly scene feature library. The anomaly type identifier, anomaly cause tracing information, and processing suggestions are encapsulated to generate anomaly event results.

[0122] In this embodiment, the abnormal event result refers to the standardized structured data output after multi-dimensional heterogeneous data feature fusion and deep temporal similarity matching analysis, which includes the abnormal state of global network collection and its response strategy. It is used to represent the specific abnormal links and corresponding handling strategies that occur in the current distributed data monitoring system at the business crawling, underlying node operation or legal compliance level. For example, the result can be a JSON format warning payload package containing "the abnormal type is identified as 'the node IP is blocked by the target platform'", "the abnormal cause tracing information is 'the access frequency to the target platform exceeds the anti-crawling threshold within 5 consecutive minutes'", and "the handling suggestion is 'immediately suspend the current proxy IP and start the backup IP pool, while reducing the global crawling frequency by 30%'".

[0123] Specifically, the system parses and extracts business content features reflecting the success rate of data capture or the rate of missing key fields from the anonymized monitoring data containing cleaned business fields (used to indicate the quality fluctuation of upper-layer business data). In parallel, it extracts node operation evaluation parameters such as network latency spikes or connection drop frequencies from node status data reflecting the underlying network performance (used to indicate the health and physical connectivity indicators of the underlying network channel). It also extracts compliance status identifiers (i.e., Boolean values ​​or security rating error codes) from compliance verification results reflecting legal and physical security status to determine whether unauthorized transmission or overdue retention has occurred. For the three types of heterogeneous data with differences in numerical range and physical unit dimensions, scaling algorithms such as max-min normalization or Z-Score normalization are used to perform cross-dimensional normalization processing on the extracted business content features, node operation evaluation parameters, and compliance status identifiers to eliminate the dimensional barriers of the underlying data, resulting in a normalized feature vector that can be used for indiscriminate distance calculation and weight evaluation within a unified vector space.

[0124] Furthermore, to accurately capture the dynamic evolution trend and contextual relationships of network anomalies, the normalized feature vectors generated in multiple consecutive time windows (e.g., the past 10 one-minute data slices) are stacked and combined chronologically according to the system's set time step. This yields a specific data structure that serves as the multidimensional fusion feature carrying the macro-global state, namely, a multidimensional temporal feature matrix (a two-dimensional or three-dimensional numerical matrix that not only contains multidimensional static features but also incorporates the dynamic change trend of the time series). This multidimensional temporal feature matrix is ​​then input into a pre-trained network based on Long Short-Term Memory (LSTM) or Temporal Convolutional Network (TCON). In anomaly detection models built with deep learning architectures such as Network (TCN) (representing an AI detection hub with the ability to extract massive temporal features and match patterns), the algorithm uses its built-in distance metric mechanism to calculate the dynamic time warping (DTW) distance or cosine correlation between the current input multidimensional temporal feature matrix and the pre-set anomaly scenario feature library (an offline experience database that has collected and labeled massive historical anomaly feature slices such as web crawler obstruction, node downtime, or compliance breaches). This results in a series of similarity scores that quantitatively reflect the degree of matching between the current system operating state and the fluctuation trajectory of various known anomalies.

[0125] Furthermore, when the ranking comparison reveals that the highest similarity score among the calculated similarity scores is greater than or equal to a preset threshold (e.g., 0.85 or 85%) representing the lower limit of confidence in confirming a substantial anomaly, it is determined that the current system state has irreversibly fallen into a known risk pattern. The waveform template corresponding to this highest similarity score is then identified as the target baseline feature sequence. An anomaly type identifier (e.g., a specific error code representing rate limiting) strongly bound to this target baseline feature sequence is extracted from its metadata attributes. Using this extracted anomaly type identifier as the underlying mapping index, key-value pair queries are then performed. The system queries and retrieves relevant anomaly cause tracing information (such as the specific node IP path that triggered the interception, snapshots of specific fields that caused the violation, etc.) associated with the anomaly type identifier from the association table of the aforementioned anomaly scenario feature library. This information is intended to help operations and maintenance personnel locate the root cause of the problem. It also retrieves pre-prepared processing suggestions (representing standard operating procedures or adjustment instructions formulated for this specific anomaly). The system then performs a structured format assembly and encapsulation of the anomaly type identifier, anomaly cause tracing information, and the aforementioned processing suggestions to generate the aforementioned anomaly event result that can directly trigger a warning on the monitoring dashboard or drive the underlying policy self-healing component to execute a response.

[0126] In one possible implementation, before inputting the multidimensional fused features into a preset anomaly detection model for anomaly analysis, the method further includes an offline training phase for the anomaly detection model, specifically including:

[0127] A massive amount of system operation logs from historical time periods are acquired, and historical anonymized monitoring data, historical node status data, and historical compliance verification results are extracted. These are then sliced ​​according to a preset time window to construct a historical multi-dimensional temporal feature matrix sample set. An expert annotation mechanism is introduced to assign a corresponding real-state label (e.g., normal fluctuation, triggering anti-crawling interception, node link failure, unauthorized data, etc.) to each historical feature matrix in the sample set. An initial deep temporal network structure (e.g., a network model containing an input layer, multiple LSTM hidden units, and a Softmax output layer) is built. The historical multi-dimensional temporal feature matrix is ​​used as input, and the real-state label is used as the expected output. The cross-entropy loss function is used to calculate the error loss between the predicted probability distribution of the network output and the real-state label. A gradient descent algorithm (e.g., the Adam optimizer) is used for backpropagation based on this error loss, iteratively updating the connection weights and bias parameters in the network model. When the value of the loss function converges to a preset minimum threshold or reaches the maximum number of iterations, training stops, and the solidified network structure and parameters are saved as the preset anomaly recognition model that can be called online in real time.

[0128] S208: Generate corresponding early warning information based on the results of abnormal events, and update data collection rules and / or access spoofing strategies according to the results of abnormal events.

[0129] In this embodiment, the early warning information refers to the notification payload, which includes risk classification and on-site snapshot, output to the external operation and maintenance module or management terminal when there is a sudden situation in the underlying link, business logic or physical compliance of the monitoring system that deviates from the normal benchmark. It is used to indicate that the current data collection architecture is in a security alarm state that requires manual intervention or that the automatic self-healing mechanism has been activated. For example, the information can be a structured message or a visual pop-up window prompt pushed through the enterprise's internal communication interface, containing content such as "High risk: The agent cluster's North American nodes are being blocked on a large scale by Twitter's risk control system, and the capture success rate has dropped to below 30%. IP rotation has been initiated as recommended."

[0130] Specifically, after multi-dimensional time-series fusion and model calculation, the abnormal event results from the preceding steps are analyzed (i.e., a structured diagnostic package containing specific abnormality type identifiers, underlying source evidence chains, and pre-set processing suggestions). Based on the risk rating (e.g., fatal, severe, general) and traceability summary extracted from the abnormal event results, the underlying message routing middleware is invoked to generate corresponding early warning information matching the current abnormality level and scenario, and this information is asynchronously pushed to the monitoring bus or automated operation and maintenance processing queue to achieve immediate alerts for business disruptions or data compliance risks. Simultaneously, to endow the entire distributed monitoring architecture with adaptive anti-crawling game-theoretic and closed-loop repair capabilities, the specific obstruction reasons indicated in the abnormal event results are further analyzed in depth. For example, if the abnormality is caused by Google's search engine blocking CAPTCHA verification for a specific proxy IP, or by a hot update of the front-end encryption algorithm parameters of TikTok causing request signature verification failure, or by Twitter upgrading the call frequency risk control threshold for a specific API. Based on the self-healing suggestions pre-set in the results of the abnormal event, the system dynamically reconstructs and updates the data collection rules in the memory configuration center by issuing a hot update command (such as updating the XPath addressing path for extracting news content), and / or updates the access spoofing strategy (such as automatically introducing AI captcha solving components to bypass Google verification, dynamically synchronizing the latest App signature algorithm model of TikTok, or reducing the concurrent request rate for the Twitter interface), thereby quickly restoring high availability access to the target platform and accurate data capture without downtime reconstruction.

[0131] Figure 3 This is a schematic diagram of a module of an overseas data monitoring system according to an embodiment of this application. This system can be implemented through software, hardware, or a combination of both, forming all or part of a larger system. For example... Figure 3 As shown, the system includes:

[0132] The status acquisition module 301 is used to deploy a distributed agent cluster in the target network area and continuously collect the network operation characteristics of each agent node in the distributed agent cluster to generate node status data.

[0133] The request evaluation module 302 is used to obtain data collection requests for the target platform within the target network area and determine the availability status of each proxy node based on node status data. The data collection request includes access masquerading strategies and data collection rules that match the target platform.

[0134] The dynamic scheduling module 303 is used to select target agent nodes whose availability status meets preset conditions from among the agent nodes, and dynamically schedule data collection requests to the target agent nodes.

[0135] The data parsing module 304 is used to initiate access to the target platform through the target proxy node based on the access masquerading policy and receive the response data returned by the target platform. It then uses data collection rules to parse the response data to obtain the raw monitoring data.

[0136] The compliance verification module 305 is used to store the original monitoring data in the local storage space corresponding to the target network area, use a preset compliance rule library that matches the target network area to perform compliance verification on the original monitoring data to generate compliance verification results, and clean up the original monitoring data that failed the compliance verification based on the compliance verification results.

[0137] The data desensitization module 306 is used to identify the data attributes of the original monitoring data that has passed the compliance verification, match the corresponding desensitization rules according to the data attributes, and perform desensitization processing on the original monitoring data that has passed the compliance verification based on the desensitization rules to obtain desensitized monitoring data.

[0138] The fusion analysis module 307 is used to fuse desensitized monitoring data, node status data and compliance verification results to generate multi-dimensional fusion features. The multi-dimensional fusion features are then input into a preset anomaly identification model for anomaly analysis to obtain anomaly event results.

[0139] The early warning feedback module 308 is used to generate corresponding early warning information based on the results of abnormal events, and to update data collection rules and / or access spoofing strategies according to the results of abnormal events.

[0140] Based on the above embodiments, as an optional embodiment, the status acquisition module 301 is specifically used for: acquiring network connection requests for each proxy node, extracting signaling protocol features from the network connection requests, and monitoring the signaling protocol features in real time based on preset signaling protection rules; when an abnormal routing identifier or unauthorized operation instruction is identified from the signaling protocol features based on the preset signaling protection rules, the network connection request containing the signaling protocol features is determined as an abnormal connection request, the abnormal connection request is intercepted, and a security interception record corresponding to the abnormal connection request is generated; collecting the latency parameters, stability parameters, and survival parameters of each proxy node, and determining the latency parameters, stability parameters, survival parameters, and signaling security feature parameters statistically derived from the security interception records as the network operation features of each proxy node; aggregating the network operation features to obtain aggregated features, and inputting the aggregated features into a preset machine learning evaluation algorithm for dynamic evaluation of node health, generating node status data.

[0141] Based on the above embodiments, as an optional embodiment, the data parsing module 304 is specifically used to: modify the graphics rendering interface features in the access environment of the target proxy node based on the fingerprint simulation rules in the access masquerading policy to generate a fake browser fingerprint; simulate the security communication handshake features of a preset version based on the security protocol masquerading rules in the access masquerading policy to generate a fake handshake feature; encapsulate the fake browser fingerprint and the fake handshake feature into a data collection request to obtain an encapsulated data collection request; obtain the anti-crawling frequency limit corresponding to the target platform in the data collection rules, and determine the sending frequency for the encapsulated data collection request based on the anti-crawling frequency limit; send the encapsulated data collection request to the target platform through the target proxy node according to the sending frequency, and receive the response data returned by the target platform.

[0142] Based on the above embodiments, as an optional embodiment, the data parsing module 304 is specifically used for: parsing data collection rules, identifying the target monitoring scenario corresponding to the target platform, and matching the preset data extraction script corresponding to the target monitoring scenario from the preset script library; extracting structured text content from the page source code of the response data through the path language rules and preset character matching rules in the preset data extraction script; locating image elements in the response data, recognizing the text in the image elements using a preset optical character recognition algorithm, and obtaining visual presentation content; associating and mapping the structured text content with the visual presentation content to generate raw monitoring data containing multi-dimensional business characteristics.

[0143] Based on the above embodiments, as an optional embodiment, the compliance verification module 305 is specifically used for: obtaining a preset compliance template corresponding to the target network area, the preset compliance template containing multi-dimensional compliance logic verification rules for the original monitoring data, the multi-dimensional compliance logic verification rules including data collection location access rules, data transmission path encryption rules, and data retention duration limit rules; constructing a preset compliance rule library based on the preset compliance template; monitoring the temporary storage duration of the original monitoring data in the local storage space in real time, and extracting the collection node location of the target agent node and the transmission path information of the original monitoring data; inputting the temporary storage duration, collection node location, and transmission path information into the preset compliance rule library, and comparing and verifying them with the corresponding data retention duration limit rules, data collection location access rules, and data transmission path encryption rules, respectively, and generating a compliance verification result containing a risk judgment identifier; when the risk judgment identifier indicates that there is a risk of violation, interrupting the communication session between the target agent node and the target platform, and calling the storage destruction interface of the local storage space to perform a data overwrite and erase operation on the storage address corresponding to the original monitoring data.

[0144] Based on the above embodiments, as an optional embodiment, the data desensitization module 306 is specifically used to: perform field traversal comparison on the original monitoring data that has passed compliance verification using a preset sensitive data classification dictionary, wherein the sensitive data classification dictionary is configured with a first sensitivity label and a second sensitivity label based on different security protection levels; extract the field content that matches the first sensitivity label as a first data fragment, extract the field content that matches the second sensitivity label as a second data fragment, and determine the first sensitivity label and the second sensitivity label as data attributes; and map the data according to the label in the preset desensitization rule library. Based on the mapping relationship, the irreversible encryption rules bound to the first sensitivity label and the character masking rules bound to the second sensitivity label are extracted respectively, and the irreversible encryption rules and character masking rules are combined into desensitization rules; the hash salting algorithm in the irreversible encryption rules is called to perform hash calculation on the first data segment to generate an encrypted segment, and the preset masking parameters in the character masking rules are called to perform character replacement operation on the second data segment to generate a masked segment; according to the original field layout structure of the original monitoring data, the encrypted segment, masked segment and the reserved field content of the missing labels are spliced ​​and assembled to generate desensitized monitoring data.

[0145] Based on the above embodiments, as an optional embodiment, the fusion analysis module 307 is specifically used for: extracting business content features from de-identified monitoring data, extracting node operation evaluation parameters from node status data, and extracting compliance status identifiers from compliance verification results; normalizing the business content features, node operation evaluation parameters, and compliance status identifiers to obtain normalized feature vectors; concatenating and combining the normalized feature vectors under multiple consecutive time windows to obtain a multidimensional time-series feature matrix as a multidimensional fusion feature, and inputting the multidimensional time-series feature matrix into the anomaly identification model; calculating the similarity score between the multidimensional time-series feature matrix and each benchmark feature sequence in the preset anomaly scenario feature library through the anomaly identification model; when the maximum similarity score in the similarity score is greater than or equal to a preset threshold, determining the target benchmark feature sequence corresponding to the maximum similarity score, and extracting the anomaly type identifier bound to the target benchmark feature sequence; using the anomaly type identifier as a mapping index, querying and retrieving the anomaly cause tracing information and processing suggestions associated with the anomaly type identifier from the anomaly scenario feature library; and encapsulating the anomaly type identifier, anomaly cause tracing information, and processing suggestions to generate anomaly event results.

[0146] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here. Through the coordinated operation of the above modules, this system can adapt to the high-frequency data collection needs of overseas platforms such as TikTok, The New York Times, and Twitter. While ensuring that key fields such as web page content, news titles, and publishers are efficiently and covertly captured, it strictly guarantees the compliance of cross-border data in terms of physical storage and field anonymization.

[0147] This embodiment also discloses an electronic device, referring to... Figure 4 The electronic device may include: at least one processor 401, at least one communication bus 402, user interface 403, network interface 404, and at least one memory 405.

[0148] The communication bus 402 is used to enable communication between these components.

[0149] The user interface 403 may include a display screen and a camera. Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.

[0150] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0151] The processor 401 may include one or more processing cores. The processor 401 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 405, and by calling data stored in memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 401.

[0152] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for an overseas data monitoring method.

[0153] exist Figure 4 In the electronic device shown, the user interface 403 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 401 can be used to call an application program stored in the memory 405 for an overseas data monitoring method. When executed by one or more processors 401, the electronic device executes one or more methods as described in the above embodiments.

[0154] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0155] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 405 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory 405 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0160] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the disclosure in this specification. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are considered exemplary only, and the scope of this application is defined by the claims.< / h1> < / article>

Claims

1. A method for monitoring overseas data, characterized in that, The method includes: A distributed proxy cluster is deployed in the target network area, and the network operation characteristics of each proxy node in the distributed proxy cluster are continuously collected to generate node status data. Obtain data collection requests for target platforms within the target network area, and determine the availability status of each proxy node based on the node status data. The data collection requests include access masquerading strategies and data collection rules that match the target platform. Among the agent nodes, target agent nodes whose availability status meets preset conditions are selected, and the data collection request is dynamically scheduled to the target agent node; The target proxy node initiates access to the target platform based on the access masquerading policy and receives response data returned by the target platform. The response data is then parsed using the data collection rules to obtain the raw monitoring data. The original monitoring data is stored in the local storage space corresponding to the target network area. The original monitoring data is then verified for compliance using a preset compliance rule base that matches the target network area to generate a compliance verification result. Based on the compliance verification result, the original monitoring data that fails the compliance verification is then cleaned up. Identify the data attributes of the original monitoring data that have passed the compliance verification, match the corresponding de-identification rules according to the data attributes, and perform de-identification processing on the original monitoring data that has passed the compliance verification based on the de-identification rules to obtain de-identified monitoring data; The desensitized monitoring data, the node status data, and the compliance verification results are fused to generate multi-dimensional fusion features. The multi-dimensional fusion features are then input into a preset anomaly identification model for anomaly analysis to obtain anomaly event results. Based on the results of the abnormal events, corresponding early warning information is generated, and the data collection rules and / or the access spoofing strategy are updated according to the results of the abnormal events.

2. The method according to claim 1, characterized in that, The continuous collection of network operation characteristics of each agent node in the distributed agent cluster to generate node status data specifically includes: Obtain network connection requests for each of the agent nodes, extract signaling protocol features from the network connection requests, and monitor the signaling protocol features in real time based on preset signaling protection rules; When an abnormal routing identifier or unauthorized operation instruction is identified from the signaling protocol features based on the preset signaling protection rules, the network connection request containing the signaling protocol features is determined to be an abnormal connection request, the abnormal connection request is intercepted, and a security interception record corresponding to the abnormal connection request is generated. The latency parameters, stability parameters, and survival parameters of each proxy node are collected, and the latency parameters, stability parameters, survival parameters, and signaling security feature parameters statistically derived from the security interception records are determined as the network operation characteristics of each proxy node. The network operation characteristics are aggregated to obtain aggregated features, and the aggregated features are input into a preset machine learning evaluation algorithm to dynamically evaluate the node health and generate the node status data.

3. The method according to claim 1, characterized in that, The step of initiating access to the target platform through the target proxy node based on the access masquerading policy and receiving response data returned by the target platform specifically includes: Based on the fingerprint simulation rules in the access spoofing strategy, the graphics rendering interface features in the access environment of the target proxy node are modified to generate a fake browser fingerprint. Based on the security protocol spoofing rules in the access spoofing strategy, simulate the security communication handshake features of a preset version to generate spoofed handshake features; The fake browser fingerprint and the fake handshake feature are encapsulated into the data collection request to obtain the encapsulated data collection request. Obtain the anti-crawling frequency limit corresponding to the target platform in the data collection rules, and determine the sending frequency for the encapsulated data collection request based on the anti-crawling frequency limit; According to the stated sending frequency, the encapsulated data collection request is sent to the target platform through the target proxy node, and the response data returned by the target platform is received.

4. The method according to claim 3, characterized in that, The step of parsing the response data using the data acquisition rules to obtain the raw monitoring data specifically includes: The data collection rules are parsed to identify the target monitoring scenario corresponding to the target platform, and a preset data extraction script corresponding to the target monitoring scenario is matched from the preset script library. Structured text content is extracted from the page source code of the response data using the path language rules and preset character matching rules in the preset data extraction script. Locate the image elements in the response data, and use a preset optical character recognition algorithm to identify the text in the image elements to obtain the visual presentation content; The structured text content is associated and mapped with the visual presentation content to generate the original monitoring data containing multi-dimensional business characteristics.

5. The method according to claim 1, characterized in that, The process of using a preset compliance rule base that matches the target network region to perform compliance verification on the original monitoring data to generate a compliance verification result, and cleaning up the original monitoring data that failed the compliance verification based on the compliance verification result, specifically includes: Obtain a preset compliance template corresponding to the target network area. The preset compliance template contains multi-dimensional compliance logic verification rules for the original monitoring data. The multi-dimensional compliance logic verification rules include data collection location access rules, data transmission path encryption rules, and data retention duration restriction rules. Construct the preset compliance rule library based on the preset compliance template. The system monitors the temporary storage duration of the raw monitoring data in the local storage space in real time, and extracts the collection node location of the target agent node and the transmission path information of the raw monitoring data. The temporary storage duration, the location of the collection node, and the transmission path information are input into the preset compliance rule base, and compared and verified with the corresponding data retention duration limit rule, data collection location access rule, and data transmission path encryption rule, respectively, to generate the compliance verification result containing risk judgment identifier; When the risk assessment indicator indicates a risk of violation, the communication session between the target agent node and the target platform is interrupted, and the storage destruction interface of the local storage space is invoked to perform a data overwrite and erase operation on the storage address corresponding to the original monitoring data.

6. The method according to claim 1, characterized in that, The process of identifying the data attributes of the original monitoring data that has passed the compliance verification, matching corresponding de-identification rules according to the data attributes, and performing de-identification processing on the original monitoring data that has passed the compliance verification based on the de-identification rules to obtain de-identified monitoring data specifically includes: The original monitoring data that has passed the compliance verification is compared by traversing fields using a preset sensitive data classification dictionary. The sensitive data classification dictionary is configured with a first sensitivity label and a second sensitivity label based on different security protection levels. The field content that matches the first sensitivity tag is extracted as a first data fragment, the field content that matches the second sensitivity tag is extracted as a second data fragment, and the first sensitivity tag and the second sensitivity tag are determined as the data attribute; Based on the tag mapping relationship in the preset desensitization rule library, the irreversible encryption rule bound to the first sensitivity tag and the character masking rule bound to the second sensitivity tag are extracted respectively, and the irreversible encryption rule and the character masking rule are combined into the desensitization rule; The hash salting algorithm in the irreversible encryption rule is called to perform hash calculation on the first data segment to generate an encrypted segment, and the preset masking parameter in the character masking rule is called to perform character replacement operation on the second data segment to generate a masked segment; According to the original field layout structure of the original monitoring data, the encrypted fragment, the mask fragment, and the retained field content of the missing tags are spliced ​​and assembled to generate the de-identified monitoring data.

7. The method according to claim 1, characterized in that, The process involves fusing the desensitized monitoring data, the node status data, and the compliance verification results to generate multi-dimensional fused features. These multi-dimensional fused features are then input into a preset anomaly identification model for anomaly analysis to obtain anomaly event results. Specifically, this includes: Business content features are extracted from the de-identified monitoring data, node operation evaluation parameters are extracted from the node status data, and compliance status identifiers are extracted from the compliance verification results. The business content features, node operation evaluation parameters, and compliance status identifiers are then normalized to obtain a normalized feature vector. The normalized feature vectors from multiple consecutive time windows are concatenated and combined to obtain a multidimensional temporal feature matrix that serves as the multidimensional fusion feature, and the multidimensional temporal feature matrix is ​​input into the anomaly recognition model. The anomaly recognition model calculates the similarity score between the multidimensional temporal feature matrix and each benchmark feature sequence in the preset anomaly scene feature library; When the maximum similarity score in the similarity scores is greater than or equal to a preset threshold, the target benchmark feature sequence corresponding to the maximum similarity score is determined, and the anomaly type identifier bound to the target benchmark feature sequence is extracted; Using the anomaly type identifier as a mapping index, query and retrieve anomaly cause tracing information and processing suggestions associated with the anomaly type identifier from the anomaly scene feature library; The anomaly type identifier, the anomaly cause tracing information, and the processing suggestions are encapsulated to generate the anomaly event result.

8. An overseas data monitoring system, characterized in that, The system includes: The status acquisition module is used to deploy a distributed agent cluster in the target network area and continuously collect the network operation characteristics of each agent node in the distributed agent cluster to generate node status data. The request evaluation module is used to obtain data collection requests for the target platform within the target network area, and to determine the availability status of each of the proxy nodes based on the node status data. The data collection request includes an access masquerading strategy and data collection rules that match the target platform. The dynamic scheduling module is used to select target proxy nodes whose availability status meets preset conditions from among the proxy nodes, and dynamically schedule the data collection request to the target proxy node. The data parsing module is used to initiate access to the target platform through the target proxy node based on the access masquerading policy and receive the response data returned by the target platform, and to parse the response data using the data collection rules to obtain the raw monitoring data; The compliance verification module is used to store the original monitoring data in the local storage space corresponding to the target network area, perform compliance verification on the original monitoring data using a preset compliance rule library that matches the target network area to generate a compliance verification result, and clean up the original monitoring data that failed the compliance verification based on the compliance verification result. The data desensitization module is used to identify the data attributes of the original monitoring data that has passed the compliance verification, match the corresponding desensitization rules according to the data attributes, and perform desensitization processing on the original monitoring data that has passed the compliance verification based on the desensitization rules to obtain desensitized monitoring data. The fusion analysis module is used to fuse the desensitized monitoring data, the node status data, and the compliance verification results to generate multi-dimensional fusion features, and input the multi-dimensional fusion features into a preset anomaly identification model for anomaly analysis to obtain anomaly event results. The early warning feedback module is used to generate corresponding early warning information based on the abnormal event results, and update the data collection rules and / or the access spoofing strategy according to the abnormal event results.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images